Selected Search Tools for the WWW: A Guide
The information provided here has been drawn from my experience with these tools, and from the help screens for the search tools themselves. These pages provide basic information on using these few search tools--just enough to get you started. More details are available from the help provided by each of the search tools. I've also created an accompanying summary chart that presents the most essential details on using the tools.
You might want to use this page and the chart in a couple of ways. You could read the text on the search tool you're interested in and then look at the chart, or you could go first to the summary chart, get a quick idea of how the tool works, then follow the link back to the relevant text.
The tools presented here are those that I have found useful and that have also received positive reviews from a number of sources, including some of those listed on the page for More Information on Searching. The tools are listed from what I have subjectively determined is the simplest to use to the most sophisticated. This page has existed in some form since early 1996, and has changed over time as search tools have come, gone, and changed. Definitions and connections to "official" help screens are provided via links in the text and in the chart.
Search tools, search engines, and Web directories
The term search tools is used here to designate anything that you can use to search the Web. The world of search tools encompasses search engines, like Alta Vista and HotBot, and directories, like Yahoo. You should know that despite their claims, there is no search tool in existence that indexes all of the Web. Also, your searches are not run "live" on the Web, but are actually executed on the index file built by the search tool you are using. These index files may have been built as long as three or more months ago; that's why you might get an error message when you follow a link after your search.
Those search tools known as search engines provide keyword search access to their index files. These index files are built by computer programs called spiders or robots, which search the web, bring back copies of web pages, and build indexes based on words and other strings of letters and numbers that appear in a page. In keyword searching, you will find web pages containing the specific keyboard characters you type, but you will not be able to search for concepts. For example a search for the term washers will find pages that contains that term anywhere in the HTML code or the visible text of the page. You may find things about people who work as washers, machines that wash, or the flat metal rings you find in use with nuts and bolts--all these are the same to the search engines.
In Web directories, you have to use the terms provided by human beings rather than by computer programs, and you have to follow their hierarchy to get what you want. So you might go to the category Education, then follow the link to Higher Education, then Colleges and Universities then United States to get to a list of private colleges by state. This is a longer process, but it has the advantage of allowing you to browse headings and pick those that will be really helpful.
Generally, Web directories and search engines are becoming more like one another. Many directories now incorporate the ability to search what they have selected. In addition, nearly all the major search engines, including those listed here, now provide directory access to what they consider the best of the pages they index. It might be useful to think about search tools as a general category of things that help you find information on the Web, and to think about directories and search engines as providing different ways of getting at the stuff you need.
There may be significant differences between the way results are ranked and displayed among the different search tools. Many use a ranking system, called relevance ranking, based on the relative occurrence of the terms you searched within the documents found. Sometimes the relevance is based on the terms in the visible text of the documents, but often relevance is figured based on the HTML coding of the pages. This means, for example, that terms can be inserted in the header of a document--the part you don't see when you look at the page in your browser--that will influence the likelihood of a document being found and ranked highly in the results of a search. Sometimes page designers will insert terms for precisely this reason. Also, in order to generate more revenue, some search tools will offer paid inclusion or paid placement. In search tools that offer paid inclusion, a Web page producer, usually a commercial concern, is given the opportunity to pay for a greater depth of indexing, increasing the chance that a particular page will be found and ranked higher. Danny Sullivan, of Search Engine Watch, compares this to buying a greater number of tickets in the Internet search lottery (see Paid Inclusion At Search Engines Gains Ground). By contrast, with paid placement a higher ranking is guaranteed to those who will pay. So in the Web, as in other parts of life, let the buyer beware, even when it all seems free.
Created: 16 September 2000