The Major Search Engines & Their Differences

There are true “engines” that are powered by software that searches the net and then there are “directories” that are big lists, often compiled in a different way. Most people refer to both as search engines but they are different animals.

Directories (Big Lists) & True Search Engines.

Directories: Yahoo is a big
list and for the most part, it is created and
maintained by humans. Webmasters submit their sites
to Yahoo and hope that the Yahoo Gods like the site
and list it favorably. This is a double edge sword.
Changes you make within your site have nothing to do
with the listing result although they might help you
get reviewed at a later date.



Search Engines: Search engines, such as
Google, produce search results on the fly. The
software of the search engine company crawls the
Internet and refreshes their internal index. When you
request a search, the search engine looks to its
index and delivers the results to your browser. These
listings are often unfiltered by humans. When
Websites change these search engines periodically
find the changes and it affects how results come back
to the user.

Specific parts of Web pages have a significant impact
on how the page gets listed in terms of ranking, such
as titles, meta tags, body copy, location of text,
frequency etc. Google has recently produced a
directory so they are in both categories now, a
search engine and a directory.



The Hybrids that combine both: More and more search
sites combine directories and crawling the Web.

Components of a Search Engine

1. Spider: The technology
that searches the Web, often called a spider or
crawler. This “spider” of technology lands on a
Website and follows within that site, performing a
deeper crawl. Once the Website is indexed, the spider
will come back periodically to look for changes and
update its listing.



2. Index: The spider finds data and sends it
to the search engine database or index or catalog.
This takes time, from a few days to weeks.



3. Ranking: Once a user on the search engine Website
requests a search, the search engine uses its special
software to analyze the index and produces results in
the order it thinks they are most relevant, i.e.
ranking the results.

The Majors

They are many search engines but
the following are all you need to concern yourself
with at this point in time. Direct your efforts with
these and worry about the rest later: (not in order
of importance)



AOL Search http://search.aol.com/

Users can search Web content across the full
Internet. AOL uses Open Directory for its directories
(see below) and the Inktomi engine (see below) for
its crawler-based search.



AltaVista http://www.altavista.digital.com/

AltaVista is a large search site and has excellent
coverage plus a diversity of power searching
commands.

Ask Jeeves http://www.ask.com/

Ask Jeeves is powered by humans for the most part,
that the site attempts to point you to the exact page
that answers your question. We do not think it does
this particularly well. In their defense, this is a
difficult task. If the search fails, this site tries
to send you elsewhere on the Web.



Direct Hit http://www.directhit.com/

The site is kind of a popularity engine in that it
measures what people click on in the results of
searches on its site and that of its partners. The
popular sites rise to the top of the listings. They
partner with HotBot. Direct Hit provides search
results for HotBot and can also be seen on MSN. They
are owned by Ask Jeeves.



Excite http://www.excite.com/

Very popular search engine with a large index. They
also make available other information such as sports.
Excite owns Magellan and WebCrawler which still are
separate sites



FAST Search http://www.fastsearch.com/

Fast Search is on a mission to index the entire
Internet, a daunting task to say the least. It used
to be called All The Web. It is huge and claims to be
the first search engine to break the 200 million Web
page index milestone. Based in Norway



Go / Infoseek http://www.go.com/l

Formerly Infoseek. Go is part of Disney. It attempted
to be a portal site with personalization, customized
services, free E-mail and more. InfoSeek was
integrated into the site. It used to use an ESP
search algorithm which produces good results and had
a directory. On March 15th 2001 they announced they
are shutting down. They now have paid listings from
GoTo.com



GoTo.com http://www.goto.com/

Not related to Go.com. You pay to play on GoTo, in
other words, you buy your way in. The more you pay
the higher you are listed. When results come back
that are non paying sites, GoTo uses the Inktomi
engine. The University of Colorado-based World Wide
Web Worm became GoTo a few years ago but was replaced
by Inktomi technology. When sites pay to be listed on
GoTo, they show up elsewhere such as AltaVista, AOL
Search, Lycos, HotBot and Netscape Search.



Google http://www.google.com/

Critically acclaimed Google separates itself from the
pack by high relevancy, which is largely determined
by the link popularity of the listed sites. This
works well when users are searching for generic terms
such as home or vacations.

Google’s index is very large and also services
searches for Netscape and Yahoo.



HotBot http://www.hotbot.com/

Known as a power searching favorite, HotBot splits
search results with the top page being provided from
Direct Hit and the rest of the results from Inktomi.
There is also a directory, which is sourced from the
Open Directory project (see below). Lycos purchased
HotBot from Wired Digital.



IWon http://www.iwon.com/

Win a million bucks while you search. Financed by CBS
among others when venture capital money flowed like
water, iWon produces a directory powered by Inktomi.
The site actually gives away money, a strategy to buy
eyeballs (visitors). The jury is still out on their
business model, which probably cannot continue.
Search fast on this site while they are still around.
If they don’t make it, they can reinvent themselves
with a pay search site, “I-Lost” or “I’ll search if
you pay me.”



Inktomi

From its roots at UC Berkeley, the founders left to
go out on their own and the company got hot. This is
not a search site but technology that powers search
so we thought you should at least know about it.



LookSmart http://www.looksmart.com/

LookSmart is a powered by smart looking humans that
make a directory of the Web. They provide their
services to Excite and Microsoft, among others. They
use Inktomi when they do not produce a review of a
site. Formerly part of Reader's Digest.



Lycos http://www.lycos.com/

Once a spider that crawled all over the Web, it
transformed into a new animal, a directory of the
Internet jungle. Its main menu is dished up from the
Open Directory project and back up results via FAST
Search and Direct Hit. The old spider gobbled up
HotBot but let it live for the moment on its own.
Nice spider.



MSN Search http://www.msn.com/

We can not do much without the 400 pound gorilla
rearing its head. Microsoft's MSN Search service is
powered by LookSmart with a back up of Inktomi.
RealNames and Direct Hit data is also there. Within
five years, all data of the Universe will be
controlled, filtered and approved by Microsoft and
you will not need search at all because Microsoft
will just tell you what you need to know. (a little
joke…)



NBCi http://www.nbci.com/

“What you want, when you need it.” At least that’s
what their TV commercial says. Sounds more like
Microsoft to me or was that Burger King. Anyway, this
is a directory complied by humans with back up
results from Inktomi. NBCi wants to be the next Yahoo
and was the former Snap, part of Cnet and who knows
what it will be tomorrow, maybe a TV guide or
something.



Netscape Search http://www.netscape.com/

Netscape uses Open Directory and its own "Smart
Browsing" technology and does a pretty good job.
Back-up data comes from Google. Owned by AOL-Time
Warner so watch for entertainment plugs popping up.
Do a search, get free popcorn.



Northern Light http://www.northernlight.com/

Northern Light charges you for top search results. If
you are a researcher with a budget great, otherwise
it is not a great choice. They have special
collection of documents that spiders can not see
readily.



Open Directory http://dmoz.org/

Believe it or not, this is done by volunteers. Used
to be known as NewHoo as opposed to YaHoo. Netscape
bought it and it works through an open license
arrangement. Netscape was an early licensee as were
Lycos and AOL Search.



Raging Search

Owned by AltaVista and uses the same index and algorithms. It does not have the bells and whistles of Alta Vista.

Yahoo

Yahoo is a giant. They started with humans evaluating Websites and people liked the quality results. Google does assist in searching but it is dominantly controlled by Yahooites. Yahoo was a pioneer out of Stanford in 1994.

WebCrawler

Now owned by Excite but continues to run on its own. Uses crawler-based searching.

WebTop

A crawler-based engine with a huge index that is supplemented with content news sources and more. Has a Webcheck discovery tool and is owned by Bright Station.

To learn more about Search Engines and how to optimize your site for maximum rankings, please click here for more information

Copyright The How-To Network  All rights reserved

Return to Library of Business Information