SAN FRANCISCO — Hoping to attract more mass appeal for an online search engine with a cult following, AlltheWeb.com on Monday declared that it indexes more Internet information than longtime pacesetter Google.
AlltheWeb, owned by Norway-based Fast Search & Transfer, says its database spans 2.1 billion Web pages, just ahead of the 2.07 billion scanned by Google at the end of last week. AlltheWeb has been quietly building upon its previous foundation of 800 million pages over the past two months.
The rapid expansion represents the latest salvo aimed at Mountain View-based Google, whose growing popularity since its inception in 1998 has inspired a mixture of awe, jealousy and one-upmanship among its competitors.
The challenges haven’t toppled privately held Google so far. Besides operating one of the Web’s most trafficked destinations, Google also provides search results for other prominent sites, including Yahoo and AOL.
Even before it expanded its index, AlltheWeb believed it had developed a better search engine than Google’s. AlltheWeb believes its technology provides more relevant responses that guides users to the freshest information on the Web. AlltheWeb also says its scanning devices dig deeper into online content than Google.
AlltheWeb’s effectiveness has already made it a hit among scientists, librarians and other researchers looking for more obscure data.
By expanding the amount of online turf that it scans, AlltheWeb believes it will become even more useful to a broader audience.
“Our goal is to develop the best search experience possible,” said John M. Lervik, Fast Search’s chief executive officer. “We really hope Google responds to this.”
Google regards “quantity as just one component of search,” said company spokesman David Krane. “We still believe we offer the most comprehensive search experience on the Internet.”
While Google has long boasted about the breadth of its Web indexes, the company takes even greater pride in the complex formulas that it had developed to deliver quick results that list the most pertinent destinations.
Still, there is no question that size matters when it comes to search engines, said Greg Notess, a Bozeman, Mont. researcher who has been studying online search engines since 1990.
“If a page hasn’t been indexed, you won’t be able to find it no matter how good a search engine’s algorithms are,” he said. “It’s good to see someone is aggressively pushing to get to Web pages that no one may be looking at right now.”
The issue is important enough to Google that Notess predicts the company will come up with a way to surpass AlltheWeb’s new index volume very quickly
Besides expanding its index, AlltheWeb is also unveiling a new design meant to make it more attractive to a mainstream audience.
Since its launch in 1999, AlltheWeb primarily has served as a showcase and testing site for Fast Search, which makes its money licensing its results to other popular destinations such as Lycos. Fast Search also provides search results for many other sites, including IBM, eBay and FirstGov.gov.
With the AlltheWeb makeover, Fast Search is walking a fine line.
On the one hand, the company wants more people to know about AlltheWeb’s abilities. On the flip side, the company doesn’t want AlltheWeb to lure too much traffic from other sites that use its search technology.