The “kottke matrix test-tube babies” Search Engine test

Today at work we were discussing Search Engine Optimisation and wether or not search engines actually index content a long way down the page. I developed the “kottke matrix test-tube babies” test, this originated from me recalling a crazy post on Jason Kottke’s blog about the 2nd and 3rd Matrix movies. These had a ridiculous number of comments (457) and a lot of content, which I suspected would be beyond any reasonable content threshold a search engine may have implemented. I took a search term out of the final comment (test-tube babies) and a couple of terms to narrow the search (kottke matrix) and away I fired on the big search engines.

The results of the test, Google and Live are the only 2 that happen to return the page I was targeting. It’s interesting to see however, that Yahoo finds the Science page which contains the term “test tube” but not “babies”. This page is 180kb in size and “test tube” appears approximately half way down, whereas the Matrix Revolutions page is 646kb and test-tube appears at the very bottom.

What’s the moral to this story? Google and Live.com index atleast the first 646kb of a page, Yahoo indexes (at a minimum) around 100kb and Ask isn’t on the radar.