Google bots are crawling in a new way / Stephen Shankland Silicon.com, 16 April 2008
http://networks.silicon.com/webwatch/0,39024667,39188654,00.htm
"Google's search bots, which scour the web constantly for new pages, have begun a new, more active phase of their indexing jobs. The new Google indexing practice involves only "high quality" websites and doesn't run on sites with 'robots.txt' files or other standard mechanisms of warding off indexing software."