August 29, 2005
MSNbot Gone Wild !
Has anybody else noticed MSNbot going a bit wild in spidering their sites? If so, does your site include an RSS feed?
I noticed it a couple of weeks ago on a few of my sites, so just had to investigate. The common element with those that were getting hit really hard was that they all had some RSS component to the sites. Those sites that had no RSS component showed completely normal spidering patterns for MSNbot.
According to this short MSN Blog post they've evidently testing something to get the freshest content possible into their index. Which means blogs, generally speaking. Which usually have an RSS feed.
They've promised to slow it down and apparently have, since I've seen a fairly major drop off of MSNbot activity the last several days. But when it was going, it was kind of funny! On some of my smaller sites, like this one, MSNbot still accounts for nearly one-quarter of all traffic for this month. After only a week or so of crazy spidering activity.
Jeremy Zawodny's post on his blog is hilarious IMHO. "Dear MSNbot, Take a chill pill." LOL Too funny!
There you have it. Now you know what was up if you saw MSNbot practically living on your site.