Here I sit, broken hearted… my photo gallery is down. I happened to click the link this morning and the gallery was down with a 500 internal server error. I hit the logs and what do I see, i’m being spidered and it’s beating my web server down. There’s a new search engine on the block called Cuil (pronounced ‘cool’), and apparently it’s not very nice to the small web server owner. They advertise themselves and brag about how many pages they crawl and such… well, thanks for killing my damned web server jackasses. Your mega crawler just blasted my web server into oblivion. Now I have a robots.txt in place and hopefully that will shut this crap down. Funny, i’ve never had problems with Googlebot or any other search engine’s crawler until now.

Another fun day on the web.

Like

Unlike