1000 FAQs, 500 tutorials and explanatory videos. Here, there are only solutions!
Setting a crawl delay for certain crawlers
Add these commands to a robots.txt or .htaccess file in the root of your hosting space:
The range of values of Crawl-delay are as follows:
Crawl-delay setting -- Index refresh speed
No crawl delay set -- Normal
1 -- Slow
5 -- Very slow
10 -- Extremely slow
If you use Google Webmaster Tools, go to Site configuration -> Settings -> Crawl rate. The new rate will be valid for 90 days.
Link to this FAQ: