1000 FAQs, 500 tutorials and explanatory videos. Here, there are only solutions!
Error: "GET /robots.txt HTTP/1.1" 404"
404 errors for robots.txt are due to HTTP requests to existing subdomains.
crawl-12-345-67-890.googlebot.com - "GET /robots.txt HTTP/1.1" 404 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
To alleviate this issue, you may either:
- Add a separate robots.txt file for each subdomain
- set up a generic rewrite rule that redirects to a common robots.txt file.
Link to this FAQ: