This guide provides information about the robots.txt file created by default for web hosting services where this file is missing.

Introduction

• The robots.txt file acts as a guide for search engine crawlers.
• It is placed in the root directory of a website and contains specific instructions for these crawlers, indicating which directories or pages they are allowed to crawl and which they should ignore.
• However, please note that robots may choose to ignore these directives, making robots.txt a voluntary guide rather than a strict rule.

File content

If the robots.txt file is missing from an Infomaniak website, a file with the same name is automatically generated with the following directives:

User-agent: *
Crawl-delay: 10

These directives instruct bots to space out their requests by 10 seconds, which prevents unnecessary overloading of the servers.

Override the default robots.txt file

It is possible to bypass the robots.txt file by following these steps:

1. Create an empty robots.txt file (it will only serve as a placeholder so that the rules do not apply).
2. Manage the redirection of the URI (Uniform Resource Identifier) robots.txt to the file of your choice using a .htaccess file.

Example

RewriteEngine On RewriteCond %{REQUEST_URI} /robots.txt$ RewriteRule ^robots\.txt$ index.php [QSA,L]

Explanation

Explications

• The mod_rewrite module of Apache is enabled to allow redirects.
• The condition RewriteCond %{REQUEST_URI} /robots.txt$ checks if the request concerns the robots.txt file.
• The rule RewriteRule ^robots\.txt$ index.php [QSA,L] redirects all requests to robots.txt to index.php, with the [QSA] option which preserves the request parameters.

It is recommended to place these instructions at the beginning of the .htaccess file.