robots.txt Generator

Quick Presets

+ Add User-Agent Rule

Preview

What is a robots.txt File?

A robots.txt file is a text file placed in the root directory of your website that tells search engine crawlers and bots which pages they can or cannot access. It’s a standard way to manage crawl budget and control bot traffic to your site.

The robots.txt file follows the Robots Exclusion Standard and is read by all major search engines including Google, Bing, and others. It’s not a security measure—it’s advisory only. Well-behaved bots follow these rules; malicious crawlers often ignore them.

How to Use This Generator

1. Start with a preset or manually add user-agent rules using the “Add User-Agent Rule” button.
2. Configure each rule: Select a user-agent (All, Googlebot, Bingbot, or custom), add paths to allow/disallow, and set crawl delays if needed.
3. Set global sitemap URL (optional) — helps search engines discover all your pages.
4. Click “Generate robots.txt” to preview the formatted output.
5. Copy or download the file and place it in your website’s root directory (e.g., https://yourdomain.com/robots.txt).

Common Use Cases

• Block AI crawlers: Prevent GPTBot, Claude-Web, and other AI training bots from accessing your content.
• Protect admin areas: Disallow crawlers from accessing /admin, /wp-admin, or sensitive directories.
• Reduce crawl waste: Block low-priority pages like duplicate content, pagination, or tag pages.
• Allow specific bots: Permit Google and Bing but disallow smaller search engines.
• Control crawl budget: Set crawl delays to prevent server overload from aggressive bot activity.

Frequently Asked Questions

Do I need a robots.txt file?

Not strictly required, but highly recommended. Without one, all bots follow their default behavior. A robots.txt file gives you explicit control over what crawlers can access, helping manage server load and SEO crawl budget.

Where do I place the robots.txt file?

The robots.txt file must be placed in the root directory of your domain. For example: https://yourdomain.com/robots.txt. If using a subdomain, place it at the root of that subdomain.

What’s the difference between Allow and Disallow?

Disallow tells bots to NOT access a path (e.g., /admin). Allow explicitly permits access, which is useful for overriding broader disallow rules (e.g., allow /admin/public while disallowing /admin).

Can I block specific bots like GPTBot?

Yes! Use this generator’s “Block AI Crawlers” preset or add custom user-agent rules for GPTBot, Claude-Web, CCBot, and others. However, remember that malicious bots may ignore robots.txt—it’s not a security tool.

What is Crawl-Delay?

Crawl-Delay (in seconds) tells bots how long to wait between consecutive requests. For example, Crawl-delay: 5 means bots should wait 5 seconds between page requests. This helps reduce server load. Not all bots respect this directive.

Is robots.txt a security tool?

No. robots.txt is advisory only—well-behaved bots follow it, but malicious crawlers and hackers ignore it. For sensitive data, use proper authentication, HTTPS, and access controls instead of relying on robots.txt.