Skip to content

Robots.txt Generator

Create a robots.txt file that blocks paths, allows exceptions, lists sitemaps and can block AI training crawlers.

For staging or private sites only

One per line. * matches anything, $ marks the end: /admin/, /*.pdf$

Paths inside blocked ones that crawlers may still visit.

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, PerplexityBot, Bytespider, meta-externalagent. Search engines are not affected.

robots.txt

User-agent: *
Allow: /

Runs in your browser — nothing you enter is sent to a server.

How the Robots.txt Generator works

A robots.txt file at the root of your site (https://example.com/robots.txt) tells crawlers which paths they may visit. List the paths to block, any exceptions inside them, and your sitemap, then download the file and upload it to the root of your site.

The file follows RFC 9309. * matches any characters and $ marks the end of a URL, so /*.pdf$ blocks every PDF. When a crawler has its own group — like the AI crawler group — it follows only that group and ignores the * rules.

AI crawlers blocked by the option

Block AI training crawlers adds one group for these crawler names, as published by their operators: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, PerplexityBot, Bytespider, meta-externalagent.

Search engines such as Googlebot and Bingbot aren't affected: Google-Extended, for example, controls use of your content for Google's AI models without changing Google Search. New crawlers appear regularly, so review the list from time to time.

Examples

Block an admin area but keep its help pages

User-agent: *
Allow: /admin/help/
Disallow: /admin/

Allow everything and list the sitemap

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Frequently asked questions

Does robots.txt hide a page from Google?

No. It stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of search results, allow crawling and add a noindex robots meta tag (see the Meta Tag Generator).

Do all crawlers obey robots.txt?

Reputable ones do, but it's a voluntary standard, not access control. Protect private content with a login instead.

Why no crawl-delay?

Google ignores it, and other crawlers treat it differently. Control crawl rate in each search engine's webmaster tools instead.

Related tools

Sitemap Helper

Turn a list of page URLs into a valid XML sitemap, with duplicate removal and URL checks.

SEO Tools

Meta Tag Generator

Generate the title, description, canonical, robots and viewport tags for a page's <head>, with length checks.

SEO Tools

URL Encoder & Decoder

Percent-encode text for URLs and query strings, or decode %20-style encoded text back to readable text.

Developer Tools

Also in Tools for Website Owners.