robots.txt Generator
This robots.txt generator builds the file in a few clicks: choose a preset, add blocked paths and your sitemap, and optionally exclude AI crawlers. The finished file appears at once and is checked for common mistakes.
Source: IETF RFC 9309 – Robots Exclusion Protocol (groups, Allow/Disallow, wildcards * and $). Updated: .
How it is calculated
A robots.txt is a small text file in the root of a website that tells crawlers which areas they may fetch. It follows the Robots Exclusion Protocol (RFC 9309): each group starts with a User-agent and contains Allow and Disallow rules.
What the generator writes
- Preset: “Allow everything” writes an empty Disallow line, WordPress blocks
/wp-admin/and allowsadmin-ajax.php, the shop preset blocks typical cart, checkout and filter URLs. - Your own paths: paths start with
/. The character*stands for any characters,$for the end of the address. - AI crawlers: on request each known AI training crawler gets its own group with
Disallow: /. Whether a provider obeys is up to the provider. - Sitemap: the line sits outside the groups and gives the full address.
Good to know
The file must be reachable at /robots.txt of the domain and is public; it is not access control. To keep a page out of Google use noindex rather than a block, since blocked pages can still be listed without content. After uploading, the robots.txt checker tests whether crawlers read the rules as intended, and the SERP preview shows how a title and description look in search.
Frequently asked questions
How do I create a robots.txt?
Choose a preset, add blocked paths and the sitemap, copy the text and save it as robots.txt in the root of your domain. Then open your-domain.com/robots.txt and check it.
Where must the robots.txt be placed?
In the root of the domain, i.e. at https://your-domain.com/robots.txt. A file in a subfolder is ignored. Every subdomain needs its own file.
Does robots.txt keep pages out of Google?
Not reliably. It only forbids fetching. A blocked page can appear in the index with its address but no content if other pages link to it. To exclude it, use noindex.
How do I block AI crawlers?
With a separate group for each crawler and Disallow: /. The tool adds the known AI training crawlers. It is not legally binding, but reputable providers follow the file.
Do I need a robots.txt at all?
Not strictly: without the file crawlers may fetch everything. It is worthwhile to keep unimportant areas such as the cart or internal search out of crawling and to announce the sitemap.
Sources and legal basis
- IETF RFC 9309 – Robots Exclusion Protocol (groups, Allow/Disallow, wildcards * and $)
- Google Search Central – Create and submit a robots.txt file
- Google Search Central – robots.txt: supported rules (crawl-delay is not supported)
As of: · Editorial policy
Related tools
- robots.txt Checker with Sitemap ValidationIs a URL blocked for Googlebot, Bingbot or AI crawlers? This robots.txt checker shows the deciding rule, flags syntax issues and validates the sitemap.
- SERP Preview Tool With Pixel WidthEnter a title tag and meta description to preview the search result – with pixel width, character count and a note when the snippet is likely truncated.
- UTM builderAdd UTM parameters to links: utm_source, utm_medium, utm_campaign and more, correctly encoded, existing parameters kept. Campaign URL builder with typo checks.
- HTTP Status CodesLook up HTTP status codes: the meaning of 200, 301, 404, 500 and over 40 more codes explained, with search by code or keyword and a filter by class.
- What is my IP address?What is my IP? See your public IPv4 or IPv6 address at once, plus approximate location, internet provider and browser details. Free, never stored.
- Meta Tag GeneratorMeta tag generator: create the title, meta description, canonical, Open Graph and X card tags for your page head – ready to copy, with a check of the lengths.
- JSON-LD GeneratorJSON-LD generator for Schema.org: fill in an article, FAQ, product, local business or organization and copy ready-to-use code with a built-in error check.
- Cron expression generator and explainerCron expression generator and explainer: build or paste an expression, read it in plain words and see the next run times. Unix cron, Spring and Quartz, local.
- What is my user agent?See your user agent string and detect browser, version, engine, OS and device type, plus User-Agent Client Hints. Parse any other UA too. Runs locally.