— SEO & GEO

Robots.txt Generator

Build a clean, valid robots.txt — allow or block crawlers, set a user-agent, add crawl-delay and your sitemap, then copy or download.

— What it does

The Robots.txt Generator is a free tool that creates a valid robots.txt file controlling how search engine crawlers access your site. Pick a preset to allow or block everything, or add custom Disallow and Allow paths, a user-agent, an optional crawl-delay, and one or more sitemap URLs. The file is generated live in your browser and downloads as robots.txt — nothing is uploaded.

  • Presets for Allow all, Disallow all, or fully custom rules.
  • Set the user-agent, multiple Disallow and Allow paths, and an optional crawl-delay.
  • Add one or more Sitemap URLs that get appended at the bottom of the file.
  • Runs entirely in your browser — your rules are never uploaded to a server.

How to use it

1

Choose a preset

Start with Allow all, Disallow all, or Custom to write your own allow and disallow rules.

2

Add your rules and sitemap

Set the user-agent, enter Disallow and Allow paths one per line, an optional crawl-delay, and your sitemap URLs.

3

Copy or download robots.txt

The file updates as you type. Copy it or download robots.txt, then upload it to the root of your domain.

Frequently asked

Where do I put the robots.txt file?
Place it at the root of your domain so it is reachable at https://yourdomain.com/robots.txt. Crawlers only look there — a robots.txt in a subfolder is ignored.
What does Disallow all do?
The Disallow all preset writes a rule that asks compliant crawlers not to index any path on your site. Use it for staging sites, but remember robots.txt is a request, not a security control.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap line helps search engines discover all your URLs. You can list more than one sitemap, and the lines are placed at the bottom of the file where crawlers expect them.
Does this tool upload my rules anywhere?
No. The robots.txt is built entirely in your browser with JavaScript. Nothing you enter is sent to a server, so it is private and safe to use for unpublished sites.

Related tools

Robots.txt Tester

Paste your robots.txt, enter a path and a crawler, and instantly see whether it is allowed or blocked — and exactly which rule decided it.

Open tool →

XML Sitemap Generator

Paste a list of URLs and get a valid sitemap.xml — with changefreq, priority and lastmod — ready to copy or download.

Open tool →

llms.txt Generator

Build an llms.txt that points AI answer engines at your most important pages — a name, a summary, and curated link sections, ready to download.

Open tool →

← All free tools

— Built by saavos

These tools are free. So is the first version of your agent.

saavos is the AI agent that lives on your website — themed to match your design, answering visitors from your own content, and telling you what they actually want to know. Paste your URL and see it answer, before you install anything.

Make my site feel alive