robots.txt Generator & Formatter

robots.txt Generator & Formatter

Local Processing

Build robots.txt from rules or tidy one you paste.

How It Works & Processing Standards

Build robots.txt from rules or tidy one you paste.

Tool processing runs in your browser. If you create a calculator or conversion share link, its values are included in the URL. See the privacy policy for account, analytics and advertising data.

Current Limits

  • Maximum input length: 100,000 characters

Choose create or tidy under "Task". Block paths such as admin pages or internal search results from crawlers and tell them where your sitemap is.

What You Can Do

Create
Writes agent, Disallow, Allow and Sitemap lines correctly.
Tidy
Normalises inconsistent case and spacing.

How to Use robots.txt Generator & Formatter

  1. 1
    Choose a taskCreate or tidy.
  2. 2
    Enter rulesOne path per line.
  3. 3
    SaveUpload it as robots.txt at the site root.

Examples

Block admin and temp folders
Input: User-agent *, Disallow /admin, /tmp, Sitemap https://example.com/sitemap.xml
Result: User-agent: * Disallow: /admin Disallow: /tmp Sitemap: https://example.com/sitemap.xml

Common Use Cases

  • Creating robots.txt for a new site
  • Cleaning up a robots.txt flagged in Search Console

Frequently Asked Questions (FAQ)

Does Disallow remove a page from search entirely?

It only blocks crawling. If linked elsewhere, the URL can still appear; use noindex or login protection to hide it.

Have feedback or ideas for this tool? Suggest & Inquire →