Robots.txt Generator
Generate robots.txt files with a visual builder. Choose user-agents, set Allow/Disallow rules, add sitemap URLs, and use presets to block AI scrapers, bad bots, or common crawlers. Copy or download the output.
Quick Presets
robots.txt
How to use the Robots.txt Generator
Build your robots.txt file by adding user-agent blocks with Allow/Disallow rules. Use the presets for common configurations like blocking AI scrapers or setting up an SEO-friendly default. Add your sitemap URL and download the finished file.
What is robots.txt?
The robots.txt file lives at your domain root (e.g. https://example.com/robots.txt) and tells web crawlers which pages they can and cannot access. It is part of the Robots Exclusion Protocol (REP). Compliant crawlers like Googlebot respect the rules; malicious bots may ignore them.