Robots.txt Generator
Generate robots.txt
Build crawler directives for search engines and website indexing control.
Use an absolute HTTP or HTTPS URL.
About
The XAVERT Robots.txt Generator creates crawler directives directly in your browser.
Main Features
- Select a common crawler or use the wildcard user-agent.
- Add Allow and Disallow directives.
- Optionally configure Crawl-delay.
- Add an absolute Sitemap URL.
- Copy or download the generated robots.txt file.
Privacy First
robots.txt generation happens locally within your browser. Your configuration is never uploaded, transmitted, shared or stored on any server.