Robots.txt Generator
Build a robots.txt — allow / disallow rules per user-agent, optional crawl-delay, and sitemap URL.
Rules (one per line: allow:/path, disallow:/path, or useragent:Name)
More options
How to use Robots.txt Generator
- Type rules — one per line: useragent:*, disallow:/admin/, allow:/admin/public/, etc.
- Optionally add sitemap URL and crawl-delay.
- Click Generate and save as robots.txt at your site root.
What is Robots.txt Generator?
robots.txt tells search-engine crawlers which parts of your site to index and which to skip. This generator takes simple rules in the format useragent:Name, disallow:/path, and allow:/path and outputs a properly-formatted file ready to upload to your site root.
FAQ
- Where does robots.txt go?
- Always at the root of your domain: https://yoursite.com/robots.txt — not in subdirectories.
- Is robots.txt enforceable?
- No — it's a suggestion that well-behaved crawlers respect. For real access control, use authentication or server-side blocks.
Related tools
People also use
- App Icon Generator📱 Mobile App Assets
App Icon Generator — Upload one 1024×1024 (or larger) image and instantly get every icon size required by iOS, Android, Google Play and the App Store — ...
- App Store Screenshot Generator📱 Mobile App Assets
Free online app store screenshot creator. Wrap your mobile screens in iPhone & Pixel device frames with beautiful gradients and custom text headlines.
- Box Shadow Generator🗄️ Developer Tools
Box Shadow Generator — Build CSS box-shadow visually — tune offset, blur, spread, color, opacity, and inset with live preview.