a11y.skipToMainContent
1

机器人.txt 发电机

构建一个机器人.txt——允许/不允许每个用户代理的规则,可选的爬行延迟,以及站点地图URL.

Add directives below, then Generate.

Rules (one per line: allow:/path, disallow:/path, or useragent:Name)

More options
Was this tool helpful?
Send output to:
Advertisement

How to use Robots.txt Generator

  1. 类型规则——一行一:用户代理人:*,不允许:/admin/,允许:/admin/public/等.
  2. 可选添加站点映射 URL并 craw-delay.
  3. 点击生成并保存为机器人. txt .

{姓名}是什么来着?

robots.txt告诉搜索引擎爬行者您网站的哪些部分要索引,哪些部分要跳过。 此生成器在格式useragent:Name,disallow:/path,和allow:/path中采用简单的规则,并输出一个适合格式的文件,准备上传到您的站点根.

Advertisement

FAQ

机器人会去哪儿?
总是在您域名的根部:https://yoursite.com/robots.txt——不是子目录.
机器人可以执行吗?
不,这是个建议 行为良好的爬行者尊重。 对于真正的访问控制,使用认证或服务器侧块.

Related tools

Author

1S
1Stop Tools"The Toolsmith"

Editorial Team at 1Stop Tools

The 1Stop Tools editorial team tests, reviews, and curates every utility on this platform. We believe tools should be free, fast, and private, with no accounts, no uploads, and no tracking. Every tool on this site runs entirely in the browser, and we test on multiple devices and browsers to ensure reliability for everyone, everywhere, on any connection.

Advertisement