DevToolBox免费
博客

Robots.txt 生成器

生成 robots.txt 文件，包含 user-agent 规则和 sitemap 指令。

Quick Presets
Rule Group 1
No disallow rules (everything is allowed)
Used by Yandex to specify the preferred domain.
Generated robots.txt
User-agent: *
Allow: /

What is Robots.txt?

The robots.txt file tells search engine crawlers which pages they can or cannot access on your site.

𝕏 Twitterin LinkedIn

评价此工具

4.2 / 5 · 208 人评价

更多工具

🏷️Meta 标签生成器📐Schema 结构化数据生成器🔗URL 别名生成🔗URL 解析器

Recommended

VercelDeploy your frontend instantlyDigitalOceanCloud hosting from $4/mo

保持更新

获取每周开发技巧和新工具通知。

无垃圾邮件，随时退订。

Enjoy these free tools?

☕Buy Me a Coffee

How to Use

  1. Select user-agent
  2. Add Allow/Disallow rules
  3. Enter sitemap URL
  4. Use presets for quick setup
  5. Copy or download

Use Cases

  • Control crawling
  • Block private pages
  • Set crawl limits
  • Reference sitemaps

FAQ

What is robots.txt?
A text file at your website root that instructs crawlers which URLs they can access.
Does it block indexing?
No, robots.txt controls crawling, not indexing. Use noindex for that.
Where to place it?
At your domain root: https://example.com/robots.txt