Robots.txt Generator.
Generate a robots.txt file for your website. Control search engine crawling with an easy-to-use generator.
Input
Disallow:
Allow:
/Output
1 rule
User-agent: * Allow: /
Need to do this on any website, at scale?
Browzey runs this — and hundreds of other browser tasks — in plain English. No copy-paste, no limit. Install the Chrome extension and try it yourself.
About This Tool
What is robots.txt?
A robots.txt file is a text file that tells search engine crawlers which pages or sections of your website they can or cannot access. It's placed in the root directory of your website and is one of the first files crawlers look for when visiting your site.
Why Do You Need robots.txt?
- Control crawler access - Block search engines from indexing sensitive or duplicate content
- Manage crawl budget - Help search engines focus on your most important pages
- Protect server resources - Prevent aggressive bots from overloading your server
- Hide development areas - Block staging environments and admin areas
- Point to sitemap - Tell crawlers where to find your XML sitemap
robots.txt Syntax
User-agent
Specifies which crawler the rules apply to. Use * for all crawlers, or specify individual bots like Googlebot or Bingbot.
Disallow
Tells crawlers not to access specific paths. For example, Disallow: /admin/ blocks the admin directory.
Allow
Overrides a disallow directive for specific paths. Useful for allowing a subdirectory within a blocked directory.
Sitemap
Specifies the location of your XML sitemap. This helps search engines discover all your pages.
Common robots.txt Examples
Allow All Crawlers
User-agent: * Allow: /
Block All Crawlers
User-agent: * Disallow: /
Block Specific Directory
User-agent: * Disallow: /private/ Disallow: /admin/ Disallow: /temp/
Best Practices
- Always include a sitemap reference
- Be careful not to accidentally block important content
- Test your robots.txt with Google Search Console
- Remember that robots.txt is publicly accessible
- Don't rely on robots.txt for security (it's just a guideline)
- Keep rules simple and organized by user-agent
Common Crawlers
- Googlebot - Google's main crawler
- Googlebot-Image - Google Images crawler
- Bingbot - Microsoft Bing crawler
- Slurp - Yahoo crawler
- DuckDuckBot - DuckDuckGo crawler
- Baiduspider - Baidu crawler
- YandexBot - Yandex crawler
Limitations
It's important to understand that robots.txt is not a security mechanism. Malicious bots can ignore it, and the blocked URLs are visible in the file itself. For truly private content, use authentication or password protection.
Frequently Asked Questions
What is robots.txt?
robots.txt is a file that tells search engine crawlers which pages or sections of your site should not be crawled or indexed.
Does robots.txt block pages from Google?
Robots.txt only prevents crawling, not indexing. Pages can still appear in search results if linked from other sites. Use noindex meta tags to prevent indexing.
What is the User-agent directive?
User-agent specifies which crawler the rules apply to. Use * for all crawlers, or specific names like Googlebot or Bingbot for targeted rules.
Should I block CSS and JavaScript files?
No, Google recommends allowing access to CSS and JS files so it can properly render and understand your pages for better indexing.