robots.txt Generator
Create a valid robots.txt file: choose what search engines can crawl, add your sitemap, and decide which AI crawlers are allowed.
Need this for a whole website? Website SEO Audit & Broken Link Checker crawls every page and finds missing titles, long descriptions, duplicates and broken links, with an HTML report.
FAQ
Where do I put robots.txt?
In the root of your domain, so it is reachable at https://yourdomain.com/robots.txt.
Does robots.txt hide pages from Google?
No. It stops crawling, but a blocked URL can still appear in results if other sites link to it. Use a noindex meta tag to keep a page out of search.
Can I block AI crawlers?
Yes. Well-behaved AI crawlers such as GPTBot, ClaudeBot and Google-Extended follow robots.txt rules.