SiteKit Tools

robots.txt Generator

Create a valid robots.txt file: choose what search engines can crawl, add your sitemap, and decide which AI crawlers are allowed.

Need this for a whole website? Website SEO Audit & Broken Link Checker crawls every page and finds missing titles, long descriptions, duplicates and broken links, with an HTML report.

FAQ

Where do I put robots.txt?

In the root of your domain, so it is reachable at https://yourdomain.com/robots.txt.

Does robots.txt hide pages from Google?

No. It stops crawling, but a blocked URL can still appear in results if other sites link to it. Use a noindex meta tag to keep a page out of search.

Can I block AI crawlers?

Yes. Well-behaved AI crawlers such as GPTBot, ClaudeBot and Google-Extended follow robots.txt rules.