Check your robots.txt: who may read your website, AI included?
Enter your address and see whether Google, Bing and AI assistants may read your pages. The checker finds syntax errors and shows, per crawler, the rule that applies. If you like, create a new robots.txt.
Free. No signup. The analysis runs in your browser. To fetch the robots.txt our server requests the public address for you; we store nothing.
A robots.txt is a request to crawlers, not protection. Not every crawler obeys it. The classification of AI crawlers follows the vendors' statements (as of October 2026) and may change.
How the checker works
FETCH
The file is read
You enter your address and the checker fetches the robots.txt. Or you paste the text yourself, in which case nothing is transmitted at all.
CHECK
Rules as crawlers read them
Syntax errors, a missing sitemap, blocked design folders and a size over 500 KiB are reported. Rules are evaluated the way Google does it: the more specific rule wins.
AI CRAWLERS
Who may do what
For Googlebot, Bingbot and the main AI crawlers you see whether the path is allowed or blocked, and which line causes it. You decide on search and training separately.
Questions about robots.txt
What is a robots.txt?
A small text file in the root of your website. It tells crawlers which areas they may read and where your sitemap is.
Should I block AI crawlers?
That depends on your goal. Search crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot decide whether your pages can be cited in AI answers. Training crawlers such as GPTBot or ClaudeBot collect content for models. If you want to be found, keep search open and decide on training separately.
Does robots.txt keep pages out of Google?
Not reliably. A blocked address can still appear in the index if others link to it. To exclude a page, use a noindex meta tag.
How do I change the robots.txt in Webflow?
Under Site settings, SEO, robots.txt. Paste the text, save and republish the site.
Is what I enter stored?
No. For an address, our server fetches the public robots.txt and returns it to your browser. The address is not stored. Pasted text stays in your browser.
More free tools
Fifteen minutes. No presentation.
Tell us where your customers come from today. We'll tell you which of the three places you're losing the most – and whether working with us makes sense, or whether you can fix it yourself this week.
We only take on a limited number of projects at a time. If we're fully booked, we'll tell you on the call.