Free Tool
Free Robots.txt Tester
Check whether a specific page is blocked for Googlebot, GPTBot, or any crawler — the exact tool Google retired from Search Console in 2023. No signup required.
— paths tested so far
How this tool works
Fetches your live robots.txt
Not a cached copy — the actual file at your domain root, right now.
Groups by user-agent
Correctly handles multiple consecutive User-agent lines sharing one set of rules — a common pattern many simpler parsers get wrong.
Longest-match wins
Supports wildcards (*) and end-anchors ($), and applies the same specificity rules real crawlers use.
Falls back to the wildcard group
If a crawler has no specific rules, it inherits whatever applies to User-agent: *.
Shows the raw file
So you can see exactly what's driving the verdict, not just trust a black box.
No robots.txt = allowed
If a domain has no robots.txt at all, everything is crawlable by default — this tool reflects that correctly.
How to use the Robots.txt Tester
- Enter your domain, for example yourwebsite.com.
- Enter the path you want to test, such as / for the home page or /blog/ for your blog.
- Choose a crawler, such as Googlebot, GPTBot or ClaudeBot, or leave it on all crawlers.
- Click "Test" to see whether that crawler is allowed, and check the raw robots.txt file shown below the result.
What to do with your results
If an important page shows as blocked, find the Disallow rule that matches its path and remove or narrow it in your robots.txt file. Remember that robots.txt controls crawling, not indexing: to keep a page out of Google's results, use a noindex tag instead.
Frequently asked questions
Why isn't Google's own robots.txt tester available anymore?
Google removed it from the old Search Console in 2023 with no path-specific replacement in the new version — this tool fills that gap.
What does "longest matching rule wins" mean?
When multiple rules match the same path, the most specific (longest) pattern takes precedence, and Allow wins on a tie — the same logic real crawlers use.
Does blocking a page in robots.txt remove it from Google?
Not necessarily. Robots.txt stops crawling, but Google can still index a blocked URL if other sites link to it. Use a noindex tag to keep a page out of search results.
Should I allow GPTBot and other AI crawlers?
If you want tools like ChatGPT to find and cite your content, yes. Blocking them keeps your pages out of those answers.
Let's build something
Accidentally blocking your own pages?
We audit and fix crawl configuration as part of every technical SEO engagement.
Last updated: