Robots.txt Tester
See which robots.txt rule applies before you publish it.
Paste a robots.txt file, choose a crawler, and test paths locally. The report shows the matching group, winning Allow or Disallow rule, and why it won.
Everything stays in your browser. Paste up to 500 KiB of robots.txt and test up to 100 complete HTTP(S) URLs or paths beginning with /. Inputs replace prior results so the report cannot become stale.
Googlebot is the default. Use a full user-agent when checking a crawler variant.
One per line. Query strings are included in rule matching.
Rule matching follows RFC 9309 and Google’s robots.txt interpretation. Review a published file in the relevant search engine’s tools too.
About Robots.txt Tester
This tester applies the supplied robots.txt text to the URLs or paths you enter in this browser. It follows longest-match precedence, lets Allow win ties, and shows which matching user-agent groups supplied the rules. It never downloads a robots.txt file or visits your URLs.
Frequently Asked Questions
No. This is a local simulation. It does not fetch a site, verify where a robots.txt file is hosted, or confirm that any crawler will request a URL.
Matching user-agent groups are selected by the most specific agent name; groups with that same specificity are combined. The longest matching Allow or Disallow path wins. When two matching rules have equal length, Allow wins.
Yes. An asterisk matches any sequence and a trailing dollar sign requires the end of the path. Matching is case-sensitive for paths. UTF-8 and percent-encoded paths are compared in encoded form.
Robots.txt controls crawler access; it does not promise indexing or removal from search. A blocked URL can still be discovered through links. Use the appropriate noindex, access-control, or removal method for the outcome you need.
The tester reads User-agent, Allow, and Disallow. Comments are ignored. Unsupported directives such as Crawl-delay are listed as warnings because crawler support differs.