Robots.txt and llms.txt analyzer
Check whether your robots.txt and llms.txt files give crawlers and AI agents clear, useful discovery instructions.
Paste a domain or URL. The tool checks /robots.txt and /llms.txt for crawler access, sitemap hints, Markdown structure, and useful links.
Paste a homepage URL or domain. The tool checks robots.txt and llms.txt at the site root.
Switch between checks without hunting through the site.
See what crawler-style requests can read.
Compare the first HTML response with rendered content.
Score crawler access, metadata, and thin-content issues.
Check crawler rules and AI-agent discovery files.
Check up to 10 sitemap or pasted URLs.
When to use this tool
Confirm crawlers can find your sitemap and are not accidentally blocked.
Check whether llms.txt has a clear Markdown summary and useful links for AI agents.
Use this before requesting validation or submitting new pages for indexing.
Read it as crawler access, not SEO magic.
Use this result as a discovery-file sanity check. It does not replace content quality, rendered HTML checks, or Search Console validation.
It does not promise rankings, traffic, or AI citations. It tells you whether bots are getting useful HTML from the request path.
Low readable text, missing title or H1, failed bot checks, or a large raw-versus-rendered gap.
Bots receive enough readable content, important metadata is present, and the page does not depend on a blank shell.
Very low readable text, generic metadata, missing headings, blocked crawler access, or a large raw-versus-rendered gap.
If discovery files are missing or unclear
Add a simple robots.txt with a sitemap reference and an llms.txt file with clear Markdown links to important pages.