Robots.txt and llms.txt Analyzer

Use this analyzer to review crawler access rules, sitemap discovery, and the structure of your public llms.txt file.

Open the Robots.txt and llms.txt Analyzer

What it checks

For robots.txt, the analyzer reviews:

• whether the file is available
• user-agent groups and access rules
• broad crawler blocks
• sitemap references

For llms.txt, it reviews:

• whether the file is available
• Markdown headings and structure
• useful links to important public pages
• basic clarity and discovery signals

How to run the check

1. Enter a public domain or URL.
2. Run the analyzer.
3. Review crawler blocks before adjusting any other signal.
4. Confirm that sitemap references use the intended hostname.
5. Review whether llms.txt links to stable, useful public resources.

How to interpret the result

A robots rule can allow or block crawling, but it cannot make thin JavaScript HTML readable. A sitemap can help discovery, but it does not prove that page content is visible in the response.

llms.txt is a voluntary discovery file. Its presence does not guarantee that an AI system will retrieve, use, cite, or recommend the website.

Important limitations

The analyzer checks public file access and structure. It does not validate Search Console ownership, indexing status, rankings, or actual crawler-operator visits.

Be careful when changing robots rules. A broad Disallow can remove access for more pages or crawler families than intended.

Recommended next step

Use Bot View Checker to inspect the actual page response. Use the Bulk Crawler Readability Checker to sample pages referenced by the sitemap.