Troubleshooting crawler visibility
Diagnose thin crawler HTML, hostname mismatches, render failures, and routing mistakes with response evidence.
On this page
Read the result first
Start with the exact URL that is failing. Compare a normal HTTP response with a crawler-style response and write down the final URL, status code, content type, canonical URL, readable text, and any rendering headers.
Start with the disagreement you see
| What you see | What to check next |
|---|---|
| Active, but crawler HTML is still a shell | Check the final hostname, redirects, configured origin and request-layer routing. Compare the homepage with a deep link before changing DNS. |
| Pending, and the response or validation also fails | Use the exact account setup instructions for your domain. Check the authoritative DNS provider, record name and certificate-validation records. Leave unrelated records unchanged. |
| Pending, but the response looks correct | Refresh the account view and repeat the check. If they still disagree, send the status and dated response evidence to support. Further DNS edits may not be the right fix. |
| Two checker tools disagree | Compare their tested URL, redirect handling, user agent and whether JavaScript runs. Visitor and crawler-style requests can receive different responses. |
| The crawler request is blocked or times out | Check access policy, challenge pages and origin reachability before assuming a JavaScript-rendering gap. Preserve the status and response sample. |
A Pending label alone does not establish a status-sync bug. Follow the setup verification checklist to collect comparable evidence.
Common symptoms
The crawler response is still a thin shell
Confirm that the route is public and that the crawler user agent is eligible for routing. Check that the rendering layer is configured with the correct origin and that the origin itself returns the completed page when rendered.
The crawler response returns 503 or times out
Check that the render service can reach the origin over HTTPS. Look for redirect loops, slow pages, blocked assets, authentication walls, or an unavailable origin. Keep a normal visitor fallback rather than serving a misleading cached error.
One hostname works and another does not
Test the root and www hostnames independently. DNS records, redirects, certificates, and crawler routing must be valid for the hostname a crawler actually requests.
The page is public but should not be rendered
Exclude sensitive or personalized routes from the crawler path. Authentication, checkout, account, API, and dashboard pages should be protected by access control, not by robots directives alone.
Capture useful evidence
When asking support for help, include the exact public URL, check date and time, final URL after redirects, setup status, response status and headers, and a short public HTML sample from both requests. Say whether root, www and an important deep link have the same problem.
Keep API keys, authentication cookies and private page content out of screenshots or response samples. Make one change at a time and repeat the same comparison.
Instructions for your connected site
Open account setup for domain-specific values, diagnostics and copy-ready configuration. These stay private to your workspace.