Troubleshooting crawler visibility

Start with the exact URL that is failing. Compare a normal HTTP response with a crawler-style response and write down the final URL, status code, content type, canonical URL, readable text, and any rendering headers.

The crawler response is still a thin shell

Confirm that the route is public and that the crawler user agent is eligible for routing. Check that the rendering layer is configured with the correct origin and that the origin itself returns the completed page when rendered.

The crawler response returns 503 or times out

Check that the render service can reach the origin over HTTPS. Look for redirect loops, slow pages, blocked assets, authentication walls, or an unavailable origin. Keep a normal visitor fallback rather than serving a misleading cached error.

One hostname works and another does not

Test the root and www hostnames independently. DNS records, redirects, certificates, and crawler routing must be valid for the hostname a crawler actually requests.

The page is public but should not be rendered

Exclude sensitive or personalized routes from the crawler path. Authentication, checkout, account, API, and dashboard pages should be protected by access control, not by robots directives alone.

Save the response headers and a short HTML sample from both requests. This is more useful than screenshots when diagnosing routing, content type, redirects, and cache behavior. Make one change at a time, deploy it, and repeat the same comparison.