Audited Prerender Buddy edge and developer integration code recognizes crawler user-agent families and routes eligible requests to rendered HTML. This reference separates request detection, routing, controlled tests, and independent verification so those terms are not treated as the same proof.
Recognition makes a crawler eligible for the rendered-response path. It does not authenticate the operator or guarantee indexing, ranking, citation, or recommendation.
Registry reviewed 2026-07-13
29
Request-agent entries detected and routed in audited integration code.
Managed-DNS proxy detection was not separately audited in the available repositories.
5
Agents with a dated controlled crawler-style production test.
No operator verification claimed
A supplied user agent can be spoofed. Authentication requires operator-specific verification.
How to read the status
The request user-agent matches a documented family in the audited integration code.
A detected request is eligible for the rendered-response path in that integration.
Prerender Buddy can send a controlled request using that user-agent token.
A dated test authenticates or otherwise proves the stated relationship. None is currently claimed here.
Search and AI agents
Search crawlers, AI-search agents, model-development crawlers, and user-requested fetchers can come from the same operator but serve different purposes and use separate robots controls.
Crawls pages for Google Search and related Google Search features.
Request-agent token or family
GooglebotGooglebot follows Google Search robots.txt controls. User-agent strings can be spoofed; Google documents IP and reverse-DNS verification.
The available test uses a supplied Googlebot user agent. It does not authenticate Google or prove indexing.
Supports Google product crawls that are not performed by a more specific Google crawler.
Request-agent token or family
GoogleOtherGoogle-OtherReview the current Google crawler documentation before applying crawler-specific robots rules.
Recognition does not establish which Google product initiated a request or whether the request is authentic.
Microsoft
Crawls pages for Microsoft Bing search services.
Request-agent token or family
bingbotApply crawler-specific robots rules using Microsoft's current documented token.
The available test is a controlled user-agent request and does not authenticate Microsoft or prove indexing.
OpenAI
Crawls public web content for ChatGPT search features.
Request-agent token or family
OAI-SearchBotOpenAI documents OAI-SearchBot as an automatic crawler governed by its robots.txt token and published IP ranges.
Routing makes content accessible to the request; it does not guarantee inclusion, citation, or ranking in ChatGPT search.
OpenAI
Crawls content that may be used to improve and train OpenAI generative AI foundation models.
Request-agent token or family
GPTBotGPTBot has an independent robots.txt control from OAI-SearchBot.
The available test is a supplied GPTBot user agent and does not authenticate OpenAI or prove training use.
OpenAI
Fetches pages for certain user-initiated actions in ChatGPT and Custom GPTs.
Request-agent token or family
ChatGPT-UserOpenAI states that robots.txt rules may not apply because these requests are initiated by users.
ChatGPT-User is not an automatic search crawler and is not used to determine ChatGPT search inclusion.
Anthropic
Collects public web content that may contribute to Anthropic model development.
Request-agent token or family
ClaudeBotAnthropic documents a distinct robots.txt token for ClaudeBot.
The available test is a controlled user-agent request and does not authenticate Anthropic or prove training use.
Anthropic
Navigates the web to improve search result quality for Claude users.
Request-agent token or family
Claude-SearchBotAnthropic documents Claude-SearchBot separately from ClaudeBot and Claude-User.
Detection and routing do not guarantee search inclusion, citation, or recommendation.
Anthropic
Fetches web content in response to user-initiated Claude requests.
Request-agent token or family
Claude-UserAnthropic provides a separate control for user-directed retrieval.
This is a user fetcher, not an automatic search or training crawler.
Anthropic
Preserves recognition of an older Anthropic crawler identifier in accessible integration code.
Request-agent token or family
anthropic-aiUse Anthropic's current ClaudeBot, Claude-User, and Claude-SearchBot controls for new configuration.
Current Anthropic documentation emphasizes the Claude agent names; this legacy identifier should not be promoted as a current crawler.
Perplexity
Crawls content for discovery and links in Perplexity search results.
Request-agent token or family
PerplexityBotPerplexity publishes separate robots controls and IP ranges for PerplexityBot.
The available test is a controlled user-agent request and does not authenticate Perplexity or guarantee result inclusion.
Perplexity
Fetches pages in response to user actions within Perplexity.
Request-agent token or family
Perplexity-UserPerplexity states that this user-initiated fetcher generally ignores robots.txt and publishes separate IP ranges.
This is not an automatic indexing or model-training crawler.
Apple
Crawls and renders pages for Apple search experiences such as Spotlight, Siri, and Safari.
Request-agent token or family
ApplebotApplebot follows robots.txt for general search crawling and Apple documents IP and reverse-DNS verification.
Applebot-Extended is a separate control token that does not crawl pages and is not detected by the accessible request pattern.
DuckDuckGo
Crawls pages to support DuckDuckGo search results.
Request-agent token or family
DuckDuckBotDuckDuckGo publishes its user-agent format and current IP list.
Detection does not authenticate DuckDuckGo or prove search inclusion.
DuckDuckGo
Represents DuckDuckGo's AI-assisted retrieval agent family in accessible routing code.
Request-agent token or family
DuckAssistBotNo current primary operator reference was recorded during this model pass.
Keep this entry informational until its current purpose and operator guidance are confirmed from a primary source.
Yahoo
Represents Yahoo search crawler requests recognized by accessible routing code.
Request-agent token or family
SlurpNo current primary operator reference was recorded during this model pass.
Recognition does not establish current crawl volume, authenticity, or search inclusion.
Baidu
Represents Baidu search crawler requests recognized by accessible routing code.
Request-agent token or family
BaiduspiderNo current English-language primary operator reference was recorded during this model pass.
Recognition does not establish authenticity or indexing behavior.
Yandex
Represents Yandex search crawler requests recognized by accessible routing code.
Request-agent token or family
YandexThe accessible detector uses a broad family token; primary documentation should be confirmed before publishing detailed subtypes.
The broad token is not precise enough to distinguish individual Yandex agents.
Amazon
Crawls public pages to improve Amazon products and services and may contribute to model training.
Request-agent token or family
AmazonbotAmazon documents separate Amazonbot, Amzn-SearchBot, and Amzn-User agents; accessible routing code explicitly recognizes only Amazonbot.
Do not imply that Amzn-SearchBot or Amzn-User are routed until production detection explicitly includes them.
Preview and discovery agents
Link previews, advertising crawlers, public-web datasets, and archival agents can also request public pages. Detection does not guarantee how their downstream product will use or display the response.
Fetches pages for Google advertising and publisher products.
Request-agent token or family
AdsBot-GoogleMediapartners-GoogleSpecial-case Google crawlers can have product-specific robots behavior.
These are not Google Search indexing crawlers and should not be described as AI-search agents.
Microsoft
Fetches page information used by Microsoft preview experiences.
Request-agent token or family
BingPreviewPreview behavior should not be treated as equivalent to Bing search indexing.
Recognition does not prove that a preview was generated or displayed.
Meta
Fetches public page metadata and content for Meta link and sharing experiences.
Request-agent token or family
facebookexternalhitmeta-externalagentPreview and external-agent behavior can differ; confirm current Meta guidance before publishing crawler-specific robots advice.
A successful fetch does not guarantee the final preview format or distribution.
X
Fetches page metadata for link cards and previews on X.
Request-agent token or family
TwitterbotNo current primary operator crawler reference was recorded during this model pass.
Routing readable HTML does not guarantee a card will be generated or displayed.
Fetches page metadata for LinkedIn sharing previews.
Request-agent token or family
LinkedInBotNo current primary operator crawler reference was recorded during this model pass.
Routing readable HTML does not guarantee a preview will be generated or refreshed.
Slack, Discord, Telegram, and WhatsApp
Fetches page metadata and content for messaging link previews.
Request-agent token or family
SlackbotDiscordbotTelegramBotWhatsAppEach platform has separate fetch and caching behavior; this entry records routing recognition only.
The grouped entry must not imply identical robots, caching, or preview behavior across platforms.
Crawls public pages and refreshes metadata used by Pinterest Pins.
Request-agent token or family
PinterestbotPinterest documents robots.txt controls and reverse-DNS verification guidance.
Routing does not guarantee distribution, preview presentation, or traffic.
Common Crawl
Crawls public pages for the Common Crawl open web corpus.
Request-agent token or family
CCBotCommon Crawl documents robots.txt controls, dedicated IP ranges, and reverse-DNS verification.
Common Crawl inclusion does not imply use by any particular downstream search or AI product.
ByteDance
Represents ByteDance crawler requests recognized by accessible routing code.
Request-agent token or family
BytespiderNo current primary operator reference was recorded during this model pass.
Do not assign a narrower search or training purpose without a current primary source.
Internet Archive
Represents archival crawler requests recognized by accessible routing code.
Request-agent token or family
ia_archiverNo current primary operator crawler reference was recorded during this model pass.
Routing does not guarantee archival capture or retention.
Robots controls
Some names exist only to express preferences in robots.txt. They should not be presented as separately observable request agents.
A robots.txt product token controlling certain Gemini model training and grounding uses of content crawled by other Google agents.
Robots.txt token
Google-ExtendedGoogle-Extended is a robots.txt control token and does not have a separate HTTP request user agent.
It must not be presented as a crawler that Prerender Buddy can observe or independently route.
Gemini
Google documents Google-Extended as a robots.txt product token for certain Gemini uses. It has no separate HTTP request user agent; crawling is performed with existing Google agents.
Google’s documentationGrok
The audited routing configuration does not contain a distinct Grok or xAI request-agent token, and no current official crawler identity was verified for this reference. Grok is therefore not labelled as detected or routed.
Status: crawler identity not independently verified
Detection does not authenticate who sent a request.
Routing does not guarantee crawling or indexing.
Indexing does not guarantee rankings or traffic.
AI crawler access does not guarantee citation or recommendation.
Robots.txt permission does not force an operator to retrieve a page.
Controlled tests simulate user agents; they do not impersonate verified operators.
Compare the raw response first. If the important content is missing, inspect the rendered result before choosing a setup path.