Exact request agents, with the status explained.
See which search, AI, preview, and public-web crawler user-agent families Prerender Buddy detects and routes, with precise testing and verification status.
On this page
Overview
Crawler support
Audited Prerender Buddy edge and developer integration code recognizes crawler user-agent families and routes eligible requests to rendered HTML. This reference separates request detection, routing, controlled tests, and independent verification so those terms are not treated as the same proof.
Recognition makes a crawler eligible for the rendered-response path. It does not authenticate the operator or guarantee indexing, ranking, citation, or recommendation.
Registry reviewed 2026-07-13
29
Request-agent entries detected and routed in audited integration code.
Managed-DNS proxy detection was not separately audited in the available repositories.
5
Agents with a dated controlled crawler-style production test.
No operator verification claimed
A supplied user agent can be spoofed. Authentication requires operator-specific verification.
How to read the status
Four terms, four different claims.
Detected
The request user-agent matches a documented family in the audited integration code.
Routed
A detected request is eligible for the rendered-response path in that integration.
Tested
Prerender Buddy can send a controlled request using that user-agent token.
Verified
A dated test authenticates or otherwise proves the stated relationship. None is currently claimed here.
Search and AI agents
Keep each crawler purpose distinct.
Search crawlers, AI-search agents, model-development crawlers, and user-requested fetchers can come from the same operator but serve different purposes and use separate robots controls.
Googlebot
Search
Crawls pages for Google Search and related Google Search features.
Request-agent token or family
Googlebot
Detected and routed Controlled test: 2026-07-13 Not independently verified
Googlebot follows Google Search robots.txt controls. User-agent strings can be spoofed; Google documents IP and reverse-DNS verification.
The available test uses a supplied Googlebot user agent. It does not authenticate Google or prove indexing.
GoogleOther
Search
Supports Google product crawls that are not performed by a more specific Google crawler.
Request-agent token or family
GoogleOther Google-Other
Detected and routed No controlled test preset Not independently verified
Review the current Google crawler documentation before applying crawler-specific robots rules.
Recognition does not establish which Google product initiated a request or whether the request is authentic.
Microsoft
Bingbot
Search
Crawls pages for Microsoft Bing search services.
Request-agent token or family
bingbot
Detected and routed Controlled test: 2026-07-13 Not independently verified
Apply crawler-specific robots rules using Microsoft's current documented token.
The available test is a controlled user-agent request and does not authenticate Microsoft or prove indexing.
OpenAI
OAI-SearchBot
AI search
Crawls public web content for ChatGPT search features.
Request-agent token or family
OAI-SearchBot
Detected and routed No controlled test preset Not independently verified
OpenAI documents OAI-SearchBot as an automatic crawler governed by its robots.txt token and published IP ranges.
Routing makes content accessible to the request; it does not guarantee inclusion, citation, or ranking in ChatGPT search.
OpenAI
GPTBot
AI model development
Crawls content that may be used to improve and train OpenAI generative AI foundation models.
Request-agent token or family
GPTBot
Detected and routed Controlled test: 2026-07-13 Not independently verified
GPTBot has an independent robots.txt control from OAI-SearchBot.
The available test is a supplied GPTBot user agent and does not authenticate OpenAI or prove training use.
OpenAI
ChatGPT-User
User-requested fetch
Fetches pages for certain user-initiated actions in ChatGPT and Custom GPTs.
Request-agent token or family
ChatGPT-User
Detected and routed No controlled test preset Not independently verified
OpenAI states that robots.txt rules may not apply because these requests are initiated by users.
ChatGPT-User is not an automatic search crawler and is not used to determine ChatGPT search inclusion.
Anthropic
ClaudeBot
AI model development
Collects public web content that may contribute to Anthropic model development.
Request-agent token or family
ClaudeBot
Detected and routed Controlled test: 2026-07-13 Not independently verified
Anthropic documents a distinct robots.txt token for ClaudeBot.
The available test is a controlled user-agent request and does not authenticate Anthropic or prove training use.
Anthropic
Claude-SearchBot
AI search
Navigates the web to improve search result quality for Claude users.
Request-agent token or family
Claude-SearchBot
Detected and routed No controlled test preset Not independently verified
Anthropic documents Claude-SearchBot separately from ClaudeBot and Claude-User.
Detection and routing do not guarantee search inclusion, citation, or recommendation.
Anthropic
Claude-User
User-requested fetch
Fetches web content in response to user-initiated Claude requests.
Request-agent token or family
Claude-User
Detected and routed No controlled test preset Not independently verified
Anthropic provides a separate control for user-directed retrieval.
This is a user fetcher, not an automatic search or training crawler.
Anthropic
Anthropic legacy agent family
AI model development
Preserves recognition of an older Anthropic crawler identifier in accessible integration code.
Request-agent token or family
anthropic-ai
Detected and routed No controlled test preset Not independently verified
Use Anthropic's current ClaudeBot, Claude-User, and Claude-SearchBot controls for new configuration.
Current Anthropic documentation emphasizes the Claude agent names; this legacy identifier should not be promoted as a current crawler.
Current Anthropic crawler controls
Perplexity
PerplexityBot
AI search
Crawls content for discovery and links in Perplexity search results.
Request-agent token or family
PerplexityBot
Detected and routed Controlled test: 2026-07-13 Not independently verified
Perplexity publishes separate robots controls and IP ranges for PerplexityBot.
The available test is a controlled user-agent request and does not authenticate Perplexity or guarantee result inclusion.
Perplexity
Perplexity-User
User-requested fetch
Fetches pages in response to user actions within Perplexity.
Request-agent token or family
Perplexity-User
Detected and routed No controlled test preset Not independently verified
Perplexity states that this user-initiated fetcher generally ignores robots.txt and publishes separate IP ranges.
This is not an automatic indexing or model-training crawler.
Apple
Applebot
Search
Crawls and renders pages for Apple search experiences such as Spotlight, Siri, and Safari.
Request-agent token or family
Applebot
Detected and routed No controlled test preset Not independently verified
Applebot follows robots.txt for general search crawling and Apple documents IP and reverse-DNS verification.
Applebot-Extended is a separate control token that does not crawl pages and is not detected by the accessible request pattern.
DuckDuckGo
DuckDuckBot
Search
Crawls pages to support DuckDuckGo search results.
Request-agent token or family
DuckDuckBot
Detected and routed No controlled test preset Not independently verified
DuckDuckGo publishes its user-agent format and current IP list.
Detection does not authenticate DuckDuckGo or prove search inclusion.
DuckDuckGo
DuckAssistBot
AI search
Represents DuckDuckGo's AI-assisted retrieval agent family in accessible routing code.
Request-agent token or family
DuckAssistBot
Detected and routed No controlled test preset Not independently verified
No current primary operator reference was recorded during this model pass.
Keep this entry informational until its current purpose and operator guidance are confirmed from a primary source.
Yahoo
Yahoo Slurp
Search
Represents Yahoo search crawler requests recognized by accessible routing code.
Request-agent token or family
Slurp
Detected and routed No controlled test preset Not independently verified
No current primary operator reference was recorded during this model pass.
Recognition does not establish current crawl volume, authenticity, or search inclusion.
Baidu
Baiduspider
Search
Represents Baidu search crawler requests recognized by accessible routing code.
Request-agent token or family
Baiduspider
Detected and routed No controlled test preset Not independently verified
No current English-language primary operator reference was recorded during this model pass.
Recognition does not establish authenticity or indexing behavior.
Yandex
Yandex crawler family
Search
Represents Yandex search crawler requests recognized by accessible routing code.
Request-agent token or family
Yandex
Detected and routed No controlled test preset Not independently verified
The accessible detector uses a broad family token; primary documentation should be confirmed before publishing detailed subtypes.
The broad token is not precise enough to distinguish individual Yandex agents.
Amazon
Amazonbot
AI model development
Crawls public pages to improve Amazon products and services and may contribute to model training.
Request-agent token or family
Amazonbot
Detected and routed No controlled test preset Not independently verified
Amazon documents separate Amazonbot, Amzn-SearchBot, and Amzn-User agents; accessible routing code explicitly recognizes only Amazonbot.
Do not imply that Amzn-SearchBot or Amzn-User are routed until production detection explicitly includes them.
Preview and discovery agents
Readable HTML matters beyond search.
Link previews, advertising crawlers, public-web datasets, and archival agents can also request public pages. Detection does not guarantee how their downstream product will use or display the response.
Google advertising crawlers
Advertising
Fetches pages for Google advertising and publisher products.
Request-agent token or family
AdsBot-Google Mediapartners-Google
Detected and routed No controlled test preset Not independently verified
Special-case Google crawlers can have product-specific robots behavior.
These are not Google Search indexing crawlers and should not be described as AI-search agents.
Microsoft
BingPreview
Link preview
Fetches page information used by Microsoft preview experiences.
Request-agent token or family
BingPreview
Detected and routed No controlled test preset Not independently verified
Preview behavior should not be treated as equivalent to Bing search indexing.
Recognition does not prove that a preview was generated or displayed.
Meta
Meta link preview agents
Link preview
Fetches public page metadata and content for Meta link and sharing experiences.
Request-agent token or family
facebookexternalhit meta-externalagent
Detected and routed No controlled test preset Not independently verified
Preview and external-agent behavior can differ; confirm current Meta guidance before publishing crawler-specific robots advice.
A successful fetch does not guarantee the final preview format or distribution.
X
Twitterbot
Link preview
Fetches page metadata for link cards and previews on X.
Request-agent token or family
Twitterbot
Detected and routed No controlled test preset Not independently verified
No current primary operator crawler reference was recorded during this model pass.
Routing readable HTML does not guarantee a card will be generated or displayed.
LinkedInBot
Link preview
Fetches page metadata for LinkedIn sharing previews.
Request-agent token or family
LinkedInBot
Detected and routed No controlled test preset Not independently verified
No current primary operator crawler reference was recorded during this model pass.
Routing readable HTML does not guarantee a preview will be generated or refreshed.
Slack, Discord, Telegram, and WhatsApp
Messaging preview agents
Link preview
Fetches page metadata and content for messaging link previews.
Request-agent token or family
Slackbot Discordbot TelegramBot WhatsApp
Detected and routed No controlled test preset Not independently verified
Each platform has separate fetch and caching behavior; this entry records routing recognition only.
The grouped entry must not imply identical robots, caching, or preview behavior across platforms.
Pinterestbot
Link preview
Crawls public pages and refreshes metadata used by Pinterest Pins.
Request-agent token or family
Pinterestbot
Detected and routed No controlled test preset Not independently verified
Pinterest documents robots.txt controls and reverse-DNS verification guidance.
Routing does not guarantee distribution, preview presentation, or traffic.
Common Crawl
CCBot
Public web
Crawls public pages for the Common Crawl open web corpus.
Request-agent token or family
CCBot
Detected and routed No controlled test preset Not independently verified
Common Crawl documents robots.txt controls, dedicated IP ranges, and reverse-DNS verification.
Common Crawl inclusion does not imply use by any particular downstream search or AI product.
ByteDance
Bytespider
Public web
Represents ByteDance crawler requests recognized by accessible routing code.
Request-agent token or family
Bytespider
Detected and routed No controlled test preset Not independently verified
No current primary operator reference was recorded during this model pass.
Do not assign a narrower search or training purpose without a current primary source.
Internet Archive
Internet Archive crawler
Archive
Represents archival crawler requests recognized by accessible routing code.
Request-agent token or family
ia_archiver
Detected and routed No controlled test preset Not independently verified
No current primary operator crawler reference was recorded during this model pass.
Routing does not guarantee archival capture or retention.
Robots controls
A product token is not always a crawler.
Some names exist only to express preferences in robots.txt. They should not be presented as separately observable request agents.
Google-Extended
Robots control
A robots.txt product token controlling certain Gemini model training and grounding uses of content crawled by other Google agents.
Robots.txt token
Google-Extended
Control token only No controlled test preset Not independently verified
Google-Extended is a robots.txt control token and does not have a separate HTTP request user agent.
It must not be presented as a crawler that Prerender Buddy can observe or independently route.
Gemini
Covered through Google’s crawler ecosystem, not a separate Gemini crawler.
Google documents Google-Extended as a robots.txt product token for certain Gemini uses. It has no separate HTTP request user agent; crawling is performed with existing Google agents.
Grok
Product visibility without an unverified crawler claim.
The audited routing configuration does not contain a distinct Grok or xAI request-agent token, and no current official crawler identity was verified for this reference. Grok is therefore not labelled as detected or routed.
Status: crawler identity not independently verified
What this page does not promise
Detection does not authenticate who sent a request.
Routing does not guarantee crawling or indexing.
Indexing does not guarantee rankings or traffic.
AI crawler access does not guarantee citation or recommendation.
Robots.txt permission does not force an operator to retrieve a page.
Controlled tests simulate user agents; they do not impersonate verified operators.
Test what a crawler-style request receives.
Compare the raw response first. If the important content is missing, inspect the rendered result before choosing a setup path.