One of the most critical parts of prerendering is bot detection. You need to serve prerendered HTML to search engine crawlers and AI bots, but serve the normal React app to humans. Get that distinction wrong and you either expose your site to bad crawlers or you slow down real users.
Today we are publishing the complete, unfiltered list of all 43+ bot signatures that Crawllify detects and serves prerendered content to. This is the reference list for exactly which crawlers we handle.
Why Bot Detection Matters
Not all bots are created equal. Some bots you want to serve prerendered content to — like Googlebot and GPTBot. Other bots you might want to block entirely — like aggressive scrapers or malicious crawlers.
The key is accuracy. A false positive means you are serving prerendered content to a human user, which hurts performance. A false negative means a legitimate crawler gets the empty React shell and you lose indexing opportunity.
Crawllify’s bot detection pipeline identifies crawlers by their User-Agent header and only serves prerendered HTML to the ones we know are legitimate and important. Human traffic always gets the normal React app, unaffected.
The Complete List: 43+ Bot Signatures Crawllify Detects
Search Engines (4)
- Googlebot (all variants: desktop, mobile, news)
- Bingbot (all variants: desktop, mobile, preview)
- Slurp (Yahoo crawler)
- DuckDuckGoBot
AI Search and AI Crawlers (6)
- GPTBot (OpenAI ChatGPT crawler)
- PerplexityBot (Perplexity crawler)
- Claude-Web (Anthropic crawler)
- Googlebot-Extended (Google’s AI training crawler)
- YouBot (You.com search crawler)
- FriendlyCrawler (Friendly search crawler)
Social Media and Link Preview Crawlers (8)
- facebookexternalhit (Facebook link preview)
- Twitterbot (Twitter/X link preview)
- LinkedInBot (LinkedIn link preview)
- WhatsApp (WhatsApp link preview)
- Slackbot (Slack link preview)
- Telegrambot (Telegram link preview)
- Discordbot (Discord link preview)
- Pinterestbot (Pinterest link preview)
News Aggregators and Content Crawlers (5)
- newsbot (Google News crawler)
- AppleNewsBot (Apple News crawler)
- RSSRadio
- Feedly crawler
- Inoreader bot
Accessibility and Usability Crawlers (4)
- AXECrawler (Accessibility evaluation)
- W3C_Validator (W3C validation)
- SiteImproveBot (Site quality checks)
- ContentKing
Analytics and Monitoring (5)
- Semrush bot
- AhrefsSiteAudit
- MJ12bot (Majestic crawler)
- DotBot (Moz crawler)
- SemrushBot-SA
Other Important Crawlers (11)
- Applebot (Apple Siri and search)
- bingbot (Microsoft search)
- YandexBot (Yandex search, Cyrillic web)
- Sogou (Chinese search engine)
- Baidu Spider (Chinese search engine)
- Bytespider (ByteDance crawler)
- QwantBot (Qwant search engine)
- nu
- MobileBot
- Exabot
- ia_archiver (Internet Archive crawler)
Why This Matters for Your Site
Every bot on this list represents a distribution channel for your content. Googlebot means Google search rankings. GPTBot means ChatGPT can recommend you. PerplexityBot means Perplexity users see your content. Twitterbot means your links look good when shared on Twitter.
Without proper bot detection, you are invisible to all of them.
With prerendering enabled on Crawllify, every single bot on this list gets the fully rendered HTML version of your site. Your content is visible and indexable by every major search engine, AI search engine, social platform and aggregator simultaneously.
How Crawllify Handles This Detection
Behind the scenes, Crawllify checks the User-Agent header of every request and matches it against our maintained list of known bot signatures. If the request matches, we serve prerendered HTML. If it does not match, we serve the normal React app.
This happens in milliseconds. Humans never notice. Bots get what they need. Performance stays fast.
We update this list regularly as new crawlers emerge and existing ones change their signatures. This is why using a prerendering service is better than building it yourself — maintaining bot detection is an ongoing task.
What About Unrecognised Bots?
Occasionally a new bot emerges or an existing bot changes its User-Agent. If Crawllify does not recognise it, we apply conservative defaults: we do not serve prerendered content unless we can verify the bot is legitimate.
This protects your site from serving prerendered content to fake bots or aggressive crawlers. If you encounter a legitimate bot that Crawllify is not detecting, let us know and we will add it to the list.
Comparing Bot Coverage
Other prerendering services often claim to cover ‘major crawlers.’ What does that actually mean?
- Major crawlers only = 4 to 6 bots (Google, Bing, maybe ChatGPT)
- Crawllify full coverage = 43+ bots across search, AI, social, analytics and more
The difference matters. A site invisible to ChatGPT but visible to Google is only half-solved. A site invisible to social preview crawlers has broken link previews on Twitter and LinkedIn.
Crawllify handles all of it.
FAQ
Do I need to configure which bots to detect?
No. Crawllify detects and serves all 43+ bots automatically. You do not configure anything. Just set up prerendering once and we handle the rest.
What if I want to block a specific bot?
Most of the bots on this list are legitimate and valuable. If you want to block a specific bot, you can add it to robots.txt and Crawllify will respect that directive. But we recommend allowing all of these — they only help your visibility.
Does detecting more bots slow down my site?
No. Bot detection happens before rendering. The lookup is instant. There is zero performance impact for humans or bots.
Why does Crawllify detect Chinese search engines like Baidu and Sogou?
If you serve a global audience, you want visibility in Chinese search engines. Even if you are not targeting China, these crawlers are fast and efficient. We detect them so you have the option. You can always block them in robots.txt if you prefer.
Will this list grow?
Yes. As new crawlers emerge and AI search engines evolve, we will add them to Crawllify’s detection pipeline. This list is current as of June 2026.
The Bottom Line
Comprehensive bot detection is not a nice-to-have. It is the foundation of effective prerendering. Crawllify detects and serves 43+ bots — covering every major search engine, AI crawler, social platform and analytics tool.
Set it up once. Get indexed everywhere. No configuration needed.
Get Your Site Visible to All 43+ Crawlers
Automatic bot detection. Prerendered HTML for search engines, AI crawlers, social platforms. Setup in minutes.
→ Get started at crawllify.com