AI crawler configuration:
robots.txt, Cloudflare, and the 14 bots that matter.

An AI crawler is a bot identified by a specific User-Agent string that fetches your web pages on behalf of an AI company. Some are training crawlers (they read your site to build language-model training datasets). Some are live-fetch crawlers (they read your site when a user asks a question, in real time). Some are both.

AI crawlers are NOT the same thing as Googlebot. Googlebot has been crawling the web since 1998 and feeds Google Search. The newer AI crawlers (GPTBot, ClaudeBot, PerplexityBot) emerged in 2023–2024 and feed AI answer engines. Allowing one does not imply allowing the other.

You configure access through three layers: robots.txt (the polite request layer), edge rules at your CDN (Cloudflare, Vercel, Fastly: the enforcement layer), and meta tags or HTTP headers (the page-level override). All three need to agree. If they disagree, the most restrictive wins, and the crawler usually loses.

These are the 14 User-Agent strings worth caring about in 2026. Newer crawlers will appear; this list will get updated. Citably tracks the canonical list internally and re-checks every Retainer client’s robots.txt against it quarterly.

“If you wouldn’t block Googlebot, you probably shouldn’t block GPTBot. Same buyer behavior, different decade.”

Copy this. Adjust the disallows for your routes (API, admin, build artifacts). Drop the Bytespider block if you serve a B2B audience where TikTok-adjacent AI training is sensitive. Everything else stays.

# robots.txt for B2B SaaS (Southeast US example)
# Hand-written. Updated 2026-06-03.

User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /_next/

# OpenAI
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Google
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

# Apple
User-agent: Applebot-Extended
Allow: /

# Microsoft
User-agent: Bingbot
Allow: /

# Common Crawl (most LLMs train on this)
User-agent: CCBot
Allow: /

# ByteDance — optional, depends on your audience tolerance for TikTok-adjacent AI
# User-agent: Bytespider
# Allow: /

Sitemap: https://www.citably.co/sitemap.xml

Put it at public/robots.txt (Next.js) or wherever your static-asset directory lives. Verify it’s served at https://yourdomain.com/robots.txt with content-type text/plain.

Cloudflare’s Bot Fight Mode and Super Bot Fight Mode sit in front of your origin. They use IP reputation, JS challenges, and machine learning to decide whether a request is legitimate. They do NOT ask robots.txt. They make their own decision.

We’ve seen ClaudeBot get 403’d from sites with a wide-open robots.txt because the Bot Fight rules didn’t recognize the User-Agent. The fix is to add an allow rule in Cloudflare for the 14 crawler User-Agents.

In Cloudflare: Dashboard → Security → WAF → Custom rules → Create. Match expression: (http.user_agent contains "GPTBot") or (http.user_agent contains "ChatGPT-User") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "Google-Extended") ... continue with the rest. Action: Skip → All remaining custom rules + Bot Fight Mode.

Cloudflare also ships a built-in “AI Scrapers and Crawlers” toggle (Security → Bots → Configure → AI Scrapers). Per the August 2025 default, new Cloudflare sites have AI training crawlers BLOCKED by default. If you signed up for Cloudflare after that date, your robots.txt is irrelevant until you flip that toggle.

Vercel ships a Firewall with bot-protection rules. Per default these are conservative: they allow Googlebot and Bingbot but flag aggressive crawl patterns. For AI crawlers, Vercel does not block by default in 2026, but its Web Application Firewall (paid tier) can be configured to challenge or rate-limit.

If you’re on Vercel Pro and have Firewall rules: Dashboard → Project → Firewall → Add rule. Pattern: request.headers["user-agent"] contains "GPTBot". Action: Allow. Repeat for the 14 User-Agents.

Citably’s own site runs on Vercel free tier with the default Firewall, and ships the robots.txt above. We verify weekly that all 14 crawlers get HTTP 200 on the homepage.

Three checks. One curl loop. No signups. Run quarterly.

# 1. Confirm robots.txt is served
curl -sI https://yourdomain.com/robots.txt | grep -iE "(HTTP|content-type)"

# 2. Confirm each crawler can fetch the homepage (200 expected)
for ua in "GPTBot/1.0" "ChatGPT-User/1.0" "OAI-SearchBot/1.0" \
          "ClaudeBot/1.0" "anthropic-ai/1.0" "Claude-Web/1.0" \
          "PerplexityBot/1.0" "Perplexity-User/1.0" \
          "Mozilla/5.0 (compatible; Google-Extended/1.0)" \
          "Mozilla/5.0 (compatible; Applebot-Extended)" \
          "CCBot/2.0" "Bingbot/2.0"; do
  status=$(curl -sLo /dev/null -w "%{http_code}" -A "$ua" https://yourdomain.com/)
  echo "$status  $ua"
done

# 3. Confirm robots.txt parsing
curl -s https://yourdomain.com/robots.txt | grep -iE "User-agent:|Allow:|Disallow:" | head -30

Every line in step 2 should print 200. If any print 403, 429, or 503, your edge layer (Cloudflare, Vercel Firewall, or your CMS) is overriding your robots.txt. Fix at the edge, not in robots.txt.

Copy the robots.txt above into your repo. Update the disallows for your specific routes. Run the curl loop. Add the Cloudflare allowlist if you’re on Cloudflare. Re-run the curl loop. Every line should be 200.

Then schedule it quarterly. New crawlers appear. Vercel and Cloudflare push edge updates that change defaults. The robots.txt that worked in March may be silently overridden in September. Five minutes a quarter prevents a six-month invisibility gap.

Or have Citably do it as part of a Sprint. We ship the robots.txt, the Cloudflare allowlist, the curl test, and a quarterly re-test calendar. Fixed scope, 14-day delivery, scoped on the call. See the offer.