ai-crawler-index 1.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e6a2d0be0c73f728584d587a1dbff0ed65acd7e8aa5174539896d521106a2d1e
+  data.tar.gz: d2d10ed8c72ab01f6c7183c5961b3bfe34868816e6c9fc00aa7cb1a53a1496e2
+SHA512:
+  metadata.gz: 4f0fcfe0511e4a2e8e7b40fa0eda74ba1fdce21b31c4213a6f52c4bd6a4c49a739afaca6b91fcca9d3b71890529b6a2fcb4c36629792b946072a4c9b50dc8151
+  data.tar.gz: e4d86259e669717e36ba023ed958cd1a9edf751562dc948bb521cae70df0c15bfbce7a0df34f8e46e6d37402b826f2ae87adb3dd8ec1bbae5d979913f08c15f1

data/LICENSE

@@ -0,0 +1,28 @@
+MIT License
+
+Copyright (c) 2026 Pathwren
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+---
+
+The crawler table shipped in data.json is dedicated to the public domain under
+CC0-1.0. Robots tokens, user-agent strings and documentation URLs are taken
+from each operator's own published documentation; the categories and the prose
+are the index's own.

data/README.md

@@ -0,0 +1,135 @@
+# ai-crawler-index
+
+Offline classifier for AI-crawler and bot user-agents, in Ruby. Give it a
+`User-Agent` string, get back what it is:
+
+```ruby
+require "ai_crawler_index"
+
+AiCrawlerIndex.ai_crawler?("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot")
+# => true
+
+AiCrawlerIndex.identify("Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)")
+# => {"slug"=>"chatgpt-user",
+#     "name"=>"ChatGPT-User",
+#     "operator"=>"OpenAI",
+#     "category"=>"user-fetch",
+#     "verification_method"=>"published-ranges", ...}
+```
+
+**Zero dependencies, stdlib only, no network.** The table of 56 crawlers
+is bundled as one 45.4 KB JSON file and every method is a pure function of
+the string you pass in, so it runs inside a Rack request, a Sidekiq job, a
+Lambda, or on a machine with no route to the internet at all. One HTTP call per
+request to classify a user-agent is not an acceptable design; this exists so you
+do not have to make one.
+
+## Install
+
+```sh
+gem install ai-crawler-index
+```
+
+or in a `Gemfile`:
+
+```ruby
+gem "ai-crawler-index"
+```
+
+Ruby >= 2.6. No dependencies.
+
+## API
+
+| Method | Returns |
+| --- | --- |
+| `AiCrawlerIndex.ai_crawler?(ua)` | `true` for AI training, AI search, user-triggered fetch and dataset crawlers |
+| `AiCrawlerIndex.identify(ua)` | the full record, or `nil` — `{name, operator, category, verification_method, ...}` |
+| `AiCrawlerIndex.crawler?(ua)` | `true` for **any** known automated client, AI or not (search engines, SEO, archives, tools) |
+| `AiCrawlerIndex.category_of(ua)` / `.operator_of(ua)` | `"ai-training"` / `"OpenAI"`, or `nil` |
+| `AiCrawlerIndex.match_all(ua)` | every matching record, most specific first |
+| `AiCrawlerIndex.list(category: nil)` | the whole table, or one category of it |
+| `AiCrawlerIndex.get(slug)` | one record by slug, e.g. `get("gptbot")` |
+| `AiCrawlerIndex.robots_txt(stance)` | a `robots.txt` body for `"block-ai-training"`, `"block-all-ai"` or `"block-none"` |
+| `PATTERNS`, `CATEGORIES`, `META` | compiled alternations, category descriptions, and what this snapshot is |
+| `AiCrawlerIndex.refresh` | **optional, the only network path** — fetches the current table and returns a new object |
+
+In Rack, without middleware and without a request to anywhere:
+
+```ruby
+ua = env["HTTP_USER_AGENT"]
+if AiCrawlerIndex.category_of(ua) == "ai-training"
+  return [403, { "content-type" => "text/plain" }, ["no"]]
+end
+```
+
+There is a command line too, installed as `ai-crawler-index`:
+
+```sh
+ai-crawler-index "Mozilla/5.0 (compatible; GPTBot/1.2)"   # JSON verdict, exit 0 if known
+ai-crawler-index --robots block-ai-training               # a robots.txt on stdout
+ai-crawler-index --list ai-search                         # one category of the table
+```
+
+Matching is case-insensitive substring, most specific token first, so
+`Googlebot-Image` beats `Googlebot` and `Claude-SearchBot` beats `ClaudeBot`.
+Unknown, empty and non-string input returns `nil` / `false` and never raises.
+
+## A user-agent is a claim, not evidence
+
+This gem tells you what a client *says* it is. Whether the claim is true is a
+question about its IP address, and this gem deliberately does not pretend to
+answer it offline: IP ranges rotate, and a stale range list baked into a package
+is worse than no check at all. For operators that publish ranges, each record
+carries an `ip_ranges` URL — check the address before you act on the name.
+
+### The categories
+
+| Category | What it means | What blocking it costs you |
+| --- | --- | --- |
+| `ai-training` | bulk collection for training a model | your pages are excluded from future training sets; nothing a user sees today changes |
+| `ai-search` | builds the index an assistant answers and cites from | this is the class that sends you traffic; blocking it is the expensive mistake |
+| `user-fetch` | one page, right now, because a person asked for it | a visible error for a real reader |
+| `dataset` | crawls into a published or resold corpus | highest leverage per block, longest delay before any effect |
+| `search`, `seo`, `archive`, `tool`, `preview` | classic crawlers | ordinary search and tooling consequences |
+
+`verification_method` tells you how far a claim can be trusted:
+`published-ranges` (the operator publishes the IP ranges it crawls from),
+`reverse-dns`, or `none`.
+
+## The data
+
+Generated 2026-09-01T21:09:27+00:00 from the [AI Crawler Index](https://www.pathwren.workers.dev/c/rubygems-registry/) —
+56 crawlers from 30 operators, each reviewed against its
+operator's own published documentation. Robots tokens, user-agent strings and
+documentation URLs come from those operator pages (cited per record); the
+categories and the prose are the index's own.
+
+- Source of truth: `https://www.pathwren.workers.dev/c/rubygems-registry/data/agents.json` — regenerated every six hours.
+- This bundle is a **snapshot of that file taken at 2026-09-01T21:09:27+00:00**, not a
+  live feed. Crawlers appear and change names; a gem published last month cannot
+  know about a bot announced last week.
+- Data licence: **CC0-1.0**. Code licence: MIT.
+- Version scheme: the patch number moves when the table changes, the minor
+  number when a crawler is added or removed, the major number only for an API
+  change.
+
+If a bot is missing, wrong or misfiled, corrections are welcome and get applied
+to the index — it is a public reference and it is meant to be argued with.
+
+### Staying current without upgrading
+
+```ruby
+live = AiCrawlerIndex.refresh   # one HTTPS GET, explicit, never automatic
+live.identify(ua)
+```
+
+`refresh` is the only method that touches the network, it is never called for
+you, and it returns a *new* object rather than mutating the bundled table.
+Everything else works with the network unplugged.
+
+## What this is
+
+[Pathwren](https://www.pathwren.workers.dev/c/rubygems-registry/) is an independent, non-commercial project. It is run by
+automation and says so wherever it introduces itself; it is not affiliated with
+any of the operators listed, and it sells nothing. The index behind this gem is
+static files, CC0, no signup: JSON, CSV, robots.txt and regex at https://www.pathwren.workers.dev/c/rubygems-registry/.

data/exe/ai-crawler-index

@@ -0,0 +1,59 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# ai-crawler-index — command line.
+#
+#   ai-crawler-index "Mozilla/5.0 (compatible; GPTBot/1.2)"   # JSON verdict, exit 0 if known
+#   ai-crawler-index --robots block-ai-training               # a robots.txt on stdout
+#   ai-crawler-index --list ai-search                         # the table, or one category of it
+#   ai-crawler-index --version
+
+require "json"
+require "ai_crawler_index"
+
+def usage
+  warn <<~TEXT
+    ai-crawler-index #{AiCrawlerIndex::VERSION} — offline AI-crawler user-agent classifier
+
+      ai-crawler-index "<user-agent>"        classify one user-agent (JSON; exit 1 if unknown)
+      ai-crawler-index --robots <stance>     robots.txt for block-ai-training | block-all-ai | block-none
+      ai-crawler-index --list [category]     the whole table, or one category, as JSON
+      ai-crawler-index --meta                what this snapshot is
+      ai-crawler-index --version
+
+    #{AiCrawlerIndex::META['source']}
+  TEXT
+end
+
+arg = ARGV[0]
+
+case arg
+when nil, "-h", "--help"
+  usage
+  exit 2
+when "--version", "-V"
+  puts AiCrawlerIndex::VERSION
+when "--meta"
+  puts JSON.pretty_generate(AiCrawlerIndex::META)
+when "--robots"
+  begin
+    puts AiCrawlerIndex.robots_txt(ARGV[1] || "block-ai-training")
+  rescue ArgumentError => e
+    warn e.message
+    exit 2
+  end
+when "--list"
+  rows = AiCrawlerIndex.list(category: ARGV[1])
+  if rows.empty?
+    warn "no crawlers in category #{ARGV[1].inspect}; known: #{AiCrawlerIndex::CATEGORIES.keys.join(', ')}"
+    exit 1
+  end
+  puts JSON.pretty_generate(rows)
+else
+  hit = AiCrawlerIndex.identify(arg)
+  if hit.nil?
+    puts JSON.generate({ "matched" => false, "ai_crawler" => false, "ua" => arg })
+    exit 1
+  end
+  puts JSON.pretty_generate(hit.merge("ai_crawler" => AiCrawlerIndex.ai_crawler?(arg)))
+end

data/lib/ai-crawler-index.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+# The gem is named ai-crawler-index; some people will type the gem name and some
+# the module path. Both work, and they are the same object.
+require_relative "ai_crawler_index"

data/lib/ai_crawler_index/data.json

@@ -0,0 +1 @@
+{"version":"1.1.0","generated_at":"2026-09-01T21:09:27+00:00","source":"AI Crawler Index — https://www.pathwren.workers.dev — independent, non-commercial; data CC0-1.0","data_url":"https://www.pathwren.workers.dev/c/rubygems-registry/data/agents.json","window":"Table reviewed 2026-09-01; every record checked against its operator's own published documentation. IP-range mirrors behind the index refresh every six hours; this bundle is a snapshot, not a live feed.","license":{"code":"MIT","data":"CC0-1.0"},"categories":{"ai-training":{"label":"AI training crawlers","description":"Collect pages in bulk so that a model can be trained or fine-tuned on them. Blocking these removes you from future training sets and changes nothing a user sees today."},"ai-search":{"label":"AI search crawlers","description":"Build the retrieval index an assistant answers and cites from. These are the crawlers that send you traffic; blocking them is the expensive mistake in this space."},"user-fetch":{"label":"User-triggered fetchers","description":"Fetch one page because a person asked for it, right then. One human intent, one request. Blocking them produces a visible error for a real reader."},"dataset":{"label":"Corpus and dataset builders","description":"Crawl the web into a published or resold dataset that other people train on. Highest leverage per block, longest delay before any effect."},"search":{"label":"Search engines","description":"Classic index-and-rank crawlers. Several also feed their operator's generative answers, which is why the AI opt-out for Google and Apple is a token rather than a block."},"seo":{"label":"SEO and backlink crawlers","description":"Commercial link-graph tooling. No user-facing effect either way, and usually a large share of your bot bandwidth."},"archive":{"label":"Archivers","description":"Preservation crawlers. Their output is public and permanent, which makes them a separate decision from the AI one."},"preview":{"label":"Link preview fetchers","description":"Read your Open Graph tags when someone shares a link. Blocking these is almost always an accident."},"tool":{"label":"Tools and frameworks","description":"Not operators: crawling software anyone can run. The party behind the request is unknown, so treat them as a rate-limit question rather than a consent question."}},"ai_categories":["ai-training","ai-search","user-fetch","dataset"],"regex":{"all":"(AhrefsBot|AI2Bot|Ai2Bot\\-Dolma|Amazonbot|anthropic\\-ai|Applebot|archive\\.org_bot|Baiduspider|bingbot|Bytespider|CCBot|ChatGPT\\-User|Claude\\-SearchBot|Claude\\-User|Claude\\-Web|ClaudeBot|cohere\\-ai|cohere\\-training\\-data\\-crawler|Diffbot|DuckAssistBot|DuckDuckBot|FacebookBot|facebookexternalhit|FirecrawlAgent|Google\\-CloudVertexBot|Google\\-InspectionTool|Googlebot|Googlebot\\-Image|Googlebot\\-News|GoogleOther|GPTBot|ia_archiver|ImagesiftBot|img2dataset|meta\\-externalagent|meta\\-externalfetcher|MistralAI\\-User|OAI\\-SearchBot|omgili|omgilibot|Perplexity\\-User|PerplexityBot|PetalBot|Scrapy|SemrushBot|SemrushBot\\-OCOB|SeznamBot|Storebot\\-Google|TikTokSpider|Timpibot|Webzio\\-Extended|YandexBot|Yeti|YouBot)","ai_only":"(AI2Bot|Ai2Bot\\-Dolma|Amazonbot|anthropic\\-ai|Bytespider|CCBot|ChatGPT\\-User|Claude\\-SearchBot|Claude\\-User|Claude\\-Web|ClaudeBot|cohere\\-ai|cohere\\-training\\-data\\-crawler|Diffbot|DuckAssistBot|FacebookBot|Google\\-CloudVertexBot|GoogleOther|GPTBot|ImagesiftBot|img2dataset|meta\\-externalagent|meta\\-externalfetcher|MistralAI\\-User|OAI\\-SearchBot|omgili|omgilibot|Perplexity\\-User|PerplexityBot|SemrushBot\\-OCOB|TikTokSpider|Webzio\\-Extended|YouBot)","by_category":{"ai-training":"(anthropic\\-ai|Bytespider|ClaudeBot|cohere\\-training\\-data\\-crawler|FacebookBot|GoogleOther|GPTBot|meta\\-externalagent|SemrushBot\\-OCOB|TikTokSpider|Webzio\\-Extended)","ai-search":"(Amazonbot|Claude\\-SearchBot|Claude\\-Web|DuckAssistBot|Google\\-CloudVertexBot|OAI\\-SearchBot|PerplexityBot|YouBot)","user-fetch":"(ChatGPT\\-User|Claude\\-User|cohere\\-ai|meta\\-externalfetcher|MistralAI\\-User|Perplexity\\-User)","search":"(Applebot|Baiduspider|bingbot|DuckDuckBot|Googlebot|Googlebot\\-Image|Googlebot\\-News|PetalBot|SeznamBot|Storebot\\-Google|Timpibot|YandexBot|Yeti)","tool":"(FirecrawlAgent|Google\\-InspectionTool|Scrapy)","dataset":"(AI2Bot|Ai2Bot\\-Dolma|CCBot|Diffbot|ImagesiftBot|img2dataset|omgili|omgilibot)","preview":"(facebookexternalhit)","seo":"(AhrefsBot|SemrushBot)","archive":"(archive\\.org_bot|ia_archiver)"}},"crawlers":[{"slug":"ahrefsbot","name":"AhrefsBot","operator":"Ahrefs","category":"seo","category_label":"SEO and backlink crawlers","ua":"AhrefsBot","robots_token":"AhrefsBot","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://ahrefs.com/robot","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/ahrefsbot.html","what_it_is":"Ahrefs' backlink crawler, and one of the largest non-search crawlers on the web by request volume.","cost_of_blocking":"No user-facing effect. Ahrefs honours Crawl-delay, so rate-limiting is usually better than blocking."},{"slug":"ai2bot","name":"AI2Bot","operator":"Allen Institute for AI","category":"dataset","category_label":"Corpus and dataset builders","ua":"AI2Bot","robots_token":"AI2Bot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://allenai.org/crawler","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/ai2bot.html","what_it_is":"The Allen Institute's crawler, gathering pages for open research corpora such as Dolma that underpin fully open models like OLMo.","cost_of_blocking":"Excluded from open research datasets. Worth a deliberate decision: this is the category where 'blocking AI' also blocks the open, auditable end of it."},{"slug":"ai2bot-dolma","name":"Ai2Bot-Dolma","operator":"Allen Institute for AI","category":"dataset","category_label":"Corpus and dataset builders","ua":"Ai2Bot-Dolma","robots_token":"Ai2Bot-Dolma","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://allenai.org/crawler","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/ai2bot-dolma.html","what_it_is":"The variant of AI2's crawler named for the Dolma corpus specifically.","cost_of_blocking":"Same as AI2Bot: exclusion from an open, published training corpus."},{"slug":"amazonbot","name":"Amazonbot","operator":"Amazon","category":"ai-search","category_label":"AI search crawlers","ua":"Amazonbot","robots_token":"Amazonbot","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://developer.amazon.com/amazonbot","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/amazonbot.html","what_it_is":"Amazon's crawler, feeding Alexa's ability to answer questions from the web and Amazon's own search and assistant products.","cost_of_blocking":"Alexa and Amazon's assistants stop answering from your pages. Verify with reverse DNS to crawl.amazonbot.amazon before trusting the user-agent."},{"slug":"anthropic-ai","name":"anthropic-ai","operator":"Anthropic","category":"ai-training","category_label":"AI training crawlers","ua":"anthropic-ai","robots_token":"anthropic-ai","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"n-a","docs":"https://support.anthropic.com/en/articles/8896518","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/anthropic-ai.html","what_it_is":"A legacy robots.txt token from before Anthropic consolidated on ClaudeBot. It is still widely present in robots.txt files and costs nothing to keep, but it is a control token rather than a bot you will see in logs.","cost_of_blocking":"None. Nothing crawls under this name today; keeping the rule is harmless insurance."},{"slug":"applebot","name":"Applebot","operator":"Apple","category":"search","category_label":"Search engines","ua":"Applebot","robots_token":"Applebot","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://support.apple.com/en-us/119829","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/apple-applebot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/applebot.html","what_it_is":"Powers Siri, Spotlight and Safari suggestions. Blocking it is a search decision, not an AI decision — the AI decision has its own token.","cost_of_blocking":"You disappear from Siri, Spotlight and Safari search suggestions across Apple's install base."},{"slug":"applebot-extended","name":"Applebot-Extended","operator":"Apple","category":"ai-training","category_label":"AI training crawlers","ua":"(control token only — no crawler)","robots_token":"Applebot-Extended","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"n-a","docs":"https://support.apple.com/en-us/119829","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/applebot-extended.html","what_it_is":"Apple's counterpart to Google-Extended: a robots.txt token that withdraws consent for Apple Intelligence and Apple foundation-model training, without touching Applebot's search crawl.","cost_of_blocking":"Excluded from Apple Intelligence training. Siri, Spotlight and Safari suggestions are unaffected."},{"slug":"archive-org-bot","name":"archive.org_bot","operator":"Internet Archive","category":"archive","category_label":"Archivers","ua":"archive.org_bot","robots_token":"archive.org_bot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://archive.org/details/archive.org_bot","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/archive-org-bot.html","what_it_is":"The Wayback Machine's crawler. Preservation rather than AI, but it lands in the same 'is this bot welcome' decision and its output is a public corpus.","cost_of_blocking":"Your site stops being preserved. When it dies, it is gone. Consider this one separately from the AI question."},{"slug":"baiduspider","name":"Baiduspider","operator":"Baidu","category":"search","category_label":"Search engines","ua":"Baiduspider","robots_token":"Baiduspider","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://help.baidu.com/question?prod_id=99&class=0&id=3001","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/baiduspider.html","what_it_is":"Baidu's search crawler, and the ingest path for Baidu's Ernie-backed answers.","cost_of_blocking":"Removal from Baidu Search, which matters only if you want Chinese-language traffic."},{"slug":"bingbot","name":"bingbot","operator":"Microsoft","category":"search","category_label":"Search engines","ua":"bingbot","robots_token":"bingbot","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/bing-bingbot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/bingbot.html","what_it_is":"Bing's only crawler, and therefore also the crawler behind Microsoft Copilot's grounding. Microsoft's documented way to keep search indexing while refusing generative reuse is the nocache / noarchive robots meta directive, not a separate user-agent.","cost_of_blocking":"Very high and very wide: Bing, Copilot, DuckDuckGo and several assistants that resell Bing's index all lose you at once. Use nocache/noarchive rather than blocking."},{"slug":"bytespider","name":"Bytespider","operator":"ByteDance","category":"ai-training","category_label":"AI training crawlers","ua":"Bytespider","robots_token":"Bytespider","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"disputed","docs":"https://www.bytespider.net/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/bytespider.html","what_it_is":"ByteDance's crawler, associated with training data collection for Doubao and related models. Repeatedly reported by CDNs and site operators as the highest-volume AI crawler on the web and as inconsistent about robots.txt.","cost_of_blocking":"Little to lose. If you want it gone, expect to block by user-agent at the edge rather than to ask politely in robots.txt."},{"slug":"ccbot","name":"CCBot","operator":"Common Crawl","category":"dataset","category_label":"Corpus and dataset builders","ua":"CCBot","robots_token":"CCBot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://commoncrawl.org/faq","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/ccbot.html","what_it_is":"Common Crawl's corpus builder. It trains nothing itself, but its archive is an input to most open and many closed LLM training sets, which makes it the highest-leverage single entry on this list.","cost_of_blocking":"Future Common Crawl snapshots exclude you, so downstream training sets lose you too — but only going forward. Existing snapshots are permanent and blocking today does not retract them."},{"slug":"chatgpt-user","name":"ChatGPT-User","operator":"OpenAI","category":"user-fetch","category_label":"User-triggered fetchers","ua":"ChatGPT-User","robots_token":"ChatGPT-User","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://platform.openai.com/docs/bots","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/openai-chatgpt-user.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/chatgpt-user.html","what_it_is":"Fetches a single page at the moment a user or a ChatGPT agent asks for it — a pasted link, a browsing step, an Operator task. One human intent, one request. OpenAI states these fetches are not used for training.","cost_of_blocking":"ChatGPT cannot open your pages when a user explicitly asks it to. The user sees a fetch failure. This is usually the last bot anyone means to block."},{"slug":"claude-searchbot","name":"Claude-SearchBot","operator":"Anthropic","category":"ai-search","category_label":"AI search crawlers","ua":"Claude-SearchBot","robots_token":"Claude-SearchBot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://support.anthropic.com/en/articles/8896518","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/claude-searchbot.html","what_it_is":"Indexes pages so Claude's web search can find and cite them. Separate token from the training crawler, so search visibility and training consent are independent decisions.","cost_of_blocking":"You stop appearing in Claude's search results and citations."},{"slug":"claude-user","name":"Claude-User","operator":"Anthropic","category":"user-fetch","category_label":"User-triggered fetchers","ua":"Claude-User","robots_token":"Claude-User","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://support.anthropic.com/en/articles/8896518","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/claude-user.html","what_it_is":"Fetches a page because a Claude user asked Claude to read it, at that moment.","cost_of_blocking":"Claude reports a fetch failure to a user who asked for your page by name."},{"slug":"claude-web","name":"Claude-Web","operator":"Anthropic","category":"ai-search","category_label":"AI search crawlers","ua":"Claude-Web","robots_token":"Claude-Web","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"n-a","docs":"https://support.anthropic.com/en/articles/8896518","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/claude-web.html","what_it_is":"An earlier Anthropic token for user-facing web access, superseded by Claude-User and Claude-SearchBot. Kept here because it appears in most published robots.txt templates.","cost_of_blocking":"None in practice. Retain the rule; expect no traffic."},{"slug":"claudebot","name":"ClaudeBot","operator":"Anthropic","category":"ai-training","category_label":"AI training crawlers","ua":"ClaudeBot","robots_token":"ClaudeBot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://support.anthropic.com/en/articles/8896518","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/claudebot.html","what_it_is":"Anthropic's bulk crawler, gathering pages that may be used to train Claude models.","cost_of_blocking":"Content excluded from training data for future Claude models. No effect on Claude's ability to fetch a link a user gives it."},{"slug":"cohere-ai","name":"cohere-ai","operator":"Cohere","category":"user-fetch","category_label":"User-triggered fetchers","ua":"cohere-ai","robots_token":"cohere-ai","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://cohere.com/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/cohere-ai.html","what_it_is":"Cohere's fetcher, used when its assistant products need a page.","cost_of_blocking":"Cohere-powered assistants cannot read your pages on request."},{"slug":"cohere-training-data-crawler","name":"cohere-training-data-crawler","operator":"Cohere","category":"ai-training","category_label":"AI training crawlers","ua":"cohere-training-data-crawler","robots_token":"cohere-training-data-crawler","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://cohere.com/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/cohere-training-data-crawler.html","what_it_is":"Cohere's separately-named bulk crawler for model training data, split out so consent for training and consent for retrieval can differ.","cost_of_blocking":"Excluded from Cohere model training."},{"slug":"diffbot","name":"Diffbot","operator":"Diffbot","category":"dataset","category_label":"Corpus and dataset builders","ua":"Diffbot","robots_token":"Diffbot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://docs.diffbot.com/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/diffbot.html","what_it_is":"Extracts structured records from pages to build a commercial knowledge graph that is resold and used for retrieval and training.","cost_of_blocking":"Your facts stop entering a widely-licensed knowledge graph. Whether that is a loss depends on whether you want to be a machine-readable entity."},{"slug":"duckassistbot","name":"DuckAssistBot","operator":"DuckDuckGo","category":"ai-search","category_label":"AI search crawlers","ua":"DuckAssistBot","robots_token":"DuckAssistBot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/duckassistbot.html","what_it_is":"Fetches pages so DuckAssist can generate and cite answers inside DuckDuckGo.","cost_of_blocking":"No DuckAssist answers or citations from your site. Ordinary DuckDuckGo results are unaffected."},{"slug":"duckduckbot","name":"DuckDuckBot","operator":"DuckDuckGo","category":"search","category_label":"Search engines","ua":"DuckDuckBot","robots_token":"DuckDuckBot","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/duckduckgo-duckduckbot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/duckduckbot.html","what_it_is":"DuckDuckGo's own crawler. Note that the bulk of DuckDuckGo's web results come from Bing, so blocking bingbot removes you from DuckDuckGo whether or not you allow this one.","cost_of_blocking":"Limited on its own; the real DuckDuckGo lever is bingbot."},{"slug":"facebookbot","name":"FacebookBot","operator":"Meta","category":"ai-training","category_label":"AI training crawlers","ua":"FacebookBot","robots_token":"FacebookBot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/facebookbot.html","what_it_is":"Meta's older speech- and language-corpus crawler, largely superseded by meta-externalagent but still listed as a valid robots token.","cost_of_blocking":"Negligible today. Keep the rule; expect little traffic."},{"slug":"facebookexternalhit","name":"facebookexternalhit","operator":"Meta","category":"preview","category_label":"Link preview fetchers","ua":"facebookexternalhit","robots_token":"facebookexternalhit","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/facebookexternalhit.html","what_it_is":"The link unfurler: it reads your Open Graph tags when somebody shares your URL on a Meta property.","cost_of_blocking":"Severe and usually accidental. Your links share as bare grey boxes with no title, image or description across Facebook, Instagram, Messenger and WhatsApp. Almost nobody means to block this."},{"slug":"firecrawlagent","name":"FirecrawlAgent","operator":"Firecrawl","category":"tool","category_label":"Tools and frameworks","ua":"FirecrawlAgent","robots_token":"FirecrawlAgent","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://docs.firecrawl.dev/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/firecrawlagent.html","what_it_is":"A hosted scrape-to-markdown service that LLM applications call to read pages. The requester is whoever is building on it, not Firecrawl itself, so volume and intent vary wildly.","cost_of_blocking":"Applications built on Firecrawl cannot read your pages. This is increasingly how agents fetch the web, so it is a bigger block than its name suggests."},{"slug":"google-cloudvertexbot","name":"Google-CloudVertexBot","operator":"Google","category":"ai-search","category_label":"AI search crawlers","ua":"Google-CloudVertexBot","robots_token":"Google-CloudVertexBot","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/google-special.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/google-cloudvertexbot.html","what_it_is":"Crawls a site on behalf of a Vertex AI Agent Builder customer who is building an agent over that site. It only visits sites the customer has asked it to.","cost_of_blocking":"Third parties can no longer build Vertex AI agents that read your site. Irrelevant to Google Search."},{"slug":"google-extended","name":"Google-Extended","operator":"Google","category":"ai-training","category_label":"AI training crawlers","ua":"(control token only — no crawler)","robots_token":"Google-Extended","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"n-a","docs":"https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/google-extended.html","what_it_is":"Not a crawler. A robots.txt token that tells Google whether pages Googlebot already fetched may be used to train and ground Gemini. You will never see it in an access log; disallowing it changes what Google does with content it fetched under a different name.","cost_of_blocking":"You are excluded from Gemini grounding and Gemini training. Google Search ranking and indexing are explicitly unaffected. This is the cleanest 'no training, keep my search traffic' lever that exists."},{"slug":"google-inspectiontool","name":"Google-InspectionTool","operator":"Google","category":"tool","category_label":"Tools and frameworks","ua":"Google-InspectionTool","robots_token":"Google-InspectionTool","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/google-special.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/google-inspectiontool.html","what_it_is":"The fetcher behind Search Console's URL Inspection and the Rich Results Test. It runs when a site owner clicks a button.","cost_of_blocking":"Your own Search Console live tests stop working. Blocking this only hurts you."},{"slug":"googlebot","name":"Googlebot","operator":"Google","category":"search","category_label":"Search engines","ua":"Googlebot","robots_token":"Googlebot","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/google-googlebot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/googlebot.html","what_it_is":"The classic search crawler. It is also the crawler behind AI Overviews: Google does not run a separate bot for them, which is why the only AI opt-out is the Google-Extended token and not a Googlebot block.","cost_of_blocking":"Total. You leave Google Search. Never block this to avoid AI use; use Google-Extended instead."},{"slug":"googlebot-image","name":"Googlebot-Image","operator":"Google","category":"search","category_label":"Search engines","ua":"Googlebot-Image","robots_token":"Googlebot-Image","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/google-googlebot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/googlebot-image.html","what_it_is":"Image indexing for Google Images. A separate token so you can leave images out of search without leaving search.","cost_of_blocking":"Your images stop appearing in Google Images."},{"slug":"googlebot-news","name":"Googlebot-News","operator":"Google","category":"search","category_label":"Search engines","ua":"Googlebot-News","robots_token":"Googlebot-News","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/google-googlebot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/googlebot-news.html","what_it_is":"A robots.txt token controlling inclusion in Google News. It does not have its own user-agent string; the fetch arrives as Googlebot.","cost_of_blocking":"Removal from Google News, with normal Search unaffected."},{"slug":"googleother","name":"GoogleOther","operator":"Google","category":"ai-training","category_label":"AI training crawlers","ua":"GoogleOther","robots_token":"GoogleOther","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/google-special.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/googleother.html","what_it_is":"A generic fetcher used by Google product teams for one-off crawls and research, including data collection that does not belong to Search.","cost_of_blocking":"No effect on Search indexing. Blocks internal Google research and product fetches."},{"slug":"gptbot","name":"GPTBot","operator":"OpenAI","category":"ai-training","category_label":"AI training crawlers","ua":"GPTBot","robots_token":"GPTBot","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://platform.openai.com/docs/bots","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/openai-gptbot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/gptbot.html","what_it_is":"OpenAI's bulk crawler. Pages it fetches may be used to train future OpenAI foundation models. It is not the bot that puts you in ChatGPT's search results, and blocking it does not remove you from them.","cost_of_blocking":"Your content is excluded from training data for future OpenAI models. No effect on ChatGPT search visibility, on citations, or on links a user pastes into ChatGPT."},{"slug":"ia-archiver","name":"ia_archiver","operator":"Internet Archive","category":"archive","category_label":"Archivers","ua":"ia_archiver","robots_token":"ia_archiver","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://archive.org/details/archive.org_bot","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/ia-archiver.html","what_it_is":"The legacy Alexa/Internet Archive token, still present in most robots.txt files and still occasionally honoured.","cost_of_blocking":"Negligible today; retain for tidiness."},{"slug":"imagesiftbot","name":"ImagesiftBot","operator":"Hive AI","category":"dataset","category_label":"Corpus and dataset builders","ua":"ImagesiftBot","robots_token":"ImagesiftBot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://imagesift.com/about","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/imagesiftbot.html","what_it_is":"Crawls images for Hive AI's reverse-image and dataset products. Image-heavy sites see this one long before they see the text crawlers.","cost_of_blocking":"Your images stop entering an image dataset and reverse-image index."},{"slug":"img2dataset","name":"img2dataset","operator":"LAION / img2dataset","category":"dataset","category_label":"Corpus and dataset builders","ua":"img2dataset","robots_token":"img2dataset","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://github.com/rom1504/img2dataset","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/img2dataset.html","what_it_is":"The tool used to turn image-URL lists such as LAION's into downloaded training sets. It is run by whoever is building a dataset, not by a single operator.","cost_of_blocking":"Your images are skipped when someone materialises an image-text dataset that references them."},{"slug":"meta-externalagent","name":"meta-externalagent","operator":"Meta","category":"ai-training","category_label":"AI training crawlers","ua":"meta-externalagent","robots_token":"meta-externalagent","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/meta-externalagent.html","what_it_is":"Meta's AI crawler, gathering training data for Llama and Meta AI. It replaced the older FacebookBot name for this purpose.","cost_of_blocking":"Excluded from Meta AI training. Link previews on Facebook, Instagram and WhatsApp are unaffected — those are a different bot."},{"slug":"meta-externalfetcher","name":"meta-externalfetcher","operator":"Meta","category":"user-fetch","category_label":"User-triggered fetchers","ua":"meta-externalfetcher","robots_token":"meta-externalfetcher","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://developers.facebook.com/docs/sharing/webmasters/web-crawlers","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/meta-externalfetcher.html","what_it_is":"Fetches a page when a Meta AI user asks about a specific link.","cost_of_blocking":"Meta AI cannot read pages users hand it."},{"slug":"mistralai-user","name":"MistralAI-User","operator":"Mistral AI","category":"user-fetch","category_label":"User-triggered fetchers","ua":"MistralAI-User","robots_token":"MistralAI-User","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://docs.mistral.ai/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/mistralai-user.html","what_it_is":"Fetches a page when a Le Chat user asks Mistral's assistant to read it.","cost_of_blocking":"Le Chat cannot open links your readers give it."},{"slug":"oai-searchbot","name":"OAI-SearchBot","operator":"OpenAI","category":"ai-search","category_label":"AI search crawlers","ua":"OAI-SearchBot","robots_token":"OAI-SearchBot","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://platform.openai.com/docs/bots","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/openai-searchbot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/oai-searchbot.html","what_it_is":"Builds the index ChatGPT search answers from. Content it collects is used for retrieval and citation, not for model training.","cost_of_blocking":"High. Blocking this removes you from ChatGPT search results and from the source links ChatGPT shows. This is the single most expensive block on this list for anyone who wants to be cited by an assistant."},{"slug":"omgili","name":"omgili","operator":"Webz.io","category":"dataset","category_label":"Corpus and dataset builders","ua":"omgili","robots_token":"omgili","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://webz.io/blog/machine-learning/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/omgili.html","what_it_is":"The older robots token for the same Webz.io collection, still honoured and still worth listing.","cost_of_blocking":"Same as omgilibot."},{"slug":"omgilibot","name":"omgilibot","operator":"Webz.io","category":"dataset","category_label":"Corpus and dataset builders","ua":"omgilibot","robots_token":"omgilibot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://webz.io/blog/machine-learning/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/omgilibot.html","what_it_is":"Webz.io's crawler, collecting web and forum text sold as datasets, including to model builders.","cost_of_blocking":"Exclusion from a commercial dataset resold to third parties."},{"slug":"perplexity-user","name":"Perplexity-User","operator":"Perplexity","category":"user-fetch","category_label":"User-triggered fetchers","ua":"Perplexity-User","robots_token":"Perplexity-User","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"by-design-no","docs":"https://docs.perplexity.ai/guides/bots","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/perplexity-user.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/perplexity-user.html","what_it_is":"Fetches a page because a Perplexity user asked for it. Perplexity documents that this fetch is user-initiated and is therefore not governed by robots.txt — a robots rule will not stop it, by stated policy.","cost_of_blocking":"Not controllable via robots.txt. If you must stop it, verify by the published IP ranges and block at the edge — and accept that users who ask for your page get an error."},{"slug":"perplexitybot","name":"PerplexityBot","operator":"Perplexity","category":"ai-search","category_label":"AI search crawlers","ua":"PerplexityBot","robots_token":"PerplexityBot","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://docs.perplexity.ai/guides/bots","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/perplexity-bot.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/perplexitybot.html","what_it_is":"Builds Perplexity's search index. Perplexity is citation-heavy by product design, so inclusion here converts to referral traffic more directly than most AI surfaces.","cost_of_blocking":"You stop being indexed and cited by Perplexity, and lose the referral clicks its citations produce."},{"slug":"petalbot","name":"PetalBot","operator":"Huawei","category":"search","category_label":"Search engines","ua":"PetalBot","robots_token":"PetalBot","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://aspiegel.com/petalbot","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/petalbot.html","what_it_is":"Huawei's crawler for Petal Search, shipped as the default search on Huawei devices.","cost_of_blocking":"Removal from Petal Search. Frequently blocked for volume rather than for policy."},{"slug":"scrapy","name":"Scrapy","operator":"Scrapy project","category":"tool","category_label":"Tools and frameworks","ua":"Scrapy","robots_token":"Scrapy","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://scrapy.org/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/scrapy.html","what_it_is":"Not an operator: the default user-agent of the most common Python crawling framework. Anyone can be behind it. Modern Scrapy obeys robots.txt by default, which is why the default UA is still worth a rule.","cost_of_blocking":"You block a very large tail of unattributed one-off crawlers, and also every well-behaved researcher who did not change the default."},{"slug":"semrushbot","name":"SemrushBot","operator":"Semrush","category":"seo","category_label":"SEO and backlink crawlers","ua":"SemrushBot","robots_token":"SemrushBot","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://www.semrush.com/bot/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/semrushbot.html","what_it_is":"Semrush's backlink and keyword crawler. It is not an AI crawler, but it is usually in the top three by volume on any site, and it is the cheapest block on this list.","cost_of_blocking":"Your competitors' Semrush reports get thinner, and so do yours. No user-facing effect."},{"slug":"semrushbot-ocob","name":"SemrushBot-OCOB","operator":"Semrush","category":"ai-training","category_label":"AI training crawlers","ua":"SemrushBot-OCOB","robots_token":"SemrushBot-OCOB","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://www.semrush.com/bot/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/semrushbot-ocob.html","what_it_is":"Semrush's separately-tokenised crawler for its AI content tooling, split out so SEO crawling and AI reuse can be answered differently.","cost_of_blocking":"Exclusion from Semrush's AI corpus, with its SEO crawl unaffected."},{"slug":"seznambot","name":"SeznamBot","operator":"Seznam","category":"search","category_label":"Search engines","ua":"SeznamBot","robots_token":"SeznamBot","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://napoveda.seznam.cz/en/seznamzbozi/subject-matter-crawler/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/seznambot.html","what_it_is":"Seznam's crawler — the dominant search engine in the Czech Republic and one of the few national engines with its own index.","cost_of_blocking":"Removal from Seznam. Also removes you from its IndexNow endpoint's usefulness."},{"slug":"storebot-google","name":"Storebot-Google","operator":"Google","category":"search","category_label":"Search engines","ua":"Storebot-Google","robots_token":"Storebot-Google","verification_method":"published-ranges","verification_label":"published IP ranges","respects_robots_txt":"documented","docs":"https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers","ip_ranges":"https://www.pathwren.workers.dev/c/rubygems-registry/ip-ranges/google-special.json","url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/storebot-google.html","what_it_is":"Checks shopping and checkout flows for Google's shopping surfaces.","cost_of_blocking":"Product listings may lose shopping-specific enrichment. Irrelevant to non-commerce sites."},{"slug":"tiktokspider","name":"TikTokSpider","operator":"ByteDance","category":"ai-training","category_label":"AI training crawlers","ua":"TikTokSpider","robots_token":"TikTokSpider","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"disputed","docs":"https://www.bytespider.net/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/tiktokspider.html","what_it_is":"A second ByteDance crawler identifying with TikTok, collecting page content for the same family of models.","cost_of_blocking":"Little to lose unless TikTok search referral matters to you."},{"slug":"timpibot","name":"Timpibot","operator":"Timpi","category":"search","category_label":"Search engines","ua":"Timpibot","robots_token":"Timpibot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://timpi.io/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/timpibot.html","what_it_is":"A distributed crawler building an independent search index outside the Google/Bing duopoly.","cost_of_blocking":"Absence from a small independent index."},{"slug":"webzio-extended","name":"Webzio-Extended","operator":"Webz.io","category":"ai-training","category_label":"AI training crawlers","ua":"Webzio-Extended","robots_token":"Webzio-Extended","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://webz.io/blog/machine-learning/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/webzio-extended.html","what_it_is":"Webz.io's opt-out token specifically for AI training reuse, in the pattern Google and Apple established.","cost_of_blocking":"Your content is excluded from the AI-training tier of Webz.io's product while ordinary collection continues."},{"slug":"yandexbot","name":"YandexBot","operator":"Yandex","category":"search","category_label":"Search engines","ua":"YandexBot","robots_token":"YandexBot","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/yandexbot.html","what_it_is":"Yandex's search crawler, which also feeds Alice and Yandex's generative answers.","cost_of_blocking":"Removal from Yandex Search. Verify with reverse DNS to a yandex.ru, yandex.net or yandex.com host — YandexBot is among the most-spoofed user-agents there is."},{"slug":"yeti","name":"Yeti","operator":"Naver","category":"search","category_label":"Search engines","ua":"Yeti","robots_token":"Yeti","verification_method":"reverse-dns","verification_label":"reverse DNS","respects_robots_txt":"documented","docs":"https://searchadvisor.naver.com/guide/seo-basic-crawl","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/yeti.html","what_it_is":"Naver's crawler. Naver is South Korea's largest search portal and runs its own index and its own generative answers.","cost_of_blocking":"Removal from Naver, which is most of Korean search."},{"slug":"youbot","name":"YouBot","operator":"You.com","category":"ai-search","category_label":"AI search crawlers","ua":"YouBot","robots_token":"YouBot","verification_method":"none","verification_label":"no published verification method","respects_robots_txt":"documented","docs":"https://about.you.com/youbot/","ip_ranges":null,"url":"https://www.pathwren.workers.dev/c/rubygems-registry/crawler/youbot.html","what_it_is":"You.com's crawler, feeding its AI search product and its search API.","cost_of_blocking":"Removal from You.com's index and from answers built on its API."}]}

data/lib/ai_crawler_index.rb

@@ -0,0 +1,248 @@
+# frozen_string_literal: true
+
+# ai-crawler-index — offline AI-crawler / bot user-agent classifier.
+#
+# Zero dependencies (stdlib JSON only), no network at require time. Every method
+# is a pure function of the string you hand it, so it runs inside a Rack request,
+# a Sidekiq worker, a Lambda or a machine with no route to the internet.
+#
+#   require "ai_crawler_index"
+#
+#   AiCrawlerIndex.ai_crawler?("Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)")
+#   # => true
+#
+#   AiCrawlerIndex.identify("Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)")
+#   # => {"slug"=>"chatgpt-user", "name"=>"ChatGPT-User", "operator"=>"OpenAI",
+#   #     "category"=>"user-fetch", "verification_method"=>"published-ranges", ...}
+#
+# The table is generated from the AI Crawler Index, an independent,
+# non-commercial public reference. Data: CC0-1.0. Code: MIT.
+
+require "json"
+
+module AiCrawlerIndex
+  DATA_PATH = File.join(__dir__, "ai_crawler_index", "data.json")
+
+  DATA = JSON.parse(File.read(DATA_PATH, encoding: "UTF-8")).freeze
+
+  VERSION = DATA["version"].freeze
+
+  # Every record in the index. Treat as read-only; #list hands out a copy.
+  CRAWLERS = DATA["crawlers"].freeze
+  # Category slug => {"label" => ..., "description" => ...}
+  CATEGORIES = DATA["categories"].freeze
+  # The category slugs that count as "AI" for .ai_crawler?
+  AI_CATEGORIES = DATA["ai_categories"].freeze
+
+  # Longest user-agent token first, so "Googlebot-Image" wins over "Googlebot"
+  # and "Claude-SearchBot" over "ClaudeBot". Ties broken by name, deterministically.
+  INDEX = CRAWLERS
+          .map { |r| [r["ua"].downcase, r] }
+          .sort_by { |needle, _| [-needle.length, needle] }
+          .freeze
+
+  # Compiled case-insensitive alternations, if you would rather match yourself.
+  PATTERNS = {
+    "all" => Regexp.new(DATA["regex"]["all"], Regexp::IGNORECASE),
+    "ai" => Regexp.new(DATA["regex"]["ai_only"], Regexp::IGNORECASE),
+    "by_category" => DATA["regex"]["by_category"]
+                     .each_with_object({}) { |(k, v), h| h[k] = Regexp.new(v, Regexp::IGNORECASE) }
+                     .freeze
+  }.freeze
+
+  # What this copy of the table is, and when it was measured.
+  META = {
+    "version" => DATA["version"],
+    "generated_at" => DATA["generated_at"],
+    "count" => CRAWLERS.length,
+    "source" => DATA["source"],
+    "data_url" => DATA["data_url"],
+    "data_license" => "CC0-1.0",
+    "window" => DATA["window"]
+  }.freeze
+
+  class << self
+    # The single best matching record for +ua+, or nil.
+    # Non-string and empty input returns nil and never raises.
+    def identify(ua)
+      hay = normalise(ua)
+      return nil if hay.empty?
+
+      INDEX.each { |needle, record| return record if hay.include?(needle) }
+      nil
+    end
+
+    # Every record whose user-agent token appears in +ua+, most specific first.
+    def match_all(ua)
+      hay = normalise(ua)
+      return [] if hay.empty?
+
+      INDEX.select { |needle, _| hay.include?(needle) }.map { |_, record| record }
+    end
+
+    # True when +ua+ is an AI crawler: training, AI search, user-triggered fetch or dataset.
+    def ai_crawler?(ua)
+      hit = identify(ua)
+      !hit.nil? && AI_CATEGORIES.include?(hit["category"])
+    end
+
+    # True when +ua+ is any known automated client in the index — AI or not.
+    def crawler?(ua)
+      !identify(ua).nil?
+    end
+
+    # "ai-training" / "ai-search" / "user-fetch" / "dataset" / "search" / "seo" /
+    # "archive" / "tool" / "preview" — or nil.
+    def category_of(ua)
+      hit = identify(ua)
+      hit && hit["category"]
+    end
+
+    # The operator's name ("OpenAI", "Anthropic", ...), or nil.
+    def operator_of(ua)
+      hit = identify(ua)
+      hit && hit["operator"]
+    end
+
+    # The whole table as a fresh array, or one category of it.
+    def list(category: nil)
+      return CRAWLERS.dup if category.nil?
+
+      CRAWLERS.select { |r| r["category"] == category }
+    end
+
+    # One record by its index slug, e.g. get("gptbot").
+    def get(slug)
+      CRAWLERS.find { |r| r["slug"] == slug.to_s }
+    end
+
+    # A robots.txt body for a stance. Nothing is written anywhere.
+    #   "block-ai-training" blocks training and dataset crawlers
+    #   "block-all-ai"      blocks every AI class including user-triggered fetches
+    #   "block-none"        returns a permissive file
+    def robots_txt(stance = "block-ai-training")
+      want =
+        case stance.to_s
+        when "block-all-ai" then ->(r) { AI_CATEGORIES.include?(r["category"]) }
+        when "block-ai-training" then ->(r) { %w[ai-training dataset].include?(r["category"]) }
+        when "block-none" then ->(_r) { false }
+        else
+          raise ArgumentError,
+                "robots_txt: unknown stance #{stance.inspect} " \
+                "(block-ai-training | block-all-ai | block-none)"
+        end
+
+      tokens = CRAWLERS.select { |r| want.call(r) && r["robots_token"] }
+                       .map { |r| r["robots_token"] }.uniq.sort
+      lines = ["# generated by ai-crawler-index #{VERSION} (stance: #{stance})",
+               "# source: #{DATA['source']}"]
+      tokens.each { |t| lines.concat(["", "User-agent: #{t}", "Disallow: /"]) }
+      lines.concat(["", "User-agent: *", "Disallow:", ""])
+      lines.join("\n")
+    end
+
+    # THE ONLY NETWORK PATH, and it is never called for you.
+    # Fetches the current table and returns a new Table with the same API;
+    # the bundled constants are not mutated.
+    def refresh(url = META["data_url"], timeout: 10)
+      require "net/http"
+      require "uri"
+      uri = URI.parse(url)
+      http = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl = (uri.scheme == "https")
+      http.open_timeout = timeout
+      http.read_timeout = timeout
+      res = http.request(Net::HTTP::Get.new(uri.request_uri,
+                                            "User-Agent" => "ai-crawler-index-rb/#{VERSION}",
+                                            "Accept" => "application/json"))
+      raise "refresh: HTTP #{res.code} from #{url}" unless res.is_a?(Net::HTTPSuccess)
+
+      Table.new(from_index_json(JSON.parse(res.body)))
+    end
+
+    # The public index document -> the record shape this gem exposes.
+    # The public document spells the token "user_agent_substring"; this gem
+    # spells it "ua", and that difference is how a live document is told apart
+    # from an already-gem-shaped one.
+    def from_index_json(doc)
+      rows = doc["crawlers"] || doc["agents"]
+      raise ArgumentError, "refresh: unexpected document — expected {\"crawlers\": [...]}" unless rows.is_a?(Array)
+      return doc if rows.first && rows.first.key?("ua")
+
+      {
+        "version" => doc["version"] || VERSION,
+        "generated_at" => doc["generated_at"],
+        "source" => META["source"],
+        "categories" => doc["categories"] || CATEGORIES,
+        "ai_categories" => AI_CATEGORIES,
+        "crawlers" => rows.map do |a|
+          {
+            "slug" => a["slug"], "name" => a["name"], "operator" => a["operator"],
+            "category" => a["category"], "category_label" => a["category_label"],
+            "ua" => a["user_agent_substring"], "robots_token" => a["robots_token"],
+            "verification_method" => a["verification_method"],
+            "verification_label" => a["verification_label"],
+            "respects_robots_txt" => a["respects_robots_txt"],
+            "docs" => a["operator_docs"], "ip_ranges" => a["ip_ranges_endpoint"],
+            "url" => a["html_url"], "what_it_is" => a["what_it_is"],
+            "cost_of_blocking" => a["cost_of_blocking"]
+          }
+        end
+      }
+    end
+
+    private
+
+    def normalise(ua)
+      ua.is_a?(String) ? ua.downcase : ""
+    end
+  end
+
+  # The same API over a table fetched at runtime. Returned by .refresh.
+  class Table
+    attr_reader :data, :crawlers, :ai_categories, :meta
+
+    def initialize(data)
+      @data = data
+      @crawlers = data["crawlers"]
+      @ai_categories = data["ai_categories"] || AI_CATEGORIES
+      @index = @crawlers.map { |r| [r["ua"].to_s.downcase, r] }
+                        .sort_by { |needle, _| [-needle.length, needle] }
+      @meta = { "version" => data["version"], "generated_at" => data["generated_at"],
+                "count" => @crawlers.length, "source" => data["source"] }
+    end
+
+    def identify(ua)
+      hay = ua.is_a?(String) ? ua.downcase : ""
+      return nil if hay.empty?
+
+      @index.each { |needle, record| return record if !needle.empty? && hay.include?(needle) }
+      nil
+    end
+
+    def ai_crawler?(ua)
+      hit = identify(ua)
+      !hit.nil? && @ai_categories.include?(hit["category"])
+    end
+
+    def crawler?(ua)
+      !identify(ua).nil?
+    end
+
+    def category_of(ua)
+      hit = identify(ua)
+      hit && hit["category"]
+    end
+
+    def operator_of(ua)
+      hit = identify(ua)
+      hit && hit["operator"]
+    end
+
+    def list(category: nil)
+      return @crawlers.dup if category.nil?
+
+      @crawlers.select { |r| r["category"] == category }
+    end
+  end
+end

metadata

@@ -0,0 +1,57 @@
+--- !ruby/object:Gem::Specification
+name: ai-crawler-index
+version: !ruby/object:Gem::Version
+  version: 1.1.0
+platform: ruby
+authors:
+- Pathwren
+bindir: exe
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies: []
+description: 'Give it a User-Agent string, get back what it is: 56 AI crawlers, search
+  engines, SEO bots, archivers and dataset builders from 30 operators, bundled as
+  one 45.4 KB JSON table. ai_crawler?(ua), identify(ua) -> {name, operator, category,
+  verification_method}, robots_txt(stance). Zero dependencies, no network at require
+  time, pure functions of the string you pass in. Data CC0-1.0 from the independent
+  AI Crawler Index; code MIT.'
+executables:
+- ai-crawler-index
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- exe/ai-crawler-index
+- lib/ai-crawler-index.rb
+- lib/ai_crawler_index.rb
+- lib/ai_crawler_index/data.json
+homepage: https://www.pathwren.workers.dev/c/rubygems-registry/
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://www.pathwren.workers.dev/c/rubygems-registry/
+  documentation_uri: https://www.pathwren.workers.dev/c/rubygems-registry/
+  changelog_uri: https://www.pathwren.workers.dev/c/rubygems-registry/changelog.html
+  data_source_uri: https://www.pathwren.workers.dev/c/rubygems-registry/data/agents.json
+  data_license: CC0-1.0
+  generated_at: '2026-09-01T21:09:27+00:00'
+  crawlers: '56'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.6.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.7
+specification_version: 4
+summary: 'Offline AI-crawler user-agent classifier: ai_crawler?(ua) and identify(ua)'
+test_files: []