browserx 1.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b8b37f466da96715f58be6f7581204d5a46fe12ed3ea96995b46cbcc48a95e1f
-  data.tar.gz: ff7419bb6d7162a3abcf82dfac7e851dd50d55605dbea351eac97b11d819cacb
+  metadata.gz: bcb687682794dba3dbcdfa71717f8e2f58094986dc9533e3d7dc5fd4da64d5e4
+  data.tar.gz: 868d50297677fa812d4cc5dd0cdbe45bf273a8c97ffea3fe6fee8d75e841fe66
 SHA512:
-  metadata.gz: be7daa1b157872be556fe319352180b12f978f42aedf23e9e43eb880905e4d6d849cf7ea268281ca86cf56d3baf46707f4aafd785ab0c589a5fdef57d99e650f
-  data.tar.gz: 19d21da9ee4eb84763cb4609c2b160f9ac71f6656a783ec4d31d87c2558ee71a7d27e99f4e5437119998dcc2f13c2c00abb8ce478e8d8314b1012c76773cc1fd
+  metadata.gz: cc1313014078805ac0e6710454a2a1aa8436ccd6aa568a789186675094df3b256a19cc4b0d6d27eff83ff48cae5ccb7bd2bafca49896f0fc0b4e9cbe31915bb8
+  data.tar.gz: 7c22cbb3fc05f9b1a563008c90399512783584cd1d4e64fe4a0aac4b00cccbb90bab7ae37c4410fca316f7d4a70045ea1c0f7eff518d49801afaf9fcfa671868

data/CHANGELOG.md

@@ -2,6 +2,60 @@
 
 ## [Unreleased]
 
+## [2.1.0] — 2026-11-05
+
+The first request of each process gets cheaper: the lists ship as JSON, and `Browser.eager_load!` builds them
+before a server forks.
+
+**New**
+- `Browser.eager_load!` loads the lists and builds the matchers and bot keyword tries up front, so a server that
+  preloads its app before forking (Puma, Unicorn, Pitchfork) shares them with every worker. Rails calls it on its
+  own when it eager loads, which production does by default. A worker's first request spent 12.9 ms building
+  them, 0.2 ms once built.
+
+**Faster**
+- The lists ship as JSON: without Bootsnap's cache, a process's first request is 3× faster (12.4 → 4.0 ms with
+  YJIT, 7.5 → 2.7 ms without), and `require "browser"` no longer loads Psych (32 → 21 ms with YJIT, 28 → 22 ms
+  without). With Bootsnap, nothing changes. Code that used `YAML` without requiring it now needs its own
+  `require "yaml"`.
+
+## [2.0.0] — 2026-11-04
+
+A repeated user agent is now answered in 2 µs, about 600 more bots are recognised, and bots, Cloudflare headers and
+Client Hints can be checked against the request. Without the new headers, every answer stays browser 6.2.0's,
+except bot detection, which finds more bots.
+
+**Breaking**
+- `bot?` is true for about 600 more user agents: bots from crawler-user-agents and ai.robots.txt, synced weekly,
+  add 1,247 entries to `bots.yml`. Of the crawlers added there since 2024, browser misses 353 out of 882 and
+  BrowserX 11. Analytics that filter on `bot?` will count fewer humans from the day you switch.
+- The Rails helper and the middleware now pass the request's headers, so Client Hints refine the answers on
+  Chromium: Windows 11 is no longer `windows10?`, and Chrome for Android in desktop mode is an Android tablet,
+  not a Linux desktop.
+
+**New**
+- `bot.category` (16 categories, close to Cloudflare's verified bot categories), `bot.ai?` and
+  `bot.user_triggered?`. Test drivers (Playwright, Puppeteer, Selenium, headless Chrome) get the
+  `browser_automation` category but are not bots, so system tests keep running as humans.
+- `bot.verified?(ip)` checks a crawler against the IP ranges its owner publishes (Google, Bing, OpenAI, Apple,
+  Perplexity, DuckDuckGo), shipped with the gem and refreshable at runtime; reverse DNS is opt-in. `bot.owner`
+  names the owner.
+- `Browser.new(ua, headers: request.env)` reads Cloudflare's bot headers (`browser.cloudflare`: verified bot,
+  category, score, country; header names configurable) and Chromium's Client Hints (`browser.client_hints`,
+  `platform.windows11?`).
+- `Browser.cache_size` (default 1,000, `0` turns it off) bounds the new per-user-agent cache: about 1 KB per
+  entry, so about 1 MB per process.
+
+**Faster**
+- Answers are cached per user agent, matchers that can't match are skipped, and bots are found through a byte
+  trie. Ruby 4.0.9: a repeated user agent takes 1.8 µs instead of 57.5 with YJIT and 3.6 µs instead of 72.5
+  without; 2,000 distinct user agents take 24.2 µs instead of 62.4 with YJIT and 43.9 µs instead of 77.3
+  without. Objects per request: 9 instead of 116 repeated, 50 instead of 125 distinct.
+
+**Fixed**
+- Loading `browser` and `browserx` together raises a clear `LoadError` instead of one silently overriding the
+  other.
+
 ## [1.0.0] — 2026-11-04
 
 First release as `browserx`, forked from browser 6.2.0 (`dd88933`). Same API: `gem "browserx"` replaces
@@ -25,5 +79,7 @@ First release as `browserx`, forked from browser 6.2.0 (`dd88933`). Same API: `g
 - A corpus of 20,328 user agents, with snapshot and diff scripts proving answers match upstream, and a
   request benchmark.
 
-[Unreleased]: https://github.com/next-gen-gems/browserx/compare/v1.0.0...HEAD
+[Unreleased]: https://github.com/next-gen-gems/browserx/compare/v2.1.0...HEAD
+[2.1.0]: https://github.com/next-gen-gems/browserx/compare/v2.0.0...v2.1.0
+[2.0.0]: https://github.com/next-gen-gems/browserx/compare/v1.0.0...v2.0.0
 [1.0.0]: https://github.com/next-gen-gems/browserx/releases/tag/v1.0.0

data/README.md

@@ -1,12 +1,19 @@
 # BrowserX
 
-Browser detection for Ruby, with ActionController integration. BrowserX is a drop-in replacement for the
-[browser](https://rubygems.org/gems/browser) gem: put `gem "browserx"` in your Gemfile instead of
-`gem "browser"`, and every call site keeps working — same `Browser` namespace, same `require "browser"`, same
-predicates. It answers nearly twice as fast (61 µs and 129 objects per request instead of 110 µs and 245 on
-Ruby 4.0 with YJIT), and its bot list is meant to be kept fresh from permissively licensed sources
-([crawler-user-agents](https://github.com/monperrus/crawler-user-agents),
-[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt)).
+Browser detection for Ruby, with ActionController integration: the API of the
+[browser](https://rubygems.org/gems/browser) gem, faster, and a bot list that stays fresh. Put `gem "browserx"` in
+your Gemfile instead of `gem "browser"`, and every call site keeps working — same `Browser` namespace, same
+`require "browser"`, same predicates.
+
+- **Faster**: 25 µs per request instead of 126 µs with browser 6.2.0, 2 µs once a user agent was seen (Ruby 4.0.9,
+  YJIT, `bench/request.rb`).
+- **Fresher bots**: synced every week from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents)
+  and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt), with categories (`bot.category`, `bot.ai?`).
+  Of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out),
+  browser misses 353, BrowserX 11.
+- **Verified signals**: `bot.verified?(ip)` checks a crawler against the IP ranges Google, Bing, OpenAI, Apple,
+  Perplexity and DuckDuckGo publish, `browser.cloudflare` reads the headers of a Cloudflare Transform Rule, and
+  Chromium's Client Hints tell Windows 11 from Windows 10.
 
 Site: https://next-gen-gems.github.io/browserx/
 
@@ -64,6 +71,9 @@ browser.epiphany?
 # Get bot info
 browser.bot.name
 browser.bot.search_engine?
+browser.bot.category        # :ai_crawler, :search_engine_crawler, :scanner… (nil for a browser)
+browser.bot.ai?
+browser.bot.user_triggered? # an AI agent acting for a person right now (ChatGPT-User, Claude-User)
 browser.bot?
 browser.bot.why? # shows which matcher detected this user agent as a bot.
 Browser::Bot.why?(ua)
@@ -187,9 +197,17 @@ end
 ### Migrating from browser
 
 Replace `gem "browser"` with `gem "browserx"` in the Gemfile, nothing else: `require "browser"`, the `Browser`
-namespace and every predicate stay. To roll back, put `gem "browser"` back. BrowserX requires Ruby 3.3 or later
+namespace and every predicate stay. Both gems define `Browser`, so BrowserX refuses to load next to `browser`,
+with an error saying so, rather than letting one silently override the other. BrowserX requires Ruby 3.3 or later
 and is tested on Rails 8.0 and 8.1.
 
+What changes on its own: the bots of the weekly sync are bots from the day you switch (600 user agents of the
+corpus that browser took for humans), so analytics that leave bots out count fewer visitors from that day; 792
+user agents named `Generic Bot` get a name; no browser, platform or device answer moves, and test drivers stay out
+of `bot?`. To roll back, put `gem "browser"` back, after removing the calls to what browser lacks
+(`Browser.cache_size=`, `bot.category`, `bot.ai?`, `bot.user_triggered?`). The
+[migration guide](https://next-gen-gems.github.io/browserx/guide.html#migrating) has the details.
+
 ### Rails integration
 
 Just add it to the Gemfile.
@@ -310,6 +328,27 @@ following requirements will be considered a bot.
 - Any known bot listed under
   [bots.yml](bots.yml)
 
+Every week, `script/sync_bots.rb` adds the bots of
+[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) and
+[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (both MIT) to the end of `bots.yml`, and
+categorizes them in `bot_categories.yml`, through a pull request that lists the answers that change. How many of
+those lists' bots browser and BrowserX detect, per category, is on the
+[Bots page](https://next-gen-gems.github.io/browserx/bots.html#coverage).
+
+`browser.bot.category` names what the bot declares it is, with the names of Cloudflare's verified bot categories
+where one fits (`search_engine_crawler`, `ai_crawler`, `ai_search`, `ai_assistant`, `page_preview`,
+`feed_fetcher`, `monitoring_and_analytics`, `advertising_and_marketing`, `archiver`, `security`…), plus
+`ai_agent`, `http_library`, `scanner` and `browser_automation`. `bot.ai?` is true for the four AI categories,
+`bot.user_triggered?` for `ai_assistant` and `ai_agent`. The category is `nil` for a browser, a bot exception
+and a bot no list names.
+
+HTTP libraries (`curl`, `got`, `python-requests`) and scanners (sqlmap, Nikto, Censys) are bots, as `curl` and
+`python-requests` already were in browser. Test drivers (headless Chrome, Playwright, Puppeteer, Selenium) are
+not: an app's system tests drive a real browser through them, and the headless Chrome of Playwright and Selenium
+sends `HeadlessChrome`. They get the `browser_automation` category instead, so an app can tell them apart in
+production (`browser.bot? || browser.bot.category == :browser_automation`). PhantomJS stays a bot, as in
+browser.
+
 To add custom matchers, you can add a callable object to
 `Browser::Bot.matchers`. The following example matches everything that has a
 `externalhit` substring on it. The bot name will always be `General Bot`.
@@ -339,8 +378,34 @@ To extend the bot list, you can manipulate the methods below:
 Browser::Bot.bots.merge!(new_bots_hash)
 Browser::Bot.bot_exceptions += new_exceptions
 Browser::Bot.search_engines.merge!(new_search_engines_hash)
+Browser::Bot.category_by_keyword["faraday"] = :http_library
+```
+
+### Verified crawlers
+
+Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish their crawlers' IP ranges. They ship in
+`bot_ip_ranges.yml` (as JSON in the gem), rewritten by the weekly sync (`script/sync_ip_ranges.rb`). The lookup is a binary search, no
+network call; an IP that does not parse answers `false`.
+
+```ruby
+browser.bot.owner                                            # => :google for a user agent claiming Googlebot
+browser.bot.verified?(request.remote_ip)                     # => whether the IP is in Google's ranges
+browser.bot.verified?(request.remote_ip, reverse_dns: true)  # also Google's DNS check: two DNS queries
+Browser::Bot::IPRanges.refresh                               # fetches the ranges again, in a long-lived process
 ```
 
+### Cloudflare and Client Hints
+
+`Browser.new(user_agent, accept_language:, headers: rack_env)` also reads the request's headers; the Rails helper
+and the middleware pass them. `browser.cloudflare` reads the headers a Cloudflare Transform Rule sets
+(`verified_bot?`, `bot_category`, `bot_score`) and `CF-IPCountry` (`country`); their names are configurable with
+`Browser::Cloudflare.header_names = {bot_score: "X-My-Score"}`. Chromium's Client Hints (`Sec-CH-UA`,
+`Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`, `Sec-CH-UA-Platform-Version`) refine the platform and the device, and
+`browser.platform.windows11?` needs them. Without `headers:`, every answer is the user agent's, as before. The
+rule to create and the details are in the [guide](https://next-gen-gems.github.io/browserx/guide.html#cloudflare).
+
+`bench/signals.rb` measures what they cost.
+
 ### Middleware
 
 You can use the `Browser::Middleware` to redirect user agents.
@@ -383,17 +448,44 @@ Browser.user_agent_size_limit = 4096
 Browser.accept_language_size_limit = 4096
 ```
 
+### Cache
+
+Each process keeps what it detected for the last 1,000 distinct user agents (browser, platform, device, bot
+answers) and the last 1,000 Accept-Language headers it parsed, about 1 KB per user agent: a repeated user agent
+costs about 2 µs. The cache empties itself when full, and when `Browser.matchers`, `Browser::Bot.matchers`,
+`Browser::Platform.matchers`, `Browser::Device.matchers`, the bot list or its exceptions change. Setting
+`Browser.cache_size` empties it too, for example after renaming an entry of `Browser::Bot.bots` in place.
+
+```ruby
+Browser.cache_size = 5_000 # 0 turns it off
+```
+
+### Preloading before a fork
+
+The lists, the matchers and the bot keyword tries are built on first use: the first request of each process pays
+about 13 ms for them, the next ones 0.2 ms. `Browser.eager_load!` builds them up front, so a server that preloads
+its app before forking (Puma's `preload_app!`, Unicorn, Pitchfork) shares them with every worker. Rails calls it on
+its own when it eager loads, which production does by default. Elsewhere, call it before the fork:
+
+```ruby
+Browser.eager_load!
+```
+
 ## Performance
 
 `bench/request.rb` replays what a web request asks (name, version, `bot?`, platform, device, accept-language) over
 2,000 distinct user agents of the corpus and over 20 repeated ones, median of 7. On Ruby 4.0.9 with YJIT, browser
-6.2.0 takes 110 µs and 245 objects per request, BrowserX 1.0.0 61 µs and 129 objects.
+6.2.0 takes 126 µs and 245 objects per request, BrowserX 25 µs and 52 objects on a new user agent, 2 µs and 12
+objects on a repeated one. All four runs, with and without YJIT, are on the
+[benchmarks page](https://next-gen-gems.github.io/browserx/benchmarks.html).
 
 ```bash
-ruby --yjit -I path/to/upstream/lib bench/request.rb
-ruby --yjit -I lib bench/request.rb
+bundle exec rake bench UPSTREAM=path/to/upstream # fnando/browser at dd88933
 ```
 
+It runs `ruby [--yjit] -I lib bench/request.rb --write` for each gem and JIT, which records the run (gem, commit,
+Ruby, YJIT, machine, date) in `docs/benchmarks.json`, then `rake docs:render` rewrites the site's numbers from it.
+
 ## Development
 
 ```bash
@@ -403,7 +495,7 @@ bundle exec rubocop
 ```
 
 A change that is not meant to change an answer proves it on the corpus of 20,000 user agents, against a plain
-clone of [fnando/browser](https://github.com/fnando/browser) at `dd88933`:
+clone of [fnando/browser](https://github.com/fnando/browser) at `dd88933`, as CI does on every pull request:
 
 ```bash
 ruby -I path/to/upstream/lib script/corpus_snapshot.rb > /tmp/upstream.jsonl
@@ -411,28 +503,50 @@ ruby -I lib script/corpus_snapshot.rb > /tmp/browserx.jsonl
 ruby script/corpus_diff.rb /tmp/upstream.jsonl /tmp/browserx.jsonl
 ```
 
-To add a bot, add its user agent to `test/ua_bots.yml` and its readable name to `bots.yml`, keyed by a
-lowercased string the user agent contains. A search engine goes in `test/ua_search_engines.yml`,
+The diff passes when the answers that differ are exactly those of `corpus/expected_differences.jsonl`: a change
+meant to change answers rewrites it with `ruby script/corpus_diff.rb --write …` and shows its diff.
+
+The weekly bot sync (`.github/workflows/bot-sync.yml`) runs the same steps by hand:
+
+```bash
+ruby script/sync_bots.rb                         # bots.yml and bot_categories.yml
+ruby -I lib script/sync_ip_ranges.rb             # bot_ip_ranges.yml
+ruby script/build_data.rb                        # data/*.json, what the gem reads, from the *.yml
+ruby script/bot_coverage.rb path/to/upstream/lib # docs/bot_coverage.json, and its Markdown table
+ruby script/render_docs.rb                       # the site's numbers, from docs/*.json and the lists
+```
+
+The numbers of `docs/*.html` sit between `<!-- render:… -->` markers and come from `docs/benchmarks.json`,
+`docs/bot_coverage.json`, the bot lists and `corpus/expected_differences.jsonl`: CI fails when a page differs from
+what `bundle exec rake docs:render` writes.
+
+To add a bot, add its user agent to `test/ua_bots.yml` and its readable name to `bots.yml` above the synced
+section, keyed by a lowercased string the user agent contains. A search engine goes in `test/ua_search_engines.yml`,
 `test/ua_bots.yml` and `search_engines.yml`.
 
 ### Releasing
 
-Changes go under `## [Unreleased]` in `CHANGELOG.md` as they land. To release them, from an up-to-date `main`
-whose CI is green:
+Pull requests leave `CHANGELOG.md` alone: a release starts by writing its notes under `## [Unreleased]`, from the
+pull requests merged since the last tag, without committing them. Then, from an up-to-date `main` whose CI is
+green:
 
 ```sh
 bin/release minor --dry-run   # the notes and the changes, nothing written
 bin/release minor             # or patch, major, or an exact version such as 1.2.0
 ```
 
-`bin/release` dates the Unreleased section, bumps the version, commits, and pushes the commit with a signed tag
-`vX.Y.Z`. The tag starts the Release workflow, which publishes the gem to RubyGems through trusted publishing and
-creates the GitHub release with the version's notes and the `.gem`.
+`main` takes no direct push, so `bin/release` dates the Unreleased section and bumps the version on a branch
+`release/vX.Y.Z`, opens its pull request, which merges itself once CI is green, and waits for it. It then pushes a
+signed tag `vX.Y.Z` on the merge commit (`bin/release --tag X.Y.Z` does it if you stopped waiting). The tag starts
+the Release workflow, which publishes the gem to RubyGems through trusted publishing and creates the GitHub release
+with the version's notes and the `.gem`.
 
 ## License
 
 MIT, by Level UP Solutions, see [LICENSE.md](LICENSE.md). Originally written by Nando Vieira as
 [fnando/browser](https://github.com/fnando/browser) (MIT). The corpus gathers user agents from
 [ua-parser/uap-core](https://github.com/ua-parser/uap-core) (Apache 2.0),
-[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT) and browser's test fixtures: see
-[corpus/SOURCES.md](corpus/SOURCES.md).
+[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT) and browser's test fixtures; the bot
+list and its categories come from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT)
+and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (MIT); the crawler IP ranges are those their
+owners publish for that purpose: see [corpus/SOURCES.md](corpus/SOURCES.md).

data/data/bot_categories.json

@@ -0,0 +1 @@
+{"academic_research":["a patent crawler","academicbotrtu","aranea","autoconfig test from ustc","blackboard","bubing","cdscbot","cispa webcrawler","citeseerxbot","citoid","classla-web","easybib autocite","fedineko","gaisbot","gulper web bot","hgfalphaxcrawl/","icc-crawler","integromedb","internetmeasurement","irlbot","macocu","mithril-crawler","researchbot","researchscan.comsys.rwth-aachen.de","semanticscholarbot","spiderling","thesis-research-bot","thinklab","turnitin","turnitinbot","ugaresearchagent","vu-server-health-scanner/","web measure/","zoterotranslationserver/wmf"],"advertising_and_marketing":[";owler","a360-search","adbeat.com","adbeat_bot","adkerneltopiccrawler","admantx","adsbot-google","adsbot-iab","adscanner","adstxtcrawler","advbot","amazing-searchbot","amazonadbot","audigentadbot","bidswitchbot","centro ads.txt crawler","channable","clickagy","contextad bot","contextualbot","criteobot","cxensebot","dvbot","ecovadissustainabilitybot","ezlynx","ezoicbot","facebookcatalog/","google-ads-conversions","google-adwords-express","google-adwords-instant","grapeshotcrawler","groovinaadsbot/","ias crawler","impact.com agent","kargobot-artemis","mediapartners-google","mediavinemetadataparser/","meta-externalads/","mgidbot","mixrankbot","moatbot","nanointeractive","netseer crawler","netshelter contentscan","newsroom.bi","nostocrawlerbot","outbrain","outclicksbot","peer39_crawler","pixalate.com","productsup.io/crawler","proximic","pulsepoint","quantcastbot/","qwarrybot","scope3","scraping@nytimes.com","smartologybot/","springservebot","tracking-quality-spider","ttd-content","uipbot","verity/","voluumdsp-content-bot","webgains-bot","weborama-fetcher","webspidermount","wknd-bot","yahoo ad monitoring","yahoo-ad-monitoring"],"ai_agent":["agi agent","aranet-searchbot","cartai agentic commerce","chatgpt agent","claude-code","claude-web","crawl4ai","grok bot","harkbot","iaskbot","iaskspider","kimi-agent","kunatocrawler","novaact","opencode","payhawk invoice fetching agent","qodercli","reflectionbot"],"ai_assistant":["amazon-qbusiness","amzn-user","bigsur.ai","chatgpt-user","claude-user","cohere-ai","duckassistbot","elevenlabsbot","exauser","geisthaus-pagefetcher","gemini-deep-research","googleagent-urlcontext","henkbot","kagi-fetcher","kimi-user","klaviyoaibot","linerbot","meta-externalfetcher","mistralai-user","notebooklm","perplexity-user","phindbot","poggio-citations","qualifiedbot","shap-user","yiyanbot"],"ai_crawler":["agenttimes","ai2bot","aisearchbot","anthropic-ai","apifybot","apifywebsitecontentcrawler","awario","bedrockbot","bixelbot","brightbot","bytespider","chatglm-spider","claudebot","cloudvertexbot","cragcrawler","crawlspace","datenbank crawler","diffbot","doubaobot","echobot bot","echoboxbot","elevenlabscrawler","erniebot","extecontextcrawl","facebookbot","firecrawlagent","flyriverbot","friendlycrawler","google-firebase","gptbot","hunyuan","imagemind","imagespider","img2dataset","isonai-user","isscyberriskcrawler","kangaroo bot","kimibot","laion-huggingface-processor","linkreader/","meta-externalagent","moonshotbot","mozilla-tabstack","mycentralaiscraperbot","nagetbot","netestate imprint crawler","newsai","novellum","oai-adsbot","perplexityuser","quillbot","qwenbot","sbintuitionsbot","scrybot","semantic-visions","spider.com","taragroup intelligent bot","the knowledge ai","thinkbot","tiktokspider","tsm-turingos","wardbot","wpbot","xai-grok","yibot"],"ai_search":["aiwebindex","amazon-kendra","amzn-searchbot","andibot","anomura","atlassian-bot","azureai-searchbot","ccbot","channel3bot","claude-searchbot","commoncrawl","exasearchbot","google-extended","grokappandroid","grokbot","keenablebot","kimi-searchbot","laiondownloader","linkupbot","mistralai-index","oai-searchbot","perplexitybot","poseidon research crawler","querit-searchbot","queritbot","shapbot","tavilybot","terra cotta","terracotta"],"archiver":["archive-it","archive.org_bot","archivebot","archivebox","archiveteam archivebot","arquivo-web-crawler","arquivobot","authory","bibliotheque nacional de france crawler","bl.uk_lddc_bot","bl.uk_ldfc_bot","bne.es_bot","bnf.fr_bot","bnfbot","cc metadata scaper","cyotekwebcopy","europarchive.org","heritrix","ia_archiver","kaikki.org-digital-archive","kb.dk_bot","lac_iaharvester","library of congress web archiving","linkace/","linkarchiver","loc.gov/programs/web-archiving","miniature.io/","mirrorweb.com","mirrorwebcrawler","nlnz_iaharvester","nlux_iaharvester","offline explorer","sitesucker","smarshbot","trove","wayback","web-archive-net.com.bot","webcapture","webcopier","webzip","xy-archive-compliance"],"browser_automation":["amazon-bedrock-agentcore-browser","amazonbuyforme","anchor browser","fluid","ghost inspector","google-agent","google-gemini-cli","googleagent-mariner","headlesschrome","lightpanda","manus-user","opencode-smartfetch","phantomjs","playwright","puppeteer","selenium","splash version/","testlocally/","trae/","twinagent"],"feed_fetcher":["allafrica","artemis web reader","atom feed robot","bazqux","bestchange","blingerp","bloglines","blogtrottr","buttondown","capitalonebot","collapsify","commafeed","cozi-icalendar-feedreader","dlvr.it","everyfeed-spider","fedoraplanet","feed-fetcher","feedbin","feedbot","feedburner","feeder /","feeder.co","feedfetcher-google","feedflow","feedly","feedpresso content index bot","feedsearch bot","feedsearch-crawler","feedspot","feedwind","fever","flipboardrss","freshrss","g2reader-bot","innologica","inoreader","magpierss","mail.ru","mailchimp","miniflux/","monitorss/","naver blog rssbot","netvibes","newsblur","nicheindex","openrss","overcast/","page2rss","pocketcasts/","podchaserparser","podimo","protopage/","qcbot","rss reader","rss-is-dead.lol web bot","rss-parser","rss.social","rss2tg","rssapi/","rssbot/","rssingbot","rssmicro","rssreaderbot","shortpixel","simplepie","sottopopnone","specificfeeds","superfeedr","theoldreader.com","tiny tiny rss","upday/","yandorirssbot","yelpspider"],"http_library":["aiohttp","alittle client","amazon cloudfront","anyevent","apache-httpclient","apachebench","asynchttpclient","axios","btwebclient","coldfusion","colly","crawler4j","curl","dap/nethttp","download ninja","eventmachine httpclient","go 1.1 package http","go http package","go-http-client","goparserbot","htmlparser","http_get","http_request2","httpclient","httpie","httpunit","httpurlconnection","httpx","httrack","jaunt","jetty","larbin","libwww-perl","lwp-trivial","mechanize","newspaper/","node-fetch","okhttp","pcore-http","phpcrawl","pycurl","python-httplib2","python-opengraph","python-requests","python-urllib","rawweb-bot","salesforce.com","scrapy","siege","simplecrawler","sindresorhus/got","trafilatura","typhoeus","wget"],"monitoring_and_analytics":["abevalbot","accessible web bot","accessstatus","acquia optimize","activecomply","alertsite","appinsights","appsignal","atvowbot","automaton","better uptime bot","betteruptimebot","binarycanary","blackbox exporter/","bling erp","blockaid","blogvault","buddybot","bugsnag","capsulechecker","carbon-umbrella-bot","catchpoint","caveman-hunter","changedetection","check_http","checkhost","checkly","checkmarknetwork/","chirp","chrome-lighthouse","cloudflare","cludo","corporama matcher","crazy egg","critical css bot","cron-job.org","cronless","curebot","dareboost","dark visitor","datadog agent","datadog synthetic","datadogsynthetics","deepnoc","determ","dmbrowser","dominicbot","dotcom-monitor","dow jones searchbot","downnotifier.com","downtimedetector/","drataautopilot","dreamhost data team","drupalbot","dubbotbot","easycron/","easydns monitoring","edgewatch","elb-healthchecker","elmah.io uptime monitoring","elmahio-uptimebot","emoney advisor","evouptimebot","experibot","fastdast","fedidb","fedilist agent","fedistatscrawler","fedreporter bot for ffiec","feedvalidator","freshbot","freshpingbot","genomecrawlerd","google page speed insights","google trust services","googlestackdrivermonitoring","googlestackdrivermonitoring-uptimechecks","gotosocial","gotsitemonitor.com","gtmetrix","halobot","hanaleibot","headonlyscraper","hetrixtools","honeybadger uptime check","honeybadgerbot","hosttracker/","hydrozen.io","infosifter","internetvista monitor","irokez.cz monitoring","isdownbot/","jetmon/","keycdn-tools/","known agent","linko","linkpadbot","loadtimebot","logicmonitor","loopimprovements.com","magnetmebot","magus bot","mediatoolkitbot","minicrawler","minoru's fediverse crawler","modatscanner","monitis -","monitoring360bot/","monsidobot","monspark/","montastic-monitor","montasticmonitor","mxtoolbox","netumo","netvigie","new york times newsgathering","newrelicmonitor","newrelicpinger","newrelicsynthetics","newsify feed fetcher","nixstatsbot","noahbot","nodeping","noibu","observepoint","ohdear/","onetrust","onlineornot.com_bot","owler-w","paessler","pagespeed/","panopta","parselysharesbot","pingadmin.ru/","pingdom","pingping.io/","plagawarebot","poduptime","pomothy-bot","prtg network monitor","prtgcloudbot/","pulsetic.com","qualys","rackspace monitoring/","rakuten image extraction bot","rigor)","ruxitsynthetic","scouturlmonitor","sentinel-linkcheck","sentry/","sequelwp","server density service monitoring","serverdensity","silktide","sindup","site24x7","siteimprove.com","sitelock","siteuptime","sleepbot","solarwinds/","spectate/","splunk","sqwatcher","statuscake","streamline3bot/","sv-watchagent","synthesibot","synthetic-monitoring-agent/","termly","thousandeyes","tsmbot","updown","uptime-kuma","uptime/","uptimebot.org","uptimebot/","uptimedoctor","uptimerobot","uptimestatistics","uptimia","uptrends","urlsuma.de crawler","validator.nu","videootv bot","vigil","vmcrawl","w3c-checklink","w3c-mobileok","w3c_css_validator","w3c_i18n-checker","w3c_unicorn","w3c_validator","wasalive-bot","watchbot monitoring robot","watchforbot","watchful","watchmouse","webmon","websiteops","websitepulse","wedos online","wheresitup.com/","wisebot","wormly","wovn crawler","wpmu dev hub","wpmudev uptime monitor","wpumbrella","www.uptime.com","ylt chrome","zabbix"],"page_preview":["aasa-bot","addthis","arenaunfurlbot","awe.sm","bingpreview","bluesky-domain-status-classifier","bluesky/","blueskypreviewbot","bufferbot","bufferlinkpreviewbot","chatwork linkpreview","clickuplinkunfurler","dazzle bluesky bot","dingtalkbot-linkservice","discordbot","discourse forum onebox","embedly","evernoterichlinkbot","everyonesocialbot","facebookexternalhit","facebot","fastmailua","flipboardproxy","friendica","gg peekbot","github-camo","goodreads","google web preview","googleimageproxy","groupmebot","hatena","hatena::bookmark","iframely","jakarta commons","jetslide","keybasebot","lemmy","linespider","linkedinbot","listgembot","longurl","loomlybot","mastodon","mattermost-bot/","meta-externalhit/","meta-webindexer","metaiab facebook","microsoftpreview/","newsharecounts","notion/","odklbot/","opengraph.io/","orlo-linkpreview/","pinterest","quicklook","quora-bot","redditbot","shortlinktranslate","shortwave image fetcher","skypeuripreview","slack","slack-imgproxy","slack-linkexpanding","slackbot","snap url preview service","snapchat","snapurlpreview/","summalybot","supabase paired crawler","synapse","telegrambot","tumblr/","tweetmemebot","twilioproxy/","twitterbot","valve/steam","vkrobot","vkshare","whatsapp","xing bot","yahoo link preview","yahoomailproxy"],"scanner":["abuse.xmco.fr","acunetix","akismetbot","aliyunsecbot","antibot","artemisbot","asnriskscorer","bad neighborhood header detector","binlar","bitdiscovery","bitsightbot/","blackduck-fd","browserbot-observer","censysinspect/","cert pl","certchief","cookie-maestro","cookiebot","cookiehub bot","cookiehubscan","cookiehubverify","cookieyesbot","csirt.cz","cyberpatrol","detectify","digicert dcv","dirbuster","dnsscanner","do not track verifier","ec2linkfinder","emailwolf","expanse","feed image audit","filterdb.iss.net/crawler","foregenix","google-trust-services/","hardenize","hstspreload-bot","http banner detection","ip-web-crawler.com","iubenda-radar/","jugendschutzprogramm-crawler","k7mlwcbot","keydrop.io","krawlerbot","l9explore","l9scan","legalmonster","let's encrypt","lightspeedsystemscrawler","ltx71","masscan","mrgbot/","nessus","netcraft","netsystemsresearch","nikto","ning","ninja bot","nmap scripting engine","nstoolsbot/","openvas","probelyspdr/","projectshield-urlcheck","reverseengineeringbot","riddler","safednsbot","sansec security monitor/","sbl-bot","securityheaders","software-security-research","sqlmap","ssl labs","sucuri","test certificate info","tls tester","twinwavescanner","unshorten.it!","urlbeebot","ut-dorkbot","vaultpress","verispider","virustotal","visionheight.com/scan","wanscannerbot","watchtowr","whatweb/","wpscan","wpsec/","wtotem","zgrab"],"search_engine_crawler":["360spider","7siters","a6-indexer","aboundex","acoonbot","addsearchbot","adidxbot","aihitbot","alexa","algolia crawler","amazonbot","amazonproductdiscovery","amazonsellerinitiatedlisting","apis-google","appengine-google","applebot","ask jeeves","aspiegelbot","baidu","biglotron","bingbot","blekkobot","bot.araturka.com","br-crawler","bravebot","cliqzbot","coccoc","convera","coveobot","daumoa","deusu/","developers.google.com/+/web/snippet","developers.google.com/+/web/snippet/","discobot","duckduckbot","duckduckgo-favicons-bot","edisterbot","exabot","fast enterprise crawler","fast-webcrawler","fediindex","femtosearchbot","findfiles.net","findthatfile","findxbot","fiperbot","flamingo_searchengine","fleebsbot","fluffy","fr-crawler","freespoke/","funnelback","fyndsearchengine-crawler","fyndsearchengine-recrawler","geedobot","geedoproductsearch","geedoshopproductfinder","gigablast","gigabot","globalwebsearch","gluten free crawler/","golfe/","google favicon","google-certificates-bridge","google-inspectiontool","google-physicalweb","google-read-aloud","google-safety","google-site-verification","google-structured-data-testing-tool","google-xrawler","googlebot","googleother","googleproducer","greppr web crawler","grub.org","gsa-crawler","https://developers.google.com/+/web/snippet","iboubot","ichiro","imagesiftbot","indeedbot","indeedjobbot","inetdex-bot","intelx.io_bot","isonsearchbot","istellabot","jobboersebot","jobswithgptcom-bot","jooble","jyxobot","kagibot/","lb-spider","level9searchbot/","linguee bot","lyonl-asset-proxy","lyonl-crawler","magibot","mail.ru_bot","matchorysearch","mojeekbot","msnbot","msnbot-media","msrbot","napbot","naverbot","nestdaddybot","nicecrawler","ntentbot","nutch","officestorebot","oicrawler","openhosebot","openindexspider","opentheboxbot","orangebot","owler (ows.eu","pangubot","petalbot","phxbot","playstore-google","psbot","psbot-page","purebot","qwantbot","qwantify","safesearch microdata crawler","scribdbot","scrubby","search.marginalia.nu","searcherweb","searcherxweb","searchexpress","seekbot","seekport","seznambot","seznamhomepagecrawler/","sitesearch360/","slurp","smtnetpmbot","sogou","sosospider","startpageprivateimageproxy/","stepstonecrawlbot","storebot-google","stractbot","swiftbot","swisscows","tagoobot","teoma","theinternetsearchx","timpibot","top100.rambler.ru crawler","toutiaospider","trovitbot","umbot","usinenouvellecrawler","voilabot","vsusearchspider/","wadoobot","wbsearchbot","wepchsearchengine","wesee","wikido","wocbot","wordupinfosearch","woriobot","wotbox","yacybot","yandex","yanga","yeti","yisouspider","youbot","youdaobot","zealbot","zoominfo-","zyborg"],"search_engine_optimization":["007ac9 crawler","2ip bot","2ip.ru","abot","acapbot","adagiobot","adminlabs","advanced_crawler","adventurer","agakidsbot","agencyanalyticsbot","ahrefsbot","ahrefssiteaudit","alienfarm","allorigins","anderspink","apercite","appsiteassociation","aragog/","asana/","attracta","audisto crawler","audistobot","awesomecrawler","b2b bot","backlinkcrawler","barkrowler","bdbot/","bdcbot","bdfetch","beeperbot","behloolbot","bitbot","bitbucket-webhooks","bitlybot","blexbot","blp_bbot","boardgamepricesbot","bot-pge.chlooe.com","botify","botpoke","boxcarbot","brainobot","brandonbot","brandverity","brandwatch","brightedge crawler","brokenlinkcheck.com","bublupbot","bushbaby","butterfly","buzzbot","caliberbot","caliperbot","capitaloneshopping","careerbot","centcombot/","centuryb.o.t9","centurybot","certytags","chargebeebot","charlotte","chrome privacy preserving prefetch proxy","cincraw","cipacrawler","cirrusexplorer","claritybot/","clearscopebot","cloudtrellis","cmschecker","cocolyzebot","cognitiveseo bot","cognitiveseo.com","collection@infegy.com","companybook-crawler","content crawler spider","contentkingapp","contxbot","convermax","cotoyogi","crawler_eb_germany","crawlson","crawlyprojectcrawler","criticalcss.com","crunchbot","crusty/","crystalsemanticsbot","cutbot","cxk_bot","daspeedbot","datafeedwatch","dataforseobot","datagnionbot","datanyze","dataprovider.com","dcrawl","dead link checker","deadlinkchecker","deepcrawl","deskyobot","df bot","digg deeper","digincore bot","disqus","dmbot","dnbcrawler","dnyzbot","domain re-animator bot","domaincrawler","domains project/","domainsbot","domainstatsbot","domcopbot","dotbot","dragonbot","drupact","e.ventures investment crawler","easybill-importmanager","edansbot","electricmonk","emoneybot","epicbot","epivozcrawler","erepublik.tools","eright","ev-crawler/","evc-batch","exodusmovement","extlinksbot","ezgif","ezooms","factset_spyderbot","fdl stats bot","fetch/","fidget-spinner-bot","finditanswersbot","findlink","firmograph","friendly testing bot","friendly-spider","fuelbot","fullstorybot/","fuseonbot/","fyrebot","g00g1e.net","g2 web services","garlik","gdnplus.com","genieo","getthit.com","giftedvisitor scan","gingercrawler","gnam gnam spider","gnowitnewsbot","goodzer","google-apps-script","google-area120","googleassociationservice","googledocs","gopay","gowikibot","grammarly/","gregcrawler","grobbot","grouphigh/","grover/","gslfbot","guestpostsbot/","happywing","harsilbot/","hawaiibot","hcardvalidator","hello world","helloworkjobpostingbot/","hifibot/","hlidam.to robot","hotjar","hubspot","hype machine/","hypestat","hyscore.io","ibm crawler","icbot/","idg/eu","ifttt/","imagefetcher/","imgproxy/","impendoom-bot/","imrbot","indy library","infoobot","innguma/","instapaper","integromat/","intelium_bot","interfaxscanbot","ioncrawl","ips-agent","iskanie","it2media-domain-crawler","james bot","jamie's spider","java","jpg-newsbot","jumio","kangaroobot/","kazbtbot/","kemvibot","keys-so-bot","kinsta-bot","klaviyo/","komodiabot","konturbot/","kosmiobot","kstandbot","kukei.eu-bot/","landau-media-spider","laserlikebot","lastmodbot/","leikibot","linkcheckerbot","linkdex","linkdexbot","linkisbot","linksindexerbot/","linktiger","linkwalker/","lipperhey","livelapbot","lssbot","lssrocketcrawler","luminator-robots","macrobondbot","madbbot/","magellan","magicsearchdev/","magnet.me-web/","magpie-crawler","mainwp/","managewp","mappydata","marketgoo/","marketingminer","mauibot","mavifinds","mb-linkchecker","mbcrawler","mediamonitoringbot/","megaindex","megaindex.ru","meltwaternews","memorybot","mergadobot","metainspector","metajobbot","metauri","metorik","mindupbot","missinglettrbot/","mixnodecache/","mj12bot","mlbot","modernizebot","modularconnector/","mollie http client","monibot","montools.com","moodlebot","moreover","motominerbot/","mtrobot","muckrack","multiviewbot","my-tiny-bot","mybot/","nbertaupete95","nerdbynature.bot","nerdybot","netapi","netestate ne crawler","neticle crawler","netpeakcheckerbot/","netresearchserver","newsnow/","niki-bot","nimbostratus-bot","nitro-","nitrobot","nomore404.com robot","noorobot","nooshub/","nuzzel","nvdorz","nyt_scraping","omgili","omnisend/","oncrawl/","online domain tools","opengraphcheck","operator/","optimizer)","orbbot/","outsellurlvalidator","page-preview-tool","pagepeeker/","pagething","pandalytics","panscient","paperlibot","paqlebot/","parse.ly scraper/","paypal/","pdf24 url to pdf","penthouse critical path css generator","piplbot","plesk screenshot bot","pocketparser","postrank","potions/","pr-cy.ru","pressenginebot","pricedroneshoppingbot/","priecobot/","primalbot","printfriendly.com","privacyawarebot","pro-sitemaps/","ps_daily/","pwabuilderhttpagent","pywikibot/","rankactivelinkbot","rankurbot/","ravencrawler/","re-animator","readable/","reelevant/","refindbot","regionstuttgartbot","remove.bg/","repolookoutbot","retool/","retrevopageanalyzer","retrolistecom/","revvimgort/","reward-gateway","ridderbot","rivva","robotschecker/","rogerbot","rsiteauditor","ruby","rytebot","s4a-probe-bot/","sandobacrawler/","schema-markup-validator","scoop.it/","scoutjet","scrapeherobot/","screaming frog seo spider","screeenly-bot","searchatlas","sebot-wa","seewithkids","semaltbot/","semanticbot","sempi.tech","semrushbot","sendgrid event api","sentibot","senutobot","seo-audit","seo4ajax.com","seobilitybot","seokicks","seolizer","seolyt","seoscanners","seostar.co","seoulbot","serankingbacklinksbot","serendeputybot","serpstatbot/","serptimizerbot","serverhunterspider/","sfdc-callout/","shopify-captain-hook","sidetrade indexer bot","simplescraper","sincerasyntheticuser/","sistrix","siteauditbot/","sitebot","sitebulb","sitecheck","siteexplorer.info","sitescorebot","sitesoverpagesbot","skroutz imagebot","skroutzbot","skyworkspider","slaccalebot","slickbot/","smtbot","snacktory","snipcart/","socialrank","sora pos/","sparkpost","sparkshipping","spbot","ssssbot/","stape/","startmebot/","statistikaustria/","statsdronebot","statusnestbacklinkspider","storygizebot","stripe/","summify","surdotlybot","surveybot","sustainabilitycrawler","svix-webhooks/","swifteqlinkchecker","swimgbot","sysomos","t3versionsbot","tactiscout/","tangibleebot","tchelebi/","test-bot","testcrawler","testuri","textrazor","tigerbot","tineye","tombapublicwebcrawler","toplistbot","traackr","tracemyfile","trellis-services","trendictionbot","trendsmap","trentwil.es","trustly/","tweakers","tweetedtimes","twingly","twurly","uaslinkchecker/","upflow","urlappendbot","urlcheckr/","useebookchecker/","vagabondo","vebidoobot","velenpublicwebcrawler","veoozbot","vercel-screenshot","vercelbot","videootvbot","vuhuvbot","web screen service by hyperhost","webceo","webcompanycrawler","webdatastats","webflow","webprosbot/","website-info.net-robot","websniffer/","webwikibot/","webz.io","webzio","wellknownbot","whatsmyip.org","wheregoes.com","wirereaderbot/","wjhro/","wmtips.com/","woobot","woorank","wordcountbot/","wordpress","wordup-1","workona-favicon-service/","worldbot","wovncrawler/","wowlink crawler/","wp time capsule","wpmu dev broken link checker","wrtnbot","xenu link sleuth","xml sitemaps generator","xovibot","xovionpagecrawler","yahoocachesystem","yellowbrandprotectionbot","yextbot/","yokoygroupag/","yoozbot","yuuperbot","zapier","zebra-v2-bot","zenback bot","zendesk webhook","zombiebot/","zoombot","zoominfobot","zuperlistbot/","zzhbot"],"security":["comodo ssl checker","comodo-webinspector-crawler"]}

data/data/bot_exceptions.json

@@ -0,0 +1 @@
+["ruby build","pinterest/android","pinterest/ios","yandexsearchbrowser"]