browserx 1.0.0 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b8b37f466da96715f58be6f7581204d5a46fe12ed3ea96995b46cbcc48a95e1f
-  data.tar.gz: ff7419bb6d7162a3abcf82dfac7e851dd50d55605dbea351eac97b11d819cacb
+  metadata.gz: 5b7ec2cc79907813c061fbd0fd29de46378a16b55266853496a4bc9ed704a0b3
+  data.tar.gz: 624eddba5541547f0e43e4de1d70d12082e40fb4f1f9267423915b7db00c5b1a
 SHA512:
-  metadata.gz: be7daa1b157872be556fe319352180b12f978f42aedf23e9e43eb880905e4d6d849cf7ea268281ca86cf56d3baf46707f4aafd785ab0c589a5fdef57d99e650f
-  data.tar.gz: 19d21da9ee4eb84763cb4609c2b160f9ac71f6656a783ec4d31d87c2558ee71a7d27e99f4e5437119998dcc2f13c2c00abb8ce478e8d8314b1012c76773cc1fd
+  metadata.gz: 9109d7b4da2dfc23f155c834b85fa86c8d477739b6afaab08fe4ad7bd3e65acfe24ae837b156d0d32679a3397990407055530221f54c1b0d47f01d916e3547b5
+  data.tar.gz: 93be1a0915373089c25935ef3877a7a6c10dd9d853a619480f7272c41161ebe9b25e0653280301f543d434893963949bb65cf901c35a07cc33ddcfe9e65bc2bf

data/CHANGELOG.md

@@ -2,6 +2,43 @@
 
 ## [Unreleased]
 
+## [2.0.0] — 2026-11-04
+
+A repeated user agent is now answered in 2 µs, about 600 more bots are recognised, and bots, Cloudflare headers and
+Client Hints can be checked against the request. Without the new headers, every answer stays browser 6.2.0's,
+except bot detection, which finds more bots.
+
+**Breaking**
+- `bot?` is true for about 600 more user agents: bots from crawler-user-agents and ai.robots.txt, synced weekly,
+  add 1,247 entries to `bots.yml`. Of the crawlers added there since 2024, browser misses 353 out of 882 and
+  BrowserX 11. Analytics that filter on `bot?` will count fewer humans from the day you switch.
+- The Rails helper and the middleware now pass the request's headers, so Client Hints refine the answers on
+  Chromium: Windows 11 is no longer `windows10?`, and Chrome for Android in desktop mode is an Android tablet,
+  not a Linux desktop.
+
+**New**
+- `bot.category` (16 categories, close to Cloudflare's verified bot categories), `bot.ai?` and
+  `bot.user_triggered?`. Test drivers (Playwright, Puppeteer, Selenium, headless Chrome) get the
+  `browser_automation` category but are not bots, so system tests keep running as humans.
+- `bot.verified?(ip)` checks a crawler against the IP ranges its owner publishes (Google, Bing, OpenAI, Apple,
+  Perplexity, DuckDuckGo), shipped with the gem and refreshable at runtime; reverse DNS is opt-in. `bot.owner`
+  names the owner.
+- `Browser.new(ua, headers: request.env)` reads Cloudflare's bot headers (`browser.cloudflare`: verified bot,
+  category, score, country; header names configurable) and Chromium's Client Hints (`browser.client_hints`,
+  `platform.windows11?`).
+- `Browser.cache_size` (default 1,000, `0` turns it off) bounds the new per-user-agent cache: about 1 KB per
+  entry, so about 1 MB per process.
+
+**Faster**
+- Answers are cached per user agent, matchers that can't match are skipped, and bots are found through a byte
+  trie. Ruby 4.0.9: a repeated user agent takes 1.8 µs instead of 57.5 with YJIT and 3.6 µs instead of 72.5
+  without; 2,000 distinct user agents take 24.2 µs instead of 62.4 with YJIT and 43.9 µs instead of 77.3
+  without. Objects per request: 9 instead of 116 repeated, 50 instead of 125 distinct.
+
+**Fixed**
+- Loading `browser` and `browserx` together raises a clear `LoadError` instead of one silently overriding the
+  other.
+
 ## [1.0.0] — 2026-11-04
 
 First release as `browserx`, forked from browser 6.2.0 (`dd88933`). Same API: `gem "browserx"` replaces
@@ -25,5 +62,6 @@ First release as `browserx`, forked from browser 6.2.0 (`dd88933`). Same API: `g
 - A corpus of 20,328 user agents, with snapshot and diff scripts proving answers match upstream, and a
   request benchmark.
 
-[Unreleased]: https://github.com/next-gen-gems/browserx/compare/v1.0.0...HEAD
+[Unreleased]: https://github.com/next-gen-gems/browserx/compare/v2.0.0...HEAD
+[2.0.0]: https://github.com/next-gen-gems/browserx/compare/v1.0.0...v2.0.0
 [1.0.0]: https://github.com/next-gen-gems/browserx/releases/tag/v1.0.0

data/README.md

@@ -1,12 +1,19 @@
 # BrowserX
 
-Browser detection for Ruby, with ActionController integration. BrowserX is a drop-in replacement for the
-[browser](https://rubygems.org/gems/browser) gem: put `gem "browserx"` in your Gemfile instead of
-`gem "browser"`, and every call site keeps working — same `Browser` namespace, same `require "browser"`, same
-predicates. It answers nearly twice as fast (61 µs and 129 objects per request instead of 110 µs and 245 on
-Ruby 4.0 with YJIT), and its bot list is meant to be kept fresh from permissively licensed sources
-([crawler-user-agents](https://github.com/monperrus/crawler-user-agents),
-[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt)).
+Browser detection for Ruby, with ActionController integration: the API of the
+[browser](https://rubygems.org/gems/browser) gem, faster, and a bot list that stays fresh. Put `gem "browserx"` in
+your Gemfile instead of `gem "browser"`, and every call site keeps working — same `Browser` namespace, same
+`require "browser"`, same predicates.
+
+- **Faster**: 25 µs per request instead of 126 µs with browser 6.2.0, 2 µs once a user agent was seen (Ruby 4.0.9,
+  YJIT, `bench/request.rb`).
+- **Fresher bots**: synced every week from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents)
+  and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt), with categories (`bot.category`, `bot.ai?`).
+  Of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out),
+  browser misses 353, BrowserX 11.
+- **Verified signals**: `bot.verified?(ip)` checks a crawler against the IP ranges Google, Bing, OpenAI, Apple,
+  Perplexity and DuckDuckGo publish, `browser.cloudflare` reads the headers of a Cloudflare Transform Rule, and
+  Chromium's Client Hints tell Windows 11 from Windows 10.
 
 Site: https://next-gen-gems.github.io/browserx/
 
@@ -64,6 +71,9 @@ browser.epiphany?
 # Get bot info
 browser.bot.name
 browser.bot.search_engine?
+browser.bot.category        # :ai_crawler, :search_engine_crawler, :scanner… (nil for a browser)
+browser.bot.ai?
+browser.bot.user_triggered? # an AI agent acting for a person right now (ChatGPT-User, Claude-User)
 browser.bot?
 browser.bot.why? # shows which matcher detected this user agent as a bot.
 Browser::Bot.why?(ua)
@@ -187,9 +197,17 @@ end
 ### Migrating from browser
 
 Replace `gem "browser"` with `gem "browserx"` in the Gemfile, nothing else: `require "browser"`, the `Browser`
-namespace and every predicate stay. To roll back, put `gem "browser"` back. BrowserX requires Ruby 3.3 or later
+namespace and every predicate stay. Both gems define `Browser`, so BrowserX refuses to load next to `browser`,
+with an error saying so, rather than letting one silently override the other. BrowserX requires Ruby 3.3 or later
 and is tested on Rails 8.0 and 8.1.
 
+What changes on its own: the bots of the weekly sync are bots from the day you switch (600 user agents of the
+corpus that browser took for humans), so analytics that leave bots out count fewer visitors from that day; 792
+user agents named `Generic Bot` get a name; no browser, platform or device answer moves, and test drivers stay out
+of `bot?`. To roll back, put `gem "browser"` back, after removing the calls to what browser lacks
+(`Browser.cache_size=`, `bot.category`, `bot.ai?`, `bot.user_triggered?`). The
+[migration guide](https://next-gen-gems.github.io/browserx/guide.html#migrating) has the details.
+
 ### Rails integration
 
 Just add it to the Gemfile.
@@ -310,6 +328,27 @@ following requirements will be considered a bot.
 - Any known bot listed under
   [bots.yml](bots.yml)
 
+Every week, `script/sync_bots.rb` adds the bots of
+[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) and
+[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (both MIT) to the end of `bots.yml`, and
+categorizes them in `bot_categories.yml`, through a pull request that lists the answers that change. How many of
+those lists' bots browser and BrowserX detect, per category, is on the
+[Bots page](https://next-gen-gems.github.io/browserx/bots.html#coverage).
+
+`browser.bot.category` names what the bot declares it is, with the names of Cloudflare's verified bot categories
+where one fits (`search_engine_crawler`, `ai_crawler`, `ai_search`, `ai_assistant`, `page_preview`,
+`feed_fetcher`, `monitoring_and_analytics`, `advertising_and_marketing`, `archiver`, `security`…), plus
+`ai_agent`, `http_library`, `scanner` and `browser_automation`. `bot.ai?` is true for the four AI categories,
+`bot.user_triggered?` for `ai_assistant` and `ai_agent`. The category is `nil` for a browser, a bot exception
+and a bot no list names.
+
+HTTP libraries (`curl`, `got`, `python-requests`) and scanners (sqlmap, Nikto, Censys) are bots, as `curl` and
+`python-requests` already were in browser. Test drivers (headless Chrome, Playwright, Puppeteer, Selenium) are
+not: an app's system tests drive a real browser through them, and the headless Chrome of Playwright and Selenium
+sends `HeadlessChrome`. They get the `browser_automation` category instead, so an app can tell them apart in
+production (`browser.bot? || browser.bot.category == :browser_automation`). PhantomJS stays a bot, as in
+browser.
+
 To add custom matchers, you can add a callable object to
 `Browser::Bot.matchers`. The following example matches everything that has a
 `externalhit` substring on it. The bot name will always be `General Bot`.
@@ -339,8 +378,34 @@ To extend the bot list, you can manipulate the methods below:
 Browser::Bot.bots.merge!(new_bots_hash)
 Browser::Bot.bot_exceptions += new_exceptions
 Browser::Bot.search_engines.merge!(new_search_engines_hash)
+Browser::Bot.category_by_keyword["faraday"] = :http_library
 ```
 
+### Verified crawlers
+
+Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish their crawlers' IP ranges. They ship in
+`bot_ip_ranges.yml`, rewritten by the weekly sync (`script/sync_ip_ranges.rb`). The lookup is a binary search, no
+network call; an IP that does not parse answers `false`.
+
+```ruby
+browser.bot.owner                                            # => :google for a user agent claiming Googlebot
+browser.bot.verified?(request.remote_ip)                     # => whether the IP is in Google's ranges
+browser.bot.verified?(request.remote_ip, reverse_dns: true)  # also Google's DNS check: two DNS queries
+Browser::Bot::IPRanges.refresh                               # fetches the ranges again, in a long-lived process
+```
+
+### Cloudflare and Client Hints
+
+`Browser.new(user_agent, accept_language:, headers: rack_env)` also reads the request's headers; the Rails helper
+and the middleware pass them. `browser.cloudflare` reads the headers a Cloudflare Transform Rule sets
+(`verified_bot?`, `bot_category`, `bot_score`) and `CF-IPCountry` (`country`); their names are configurable with
+`Browser::Cloudflare.header_names = {bot_score: "X-My-Score"}`. Chromium's Client Hints (`Sec-CH-UA`,
+`Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`, `Sec-CH-UA-Platform-Version`) refine the platform and the device, and
+`browser.platform.windows11?` needs them. Without `headers:`, every answer is the user agent's, as before. The
+rule to create and the details are in the [guide](https://next-gen-gems.github.io/browserx/guide.html#cloudflare).
+
+`bench/signals.rb` measures what they cost.
+
 ### Middleware
 
 You can use the `Browser::Middleware` to redirect user agents.
@@ -383,17 +448,33 @@ Browser.user_agent_size_limit = 4096
 Browser.accept_language_size_limit = 4096
 ```
 
+### Cache
+
+Each process keeps what it detected for the last 1,000 distinct user agents (browser, platform, device, bot
+answers) and the last 1,000 Accept-Language headers it parsed, about 1 KB per user agent: a repeated user agent
+costs about 2 µs. The cache empties itself when full, and when `Browser.matchers`, `Browser::Bot.matchers`,
+`Browser::Platform.matchers`, `Browser::Device.matchers`, the bot list or its exceptions change. Setting
+`Browser.cache_size` empties it too, for example after renaming an entry of `Browser::Bot.bots` in place.
+
+```ruby
+Browser.cache_size = 5_000 # 0 turns it off
+```
+
 ## Performance
 
 `bench/request.rb` replays what a web request asks (name, version, `bot?`, platform, device, accept-language) over
 2,000 distinct user agents of the corpus and over 20 repeated ones, median of 7. On Ruby 4.0.9 with YJIT, browser
-6.2.0 takes 110 µs and 245 objects per request, BrowserX 1.0.0 61 µs and 129 objects.
+6.2.0 takes 126 µs and 245 objects per request, BrowserX 25 µs and 52 objects on a new user agent, 2 µs and 12
+objects on a repeated one. All four runs, with and without YJIT, are on the
+[benchmarks page](https://next-gen-gems.github.io/browserx/benchmarks.html).
 
 ```bash
-ruby --yjit -I path/to/upstream/lib bench/request.rb
-ruby --yjit -I lib bench/request.rb
+bundle exec rake bench UPSTREAM=path/to/upstream # fnando/browser at dd88933
 ```
 
+It runs `ruby [--yjit] -I lib bench/request.rb --write` for each gem and JIT, which records the run (gem, commit,
+Ruby, YJIT, machine, date) in `docs/benchmarks.json`, then `rake docs:render` rewrites the site's numbers from it.
+
 ## Development
 
 ```bash
@@ -403,7 +484,7 @@ bundle exec rubocop
 ```
 
 A change that is not meant to change an answer proves it on the corpus of 20,000 user agents, against a plain
-clone of [fnando/browser](https://github.com/fnando/browser) at `dd88933`:
+clone of [fnando/browser](https://github.com/fnando/browser) at `dd88933`, as CI does on every pull request:
 
 ```bash
 ruby -I path/to/upstream/lib script/corpus_snapshot.rb > /tmp/upstream.jsonl
@@ -411,28 +492,48 @@ ruby -I lib script/corpus_snapshot.rb > /tmp/browserx.jsonl
 ruby script/corpus_diff.rb /tmp/upstream.jsonl /tmp/browserx.jsonl
 ```
 
-To add a bot, add its user agent to `test/ua_bots.yml` and its readable name to `bots.yml`, keyed by a
-lowercased string the user agent contains. A search engine goes in `test/ua_search_engines.yml`,
+The diff passes when the answers that differ are exactly those of `corpus/expected_differences.jsonl`: a change
+meant to change answers rewrites it with `ruby script/corpus_diff.rb --write …` and shows its diff.
+
+The weekly bot sync (`.github/workflows/bot-sync.yml`) runs the same steps by hand:
+
+```bash
+ruby script/sync_bots.rb                         # bots.yml and bot_categories.yml
+ruby script/bot_coverage.rb path/to/upstream/lib # docs/bot_coverage.json, and its Markdown table
+ruby script/render_docs.rb                       # the site's numbers, from docs/*.json and the lists
+```
+
+The numbers of `docs/*.html` sit between `<!-- render:… -->` markers and come from `docs/benchmarks.json`,
+`docs/bot_coverage.json`, the bot lists and `corpus/expected_differences.jsonl`: CI fails when a page differs from
+what `bundle exec rake docs:render` writes.
+
+To add a bot, add its user agent to `test/ua_bots.yml` and its readable name to `bots.yml` above the synced
+section, keyed by a lowercased string the user agent contains. A search engine goes in `test/ua_search_engines.yml`,
 `test/ua_bots.yml` and `search_engines.yml`.
 
 ### Releasing
 
-Changes go under `## [Unreleased]` in `CHANGELOG.md` as they land. To release them, from an up-to-date `main`
-whose CI is green:
+Pull requests leave `CHANGELOG.md` alone: a release starts by writing its notes under `## [Unreleased]`, from the
+pull requests merged since the last tag, without committing them. Then, from an up-to-date `main` whose CI is
+green:
 
 ```sh
 bin/release minor --dry-run   # the notes and the changes, nothing written
 bin/release minor             # or patch, major, or an exact version such as 1.2.0
 ```
 
-`bin/release` dates the Unreleased section, bumps the version, commits, and pushes the commit with a signed tag
-`vX.Y.Z`. The tag starts the Release workflow, which publishes the gem to RubyGems through trusted publishing and
-creates the GitHub release with the version's notes and the `.gem`.
+`main` takes no direct push, so `bin/release` dates the Unreleased section and bumps the version on a branch
+`release/vX.Y.Z`, opens its pull request, which merges itself once CI is green, and waits for it. It then pushes a
+signed tag `vX.Y.Z` on the merge commit (`bin/release --tag X.Y.Z` does it if you stopped waiting). The tag starts
+the Release workflow, which publishes the gem to RubyGems through trusted publishing and creates the GitHub release
+with the version's notes and the `.gem`.
 
 ## License
 
 MIT, by Level UP Solutions, see [LICENSE.md](LICENSE.md). Originally written by Nando Vieira as
 [fnando/browser](https://github.com/fnando/browser) (MIT). The corpus gathers user agents from
 [ua-parser/uap-core](https://github.com/ua-parser/uap-core) (Apache 2.0),
-[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT) and browser's test fixtures: see
-[corpus/SOURCES.md](corpus/SOURCES.md).
+[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT) and browser's test fixtures; the bot
+list and its categories come from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT)
+and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (MIT); the crawler IP ranges are those their
+owners publish for that purpose: see [corpus/SOURCES.md](corpus/SOURCES.md).