RubyGems - agentry - Versions diffs - 3.0.0 → 3.1.0 - Mend

agentry 3.0.0 → 3.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (7) hide show

checksums.yaml +4 -4
data/CHANGELOG.md +16 -1
data/README.md +213 -420
data/lib/browser/bot/ip_ranges.rb +105 -15
data/lib/browser/cloudflare.rb +3 -1
data/lib/browser/version.rb +1 -1
metadata +2 -2

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1a57ed8305d43ff3e55dce1770f63f0bf63c712e084ba3013fe2f2240fcc27f6
-  data.tar.gz: 145e9e764e8cbc76e3f40b985d37181b3d801a620998bb34ecbcae152739a15b
+  metadata.gz: ca414103cb27f143a4872d36801bd26e012ff0950552ac96646497be9a2bb8f3
+  data.tar.gz: 38d3bb2c5529a324160476b7c5874d645689bb81409234e14ba2ed7f01da4d61
 SHA512:
-  metadata.gz: 0a7b34836efd8a5cff0af7eac8e723d0027670a34bde7f1fd3e58763ba4a8f917094a76d19b4e0cb5e7ced62af3245a55ffc516310ac2d4002ea1fa421c4812d
-  data.tar.gz: ee6ba51155056619fada17e77f0997849fb1e686ca9af9424339ab788d74018e3e9d71097259e6286c5fbed478c7f84b560943a26495602169b3593b30f46e77
+  metadata.gz: 755ee15fab59790728e25954113018ca3d25fb1255933a6a029f6dabcdc2767d5781649592c88f0ba6f554f2772319d318df86b9a6009a31eba9926764f01bdf
+  data.tar.gz: ce39ec99a84d2326c2575c3a9ae8c76c3e2205283f4cf51d8b7f7ea5e6b1429e603078e2aac99947970323c89d0ba86902f32aebf014d76936c7d08edadfe215

data/CHANGELOG.md CHANGED Viewed

@@ -2,6 +2,20 @@
 ## [Unreleased]
+## [3.1.0] — 2026-11-06
+A Google Cloud VM no longer passes as a verified Googlebot, and a forged Googlebot can no longer stall requests on reverse DNS.
+**New**
+- `Browser::Bot::IPRanges.dns_cache_size` caches each reverse DNS verdict per owner and IP, failures included, for `dns_cache_ttl` seconds (3,600 by default), so a forged Googlebot pays its DNS wait once per TTL instead of on every request. It is 0, off, by default.
+- `dns_timeout` sets the timeout per query and per nameserver, as `Resolv` applies it. Invalid values raise `ArgumentError`.
+**Fixed**
+- Google's reverse DNS check accepts only `googlebot.com` and `google.com`. Any Google Cloud VM, named under `googleusercontent.com`, passed as a verified Googlebot; Google's user-triggered fetchers are covered by the IP ranges it publishes.
+- The DNS timeout drops from 2 seconds to 1, and the forward lookup uses an absolute name, so the resolver never retries it under the search domains of `/etc/resolv.conf`.
+- `IPRanges.refresh` and the weekly sync raise `Browser::Error` on anything implausible, and keep the current ranges: a prefix broader than /12 (IPv4) or /24 (IPv6), a private, loopback, link-local, multicast or unspecified range, a malformed document, or a redirect that leaves HTTPS or the owner's host.
+- Cloudflare header values over 128 bytes are ignored.
 ## [3.0.0] — 2026-11-05
 The gem is renamed Agentry and lives at levups/agentry; the API is untouched.
@@ -86,7 +100,8 @@ First release, forked from browser 6.2.0 (`dd88933`). Same API: `gem "agentry"`
 - A corpus of 20,328 user agents, with snapshot and diff scripts proving answers match upstream, and a
   request benchmark.
-[Unreleased]: https://github.com/levups/agentry/compare/v3.0.0...HEAD
+[Unreleased]: https://github.com/levups/agentry/compare/v3.1.0...HEAD
+[3.1.0]: https://github.com/levups/agentry/compare/v3.0.0...v3.1.0
 [3.0.0]: https://github.com/levups/agentry/compare/v2.1.0...v3.0.0
 [2.1.0]: https://github.com/levups/agentry/compare/v2.0.0...v2.1.0
 [2.0.0]: https://github.com/levups/agentry/compare/v1.0.0...v2.0.0

data/README.md CHANGED Viewed

@@ -1,432 +1,211 @@
 # Agentry
-Browser detection for Ruby, with ActionController integration: the API of the
-[browser](https://rubygems.org/gems/browser) gem, faster, and a bot list that stays fresh. Put `gem "agentry"` in
-your Gemfile instead of `gem "browser"`, and every call site keeps working — same `Browser` namespace, same
-`require "browser"`, same predicates.
-- **Faster**: 25 µs per request instead of 126 µs with browser 6.2.0, 2 µs once a user agent was seen (Ruby 4.0.9,
-  YJIT, `bench/request.rb`).
-- **Fresher bots**: synced every week from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents)
-  and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt), with categories (`bot.category`, `bot.ai?`).
-  Of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out),
-  browser misses 353, Agentry 11.
-- **Verified signals**: `bot.verified?(ip)` checks a crawler against the IP ranges Google, Bing, OpenAI, Apple,
-  Perplexity and DuckDuckGo publish, `browser.cloudflare` reads the headers of a Cloudflare Transform Rule, and
-  Chromium's Client Hints tell Windows 11 from Windows 10.
-Site: https://levups.github.io/agentry/
+Agentry is a Ruby gem that tells you who sent an HTTP request: which browser and version, on which operating system
+and device, or which bot. It reads the `User-Agent` header, a line of text every browser and crawler sends with each
+request, such as `Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) … Version/18.1 Mobile/15E148
+Safari/604.1`. Each browser and crawler writes it in its own way, and Agentry knows the formats so that your code
+can ask plain questions such as `browser.bot?` or `browser.device.mobile?`.
+Apps use the answers to leave bots out of their analytics, to turn crawlers away, to adapt a page to phones, or to
+warn visitors on an old browser. Agentry works in any Ruby program, and gives Rails controllers and views a
+`browser` helper for the current request.
+Website and guide: https://levups.github.io/agentry/
+## Why Agentry
+Agentry is a fork of [browser](https://github.com/fnando/browser), the gem Nando Vieira wrote, which has not changed
+since June 2025. Code written for browser runs on Agentry as it is: same `Browser` class, same `require "browser"`,
+same methods. Here is what is different:
+- It answers in 25 µs per request instead of 126 µs with browser 6.2.0, and in 2 µs for a user agent the process
+  has seen before (Ruby 4.0.9 with YJIT, see [Performance](#performance)).
+- Its list of bots is updated every week. Of the 882 crawler user agents added to
+  [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) since 2024 (test drivers and scanners
+  left out), browser misses 353 and Agentry 11.
+- It says what kind of bot sent the request: an AI crawler, a search engine, an assistant fetching a page for a
+  person, a vulnerability scanner.
+- It can check that a request claiming to be Googlebot, Bingbot or GPTBot comes from the IP addresses Google,
+  Microsoft or OpenAI publish for their crawlers.
+- It reads the headers Cloudflare can add to a request, and the Client Hints Chromium browsers send, which tell
+  Windows 11 from Windows 10.
 ## Installation
-```ruby
-gem "agentry"
-```
-## Usage
+Agentry needs Ruby 3.3 or later. It is tested on Rails 8.0 and 8.1, and works without Rails. Add it to your Gemfile:
 ```ruby
-require "browser"
-browser = Browser.new("Some User Agent", accept_language: "en-us")
-# General info
-browser.bot?
-browser.chrome?
-browser.chromium_based?
-browser.core_media?
-browser.duck_duck_go?
-browser.edge?                # Newest MS browser
-browser.electron?            # Electron Framework
-browser.firefox?
-browser.full_version
-browser.ie?
-browser.ie?(6)               # detect specific IE version
-browser.ie?([">8", "<10"])   # detect specific IE (IE9).
-browser.known?               # has the browser been successfully detected?
-browser.unknown?             # the browser wasn't detected.
-browser.meta                 # an array with several attributes
-browser.name                 # readable browser name
-browser.nokia?
-browser.opera?
-browser.opera_mini?
-browser.phantom_js?
-browser.quicktime?
-browser.safari?
-browser.safari_webapp_mode?
-browser.samsung_browser?
-browser.to_s            # the meta info joined by space
-browser.uc_browser?
-browser.version         # major version number
-browser.webkit?
-browser.webkit_full_version
-browser.yandex?
-browser.wechat?
-browser.qq?
-browser.weibo?
-browser.sputnik?
-browser.sougou_browser?
-browser.epiphany?
-# Get bot info
-browser.bot.name
-browser.bot.search_engine?
-browser.bot.category        # :ai_crawler, :search_engine_crawler, :scanner… (nil for a browser)
-browser.bot.ai?
-browser.bot.user_triggered? # an AI agent acting for a person right now (ChatGPT-User, Claude-User)
-browser.bot?
-browser.bot.why? # shows which matcher detected this user agent as a bot.
-Browser::Bot.why?(ua)
-# Get device info
-browser.device
-browser.device.id
-browser.device.name
-browser.device.unknown?
-browser.device.blackberry_playbook?
-browser.device.console?
-browser.device.ipad?
-browser.device.iphone?
-browser.device.ipod_touch?
-browser.device.kindle?
-browser.device.kindle_fire?
-browser.device.mobile?
-browser.device.nintendo?
-browser.device.playstation?
-browser.device.ps3?
-browser.device.ps4?
-browser.device.psp?
-browser.device.silk?
-browser.device.surface?
-browser.device.tablet?
-browser.device.tv?
-browser.device.vita?
-browser.device.wii?
-browser.device.wiiu?
-browser.device.samsung?
-browser.device.switch?
-browser.device.xbox?
-browser.device.xbox_360?
-browser.device.xbox_one?
-# Get platform info
-browser.platform
-browser.platform.id
-browser.platform.name
-browser.platform.version  # e.g. 9 (for iOS9)
-browser.platform.adobe_air?
-browser.platform.android?
-browser.platform.android?(4.2)   # detect Android Jelly Bean 4.2
-browser.platform.android_app?     # detect webview in an Android app
-browser.platform.android_webview? # alias for android_app?
-browser.platform.blackberry?
-browser.platform.blackberry?(10) # detect specific BlackBerry version
-browser.platform.chrome_os?
-browser.platform.firefox_os?
-browser.platform.ios?     # detect iOS
-browser.platform.ios?(9)  # detect specific iOS version
-browser.platform.ios_app?     # detect webview in an iOS app
-browser.platform.ios_webview? # alias for ios_app?
-browser.platform.linux?
-browser.platform.mac?
-browser.platform.unknown?
-browser.platform.windows10?
-browser.platform.windows7?
-browser.platform.windows8?
-browser.platform.windows8_1?
-browser.platform.windows?
-browser.platform.windows_mobile?
-browser.platform.windows_phone?
-browser.platform.windows_rt?
-browser.platform.windows_touchscreen_desktop?
-browser.platform.windows_vista?
-browser.platform.windows_wow64?
-browser.platform.windows_x64?
-browser.platform.windows_x64_inclusive?
-browser.platform.windows_xp?
-browser.platform.kai_os?
+gem "agentry"
 ```
-### Aliases
+If the Gemfile already has `gem "browser"`, replace that line: both gems define the `Browser` class, so Agentry
+refuses to load next to browser. Nothing else in your code changes, though some answers do (see
+[Switching from browser](#switching-from-browser)).
-To add aliases like `mobile?` and `tablet?` to the base object (e.g
-`browser.mobile?`), require the `browser/aliases` file and extend the
-Browser::Base object like the following:
+## A first example
 ```ruby
-require "browser/aliases"
-Browser::Base.include(Browser::Aliases)
-browser = Browser.new("Some user agent")
-browser.mobile? #=> false
-```
-### What's being detected?
-- For a list of platform detections, check
-  [lib/browser/platform.rb](lib/browser/platform.rb)
-- For a list of device detections, check
-  [lib/browser/device.rb](lib/browser/device.rb)
-- For a list of bot detections, check
-  [bots.yml](bots.yml)
-### Detecting modern browsers
-To detect whether a browser can be considered as modern or not, create a method
-that abstracts your versioning constraints. The following example will consider
-any of the following browsers as a modern:
+require "browser"
-```ruby
-# Expects an Browser instance,
-# like in `Browser.new(user_agent, accept_language: language)`.
-def modern_browser?(browser)
-  [
-    browser.chrome?(">= 65"),
-    browser.safari?(">= 10"),
-    browser.firefox?(">= 52"),
-    browser.ie?(">= 11") && !browser.compatibility_view?,
-    browser.edge?(">= 15"),
-    browser.opera?(">= 50"),
-    browser.facebook?
-      && browser.safari_webapp_mode?
-      && browser.webkit_full_version.to_i >= 602
-  ].any?
-end
+user_agent = "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) AppleWebKit/605.1.15 " \
+             "(KHTML, like Gecko) Version/18.1 Mobile/15E148 Safari/604.1"
+browser = Browser.new(user_agent, accept_language: "fr-FR,fr;q=0.9,en;q=0.8")
+browser.name                       # => "Safari"
+browser.version                    # => "18"
+browser.full_version               # => "18.1"
+browser.platform.name              # => "iOS (iPhone)"
+browser.device.mobile?             # => true
+browser.bot?                       # => false
+browser.accept_language.first.full # => "fr-FR"
 ```
-### Migrating from browser
-Replace `gem "browser"` with `gem "agentry"` in the Gemfile, nothing else: `require "browser"`, the `Browser`
-namespace and every predicate stay. Both gems define `Browser`, so Agentry refuses to load next to `browser`,
-with an error saying so, rather than letting one silently override the other. Agentry requires Ruby 3.3 or later
-and is tested on Rails 8.0 and 8.1.
-What changes on its own: the bots of the weekly sync are bots from the day you switch (600 user agents of the
-corpus that browser took for humans), so analytics that leave bots out count fewer visitors from that day; 792
-user agents named `Generic Bot` get a name; no browser, platform or device answer moves, and test drivers stay out
-of `bot?`. To roll back, put `gem "browser"` back, after removing the calls to what browser lacks
-(`Browser.cache_size=`, `bot.category`, `bot.ai?`, `bot.user_triggered?`). The
-[migration guide](https://levups.github.io/agentry/guide.html#migrating) has the details.
-### Rails integration
+The file to require is `browser` and the class is `Browser`, the names browser used. `accept_language:` is optional:
+it takes the `Accept-Language` header, the languages the visitor's browser asks for.
-Just add it to the Gemfile.
-```ruby
-gem "agentry"
-```
-This adds a helper method called `browser`, that inspects your current user
-agent.
+In Rails there is nothing to build. The `browser` helper reads the current request's headers:
 ```erb
-<% if browser.ie?(6) %>
-  <p class="disclaimer">You're running an older IE version. Please update it!</p>
+<% if browser.device.mobile? %>
+  <%= render "mobile_menu" %>
 <% end %>
 ```
-If you want to use Browser on your Rails app but don't want to taint your
-controller, use the following line on your Gemfile:
+Outside Rails, pass the Rack env as `headers:` so that Agentry can read Client Hints and Cloudflare headers too:
+`Browser.new(user_agent, accept_language: language, headers: env)`.
-```ruby
-gem "agentry", require: "browser/browser"
-```
+The [guide](https://levups.github.io/agentry/guide.html#usage) lists every method: browsers from Chrome and Safari
+to WeChat and Yandex, devices such as the iPad, Kindle, PlayStation and Xbox, and platforms down to each version of
+Windows. Version checks take a number or constraints, such as `browser.chrome?(">= 65")` or
+`browser.platform.ios?(9)`.
-### Accept Language
-Parses the accept-language header from an HTTP request and produces an array of
-language objects sorted by quality.
+## Bots
 ```ruby
-browser = Browser.new("Some User Agent", accept_language: "en-us")
-browser.accept_language.class
-#=> Array
+browser = Browser.new("Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)")
-language = browser.accept_language.first
-language.code
-#=> "en"
-language.region
-#=> "US"
-language.full
-#=> "en-US"
-language.quality
-#=> 1.0
-language.name
-#=> "English/United States"
+browser.bot?                # => true
+browser.bot.name            # => "ChatGPT"
+browser.bot.category        # => :ai_crawler
+browser.bot.ai?             # => true
+browser.bot.user_triggered? # => false
 ```
-Result is always sorted in quality order from highest to lowest. As per the HTTP
-spec:
+`browser.bot?` is true for an empty user agent, for one that matches `/crawl|fetch|search|monitoring|spider|bot/`,
+and for one that contains a name from [bots.yml](bots.yml). The net is wide on purpose, as it was in browser.
-- omitting the quality value implies 1.0.
-- quality value equal to zero means that is not accepted by the client.
+Every Monday, a workflow adds the bots of [crawler-user-agents](https://github.com/monperrus/crawler-user-agents)
+and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (both MIT) to `bots.yml` and files each under a
+category. It opens a pull request that lists every answer the change moves, and a maintainer merges it. The
+[Bots page](https://levups.github.io/agentry/bots.html#coverage) counts, per category, how many of those bots
+browser and Agentry detect.
-### Internet Explorer
+### Categories
-Internet Explorer has a compatibility view mode that allows newer versions
-(IE8+) to run as an older version. Browser will always return the navigator
-version, ignoring the compatibility view version, when defined. If you need to
-get the engine's version, you have to use `Browser#msie_version` and
-`Browser#msie_full_version`.
+`browser.bot.category` is what the bot says it is, named after Cloudflare's verified bot categories where one fits:
+`search_engine_crawler`, `ai_crawler`, `ai_search`, `ai_assistant`, `page_preview`, `feed_fetcher`,
+`monitoring_and_analytics`, `advertising_and_marketing`, `archiver`, `security` and a few more. Agentry adds
+`ai_agent`, `http_library`, `scanner` and `browser_automation`. `bot.ai?` is true for the four AI categories, and
+`bot.user_triggered?` for `ai_assistant` and `ai_agent`: a person is driving them at that moment, as with
+ChatGPT-User or Claude-User. The category is `nil` for a browser, for a bot exception, and for a bot no list names.
-So, let's say an user activates compatibility view in a IE11 browser. This is
-what you'll get:
+HTTP libraries (`curl`, `python-requests`) and scanners (sqlmap, Nikto) are bots. Test drivers (headless Chrome,
+Playwright, Puppeteer, Selenium) are not: your own system tests drive a real browser through them, and would
+otherwise count as bots. They get the `browser_automation` category, so production code can still catch them with
+`browser.bot? || browser.bot.category == :browser_automation`. PhantomJS stays a bot, as in browser.
-```ruby
-browser.version
-#=> 11
-browser.full_version
-#=> 11.0
+### Your own rules
-browser.msie_version
-#=> 7
+A matcher is anything that responds to `call(ua, browser)`. This one makes every user agent containing
+`externalhit` a bot:
-browser.msie_full_version
-#=> 7.0
-browser.compatibility_view?
-#=> true
+```ruby
+Browser::Bot.matchers << ->(ua, _browser) { ua.match?(/externalhit/i) }
 ```
-This behavior changed in browser v1.0.0; previously there wasn't a way of getting the
-real browser version.
+`browser.bot.why?` returns the matcher that fired. The lists can change at runtime too
+(`Browser::Bot.bots.merge!(new_bots_hash)`), and the
+[Bots page](https://levups.github.io/agentry/bots.html#matchers) shows how to remove the built-in matchers or go
+back to the detection of browser 2.x.
-### Safari
+### Verified crawlers
-iOS webviews and web apps aren't detected as Safari anymore, so be aware of that
-if that's your case. You can use a combination of platform and webkit detection
-to do whatever you want.
+Anyone can send `Googlebot` as their user agent. Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish the
+IP ranges their crawlers use, and Agentry ships them, refreshed by the weekly sync. `bot.verified?(ip)` says whether
+a request comes from the ranges of the company its user agent names:
 ```ruby
-# iPad's Safari running as web app mode.
-browser = Browser.new("Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405")
-browser.safari?
-#=> false
+browser.bot.owner                        # => :google for a user agent claiming to be Googlebot
+browser.bot.verified?(request.remote_ip) # => true when the IP is in Google's ranges
-browser.webkit?
-#=> true
-browser.platform.ios?
-#=> true
+if browser.bot.owner && !browser.bot.verified?(request.remote_ip)
+  head :forbidden # claims to be a known crawler, comes from somewhere else
+end
 ```
-### Bots
-The bot detection is quite aggressive. Anything that matches at least one of the
-following requirements will be considered a bot.
-- Empty user agent string
-- User agent that matches `/crawl|fetch|search|monitoring|spider|bot/`
-- Any known bot listed under
-  [bots.yml](bots.yml)
-Every week, `script/sync_bots.rb` adds the bots of
-[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) and
-[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (both MIT) to the end of `bots.yml`, and
-categorizes them in `bot_categories.yml`, through a pull request that lists the answers that change. How many of
-those lists' bots browser and Agentry detect, per category, is on the
-[Bots page](https://levups.github.io/agentry/bots.html#coverage).
-`browser.bot.category` names what the bot declares it is, with the names of Cloudflare's verified bot categories
-where one fits (`search_engine_crawler`, `ai_crawler`, `ai_search`, `ai_assistant`, `page_preview`,
-`feed_fetcher`, `monitoring_and_analytics`, `advertising_and_marketing`, `archiver`, `security`…), plus
-`ai_agent`, `http_library`, `scanner` and `browser_automation`. `bot.ai?` is true for the four AI categories,
-`bot.user_triggered?` for `ai_assistant` and `ai_agent`. The category is `nil` for a browser, a bot exception
-and a bot no list names.
-HTTP libraries (`curl`, `got`, `python-requests`) and scanners (sqlmap, Nikto, Censys) are bots, as `curl` and
-`python-requests` already were in browser. Test drivers (headless Chrome, Playwright, Puppeteer, Selenium) are
-not: an app's system tests drive a real browser through them, and the headless Chrome of Playwright and Selenium
-sends `HeadlessChrome`. They get the `browser_automation` category instead, so an app can tell them apart in
-production (`browser.bot? || browser.bot.category == :browser_automation`). PhantomJS stays a bot, as in
-browser.
-To add custom matchers, you can add a callable object to
-`Browser::Bot.matchers`. The following example matches everything that has a
-`externalhit` substring on it. The bot name will always be `General Bot`.
+The check searches the ranges in memory, with no network call, and an IP that does not parse answers `false`. Pass
+the IP as your proxies resolve it: `request.remote_ip` in Rails, with `trusted_proxies` set.
-```ruby
-Browser::Bot.matchers << ->(ua, _browser) { ua.match?(/externalhit/i) }
-```
-To clear all matchers, including the ones that are bundled, use
-`Browser::Bot.matchers.clear`. You can re-add built-in matchers by doing the
-following:
+`verified?(ip, reverse_dns: true)` adds the DNS check Google, Bing and Apple document, at the cost of two DNS
+queries for an IP outside the ranges. A fake Googlebot controls its own reverse DNS and can make that lookup wait
+for the timeout (1 second per nameserver of `/etc/resolv.conf`), so a cache, off by default, can remember each
+answer, failures included:
 ```ruby
-Browser::Bot.matchers += Browser::Bot.default_matchers
+Browser::Bot::IPRanges.dns_cache_size = 10_000 # 0, the default, caches nothing
+Browser::Bot::IPRanges.dns_cache_ttl = 3600    # seconds
+Browser::Bot::IPRanges.dns_timeout = 0.5       # seconds per query and nameserver
+Browser::Bot::IPRanges.refresh                 # fetches the ranges again, in a long-lived process
 ```
-To restore v2's bot detection, remove the following matchers:
+## Cloudflare and Client Hints
-```ruby
-Browser::Bot.matchers.delete(Browser::Bot::KeywordMatcher)
-Browser::Bot.matchers.delete(Browser::Bot::EmptyUserAgentMatcher)
-```
+When the Rack env is passed as `headers:` (the Rails helper and the middleware do it for you), Agentry reads two
+more sources.
-To extend the bot list, you can manipulate the methods below:
+Behind Cloudflare, a Transform Rule can copy what Cloudflare knows about a request into headers, and
+`browser.cloudflare` reads them:
 ```ruby
-Browser::Bot.bots.merge!(new_bots_hash)
-Browser::Bot.bot_exceptions += new_exceptions
-Browser::Bot.search_engines.merge!(new_search_engines_hash)
-Browser::Bot.category_by_keyword["faraday"] = :http_library
+browser.cloudflare.verified_bot? # => true for a bot Cloudflare verified
+browser.cloudflare.bot_category  # => :search_engine_crawler, with the names of bot.category
+browser.cloudflare.bot_score     # => 1 (automated) to 99 (human), with Bot Management only
+browser.cloudflare.country       # => "FR", from CF-IPCountry
 ```
-### Verified crawlers
-Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish their crawlers' IP ranges. They ship in
-`bot_ip_ranges.yml` (as JSON in the gem), rewritten by the weekly sync (`script/sync_ip_ranges.rb`). The lookup is a binary search, no
-network call; an IP that does not parse answers `false`.
+The header names can be changed with `Browser::Cloudflare.header_names = {bot_score: "X-My-Score"}`. These headers
+prove something only when every request goes through Cloudflare: if clients can reach your server directly, they
+can send the headers themselves. The [guide](https://levups.github.io/agentry/guide.html#cloudflare) has the rule to
+create and how to lock the origin.
-```ruby
-browser.bot.owner                                            # => :google for a user agent claiming Googlebot
-browser.bot.verified?(request.remote_ip)                     # => whether the IP is in Google's ranges
-browser.bot.verified?(request.remote_ip, reverse_dns: true)  # also Google's DNS check: two DNS queries
-Browser::Bot::IPRanges.refresh                               # fetches the ranges again, in a long-lived process
-```
+Chromium browsers no longer put the real Windows, macOS or Android version in their user agent. They send it in
+Client Hints (`Sec-CH-UA`, `Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`, `Sec-CH-UA-Platform-Version`), which Agentry
+uses to refine the platform and the device: `browser.platform.windows11?` needs them. Without `headers:`, every
+answer comes from the user agent alone, as in browser. The
+[guide](https://levups.github.io/agentry/guide.html#client-hints) shows how to ask browsers for the platform
+version. `bench/signals.rb` measures what reading these headers costs.
-### Cloudflare and Client Hints
+## Accept-Language
-`Browser.new(user_agent, accept_language:, headers: rack_env)` also reads the request's headers; the Rails helper
-and the middleware pass them. `browser.cloudflare` reads the headers a Cloudflare Transform Rule sets
-(`verified_bot?`, `bot_category`, `bot_score`) and `CF-IPCountry` (`country`); their names are configurable with
-`Browser::Cloudflare.header_names = {bot_score: "X-My-Score"}`. Chromium's Client Hints (`Sec-CH-UA`,
-`Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`, `Sec-CH-UA-Platform-Version`) refine the platform and the device, and
-`browser.platform.windows11?` needs them. Without `headers:`, every answer is the user agent's, as before. The
-rule to create and the details are in the [guide](https://levups.github.io/agentry/guide.html#cloudflare).
-`bench/signals.rb` measures what they cost.
-### Middleware
-You can use the `Browser::Middleware` to redirect user agents.
+`browser.accept_language` turns the `Accept-Language` header into a list of languages, the most wanted first:
 ```ruby
-use Browser::Middleware do
-  redirect_to "/upgrade" if browser.ie?
-end
+language = Browser.new(user_agent, accept_language: "en-us").accept_language.first
+language.code    # => "en"
+language.region  # => "US"
+language.full    # => "en-US"
+language.quality # => 1.0
+language.name    # => "English/United States"
 ```
-If you're using Rails, you can use the route helper methods. Just add something
-like the following to a initializer file (`config/initializers/browser.rb`).
+As the HTTP specification says, a language without a quality value has 1.0, and one with a quality of 0 is not
+accepted, so it is left out.
-```ruby
-Rails.configuration.middleware.use Browser::Middleware do
-  redirect_to upgrade_path if browser.ie?
-end
-```
+## Middleware
-If you need access to the `Rack::Request` object (e.g. to exclude a path), you
-can do so with `request`.
+`Browser::Middleware` redirects requests based on the browser, before they reach your app. In Rails, from an
+initializer such as `config/initializers/browser.rb`, route helpers work and `request` is the `Rack::Request`:
 ```ruby
 Rails.configuration.middleware.use Browser::Middleware do
@@ -434,57 +213,70 @@ Rails.configuration.middleware.use Browser::Middleware do
 end
 ```
-### Restrictions
+In another Rack app, `use Browser::Middleware do … end` takes the same block. To use Agentry in Rails without the
+`browser` helper, require only the library: `gem "agentry", require: "browser/browser"`.
-- User agent has a size limit of 2048 bytes. This can be customized through
-  `Browser.user_agent_size_limit=(size)`.
-- Accept-Language has a size limit of 2048 bytes. This can be customized through
-  `Browser.accept_language_size_limit=(size)`.
+## Cache, limits and boot
-If size is not respected, then `Browser::Error` is raised.
+Each process remembers what it found for the last 1,000 user agents (browser, platform, device and bot answers)
+and the last 1,000 `Accept-Language` headers, about 1 KB per user agent. A user agent seen before costs about 2 µs.
+The cache empties itself when it is full, and when `Browser.matchers`, `Browser::Bot.matchers`,
+`Browser::Platform.matchers`, `Browser::Device.matchers`, the bot list or its exceptions change. Setting
+`Browser.cache_size` empties it too, for example after renaming an entry of `Browser::Bot.bots` in place:
 ```ruby
-Browser.user_agent_size_limit = 4096
-Browser.accept_language_size_limit = 4096
+Browser.cache_size = 5_000 # 0 turns it off
 ```
-### Cache
+A user agent or an `Accept-Language` header longer than 2048 bytes raises `Browser::Error`. Both limits can be
+raised with `Browser.user_agent_size_limit = 4096` and `Browser.accept_language_size_limit = 4096`.
-Each process keeps what it detected for the last 1,000 distinct user agents (browser, platform, device, bot
-answers) and the last 1,000 Accept-Language headers it parsed, about 1 KB per user agent: a repeated user agent
-costs about 2 µs. The cache empties itself when full, and when `Browser.matchers`, `Browser::Bot.matchers`,
-`Browser::Platform.matchers`, `Browser::Device.matchers`, the bot list or its exceptions change. Setting
-`Browser.cache_size` empties it too, for example after renaming an entry of `Browser::Bot.bots` in place.
+The lists and matchers are built on first use: the first request of each process takes about 13 ms more, the
+following ones 0.2 ms. `Browser.eager_load!` builds them at boot, so that a server that loads the app before
+forking (Puma's `preload_app!`, Unicorn, Pitchfork) shares them with every worker. Rails calls it whenever it eager
+loads, which production does by default. Elsewhere, call `Browser.eager_load!` before the fork.
-```ruby
-Browser.cache_size = 5_000 # 0 turns it off
-```
+## Switching from browser
-### Preloading before a fork
+Replace `gem "browser"` with `gem "agentry"` in the Gemfile. Nothing else in the code changes.
-The lists, the matchers and the bot keyword tries are built on first use: the first request of each process pays
-about 13 ms for them, the next ones 0.2 ms. `Browser.eager_load!` builds them up front, so a server that preloads
-its app before forking (Puma's `preload_app!`, Unicorn, Pitchfork) shares them with every worker. Rails calls it on
-its own when it eager loads, which production does by default. Elsewhere, call it before the fork:
+Some answers change from the day you switch, because of the fresher bot list: 600 user agents of the corpus that
+browser took for people are bots in Agentry, so analytics that leave bots out will count fewer visitors, and 792
+user agents that browser called `Generic Bot` get a real name. No browser, platform or device answer changes, and
+test drivers still are not bots.
-```ruby
-Browser.eager_load!
-```
+To go back, remove the calls to what browser does not have (`Browser.cache_size=`, the `headers:` argument,
+`bot.category`, `bot.ai?`, `bot.user_triggered?`, `bot.verified?`, `browser.cloudflare`), then put
+`gem "browser"` back. The [migration guide](https://levups.github.io/agentry/guide.html#migrating) has the details.
 ## Performance
-`bench/request.rb` replays what a web request asks (name, version, `bot?`, platform, device, accept-language) over
-2,000 distinct user agents of the corpus and over 20 repeated ones, median of 7. On Ruby 4.0.9 with YJIT, browser
-6.2.0 takes 126 µs and 245 objects per request, Agentry 25 µs and 52 objects on a new user agent, 2 µs and 12
-objects on a repeated one. All four runs, with and without YJIT, are on the
-[benchmarks page](https://levups.github.io/agentry/benchmarks.html).
+`bench/request.rb` asks what a web request typically asks (name, version, `bot?`, platform, device,
+accept-language) over 2,000 distinct user agents of the corpus and over 20 repeated ones, and keeps the median of 7
+runs. On Ruby 4.0.9 with YJIT:
+| Per request | browser 6.2.0 | Agentry |
+|---|---:|---:|
+| A new user agent | 126 µs, 245 objects | 25 µs, 52 objects |
+| A user agent seen before | 114 µs, 217 objects | 2 µs, 12 objects |
+The [benchmarks page](https://levups.github.io/agentry/benchmarks.html) has the runs without YJIT, the machine, and
+why Agentry is faster. To run them again:
 ```bash
-bundle exec rake bench UPSTREAM=path/to/upstream # fnando/browser at dd88933
+bundle exec rake bench UPSTREAM=path/to/upstream # a clone of fnando/browser at dd88933
 ```
-It runs `ruby [--yjit] -I lib bench/request.rb --write` for each gem and JIT, which records the run (gem, commit,
-Ruby, YJIT, machine, date) in `docs/benchmarks.json`, then `rake docs:render` rewrites the site's numbers from it.
+It runs `ruby [--yjit] -I lib bench/request.rb --write` for each gem and JIT setting, records each run (gem,
+commit, Ruby, YJIT, machine, date) in `docs/benchmarks.json`, and `rake docs:render` then rewrites the site's
+numbers from that file.
+## How the answers are checked
+Agentry runs browser's own test suite, unmodified, next to its own tests. The repository also holds a corpus of
+20,328 user agents, gathered from uap-core, crawler-user-agents and browser's test fixtures. On every pull request,
+CI asks both browser and Agentry about each of them and compares the answers. An answer that differs is either
+listed in `corpus/expected_differences.jsonl` as an intended change, such as a newly known bot, or fails the build.
 ## Development
@@ -494,8 +286,7 @@ bundle exec rake test
 bundle exec rubocop
 ```
-A change that is not meant to change an answer proves it on the corpus of 20,000 user agents, against a plain
-clone of [fnando/browser](https://github.com/fnando/browser) at `dd88933`, as CI does on every pull request:
+To compare every answer with [fnando/browser](https://github.com/fnando/browser) at `dd88933`, as CI does:
 ```bash
 ruby -I path/to/upstream/lib script/corpus_snapshot.rb > /tmp/upstream.jsonl
@@ -503,10 +294,10 @@ ruby -I lib script/corpus_snapshot.rb > /tmp/agentry.jsonl
 ruby script/corpus_diff.rb /tmp/upstream.jsonl /tmp/agentry.jsonl
 ```
-The diff passes when the answers that differ are exactly those of `corpus/expected_differences.jsonl`: a change
-meant to change answers rewrites it with `ruby script/corpus_diff.rb --write …` and shows its diff.
+It passes when the answers that differ are exactly those in `corpus/expected_differences.jsonl`. A change meant to
+change answers rewrites that file with `ruby script/corpus_diff.rb --write …`, and its diff shows what changed.
-The weekly bot sync (`.github/workflows/bot-sync.yml`) runs the same steps by hand:
+The weekly bot sync (`.github/workflows/bot-sync.yml`) runs these steps, which also work by hand:
 ```bash
 ruby script/sync_bots.rb                         # bots.yml and bot_categories.yml
@@ -516,37 +307,39 @@ ruby script/bot_coverage.rb path/to/upstream/lib # docs/bot_coverage.json, and i
 ruby script/render_docs.rb                       # the site's numbers, from docs/*.json and the lists
 ```
-The numbers of `docs/*.html` sit between `<!-- render:… -->` markers and come from `docs/benchmarks.json`,
-`docs/bot_coverage.json`, the bot lists and `corpus/expected_differences.jsonl`: CI fails when a page differs from
-what `bundle exec rake docs:render` writes.
+The numbers on the site's pages (`docs/*.html`) sit between `<!-- render:… -->` markers and come from
+`docs/benchmarks.json`, `docs/bot_coverage.json`, the bot lists and `corpus/expected_differences.jsonl`. CI fails
+when a page differs from what `bundle exec rake docs:render` writes. A new code block on a page needs
+`bundle exec rake docs:highlight`.
-To add a bot, add its user agent to `test/ua_bots.yml` and its readable name to `bots.yml` above the synced
-section, keyed by a lowercased string the user agent contains. A search engine goes in `test/ua_search_engines.yml`,
+To add a bot, add its user agent to `test/ua_bots.yml`, and its name to `bots.yml` above the synced section, under
+a lowercase string its user agent contains. A search engine goes in `test/ua_search_engines.yml`,
 `test/ua_bots.yml` and `search_engines.yml`.
+See [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request.
 ### Releasing
-Pull requests leave `CHANGELOG.md` alone: a release starts by writing its notes under `## [Unreleased]`, from the
-pull requests merged since the last tag, without committing them. Then, from an up-to-date `main` whose CI is
-green:
+Pull requests leave `CHANGELOG.md` alone. A release starts by writing its notes under `## [Unreleased]`, from the
+pull requests merged since the last tag, without committing them. Then, from an up-to-date `main` with green CI:
 ```sh
-bin/release minor --dry-run   # the notes and the changes, nothing written
+bin/release minor --dry-run   # shows the notes and the changes, writes nothing
 bin/release minor             # or patch, major, or an exact version such as 1.2.0
 ```
-`main` takes no direct push, so `bin/release` dates the Unreleased section and bumps the version on a branch
-`release/vX.Y.Z`, opens its pull request, which merges itself once CI is green, and waits for it. It then pushes a
-signed tag `vX.Y.Z` on the merge commit (`bin/release --tag X.Y.Z` does it if you stopped waiting). The tag starts
-the Release workflow, which publishes the gem to RubyGems through trusted publishing and creates the GitHub release
-with the version's notes and the `.gem`.
+`main` takes no direct push, so `bin/release` dates the notes and bumps the version on a `release/vX.Y.Z` branch,
+opens a pull request that merges itself once CI is green, and waits for it. It then pushes a signed `vX.Y.Z` tag on
+the merge commit (`bin/release --tag X.Y.Z` does it if you stopped waiting). The tag starts the Release workflow,
+which checks that the tag is signed and on `main`, then publishes the gem to RubyGems through trusted publishing
+and creates the GitHub release with the notes and the `.gem`.
 ## License
 MIT, by Level UP Solutions, see [LICENSE.md](LICENSE.md). Originally written by Nando Vieira as
 [fnando/browser](https://github.com/fnando/browser) (MIT). The corpus gathers user agents from
 [ua-parser/uap-core](https://github.com/ua-parser/uap-core) (Apache 2.0),
-[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT) and browser's test fixtures; the bot
+[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT) and browser's test fixtures. The bot
 list and its categories come from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT)
-and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (MIT); the crawler IP ranges are those their
-owners publish for that purpose: see [corpus/SOURCES.md](corpus/SOURCES.md).
+and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (MIT). The crawler IP ranges are the ones
+their owners publish for that purpose. [corpus/SOURCES.md](corpus/SOURCES.md) lists every source.

data/lib/browser/bot/ip_ranges.rb CHANGED Viewed

@@ -15,8 +15,10 @@ module Browser
             google-read-aloud google-safety google-site-verification googleagent-urlcontext googlebot
             googleother googleproducer mediapartners-google storebot-google
           ],
-          # https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
-          domains: %w[googlebot.com google.com googleusercontent.com],
+          # https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot. Not
+          # googleusercontent.com: a Google Cloud VM's reverse DNS is x.x.x.x.bc.googleusercontent.com, and the
+          # user-triggered fetchers it also names are in the published ranges.
+          domains: %w[googlebot.com google.com],
           urls: %w[common-crawlers.json special-crawlers.json user-triggered-fetchers.json
             user-triggered-fetchers-google.json user-triggered-agents.json].map { |file| "#{GOOGLE_RANGES_URL}/#{file}" }
         },
@@ -50,7 +52,15 @@ module Browser
       FETCH_TIMEOUT_SECONDS = 10
       # Owners move their files: Google's old URLs redirect to the new ones.
       MAX_REDIRECTS = 3
-      DNS_TIMEOUT_SECONDS = 2
+      # How long each DNS query waits for each nameserver of /etc/resolv.conf: a resolver answers in milliseconds,
+      # and only a stalled one (or an attacker's own reverse zone) ever waits that long.
+      DNS_TIMEOUT_SECONDS = 1
+      # The broadest prefix a source may publish: a wider one, or one outside the public unicast space, is a broken
+      # or compromised source, never a crawler's range.
+      MIN_PREFIX_LENGTH_BY_FAMILY = {Socket::AF_INET => 12, Socket::AF_INET6 => 24}.freeze
+      # Multicast and reserved: 224.0.0.0/3 (with 240.0.0.0/4 and the broadcast address), ff00::/8.
+      NON_UNICAST_BY_FAMILY = {Socket::AF_INET => IPAddr.new("224.0.0.0/3"), Socket::AF_INET6 => IPAddr.new("ff00::/8")}
+        .freeze
       # The owner whose crawler the lowercase user agent claims to be.
       def self.owner(ua)
@@ -71,19 +81,80 @@ module Browser
         index >= 0 && number <= ends[index]
       end
+      @dns_mutex = Mutex.new
+      @dns_verdicts = {}
+      class << self
+        attr_reader :dns_timeout, :dns_cache_size, :dns_cache_ttl
+      end
+      def self.positive?(seconds)
+        seconds.is_a?(Numeric) && seconds.positive?
+      end
+      # Seconds each DNS query waits for each nameserver.
+      def self.dns_timeout=(seconds)
+        raise ArgumentError, "dns_timeout must be a positive number, got #{seconds.inspect}" unless positive?(seconds)
+        @dns_timeout = seconds
+      end
+      # How many reverse DNS verdicts to remember, the oldest dropped first: 0, the default, remembers none. A forged
+      # Googlebot then costs its DNS queries once per IP and TTL, not on every request.
+      def self.dns_cache_size=(size)
+        unless size.is_a?(Integer) && size >= 0
+          raise ArgumentError, "dns_cache_size must be an Integer >= 0, got #{size.inspect}"
+        end
+        @dns_cache_size = size
+        clear_dns_cache
+      end
+      # Seconds a verdict holds, a failed lookup's as long as any other's.
+      def self.dns_cache_ttl=(seconds)
+        raise ArgumentError, "dns_cache_ttl must be a positive number, got #{seconds.inspect}" unless positive?(seconds)
+        @dns_cache_ttl = seconds
+      end
+      def self.clear_dns_cache
+        @dns_mutex.synchronize { @dns_verdicts.clear }
+      end
+      self.dns_timeout = DNS_TIMEOUT_SECONDS
+      self.dns_cache_size = 0
+      self.dns_cache_ttl = 3600
       # Google's documented method: the reverse lookup names one of the owner's
-      # domains, and that name resolves back to the IP. Two DNS queries.
+      # domains, and that name resolves back to the IP. Two DNS queries, or
+      # none when dns_cache_size holds a fresh verdict for the IP.
       def self.verified_by_dns?(owner, ip)
         address = parse(ip)
         domains = SOURCES.dig(owner, :domains)
         return false unless address && domains&.any?
+        return resolves_to?(address, domains) if dns_cache_size.zero?
+        key = "#{owner} #{address}"
+        now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        verdict, expires_at = @dns_mutex.synchronize { @dns_verdicts[key] }
+        return verdict if expires_at && now < expires_at
+        verdict = resolves_to?(address, domains)
+        @dns_mutex.synchronize do
+          @dns_verdicts.delete(key)
+          @dns_verdicts.shift while @dns_verdicts.size >= dns_cache_size
+          @dns_verdicts[key] = [verdict, now + dns_cache_ttl]
+        end
+        verdict
+      end
+      def self.resolves_to?(address, domains)
         require "resolv"
         Resolv::DNS.open do |dns|
-          dns.timeouts = DNS_TIMEOUT_SECONDS
+          dns.timeouts = dns_timeout
           host = dns.getname(address.to_s).to_s.downcase
           domains.any? { |domain| host == domain || host.end_with?(".#{domain}") } &&
-            dns.getaddresses(host).any? { |resolved| IPAddr.new(resolved.to_s) == address }
+            dns.getaddresses(Resolv::DNS::Name.create("#{host}.")).any? { |resolved| IPAddr.new(resolved.to_s) == address }
         end
       rescue Resolv::ResolvError, Resolv::ResolvTimeout, SystemCallError
         false
@@ -114,7 +185,10 @@ module Browser
       def self.fetch
         SOURCES.to_h do |owner, source|
           addresses = source[:urls].flat_map { |url| fetch_prefixes(url) }.map do |prefix|
-            IPAddr.new(prefix)
+            address = IPAddr.new(prefix)
+            raise Error, "#{owner}: implausible prefix #{prefix.inspect}" unless plausible?(address)
+            address
           rescue ArgumentError
             raise Error, "#{owner}: invalid prefix #{prefix.inspect}"
           end
@@ -135,20 +209,36 @@ module Browser
           end
           break unless response.is_a?(Net::HTTPRedirection)
-          uri = uri.merge(response["Location"])
+          raise Error, "#{url}: HTTP #{response.code} without a Location" unless response["Location"]
+          location = uri.merge(response["Location"])
+          # Only the owner's own host, over HTTPS, publishes its ranges.
+          raise Error, "#{url}: redirected to #{location}" unless location.is_a?(URI::HTTPS) && location.host == uri.host
+          uri = location
         end
         raise Error, "#{url}: HTTP #{response.code}" unless response.is_a?(Net::HTTPSuccess)
-        prefixes = JSON.parse(response.body).fetch("prefixes").map do |prefix|
-          prefix["ipv4Prefix"] || prefix["ipv6Prefix"]
-        end
-        raise Error, "#{url}: no prefixes" if prefixes.empty?
+        document = JSON.parse(response.body)
+        entries = document.fetch("prefixes") if document.is_a?(Hash)
+        raise Error, "#{url}: no prefixes" unless entries.is_a?(Array) && entries.any?
+        entries.map do |entry|
+          prefix = entry["ipv4Prefix"] || entry["ipv6Prefix"] if entry.is_a?(Hash)
+          raise Error, "#{url}: invalid entry #{entry.inspect[0, 100]}" unless prefix.is_a?(String)
-        prefixes
-      rescue JSON::ParserError, KeyError, Timeout::Error, IOError, SystemCallError, OpenSSL::SSL::SSLError => exception
+          prefix
+        end
+      rescue JSON::ParserError, KeyError, URI::Error, Timeout::Error, IOError, SystemCallError, OpenSSL::SSL::SSLError => exception
         raise Error, "#{url}: #{exception.message}"
       end
+      def self.plausible?(address)
+        address.prefix >= MIN_PREFIX_LENGTH_BY_FAMILY.fetch(address.family) && !address.to_i.zero? &&
+          !(address.private? || address.loopback? || address.link_local?) &&
+          !NON_UNICAST_BY_FAMILY.fetch(address.family).include?(address)
+      end
       # Per owner and address family, the merged ranges as two sorted arrays of
       # integers, first and last addresses, for a binary search.
       def self.build_ranges(prefixes_by_owner)
@@ -188,7 +278,7 @@ module Browser
         nil
       end
-      private_class_method :fetch_prefixes, :build_ranges, :merge, :parse
+      private_class_method :resolves_to?, :positive?, :fetch_prefixes, :plausible?, :build_ranges, :merge, :parse
     end
   end
 end

data/lib/browser/cloudflare.rb CHANGED Viewed

@@ -5,6 +5,8 @@ module Browser
   # sets (docs/guide.html#cloudflare) and its own CF-IPCountry. Each site names
   # its headers in its rule: Browser::Cloudflare.header_names= changes them.
   class Cloudflare
+    # Cloudflare's values are a few bytes: a longer one is forged, and ignored.
+    HEADER_SIZE_LIMIT_BYTES = 128
     DEFAULT_HEADER_NAMES = {
       verified_bot: "X-Bot-Verified",
       bot_category: "X-Bot-Category",
@@ -59,7 +61,7 @@ module Browser
     private def header(signal)
       value = @headers&.[](self.class.env_key_by_signal[signal])
-      value if value.is_a?(String)
+      value if value.is_a?(String) && value.bytesize <= HEADER_SIZE_LIMIT_BYTES
     end
   end
 end

data/lib/browser/version.rb CHANGED Viewed

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module Browser
-  VERSION = "3.0.0"
+  VERSION = "3.1.0"
 end

metadata CHANGED Viewed

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: agentry
 version: !ruby/object:Gem::Version
-  version: 3.0.0
+  version: 3.1.0
 platform: ruby
 authors:
 - Nando Vieira
@@ -268,7 +268,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.24
+rubygems_version: 4.0.20
 specification_version: 4
 summary: 'Browser detection for Ruby: the browser gem''s API, faster'
 test_files: []