agentry 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (119) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +93 -0
  3. data/LICENSE.md +21 -0
  4. data/README.md +552 -0
  5. data/data/bot_categories.json +1 -0
  6. data/data/bot_exceptions.json +1 -0
  7. data/data/bot_ip_ranges.json +1 -0
  8. data/data/bots.json +1 -0
  9. data/data/languages.json +1 -0
  10. data/data/samsung.json +1 -0
  11. data/data/search_engines.json +1 -0
  12. data/lib/agentry.rb +3 -0
  13. data/lib/browser/accept_language.rb +55 -0
  14. data/lib/browser/action_controller.rb +21 -0
  15. data/lib/browser/aliases.rb +33 -0
  16. data/lib/browser/alipay.rb +21 -0
  17. data/lib/browser/base.rb +307 -0
  18. data/lib/browser/blackberry.rb +23 -0
  19. data/lib/browser/bot/empty_user_agent_matcher.rb +11 -0
  20. data/lib/browser/bot/ip_ranges.rb +194 -0
  21. data/lib/browser/bot/keyword_matcher.rb +11 -0
  22. data/lib/browser/bot/known_bots_matcher.rb +11 -0
  23. data/lib/browser/bot.rb +221 -0
  24. data/lib/browser/browser.rb +200 -0
  25. data/lib/browser/cache.rb +92 -0
  26. data/lib/browser/chrome.rb +43 -0
  27. data/lib/browser/client_hints.rb +73 -0
  28. data/lib/browser/cloudflare.rb +65 -0
  29. data/lib/browser/detect_version.rb +22 -0
  30. data/lib/browser/detection.rb +25 -0
  31. data/lib/browser/device/android.rb +20 -0
  32. data/lib/browser/device/base.rb +13 -0
  33. data/lib/browser/device/blackberry_playbook.rb +19 -0
  34. data/lib/browser/device/ipad.rb +19 -0
  35. data/lib/browser/device/iphone.rb +19 -0
  36. data/lib/browser/device/ipod_touch.rb +19 -0
  37. data/lib/browser/device/kindle.rb +19 -0
  38. data/lib/browser/device/kindle_fire.rb +19 -0
  39. data/lib/browser/device/playstation3.rb +19 -0
  40. data/lib/browser/device/playstation4.rb +19 -0
  41. data/lib/browser/device/psp.rb +19 -0
  42. data/lib/browser/device/psvita.rb +19 -0
  43. data/lib/browser/device/samsung.rb +33 -0
  44. data/lib/browser/device/surface.rb +23 -0
  45. data/lib/browser/device/switch.rb +19 -0
  46. data/lib/browser/device/tv.rb +19 -0
  47. data/lib/browser/device/unknown.rb +19 -0
  48. data/lib/browser/device/wii.rb +19 -0
  49. data/lib/browser/device/wiiu.rb +19 -0
  50. data/lib/browser/device/xbox_360.rb +19 -0
  51. data/lib/browser/device/xbox_one.rb +19 -0
  52. data/lib/browser/device.rb +246 -0
  53. data/lib/browser/duck_duck_go.rb +22 -0
  54. data/lib/browser/edge.rb +25 -0
  55. data/lib/browser/electron.rb +22 -0
  56. data/lib/browser/epiphany.rb +21 -0
  57. data/lib/browser/facebook.rb +23 -0
  58. data/lib/browser/firefox.rb +21 -0
  59. data/lib/browser/google_search_app.rb +21 -0
  60. data/lib/browser/huawei_browser.rb +21 -0
  61. data/lib/browser/instagram.rb +21 -0
  62. data/lib/browser/internet_explorer.rb +63 -0
  63. data/lib/browser/maxthon.rb +21 -0
  64. data/lib/browser/meta/base.rb +42 -0
  65. data/lib/browser/meta/device.rb +11 -0
  66. data/lib/browser/meta/generic_browser.rb +15 -0
  67. data/lib/browser/meta/id.rb +11 -0
  68. data/lib/browser/meta/ie.rb +20 -0
  69. data/lib/browser/meta/ios.rb +11 -0
  70. data/lib/browser/meta/mobile.rb +11 -0
  71. data/lib/browser/meta/platform.rb +11 -0
  72. data/lib/browser/meta/proxy.rb +11 -0
  73. data/lib/browser/meta/safari.rb +11 -0
  74. data/lib/browser/meta/tablet.rb +11 -0
  75. data/lib/browser/meta/webkit.rb +11 -0
  76. data/lib/browser/meta.rb +14 -0
  77. data/lib/browser/micro_messenger.rb +21 -0
  78. data/lib/browser/middleware/context/additions.rb +18 -0
  79. data/lib/browser/middleware/context/url_methods.rb +15 -0
  80. data/lib/browser/middleware/context.rb +23 -0
  81. data/lib/browser/middleware.rb +70 -0
  82. data/lib/browser/miui_browser.rb +21 -0
  83. data/lib/browser/nokia.rb +21 -0
  84. data/lib/browser/opera.rb +21 -0
  85. data/lib/browser/otter.rb +21 -0
  86. data/lib/browser/phantom_js.rb +21 -0
  87. data/lib/browser/platform/adobe_air.rb +23 -0
  88. data/lib/browser/platform/android.rb +23 -0
  89. data/lib/browser/platform/base.rb +18 -0
  90. data/lib/browser/platform/blackberry.rb +23 -0
  91. data/lib/browser/platform/chrome_os.rb +23 -0
  92. data/lib/browser/platform/firefox_os.rb +23 -0
  93. data/lib/browser/platform/ios.rb +36 -0
  94. data/lib/browser/platform/kai_os.rb +23 -0
  95. data/lib/browser/platform/linux.rb +23 -0
  96. data/lib/browser/platform/mac.rb +25 -0
  97. data/lib/browser/platform/unknown.rb +23 -0
  98. data/lib/browser/platform/windows.rb +23 -0
  99. data/lib/browser/platform/windows_mobile.rb +23 -0
  100. data/lib/browser/platform/windows_phone.rb +23 -0
  101. data/lib/browser/platform.rb +220 -0
  102. data/lib/browser/qq.rb +25 -0
  103. data/lib/browser/rails.rb +24 -0
  104. data/lib/browser/safari.rb +40 -0
  105. data/lib/browser/samsung_browser.rb +21 -0
  106. data/lib/browser/snapchat.rb +21 -0
  107. data/lib/browser/sougou_browser.rb +24 -0
  108. data/lib/browser/sputnik.rb +21 -0
  109. data/lib/browser/testing.rb +29 -0
  110. data/lib/browser/uc_browser.rb +21 -0
  111. data/lib/browser/unknown.rb +32 -0
  112. data/lib/browser/version.rb +5 -0
  113. data/lib/browser/weibo.rb +21 -0
  114. data/lib/browser/yandex.rb +21 -0
  115. data/lib/browser.rb +4 -0
  116. data/test/ua.yml +181 -0
  117. data/test/ua_bots.yml +121 -0
  118. data/test/ua_search_engines.yml +7 -0
  119. metadata +274 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 1a57ed8305d43ff3e55dce1770f63f0bf63c712e084ba3013fe2f2240fcc27f6
4
+ data.tar.gz: 145e9e764e8cbc76e3f40b985d37181b3d801a620998bb34ecbcae152739a15b
5
+ SHA512:
6
+ metadata.gz: 0a7b34836efd8a5cff0af7eac8e723d0027670a34bde7f1fd3e58763ba4a8f917094a76d19b4e0cb5e7ced62af3245a55ffc516310ac2d4002ea1fa421c4812d
7
+ data.tar.gz: ee6ba51155056619fada17e77f0997849fb1e686ca9af9424339ab788d74018e3e9d71097259e6286c5fbed478c7f84b560943a26495602169b3593b30f46e77
data/CHANGELOG.md ADDED
@@ -0,0 +1,93 @@
1
+ # Changelog
2
+
3
+ ## [Unreleased]
4
+
5
+ ## [3.0.0] — 2026-11-05
6
+
7
+ The gem is renamed Agentry and lives at levups/agentry; the API is untouched.
8
+
9
+ **Breaking**
10
+ - The gem is now `agentry`: replace `gem "browserx"` with `gem "agentry"` in the Gemfile (or `gem "agentry", require: "browser/browser"`). `require "browser"`, the `Browser` namespace and every predicate stay. `require "browserx"` becomes `require "agentry"`.
11
+
12
+ ## [2.1.0] — 2026-11-05
13
+
14
+ The first request of each process gets cheaper: the lists ship as JSON, and `Browser.eager_load!` builds them
15
+ before a server forks.
16
+
17
+ **New**
18
+ - `Browser.eager_load!` loads the lists and builds the matchers and bot keyword tries up front, so a server that
19
+ preloads its app before forking (Puma, Unicorn, Pitchfork) shares them with every worker. Rails calls it on its
20
+ own when it eager loads, which production does by default. A worker's first request spent 12.9 ms building
21
+ them, 0.2 ms once built.
22
+
23
+ **Faster**
24
+ - The lists ship as JSON: without Bootsnap's cache, a process's first request is 3× faster (12.4 → 4.0 ms with
25
+ YJIT, 7.5 → 2.7 ms without), and `require "browser"` no longer loads Psych (32 → 21 ms with YJIT, 28 → 22 ms
26
+ without). With Bootsnap, nothing changes. Code that used `YAML` without requiring it now needs its own
27
+ `require "yaml"`.
28
+
29
+ ## [2.0.0] — 2026-11-04
30
+
31
+ A repeated user agent is now answered in 2 µs, about 600 more bots are recognised, and bots, Cloudflare headers and
32
+ Client Hints can be checked against the request. Without the new headers, every answer stays browser 6.2.0's,
33
+ except bot detection, which finds more bots.
34
+
35
+ **Breaking**
36
+ - `bot?` is true for about 600 more user agents: bots from crawler-user-agents and ai.robots.txt, synced weekly,
37
+ add 1,247 entries to `bots.yml`. Of the crawlers added there since 2024, browser misses 353 out of 882 and
38
+ Agentry 11. Analytics that filter on `bot?` will count fewer humans from the day you switch.
39
+ - The Rails helper and the middleware now pass the request's headers, so Client Hints refine the answers on
40
+ Chromium: Windows 11 is no longer `windows10?`, and Chrome for Android in desktop mode is an Android tablet,
41
+ not a Linux desktop.
42
+
43
+ **New**
44
+ - `bot.category` (16 categories, close to Cloudflare's verified bot categories), `bot.ai?` and
45
+ `bot.user_triggered?`. Test drivers (Playwright, Puppeteer, Selenium, headless Chrome) get the
46
+ `browser_automation` category but are not bots, so system tests keep running as humans.
47
+ - `bot.verified?(ip)` checks a crawler against the IP ranges its owner publishes (Google, Bing, OpenAI, Apple,
48
+ Perplexity, DuckDuckGo), shipped with the gem and refreshable at runtime; reverse DNS is opt-in. `bot.owner`
49
+ names the owner.
50
+ - `Browser.new(ua, headers: request.env)` reads Cloudflare's bot headers (`browser.cloudflare`: verified bot,
51
+ category, score, country; header names configurable) and Chromium's Client Hints (`browser.client_hints`,
52
+ `platform.windows11?`).
53
+ - `Browser.cache_size` (default 1,000, `0` turns it off) bounds the new per-user-agent cache: about 1 KB per
54
+ entry, so about 1 MB per process.
55
+
56
+ **Faster**
57
+ - Answers are cached per user agent, matchers that can't match are skipped, and bots are found through a byte
58
+ trie. Ruby 4.0.9: a repeated user agent takes 1.8 µs instead of 57.5 with YJIT and 3.6 µs instead of 72.5
59
+ without; 2,000 distinct user agents take 24.2 µs instead of 62.4 with YJIT and 43.9 µs instead of 77.3
60
+ without. Objects per request: 9 instead of 116 repeated, 50 instead of 125 distinct.
61
+
62
+ **Fixed**
63
+ - Loading `browser` and `agentry` together raises a clear `LoadError` instead of one silently overriding the
64
+ other.
65
+
66
+ ## [1.0.0] — 2026-11-04
67
+
68
+ First release, forked from browser 6.2.0 (`dd88933`). Same API: `gem "agentry"` replaces
69
+ `gem "browser"` without touching a call site. Earlier versions are browser's: see its
70
+ [changelog](https://github.com/fnando/browser/blob/main/CHANGELOG.md).
71
+
72
+ **Breaking**
73
+ - Requires Ruby 3.3 or later: 3.2 reached its end of life on 2026-03-31.
74
+ - Tested against Rails 8.0 and 8.1 only: 7.0, 7.1 and 7.2 reached their end of life.
75
+
76
+ **Faster**
77
+ - Bots are detected through one prefix-trie regexp instead of one match per entry, and without detecting the
78
+ browser twice; platform and device matchers are built lazily. A request takes 45% less time and half the
79
+ allocations.
80
+
81
+ **Fixed**
82
+ - The railtie loads Rails, not only itself, so it works without a preloaded ActiveSupport.
83
+ - `lib/agentry.rb` exists, so Bundler's auto-require loads the gem.
84
+
85
+ **New**
86
+ - A corpus of 20,328 user agents, with snapshot and diff scripts proving answers match upstream, and a
87
+ request benchmark.
88
+
89
+ [Unreleased]: https://github.com/levups/agentry/compare/v3.0.0...HEAD
90
+ [3.0.0]: https://github.com/levups/agentry/compare/v2.1.0...v3.0.0
91
+ [2.1.0]: https://github.com/levups/agentry/compare/v2.0.0...v2.1.0
92
+ [2.0.0]: https://github.com/levups/agentry/compare/v1.0.0...v2.0.0
93
+ [1.0.0]: https://github.com/levups/agentry/releases/tag/v1.0.0
data/LICENSE.md ADDED
@@ -0,0 +1,21 @@
1
+ # The MIT License (MIT)
2
+
3
+ Copyright (c) 2010 Nando Vieira
4
+ Copyright (c) 2026 Level UP Solutions
5
+
6
+ Permission is hereby granted, free of charge, to any person obtaining a copy of
7
+ this software and associated documentation files (the "Software"), to deal in
8
+ the Software without restriction, including without limitation the rights to
9
+ use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
10
+ the Software, and to permit persons to whom the Software is furnished to do so,
11
+ subject to the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be included in all
14
+ copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
17
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
18
+ FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
19
+ COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
20
+ IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
21
+ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,552 @@
1
+ # Agentry
2
+
3
+ Browser detection for Ruby, with ActionController integration: the API of the
4
+ [browser](https://rubygems.org/gems/browser) gem, faster, and a bot list that stays fresh. Put `gem "agentry"` in
5
+ your Gemfile instead of `gem "browser"`, and every call site keeps working — same `Browser` namespace, same
6
+ `require "browser"`, same predicates.
7
+
8
+ - **Faster**: 25 µs per request instead of 126 µs with browser 6.2.0, 2 µs once a user agent was seen (Ruby 4.0.9,
9
+ YJIT, `bench/request.rb`).
10
+ - **Fresher bots**: synced every week from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents)
11
+ and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt), with categories (`bot.category`, `bot.ai?`).
12
+ Of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out),
13
+ browser misses 353, Agentry 11.
14
+ - **Verified signals**: `bot.verified?(ip)` checks a crawler against the IP ranges Google, Bing, OpenAI, Apple,
15
+ Perplexity and DuckDuckGo publish, `browser.cloudflare` reads the headers of a Cloudflare Transform Rule, and
16
+ Chromium's Client Hints tell Windows 11 from Windows 10.
17
+
18
+ Site: https://levups.github.io/agentry/
19
+
20
+ ## Installation
21
+
22
+ ```ruby
23
+ gem "agentry"
24
+ ```
25
+
26
+ ## Usage
27
+
28
+ ```ruby
29
+ require "browser"
30
+
31
+ browser = Browser.new("Some User Agent", accept_language: "en-us")
32
+
33
+ # General info
34
+ browser.bot?
35
+ browser.chrome?
36
+ browser.chromium_based?
37
+ browser.core_media?
38
+ browser.duck_duck_go?
39
+ browser.edge? # Newest MS browser
40
+ browser.electron? # Electron Framework
41
+ browser.firefox?
42
+ browser.full_version
43
+ browser.ie?
44
+ browser.ie?(6) # detect specific IE version
45
+ browser.ie?([">8", "<10"]) # detect specific IE (IE9).
46
+ browser.known? # has the browser been successfully detected?
47
+ browser.unknown? # the browser wasn't detected.
48
+ browser.meta # an array with several attributes
49
+ browser.name # readable browser name
50
+ browser.nokia?
51
+ browser.opera?
52
+ browser.opera_mini?
53
+ browser.phantom_js?
54
+ browser.quicktime?
55
+ browser.safari?
56
+ browser.safari_webapp_mode?
57
+ browser.samsung_browser?
58
+ browser.to_s # the meta info joined by space
59
+ browser.uc_browser?
60
+ browser.version # major version number
61
+ browser.webkit?
62
+ browser.webkit_full_version
63
+ browser.yandex?
64
+ browser.wechat?
65
+ browser.qq?
66
+ browser.weibo?
67
+ browser.sputnik?
68
+ browser.sougou_browser?
69
+ browser.epiphany?
70
+
71
+ # Get bot info
72
+ browser.bot.name
73
+ browser.bot.search_engine?
74
+ browser.bot.category # :ai_crawler, :search_engine_crawler, :scanner… (nil for a browser)
75
+ browser.bot.ai?
76
+ browser.bot.user_triggered? # an AI agent acting for a person right now (ChatGPT-User, Claude-User)
77
+ browser.bot?
78
+ browser.bot.why? # shows which matcher detected this user agent as a bot.
79
+ Browser::Bot.why?(ua)
80
+
81
+ # Get device info
82
+ browser.device
83
+ browser.device.id
84
+ browser.device.name
85
+ browser.device.unknown?
86
+ browser.device.blackberry_playbook?
87
+ browser.device.console?
88
+ browser.device.ipad?
89
+ browser.device.iphone?
90
+ browser.device.ipod_touch?
91
+ browser.device.kindle?
92
+ browser.device.kindle_fire?
93
+ browser.device.mobile?
94
+ browser.device.nintendo?
95
+ browser.device.playstation?
96
+ browser.device.ps3?
97
+ browser.device.ps4?
98
+ browser.device.psp?
99
+ browser.device.silk?
100
+ browser.device.surface?
101
+ browser.device.tablet?
102
+ browser.device.tv?
103
+ browser.device.vita?
104
+ browser.device.wii?
105
+ browser.device.wiiu?
106
+ browser.device.samsung?
107
+ browser.device.switch?
108
+ browser.device.xbox?
109
+ browser.device.xbox_360?
110
+ browser.device.xbox_one?
111
+
112
+ # Get platform info
113
+ browser.platform
114
+ browser.platform.id
115
+ browser.platform.name
116
+ browser.platform.version # e.g. 9 (for iOS9)
117
+ browser.platform.adobe_air?
118
+ browser.platform.android?
119
+ browser.platform.android?(4.2) # detect Android Jelly Bean 4.2
120
+ browser.platform.android_app? # detect webview in an Android app
121
+ browser.platform.android_webview? # alias for android_app?
122
+ browser.platform.blackberry?
123
+ browser.platform.blackberry?(10) # detect specific BlackBerry version
124
+ browser.platform.chrome_os?
125
+ browser.platform.firefox_os?
126
+ browser.platform.ios? # detect iOS
127
+ browser.platform.ios?(9) # detect specific iOS version
128
+ browser.platform.ios_app? # detect webview in an iOS app
129
+ browser.platform.ios_webview? # alias for ios_app?
130
+ browser.platform.linux?
131
+ browser.platform.mac?
132
+ browser.platform.unknown?
133
+ browser.platform.windows10?
134
+ browser.platform.windows7?
135
+ browser.platform.windows8?
136
+ browser.platform.windows8_1?
137
+ browser.platform.windows?
138
+ browser.platform.windows_mobile?
139
+ browser.platform.windows_phone?
140
+ browser.platform.windows_rt?
141
+ browser.platform.windows_touchscreen_desktop?
142
+ browser.platform.windows_vista?
143
+ browser.platform.windows_wow64?
144
+ browser.platform.windows_x64?
145
+ browser.platform.windows_x64_inclusive?
146
+ browser.platform.windows_xp?
147
+ browser.platform.kai_os?
148
+ ```
149
+
150
+ ### Aliases
151
+
152
+ To add aliases like `mobile?` and `tablet?` to the base object (e.g
153
+ `browser.mobile?`), require the `browser/aliases` file and extend the
154
+ Browser::Base object like the following:
155
+
156
+ ```ruby
157
+ require "browser/aliases"
158
+ Browser::Base.include(Browser::Aliases)
159
+
160
+ browser = Browser.new("Some user agent")
161
+ browser.mobile? #=> false
162
+ ```
163
+
164
+ ### What's being detected?
165
+
166
+ - For a list of platform detections, check
167
+ [lib/browser/platform.rb](lib/browser/platform.rb)
168
+ - For a list of device detections, check
169
+ [lib/browser/device.rb](lib/browser/device.rb)
170
+ - For a list of bot detections, check
171
+ [bots.yml](bots.yml)
172
+
173
+ ### Detecting modern browsers
174
+
175
+ To detect whether a browser can be considered as modern or not, create a method
176
+ that abstracts your versioning constraints. The following example will consider
177
+ any of the following browsers as a modern:
178
+
179
+ ```ruby
180
+ # Expects an Browser instance,
181
+ # like in `Browser.new(user_agent, accept_language: language)`.
182
+ def modern_browser?(browser)
183
+ [
184
+ browser.chrome?(">= 65"),
185
+ browser.safari?(">= 10"),
186
+ browser.firefox?(">= 52"),
187
+ browser.ie?(">= 11") && !browser.compatibility_view?,
188
+ browser.edge?(">= 15"),
189
+ browser.opera?(">= 50"),
190
+ browser.facebook?
191
+ && browser.safari_webapp_mode?
192
+ && browser.webkit_full_version.to_i >= 602
193
+ ].any?
194
+ end
195
+ ```
196
+
197
+ ### Migrating from browser
198
+
199
+ Replace `gem "browser"` with `gem "agentry"` in the Gemfile, nothing else: `require "browser"`, the `Browser`
200
+ namespace and every predicate stay. Both gems define `Browser`, so Agentry refuses to load next to `browser`,
201
+ with an error saying so, rather than letting one silently override the other. Agentry requires Ruby 3.3 or later
202
+ and is tested on Rails 8.0 and 8.1.
203
+
204
+ What changes on its own: the bots of the weekly sync are bots from the day you switch (600 user agents of the
205
+ corpus that browser took for humans), so analytics that leave bots out count fewer visitors from that day; 792
206
+ user agents named `Generic Bot` get a name; no browser, platform or device answer moves, and test drivers stay out
207
+ of `bot?`. To roll back, put `gem "browser"` back, after removing the calls to what browser lacks
208
+ (`Browser.cache_size=`, `bot.category`, `bot.ai?`, `bot.user_triggered?`). The
209
+ [migration guide](https://levups.github.io/agentry/guide.html#migrating) has the details.
210
+
211
+ ### Rails integration
212
+
213
+ Just add it to the Gemfile.
214
+
215
+ ```ruby
216
+ gem "agentry"
217
+ ```
218
+
219
+ This adds a helper method called `browser`, that inspects your current user
220
+ agent.
221
+
222
+ ```erb
223
+ <% if browser.ie?(6) %>
224
+ <p class="disclaimer">You're running an older IE version. Please update it!</p>
225
+ <% end %>
226
+ ```
227
+
228
+ If you want to use Browser on your Rails app but don't want to taint your
229
+ controller, use the following line on your Gemfile:
230
+
231
+ ```ruby
232
+ gem "agentry", require: "browser/browser"
233
+ ```
234
+
235
+ ### Accept Language
236
+
237
+ Parses the accept-language header from an HTTP request and produces an array of
238
+ language objects sorted by quality.
239
+
240
+ ```ruby
241
+ browser = Browser.new("Some User Agent", accept_language: "en-us")
242
+
243
+ browser.accept_language.class
244
+ #=> Array
245
+
246
+ language = browser.accept_language.first
247
+
248
+ language.code
249
+ #=> "en"
250
+
251
+ language.region
252
+ #=> "US"
253
+
254
+ language.full
255
+ #=> "en-US"
256
+
257
+ language.quality
258
+ #=> 1.0
259
+
260
+ language.name
261
+ #=> "English/United States"
262
+ ```
263
+
264
+ Result is always sorted in quality order from highest to lowest. As per the HTTP
265
+ spec:
266
+
267
+ - omitting the quality value implies 1.0.
268
+ - quality value equal to zero means that is not accepted by the client.
269
+
270
+ ### Internet Explorer
271
+
272
+ Internet Explorer has a compatibility view mode that allows newer versions
273
+ (IE8+) to run as an older version. Browser will always return the navigator
274
+ version, ignoring the compatibility view version, when defined. If you need to
275
+ get the engine's version, you have to use `Browser#msie_version` and
276
+ `Browser#msie_full_version`.
277
+
278
+ So, let's say an user activates compatibility view in a IE11 browser. This is
279
+ what you'll get:
280
+
281
+ ```ruby
282
+ browser.version
283
+ #=> 11
284
+
285
+ browser.full_version
286
+ #=> 11.0
287
+
288
+ browser.msie_version
289
+ #=> 7
290
+
291
+ browser.msie_full_version
292
+ #=> 7.0
293
+
294
+ browser.compatibility_view?
295
+ #=> true
296
+ ```
297
+
298
+ This behavior changed in browser v1.0.0; previously there wasn't a way of getting the
299
+ real browser version.
300
+
301
+ ### Safari
302
+
303
+ iOS webviews and web apps aren't detected as Safari anymore, so be aware of that
304
+ if that's your case. You can use a combination of platform and webkit detection
305
+ to do whatever you want.
306
+
307
+ ```ruby
308
+ # iPad's Safari running as web app mode.
309
+ browser = Browser.new("Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405")
310
+
311
+ browser.safari?
312
+ #=> false
313
+
314
+ browser.webkit?
315
+ #=> true
316
+
317
+ browser.platform.ios?
318
+ #=> true
319
+ ```
320
+
321
+ ### Bots
322
+
323
+ The bot detection is quite aggressive. Anything that matches at least one of the
324
+ following requirements will be considered a bot.
325
+
326
+ - Empty user agent string
327
+ - User agent that matches `/crawl|fetch|search|monitoring|spider|bot/`
328
+ - Any known bot listed under
329
+ [bots.yml](bots.yml)
330
+
331
+ Every week, `script/sync_bots.rb` adds the bots of
332
+ [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) and
333
+ [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (both MIT) to the end of `bots.yml`, and
334
+ categorizes them in `bot_categories.yml`, through a pull request that lists the answers that change. How many of
335
+ those lists' bots browser and Agentry detect, per category, is on the
336
+ [Bots page](https://levups.github.io/agentry/bots.html#coverage).
337
+
338
+ `browser.bot.category` names what the bot declares it is, with the names of Cloudflare's verified bot categories
339
+ where one fits (`search_engine_crawler`, `ai_crawler`, `ai_search`, `ai_assistant`, `page_preview`,
340
+ `feed_fetcher`, `monitoring_and_analytics`, `advertising_and_marketing`, `archiver`, `security`…), plus
341
+ `ai_agent`, `http_library`, `scanner` and `browser_automation`. `bot.ai?` is true for the four AI categories,
342
+ `bot.user_triggered?` for `ai_assistant` and `ai_agent`. The category is `nil` for a browser, a bot exception
343
+ and a bot no list names.
344
+
345
+ HTTP libraries (`curl`, `got`, `python-requests`) and scanners (sqlmap, Nikto, Censys) are bots, as `curl` and
346
+ `python-requests` already were in browser. Test drivers (headless Chrome, Playwright, Puppeteer, Selenium) are
347
+ not: an app's system tests drive a real browser through them, and the headless Chrome of Playwright and Selenium
348
+ sends `HeadlessChrome`. They get the `browser_automation` category instead, so an app can tell them apart in
349
+ production (`browser.bot? || browser.bot.category == :browser_automation`). PhantomJS stays a bot, as in
350
+ browser.
351
+
352
+ To add custom matchers, you can add a callable object to
353
+ `Browser::Bot.matchers`. The following example matches everything that has a
354
+ `externalhit` substring on it. The bot name will always be `General Bot`.
355
+
356
+ ```ruby
357
+ Browser::Bot.matchers << ->(ua, _browser) { ua.match?(/externalhit/i) }
358
+ ```
359
+
360
+ To clear all matchers, including the ones that are bundled, use
361
+ `Browser::Bot.matchers.clear`. You can re-add built-in matchers by doing the
362
+ following:
363
+
364
+ ```ruby
365
+ Browser::Bot.matchers += Browser::Bot.default_matchers
366
+ ```
367
+
368
+ To restore v2's bot detection, remove the following matchers:
369
+
370
+ ```ruby
371
+ Browser::Bot.matchers.delete(Browser::Bot::KeywordMatcher)
372
+ Browser::Bot.matchers.delete(Browser::Bot::EmptyUserAgentMatcher)
373
+ ```
374
+
375
+ To extend the bot list, you can manipulate the methods below:
376
+
377
+ ```ruby
378
+ Browser::Bot.bots.merge!(new_bots_hash)
379
+ Browser::Bot.bot_exceptions += new_exceptions
380
+ Browser::Bot.search_engines.merge!(new_search_engines_hash)
381
+ Browser::Bot.category_by_keyword["faraday"] = :http_library
382
+ ```
383
+
384
+ ### Verified crawlers
385
+
386
+ Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish their crawlers' IP ranges. They ship in
387
+ `bot_ip_ranges.yml` (as JSON in the gem), rewritten by the weekly sync (`script/sync_ip_ranges.rb`). The lookup is a binary search, no
388
+ network call; an IP that does not parse answers `false`.
389
+
390
+ ```ruby
391
+ browser.bot.owner # => :google for a user agent claiming Googlebot
392
+ browser.bot.verified?(request.remote_ip) # => whether the IP is in Google's ranges
393
+ browser.bot.verified?(request.remote_ip, reverse_dns: true) # also Google's DNS check: two DNS queries
394
+ Browser::Bot::IPRanges.refresh # fetches the ranges again, in a long-lived process
395
+ ```
396
+
397
+ ### Cloudflare and Client Hints
398
+
399
+ `Browser.new(user_agent, accept_language:, headers: rack_env)` also reads the request's headers; the Rails helper
400
+ and the middleware pass them. `browser.cloudflare` reads the headers a Cloudflare Transform Rule sets
401
+ (`verified_bot?`, `bot_category`, `bot_score`) and `CF-IPCountry` (`country`); their names are configurable with
402
+ `Browser::Cloudflare.header_names = {bot_score: "X-My-Score"}`. Chromium's Client Hints (`Sec-CH-UA`,
403
+ `Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`, `Sec-CH-UA-Platform-Version`) refine the platform and the device, and
404
+ `browser.platform.windows11?` needs them. Without `headers:`, every answer is the user agent's, as before. The
405
+ rule to create and the details are in the [guide](https://levups.github.io/agentry/guide.html#cloudflare).
406
+
407
+ `bench/signals.rb` measures what they cost.
408
+
409
+ ### Middleware
410
+
411
+ You can use the `Browser::Middleware` to redirect user agents.
412
+
413
+ ```ruby
414
+ use Browser::Middleware do
415
+ redirect_to "/upgrade" if browser.ie?
416
+ end
417
+ ```
418
+
419
+ If you're using Rails, you can use the route helper methods. Just add something
420
+ like the following to a initializer file (`config/initializers/browser.rb`).
421
+
422
+ ```ruby
423
+ Rails.configuration.middleware.use Browser::Middleware do
424
+ redirect_to upgrade_path if browser.ie?
425
+ end
426
+ ```
427
+
428
+ If you need access to the `Rack::Request` object (e.g. to exclude a path), you
429
+ can do so with `request`.
430
+
431
+ ```ruby
432
+ Rails.configuration.middleware.use Browser::Middleware do
433
+ redirect_to upgrade_path if browser.ie? && request.env["PATH_INFO"] != "/exclude_me"
434
+ end
435
+ ```
436
+
437
+ ### Restrictions
438
+
439
+ - User agent has a size limit of 2048 bytes. This can be customized through
440
+ `Browser.user_agent_size_limit=(size)`.
441
+ - Accept-Language has a size limit of 2048 bytes. This can be customized through
442
+ `Browser.accept_language_size_limit=(size)`.
443
+
444
+ If size is not respected, then `Browser::Error` is raised.
445
+
446
+ ```ruby
447
+ Browser.user_agent_size_limit = 4096
448
+ Browser.accept_language_size_limit = 4096
449
+ ```
450
+
451
+ ### Cache
452
+
453
+ Each process keeps what it detected for the last 1,000 distinct user agents (browser, platform, device, bot
454
+ answers) and the last 1,000 Accept-Language headers it parsed, about 1 KB per user agent: a repeated user agent
455
+ costs about 2 µs. The cache empties itself when full, and when `Browser.matchers`, `Browser::Bot.matchers`,
456
+ `Browser::Platform.matchers`, `Browser::Device.matchers`, the bot list or its exceptions change. Setting
457
+ `Browser.cache_size` empties it too, for example after renaming an entry of `Browser::Bot.bots` in place.
458
+
459
+ ```ruby
460
+ Browser.cache_size = 5_000 # 0 turns it off
461
+ ```
462
+
463
+ ### Preloading before a fork
464
+
465
+ The lists, the matchers and the bot keyword tries are built on first use: the first request of each process pays
466
+ about 13 ms for them, the next ones 0.2 ms. `Browser.eager_load!` builds them up front, so a server that preloads
467
+ its app before forking (Puma's `preload_app!`, Unicorn, Pitchfork) shares them with every worker. Rails calls it on
468
+ its own when it eager loads, which production does by default. Elsewhere, call it before the fork:
469
+
470
+ ```ruby
471
+ Browser.eager_load!
472
+ ```
473
+
474
+ ## Performance
475
+
476
+ `bench/request.rb` replays what a web request asks (name, version, `bot?`, platform, device, accept-language) over
477
+ 2,000 distinct user agents of the corpus and over 20 repeated ones, median of 7. On Ruby 4.0.9 with YJIT, browser
478
+ 6.2.0 takes 126 µs and 245 objects per request, Agentry 25 µs and 52 objects on a new user agent, 2 µs and 12
479
+ objects on a repeated one. All four runs, with and without YJIT, are on the
480
+ [benchmarks page](https://levups.github.io/agentry/benchmarks.html).
481
+
482
+ ```bash
483
+ bundle exec rake bench UPSTREAM=path/to/upstream # fnando/browser at dd88933
484
+ ```
485
+
486
+ It runs `ruby [--yjit] -I lib bench/request.rb --write` for each gem and JIT, which records the run (gem, commit,
487
+ Ruby, YJIT, machine, date) in `docs/benchmarks.json`, then `rake docs:render` rewrites the site's numbers from it.
488
+
489
+ ## Development
490
+
491
+ ```bash
492
+ bundle install
493
+ bundle exec rake test
494
+ bundle exec rubocop
495
+ ```
496
+
497
+ A change that is not meant to change an answer proves it on the corpus of 20,000 user agents, against a plain
498
+ clone of [fnando/browser](https://github.com/fnando/browser) at `dd88933`, as CI does on every pull request:
499
+
500
+ ```bash
501
+ ruby -I path/to/upstream/lib script/corpus_snapshot.rb > /tmp/upstream.jsonl
502
+ ruby -I lib script/corpus_snapshot.rb > /tmp/agentry.jsonl
503
+ ruby script/corpus_diff.rb /tmp/upstream.jsonl /tmp/agentry.jsonl
504
+ ```
505
+
506
+ The diff passes when the answers that differ are exactly those of `corpus/expected_differences.jsonl`: a change
507
+ meant to change answers rewrites it with `ruby script/corpus_diff.rb --write …` and shows its diff.
508
+
509
+ The weekly bot sync (`.github/workflows/bot-sync.yml`) runs the same steps by hand:
510
+
511
+ ```bash
512
+ ruby script/sync_bots.rb # bots.yml and bot_categories.yml
513
+ ruby -I lib script/sync_ip_ranges.rb # bot_ip_ranges.yml
514
+ ruby script/build_data.rb # data/*.json, what the gem reads, from the *.yml
515
+ ruby script/bot_coverage.rb path/to/upstream/lib # docs/bot_coverage.json, and its Markdown table
516
+ ruby script/render_docs.rb # the site's numbers, from docs/*.json and the lists
517
+ ```
518
+
519
+ The numbers of `docs/*.html` sit between `<!-- render:… -->` markers and come from `docs/benchmarks.json`,
520
+ `docs/bot_coverage.json`, the bot lists and `corpus/expected_differences.jsonl`: CI fails when a page differs from
521
+ what `bundle exec rake docs:render` writes.
522
+
523
+ To add a bot, add its user agent to `test/ua_bots.yml` and its readable name to `bots.yml` above the synced
524
+ section, keyed by a lowercased string the user agent contains. A search engine goes in `test/ua_search_engines.yml`,
525
+ `test/ua_bots.yml` and `search_engines.yml`.
526
+
527
+ ### Releasing
528
+
529
+ Pull requests leave `CHANGELOG.md` alone: a release starts by writing its notes under `## [Unreleased]`, from the
530
+ pull requests merged since the last tag, without committing them. Then, from an up-to-date `main` whose CI is
531
+ green:
532
+
533
+ ```sh
534
+ bin/release minor --dry-run # the notes and the changes, nothing written
535
+ bin/release minor # or patch, major, or an exact version such as 1.2.0
536
+ ```
537
+
538
+ `main` takes no direct push, so `bin/release` dates the Unreleased section and bumps the version on a branch
539
+ `release/vX.Y.Z`, opens its pull request, which merges itself once CI is green, and waits for it. It then pushes a
540
+ signed tag `vX.Y.Z` on the merge commit (`bin/release --tag X.Y.Z` does it if you stopped waiting). The tag starts
541
+ the Release workflow, which publishes the gem to RubyGems through trusted publishing and creates the GitHub release
542
+ with the version's notes and the `.gem`.
543
+
544
+ ## License
545
+
546
+ MIT, by Level UP Solutions, see [LICENSE.md](LICENSE.md). Originally written by Nando Vieira as
547
+ [fnando/browser](https://github.com/fnando/browser) (MIT). The corpus gathers user agents from
548
+ [ua-parser/uap-core](https://github.com/ua-parser/uap-core) (Apache 2.0),
549
+ [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT) and browser's test fixtures; the bot
550
+ list and its categories come from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT)
551
+ and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (MIT); the crawler IP ranges are those their
552
+ owners publish for that purpose: see [corpus/SOURCES.md](corpus/SOURCES.md).