agentry 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +93 -0
- data/LICENSE.md +21 -0
- data/README.md +552 -0
- data/data/bot_categories.json +1 -0
- data/data/bot_exceptions.json +1 -0
- data/data/bot_ip_ranges.json +1 -0
- data/data/bots.json +1 -0
- data/data/languages.json +1 -0
- data/data/samsung.json +1 -0
- data/data/search_engines.json +1 -0
- data/lib/agentry.rb +3 -0
- data/lib/browser/accept_language.rb +55 -0
- data/lib/browser/action_controller.rb +21 -0
- data/lib/browser/aliases.rb +33 -0
- data/lib/browser/alipay.rb +21 -0
- data/lib/browser/base.rb +307 -0
- data/lib/browser/blackberry.rb +23 -0
- data/lib/browser/bot/empty_user_agent_matcher.rb +11 -0
- data/lib/browser/bot/ip_ranges.rb +194 -0
- data/lib/browser/bot/keyword_matcher.rb +11 -0
- data/lib/browser/bot/known_bots_matcher.rb +11 -0
- data/lib/browser/bot.rb +221 -0
- data/lib/browser/browser.rb +200 -0
- data/lib/browser/cache.rb +92 -0
- data/lib/browser/chrome.rb +43 -0
- data/lib/browser/client_hints.rb +73 -0
- data/lib/browser/cloudflare.rb +65 -0
- data/lib/browser/detect_version.rb +22 -0
- data/lib/browser/detection.rb +25 -0
- data/lib/browser/device/android.rb +20 -0
- data/lib/browser/device/base.rb +13 -0
- data/lib/browser/device/blackberry_playbook.rb +19 -0
- data/lib/browser/device/ipad.rb +19 -0
- data/lib/browser/device/iphone.rb +19 -0
- data/lib/browser/device/ipod_touch.rb +19 -0
- data/lib/browser/device/kindle.rb +19 -0
- data/lib/browser/device/kindle_fire.rb +19 -0
- data/lib/browser/device/playstation3.rb +19 -0
- data/lib/browser/device/playstation4.rb +19 -0
- data/lib/browser/device/psp.rb +19 -0
- data/lib/browser/device/psvita.rb +19 -0
- data/lib/browser/device/samsung.rb +33 -0
- data/lib/browser/device/surface.rb +23 -0
- data/lib/browser/device/switch.rb +19 -0
- data/lib/browser/device/tv.rb +19 -0
- data/lib/browser/device/unknown.rb +19 -0
- data/lib/browser/device/wii.rb +19 -0
- data/lib/browser/device/wiiu.rb +19 -0
- data/lib/browser/device/xbox_360.rb +19 -0
- data/lib/browser/device/xbox_one.rb +19 -0
- data/lib/browser/device.rb +246 -0
- data/lib/browser/duck_duck_go.rb +22 -0
- data/lib/browser/edge.rb +25 -0
- data/lib/browser/electron.rb +22 -0
- data/lib/browser/epiphany.rb +21 -0
- data/lib/browser/facebook.rb +23 -0
- data/lib/browser/firefox.rb +21 -0
- data/lib/browser/google_search_app.rb +21 -0
- data/lib/browser/huawei_browser.rb +21 -0
- data/lib/browser/instagram.rb +21 -0
- data/lib/browser/internet_explorer.rb +63 -0
- data/lib/browser/maxthon.rb +21 -0
- data/lib/browser/meta/base.rb +42 -0
- data/lib/browser/meta/device.rb +11 -0
- data/lib/browser/meta/generic_browser.rb +15 -0
- data/lib/browser/meta/id.rb +11 -0
- data/lib/browser/meta/ie.rb +20 -0
- data/lib/browser/meta/ios.rb +11 -0
- data/lib/browser/meta/mobile.rb +11 -0
- data/lib/browser/meta/platform.rb +11 -0
- data/lib/browser/meta/proxy.rb +11 -0
- data/lib/browser/meta/safari.rb +11 -0
- data/lib/browser/meta/tablet.rb +11 -0
- data/lib/browser/meta/webkit.rb +11 -0
- data/lib/browser/meta.rb +14 -0
- data/lib/browser/micro_messenger.rb +21 -0
- data/lib/browser/middleware/context/additions.rb +18 -0
- data/lib/browser/middleware/context/url_methods.rb +15 -0
- data/lib/browser/middleware/context.rb +23 -0
- data/lib/browser/middleware.rb +70 -0
- data/lib/browser/miui_browser.rb +21 -0
- data/lib/browser/nokia.rb +21 -0
- data/lib/browser/opera.rb +21 -0
- data/lib/browser/otter.rb +21 -0
- data/lib/browser/phantom_js.rb +21 -0
- data/lib/browser/platform/adobe_air.rb +23 -0
- data/lib/browser/platform/android.rb +23 -0
- data/lib/browser/platform/base.rb +18 -0
- data/lib/browser/platform/blackberry.rb +23 -0
- data/lib/browser/platform/chrome_os.rb +23 -0
- data/lib/browser/platform/firefox_os.rb +23 -0
- data/lib/browser/platform/ios.rb +36 -0
- data/lib/browser/platform/kai_os.rb +23 -0
- data/lib/browser/platform/linux.rb +23 -0
- data/lib/browser/platform/mac.rb +25 -0
- data/lib/browser/platform/unknown.rb +23 -0
- data/lib/browser/platform/windows.rb +23 -0
- data/lib/browser/platform/windows_mobile.rb +23 -0
- data/lib/browser/platform/windows_phone.rb +23 -0
- data/lib/browser/platform.rb +220 -0
- data/lib/browser/qq.rb +25 -0
- data/lib/browser/rails.rb +24 -0
- data/lib/browser/safari.rb +40 -0
- data/lib/browser/samsung_browser.rb +21 -0
- data/lib/browser/snapchat.rb +21 -0
- data/lib/browser/sougou_browser.rb +24 -0
- data/lib/browser/sputnik.rb +21 -0
- data/lib/browser/testing.rb +29 -0
- data/lib/browser/uc_browser.rb +21 -0
- data/lib/browser/unknown.rb +32 -0
- data/lib/browser/version.rb +5 -0
- data/lib/browser/weibo.rb +21 -0
- data/lib/browser/yandex.rb +21 -0
- data/lib/browser.rb +4 -0
- data/test/ua.yml +181 -0
- data/test/ua_bots.yml +121 -0
- data/test/ua_search_engines.yml +7 -0
- metadata +274 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 1a57ed8305d43ff3e55dce1770f63f0bf63c712e084ba3013fe2f2240fcc27f6
|
|
4
|
+
data.tar.gz: 145e9e764e8cbc76e3f40b985d37181b3d801a620998bb34ecbcae152739a15b
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 0a7b34836efd8a5cff0af7eac8e723d0027670a34bde7f1fd3e58763ba4a8f917094a76d19b4e0cb5e7ced62af3245a55ffc516310ac2d4002ea1fa421c4812d
|
|
7
|
+
data.tar.gz: ee6ba51155056619fada17e77f0997849fb1e686ca9af9424339ab788d74018e3e9d71097259e6286c5fbed478c7f84b560943a26495602169b3593b30f46e77
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [Unreleased]
|
|
4
|
+
|
|
5
|
+
## [3.0.0] — 2026-11-05
|
|
6
|
+
|
|
7
|
+
The gem is renamed Agentry and lives at levups/agentry; the API is untouched.
|
|
8
|
+
|
|
9
|
+
**Breaking**
|
|
10
|
+
- The gem is now `agentry`: replace `gem "browserx"` with `gem "agentry"` in the Gemfile (or `gem "agentry", require: "browser/browser"`). `require "browser"`, the `Browser` namespace and every predicate stay. `require "browserx"` becomes `require "agentry"`.
|
|
11
|
+
|
|
12
|
+
## [2.1.0] — 2026-11-05
|
|
13
|
+
|
|
14
|
+
The first request of each process gets cheaper: the lists ship as JSON, and `Browser.eager_load!` builds them
|
|
15
|
+
before a server forks.
|
|
16
|
+
|
|
17
|
+
**New**
|
|
18
|
+
- `Browser.eager_load!` loads the lists and builds the matchers and bot keyword tries up front, so a server that
|
|
19
|
+
preloads its app before forking (Puma, Unicorn, Pitchfork) shares them with every worker. Rails calls it on its
|
|
20
|
+
own when it eager loads, which production does by default. A worker's first request spent 12.9 ms building
|
|
21
|
+
them, 0.2 ms once built.
|
|
22
|
+
|
|
23
|
+
**Faster**
|
|
24
|
+
- The lists ship as JSON: without Bootsnap's cache, a process's first request is 3× faster (12.4 → 4.0 ms with
|
|
25
|
+
YJIT, 7.5 → 2.7 ms without), and `require "browser"` no longer loads Psych (32 → 21 ms with YJIT, 28 → 22 ms
|
|
26
|
+
without). With Bootsnap, nothing changes. Code that used `YAML` without requiring it now needs its own
|
|
27
|
+
`require "yaml"`.
|
|
28
|
+
|
|
29
|
+
## [2.0.0] — 2026-11-04
|
|
30
|
+
|
|
31
|
+
A repeated user agent is now answered in 2 µs, about 600 more bots are recognised, and bots, Cloudflare headers and
|
|
32
|
+
Client Hints can be checked against the request. Without the new headers, every answer stays browser 6.2.0's,
|
|
33
|
+
except bot detection, which finds more bots.
|
|
34
|
+
|
|
35
|
+
**Breaking**
|
|
36
|
+
- `bot?` is true for about 600 more user agents: bots from crawler-user-agents and ai.robots.txt, synced weekly,
|
|
37
|
+
add 1,247 entries to `bots.yml`. Of the crawlers added there since 2024, browser misses 353 out of 882 and
|
|
38
|
+
Agentry 11. Analytics that filter on `bot?` will count fewer humans from the day you switch.
|
|
39
|
+
- The Rails helper and the middleware now pass the request's headers, so Client Hints refine the answers on
|
|
40
|
+
Chromium: Windows 11 is no longer `windows10?`, and Chrome for Android in desktop mode is an Android tablet,
|
|
41
|
+
not a Linux desktop.
|
|
42
|
+
|
|
43
|
+
**New**
|
|
44
|
+
- `bot.category` (16 categories, close to Cloudflare's verified bot categories), `bot.ai?` and
|
|
45
|
+
`bot.user_triggered?`. Test drivers (Playwright, Puppeteer, Selenium, headless Chrome) get the
|
|
46
|
+
`browser_automation` category but are not bots, so system tests keep running as humans.
|
|
47
|
+
- `bot.verified?(ip)` checks a crawler against the IP ranges its owner publishes (Google, Bing, OpenAI, Apple,
|
|
48
|
+
Perplexity, DuckDuckGo), shipped with the gem and refreshable at runtime; reverse DNS is opt-in. `bot.owner`
|
|
49
|
+
names the owner.
|
|
50
|
+
- `Browser.new(ua, headers: request.env)` reads Cloudflare's bot headers (`browser.cloudflare`: verified bot,
|
|
51
|
+
category, score, country; header names configurable) and Chromium's Client Hints (`browser.client_hints`,
|
|
52
|
+
`platform.windows11?`).
|
|
53
|
+
- `Browser.cache_size` (default 1,000, `0` turns it off) bounds the new per-user-agent cache: about 1 KB per
|
|
54
|
+
entry, so about 1 MB per process.
|
|
55
|
+
|
|
56
|
+
**Faster**
|
|
57
|
+
- Answers are cached per user agent, matchers that can't match are skipped, and bots are found through a byte
|
|
58
|
+
trie. Ruby 4.0.9: a repeated user agent takes 1.8 µs instead of 57.5 with YJIT and 3.6 µs instead of 72.5
|
|
59
|
+
without; 2,000 distinct user agents take 24.2 µs instead of 62.4 with YJIT and 43.9 µs instead of 77.3
|
|
60
|
+
without. Objects per request: 9 instead of 116 repeated, 50 instead of 125 distinct.
|
|
61
|
+
|
|
62
|
+
**Fixed**
|
|
63
|
+
- Loading `browser` and `agentry` together raises a clear `LoadError` instead of one silently overriding the
|
|
64
|
+
other.
|
|
65
|
+
|
|
66
|
+
## [1.0.0] — 2026-11-04
|
|
67
|
+
|
|
68
|
+
First release, forked from browser 6.2.0 (`dd88933`). Same API: `gem "agentry"` replaces
|
|
69
|
+
`gem "browser"` without touching a call site. Earlier versions are browser's: see its
|
|
70
|
+
[changelog](https://github.com/fnando/browser/blob/main/CHANGELOG.md).
|
|
71
|
+
|
|
72
|
+
**Breaking**
|
|
73
|
+
- Requires Ruby 3.3 or later: 3.2 reached its end of life on 2026-03-31.
|
|
74
|
+
- Tested against Rails 8.0 and 8.1 only: 7.0, 7.1 and 7.2 reached their end of life.
|
|
75
|
+
|
|
76
|
+
**Faster**
|
|
77
|
+
- Bots are detected through one prefix-trie regexp instead of one match per entry, and without detecting the
|
|
78
|
+
browser twice; platform and device matchers are built lazily. A request takes 45% less time and half the
|
|
79
|
+
allocations.
|
|
80
|
+
|
|
81
|
+
**Fixed**
|
|
82
|
+
- The railtie loads Rails, not only itself, so it works without a preloaded ActiveSupport.
|
|
83
|
+
- `lib/agentry.rb` exists, so Bundler's auto-require loads the gem.
|
|
84
|
+
|
|
85
|
+
**New**
|
|
86
|
+
- A corpus of 20,328 user agents, with snapshot and diff scripts proving answers match upstream, and a
|
|
87
|
+
request benchmark.
|
|
88
|
+
|
|
89
|
+
[Unreleased]: https://github.com/levups/agentry/compare/v3.0.0...HEAD
|
|
90
|
+
[3.0.0]: https://github.com/levups/agentry/compare/v2.1.0...v3.0.0
|
|
91
|
+
[2.1.0]: https://github.com/levups/agentry/compare/v2.0.0...v2.1.0
|
|
92
|
+
[2.0.0]: https://github.com/levups/agentry/compare/v1.0.0...v2.0.0
|
|
93
|
+
[1.0.0]: https://github.com/levups/agentry/releases/tag/v1.0.0
|
data/LICENSE.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2010 Nando Vieira
|
|
4
|
+
Copyright (c) 2026 Level UP Solutions
|
|
5
|
+
|
|
6
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
|
7
|
+
this software and associated documentation files (the "Software"), to deal in
|
|
8
|
+
the Software without restriction, including without limitation the rights to
|
|
9
|
+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
|
|
10
|
+
the Software, and to permit persons to whom the Software is furnished to do so,
|
|
11
|
+
subject to the following conditions:
|
|
12
|
+
|
|
13
|
+
The above copyright notice and this permission notice shall be included in all
|
|
14
|
+
copies or substantial portions of the Software.
|
|
15
|
+
|
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
17
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
|
18
|
+
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
|
19
|
+
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
|
20
|
+
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
|
21
|
+
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,552 @@
|
|
|
1
|
+
# Agentry
|
|
2
|
+
|
|
3
|
+
Browser detection for Ruby, with ActionController integration: the API of the
|
|
4
|
+
[browser](https://rubygems.org/gems/browser) gem, faster, and a bot list that stays fresh. Put `gem "agentry"` in
|
|
5
|
+
your Gemfile instead of `gem "browser"`, and every call site keeps working — same `Browser` namespace, same
|
|
6
|
+
`require "browser"`, same predicates.
|
|
7
|
+
|
|
8
|
+
- **Faster**: 25 µs per request instead of 126 µs with browser 6.2.0, 2 µs once a user agent was seen (Ruby 4.0.9,
|
|
9
|
+
YJIT, `bench/request.rb`).
|
|
10
|
+
- **Fresher bots**: synced every week from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents)
|
|
11
|
+
and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt), with categories (`bot.category`, `bot.ai?`).
|
|
12
|
+
Of the 882 crawler user agents added to crawler-user-agents since 2024 (test drivers and scanners left out),
|
|
13
|
+
browser misses 353, Agentry 11.
|
|
14
|
+
- **Verified signals**: `bot.verified?(ip)` checks a crawler against the IP ranges Google, Bing, OpenAI, Apple,
|
|
15
|
+
Perplexity and DuckDuckGo publish, `browser.cloudflare` reads the headers of a Cloudflare Transform Rule, and
|
|
16
|
+
Chromium's Client Hints tell Windows 11 from Windows 10.
|
|
17
|
+
|
|
18
|
+
Site: https://levups.github.io/agentry/
|
|
19
|
+
|
|
20
|
+
## Installation
|
|
21
|
+
|
|
22
|
+
```ruby
|
|
23
|
+
gem "agentry"
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
## Usage
|
|
27
|
+
|
|
28
|
+
```ruby
|
|
29
|
+
require "browser"
|
|
30
|
+
|
|
31
|
+
browser = Browser.new("Some User Agent", accept_language: "en-us")
|
|
32
|
+
|
|
33
|
+
# General info
|
|
34
|
+
browser.bot?
|
|
35
|
+
browser.chrome?
|
|
36
|
+
browser.chromium_based?
|
|
37
|
+
browser.core_media?
|
|
38
|
+
browser.duck_duck_go?
|
|
39
|
+
browser.edge? # Newest MS browser
|
|
40
|
+
browser.electron? # Electron Framework
|
|
41
|
+
browser.firefox?
|
|
42
|
+
browser.full_version
|
|
43
|
+
browser.ie?
|
|
44
|
+
browser.ie?(6) # detect specific IE version
|
|
45
|
+
browser.ie?([">8", "<10"]) # detect specific IE (IE9).
|
|
46
|
+
browser.known? # has the browser been successfully detected?
|
|
47
|
+
browser.unknown? # the browser wasn't detected.
|
|
48
|
+
browser.meta # an array with several attributes
|
|
49
|
+
browser.name # readable browser name
|
|
50
|
+
browser.nokia?
|
|
51
|
+
browser.opera?
|
|
52
|
+
browser.opera_mini?
|
|
53
|
+
browser.phantom_js?
|
|
54
|
+
browser.quicktime?
|
|
55
|
+
browser.safari?
|
|
56
|
+
browser.safari_webapp_mode?
|
|
57
|
+
browser.samsung_browser?
|
|
58
|
+
browser.to_s # the meta info joined by space
|
|
59
|
+
browser.uc_browser?
|
|
60
|
+
browser.version # major version number
|
|
61
|
+
browser.webkit?
|
|
62
|
+
browser.webkit_full_version
|
|
63
|
+
browser.yandex?
|
|
64
|
+
browser.wechat?
|
|
65
|
+
browser.qq?
|
|
66
|
+
browser.weibo?
|
|
67
|
+
browser.sputnik?
|
|
68
|
+
browser.sougou_browser?
|
|
69
|
+
browser.epiphany?
|
|
70
|
+
|
|
71
|
+
# Get bot info
|
|
72
|
+
browser.bot.name
|
|
73
|
+
browser.bot.search_engine?
|
|
74
|
+
browser.bot.category # :ai_crawler, :search_engine_crawler, :scanner… (nil for a browser)
|
|
75
|
+
browser.bot.ai?
|
|
76
|
+
browser.bot.user_triggered? # an AI agent acting for a person right now (ChatGPT-User, Claude-User)
|
|
77
|
+
browser.bot?
|
|
78
|
+
browser.bot.why? # shows which matcher detected this user agent as a bot.
|
|
79
|
+
Browser::Bot.why?(ua)
|
|
80
|
+
|
|
81
|
+
# Get device info
|
|
82
|
+
browser.device
|
|
83
|
+
browser.device.id
|
|
84
|
+
browser.device.name
|
|
85
|
+
browser.device.unknown?
|
|
86
|
+
browser.device.blackberry_playbook?
|
|
87
|
+
browser.device.console?
|
|
88
|
+
browser.device.ipad?
|
|
89
|
+
browser.device.iphone?
|
|
90
|
+
browser.device.ipod_touch?
|
|
91
|
+
browser.device.kindle?
|
|
92
|
+
browser.device.kindle_fire?
|
|
93
|
+
browser.device.mobile?
|
|
94
|
+
browser.device.nintendo?
|
|
95
|
+
browser.device.playstation?
|
|
96
|
+
browser.device.ps3?
|
|
97
|
+
browser.device.ps4?
|
|
98
|
+
browser.device.psp?
|
|
99
|
+
browser.device.silk?
|
|
100
|
+
browser.device.surface?
|
|
101
|
+
browser.device.tablet?
|
|
102
|
+
browser.device.tv?
|
|
103
|
+
browser.device.vita?
|
|
104
|
+
browser.device.wii?
|
|
105
|
+
browser.device.wiiu?
|
|
106
|
+
browser.device.samsung?
|
|
107
|
+
browser.device.switch?
|
|
108
|
+
browser.device.xbox?
|
|
109
|
+
browser.device.xbox_360?
|
|
110
|
+
browser.device.xbox_one?
|
|
111
|
+
|
|
112
|
+
# Get platform info
|
|
113
|
+
browser.platform
|
|
114
|
+
browser.platform.id
|
|
115
|
+
browser.platform.name
|
|
116
|
+
browser.platform.version # e.g. 9 (for iOS9)
|
|
117
|
+
browser.platform.adobe_air?
|
|
118
|
+
browser.platform.android?
|
|
119
|
+
browser.platform.android?(4.2) # detect Android Jelly Bean 4.2
|
|
120
|
+
browser.platform.android_app? # detect webview in an Android app
|
|
121
|
+
browser.platform.android_webview? # alias for android_app?
|
|
122
|
+
browser.platform.blackberry?
|
|
123
|
+
browser.platform.blackberry?(10) # detect specific BlackBerry version
|
|
124
|
+
browser.platform.chrome_os?
|
|
125
|
+
browser.platform.firefox_os?
|
|
126
|
+
browser.platform.ios? # detect iOS
|
|
127
|
+
browser.platform.ios?(9) # detect specific iOS version
|
|
128
|
+
browser.platform.ios_app? # detect webview in an iOS app
|
|
129
|
+
browser.platform.ios_webview? # alias for ios_app?
|
|
130
|
+
browser.platform.linux?
|
|
131
|
+
browser.platform.mac?
|
|
132
|
+
browser.platform.unknown?
|
|
133
|
+
browser.platform.windows10?
|
|
134
|
+
browser.platform.windows7?
|
|
135
|
+
browser.platform.windows8?
|
|
136
|
+
browser.platform.windows8_1?
|
|
137
|
+
browser.platform.windows?
|
|
138
|
+
browser.platform.windows_mobile?
|
|
139
|
+
browser.platform.windows_phone?
|
|
140
|
+
browser.platform.windows_rt?
|
|
141
|
+
browser.platform.windows_touchscreen_desktop?
|
|
142
|
+
browser.platform.windows_vista?
|
|
143
|
+
browser.platform.windows_wow64?
|
|
144
|
+
browser.platform.windows_x64?
|
|
145
|
+
browser.platform.windows_x64_inclusive?
|
|
146
|
+
browser.platform.windows_xp?
|
|
147
|
+
browser.platform.kai_os?
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### Aliases
|
|
151
|
+
|
|
152
|
+
To add aliases like `mobile?` and `tablet?` to the base object (e.g
|
|
153
|
+
`browser.mobile?`), require the `browser/aliases` file and extend the
|
|
154
|
+
Browser::Base object like the following:
|
|
155
|
+
|
|
156
|
+
```ruby
|
|
157
|
+
require "browser/aliases"
|
|
158
|
+
Browser::Base.include(Browser::Aliases)
|
|
159
|
+
|
|
160
|
+
browser = Browser.new("Some user agent")
|
|
161
|
+
browser.mobile? #=> false
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
### What's being detected?
|
|
165
|
+
|
|
166
|
+
- For a list of platform detections, check
|
|
167
|
+
[lib/browser/platform.rb](lib/browser/platform.rb)
|
|
168
|
+
- For a list of device detections, check
|
|
169
|
+
[lib/browser/device.rb](lib/browser/device.rb)
|
|
170
|
+
- For a list of bot detections, check
|
|
171
|
+
[bots.yml](bots.yml)
|
|
172
|
+
|
|
173
|
+
### Detecting modern browsers
|
|
174
|
+
|
|
175
|
+
To detect whether a browser can be considered as modern or not, create a method
|
|
176
|
+
that abstracts your versioning constraints. The following example will consider
|
|
177
|
+
any of the following browsers as a modern:
|
|
178
|
+
|
|
179
|
+
```ruby
|
|
180
|
+
# Expects an Browser instance,
|
|
181
|
+
# like in `Browser.new(user_agent, accept_language: language)`.
|
|
182
|
+
def modern_browser?(browser)
|
|
183
|
+
[
|
|
184
|
+
browser.chrome?(">= 65"),
|
|
185
|
+
browser.safari?(">= 10"),
|
|
186
|
+
browser.firefox?(">= 52"),
|
|
187
|
+
browser.ie?(">= 11") && !browser.compatibility_view?,
|
|
188
|
+
browser.edge?(">= 15"),
|
|
189
|
+
browser.opera?(">= 50"),
|
|
190
|
+
browser.facebook?
|
|
191
|
+
&& browser.safari_webapp_mode?
|
|
192
|
+
&& browser.webkit_full_version.to_i >= 602
|
|
193
|
+
].any?
|
|
194
|
+
end
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
### Migrating from browser
|
|
198
|
+
|
|
199
|
+
Replace `gem "browser"` with `gem "agentry"` in the Gemfile, nothing else: `require "browser"`, the `Browser`
|
|
200
|
+
namespace and every predicate stay. Both gems define `Browser`, so Agentry refuses to load next to `browser`,
|
|
201
|
+
with an error saying so, rather than letting one silently override the other. Agentry requires Ruby 3.3 or later
|
|
202
|
+
and is tested on Rails 8.0 and 8.1.
|
|
203
|
+
|
|
204
|
+
What changes on its own: the bots of the weekly sync are bots from the day you switch (600 user agents of the
|
|
205
|
+
corpus that browser took for humans), so analytics that leave bots out count fewer visitors from that day; 792
|
|
206
|
+
user agents named `Generic Bot` get a name; no browser, platform or device answer moves, and test drivers stay out
|
|
207
|
+
of `bot?`. To roll back, put `gem "browser"` back, after removing the calls to what browser lacks
|
|
208
|
+
(`Browser.cache_size=`, `bot.category`, `bot.ai?`, `bot.user_triggered?`). The
|
|
209
|
+
[migration guide](https://levups.github.io/agentry/guide.html#migrating) has the details.
|
|
210
|
+
|
|
211
|
+
### Rails integration
|
|
212
|
+
|
|
213
|
+
Just add it to the Gemfile.
|
|
214
|
+
|
|
215
|
+
```ruby
|
|
216
|
+
gem "agentry"
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
This adds a helper method called `browser`, that inspects your current user
|
|
220
|
+
agent.
|
|
221
|
+
|
|
222
|
+
```erb
|
|
223
|
+
<% if browser.ie?(6) %>
|
|
224
|
+
<p class="disclaimer">You're running an older IE version. Please update it!</p>
|
|
225
|
+
<% end %>
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
If you want to use Browser on your Rails app but don't want to taint your
|
|
229
|
+
controller, use the following line on your Gemfile:
|
|
230
|
+
|
|
231
|
+
```ruby
|
|
232
|
+
gem "agentry", require: "browser/browser"
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
### Accept Language
|
|
236
|
+
|
|
237
|
+
Parses the accept-language header from an HTTP request and produces an array of
|
|
238
|
+
language objects sorted by quality.
|
|
239
|
+
|
|
240
|
+
```ruby
|
|
241
|
+
browser = Browser.new("Some User Agent", accept_language: "en-us")
|
|
242
|
+
|
|
243
|
+
browser.accept_language.class
|
|
244
|
+
#=> Array
|
|
245
|
+
|
|
246
|
+
language = browser.accept_language.first
|
|
247
|
+
|
|
248
|
+
language.code
|
|
249
|
+
#=> "en"
|
|
250
|
+
|
|
251
|
+
language.region
|
|
252
|
+
#=> "US"
|
|
253
|
+
|
|
254
|
+
language.full
|
|
255
|
+
#=> "en-US"
|
|
256
|
+
|
|
257
|
+
language.quality
|
|
258
|
+
#=> 1.0
|
|
259
|
+
|
|
260
|
+
language.name
|
|
261
|
+
#=> "English/United States"
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
Result is always sorted in quality order from highest to lowest. As per the HTTP
|
|
265
|
+
spec:
|
|
266
|
+
|
|
267
|
+
- omitting the quality value implies 1.0.
|
|
268
|
+
- quality value equal to zero means that is not accepted by the client.
|
|
269
|
+
|
|
270
|
+
### Internet Explorer
|
|
271
|
+
|
|
272
|
+
Internet Explorer has a compatibility view mode that allows newer versions
|
|
273
|
+
(IE8+) to run as an older version. Browser will always return the navigator
|
|
274
|
+
version, ignoring the compatibility view version, when defined. If you need to
|
|
275
|
+
get the engine's version, you have to use `Browser#msie_version` and
|
|
276
|
+
`Browser#msie_full_version`.
|
|
277
|
+
|
|
278
|
+
So, let's say an user activates compatibility view in a IE11 browser. This is
|
|
279
|
+
what you'll get:
|
|
280
|
+
|
|
281
|
+
```ruby
|
|
282
|
+
browser.version
|
|
283
|
+
#=> 11
|
|
284
|
+
|
|
285
|
+
browser.full_version
|
|
286
|
+
#=> 11.0
|
|
287
|
+
|
|
288
|
+
browser.msie_version
|
|
289
|
+
#=> 7
|
|
290
|
+
|
|
291
|
+
browser.msie_full_version
|
|
292
|
+
#=> 7.0
|
|
293
|
+
|
|
294
|
+
browser.compatibility_view?
|
|
295
|
+
#=> true
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
This behavior changed in browser v1.0.0; previously there wasn't a way of getting the
|
|
299
|
+
real browser version.
|
|
300
|
+
|
|
301
|
+
### Safari
|
|
302
|
+
|
|
303
|
+
iOS webviews and web apps aren't detected as Safari anymore, so be aware of that
|
|
304
|
+
if that's your case. You can use a combination of platform and webkit detection
|
|
305
|
+
to do whatever you want.
|
|
306
|
+
|
|
307
|
+
```ruby
|
|
308
|
+
# iPad's Safari running as web app mode.
|
|
309
|
+
browser = Browser.new("Mozilla/5.0 (iPad; U; CPU OS 3_2_1 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Mobile/7B405")
|
|
310
|
+
|
|
311
|
+
browser.safari?
|
|
312
|
+
#=> false
|
|
313
|
+
|
|
314
|
+
browser.webkit?
|
|
315
|
+
#=> true
|
|
316
|
+
|
|
317
|
+
browser.platform.ios?
|
|
318
|
+
#=> true
|
|
319
|
+
```
|
|
320
|
+
|
|
321
|
+
### Bots
|
|
322
|
+
|
|
323
|
+
The bot detection is quite aggressive. Anything that matches at least one of the
|
|
324
|
+
following requirements will be considered a bot.
|
|
325
|
+
|
|
326
|
+
- Empty user agent string
|
|
327
|
+
- User agent that matches `/crawl|fetch|search|monitoring|spider|bot/`
|
|
328
|
+
- Any known bot listed under
|
|
329
|
+
[bots.yml](bots.yml)
|
|
330
|
+
|
|
331
|
+
Every week, `script/sync_bots.rb` adds the bots of
|
|
332
|
+
[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) and
|
|
333
|
+
[ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (both MIT) to the end of `bots.yml`, and
|
|
334
|
+
categorizes them in `bot_categories.yml`, through a pull request that lists the answers that change. How many of
|
|
335
|
+
those lists' bots browser and Agentry detect, per category, is on the
|
|
336
|
+
[Bots page](https://levups.github.io/agentry/bots.html#coverage).
|
|
337
|
+
|
|
338
|
+
`browser.bot.category` names what the bot declares it is, with the names of Cloudflare's verified bot categories
|
|
339
|
+
where one fits (`search_engine_crawler`, `ai_crawler`, `ai_search`, `ai_assistant`, `page_preview`,
|
|
340
|
+
`feed_fetcher`, `monitoring_and_analytics`, `advertising_and_marketing`, `archiver`, `security`…), plus
|
|
341
|
+
`ai_agent`, `http_library`, `scanner` and `browser_automation`. `bot.ai?` is true for the four AI categories,
|
|
342
|
+
`bot.user_triggered?` for `ai_assistant` and `ai_agent`. The category is `nil` for a browser, a bot exception
|
|
343
|
+
and a bot no list names.
|
|
344
|
+
|
|
345
|
+
HTTP libraries (`curl`, `got`, `python-requests`) and scanners (sqlmap, Nikto, Censys) are bots, as `curl` and
|
|
346
|
+
`python-requests` already were in browser. Test drivers (headless Chrome, Playwright, Puppeteer, Selenium) are
|
|
347
|
+
not: an app's system tests drive a real browser through them, and the headless Chrome of Playwright and Selenium
|
|
348
|
+
sends `HeadlessChrome`. They get the `browser_automation` category instead, so an app can tell them apart in
|
|
349
|
+
production (`browser.bot? || browser.bot.category == :browser_automation`). PhantomJS stays a bot, as in
|
|
350
|
+
browser.
|
|
351
|
+
|
|
352
|
+
To add custom matchers, you can add a callable object to
|
|
353
|
+
`Browser::Bot.matchers`. The following example matches everything that has a
|
|
354
|
+
`externalhit` substring on it. The bot name will always be `General Bot`.
|
|
355
|
+
|
|
356
|
+
```ruby
|
|
357
|
+
Browser::Bot.matchers << ->(ua, _browser) { ua.match?(/externalhit/i) }
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
To clear all matchers, including the ones that are bundled, use
|
|
361
|
+
`Browser::Bot.matchers.clear`. You can re-add built-in matchers by doing the
|
|
362
|
+
following:
|
|
363
|
+
|
|
364
|
+
```ruby
|
|
365
|
+
Browser::Bot.matchers += Browser::Bot.default_matchers
|
|
366
|
+
```
|
|
367
|
+
|
|
368
|
+
To restore v2's bot detection, remove the following matchers:
|
|
369
|
+
|
|
370
|
+
```ruby
|
|
371
|
+
Browser::Bot.matchers.delete(Browser::Bot::KeywordMatcher)
|
|
372
|
+
Browser::Bot.matchers.delete(Browser::Bot::EmptyUserAgentMatcher)
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
To extend the bot list, you can manipulate the methods below:
|
|
376
|
+
|
|
377
|
+
```ruby
|
|
378
|
+
Browser::Bot.bots.merge!(new_bots_hash)
|
|
379
|
+
Browser::Bot.bot_exceptions += new_exceptions
|
|
380
|
+
Browser::Bot.search_engines.merge!(new_search_engines_hash)
|
|
381
|
+
Browser::Bot.category_by_keyword["faraday"] = :http_library
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
### Verified crawlers
|
|
385
|
+
|
|
386
|
+
Google, Bing, OpenAI, Apple, Perplexity and DuckDuckGo publish their crawlers' IP ranges. They ship in
|
|
387
|
+
`bot_ip_ranges.yml` (as JSON in the gem), rewritten by the weekly sync (`script/sync_ip_ranges.rb`). The lookup is a binary search, no
|
|
388
|
+
network call; an IP that does not parse answers `false`.
|
|
389
|
+
|
|
390
|
+
```ruby
|
|
391
|
+
browser.bot.owner # => :google for a user agent claiming Googlebot
|
|
392
|
+
browser.bot.verified?(request.remote_ip) # => whether the IP is in Google's ranges
|
|
393
|
+
browser.bot.verified?(request.remote_ip, reverse_dns: true) # also Google's DNS check: two DNS queries
|
|
394
|
+
Browser::Bot::IPRanges.refresh # fetches the ranges again, in a long-lived process
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
### Cloudflare and Client Hints
|
|
398
|
+
|
|
399
|
+
`Browser.new(user_agent, accept_language:, headers: rack_env)` also reads the request's headers; the Rails helper
|
|
400
|
+
and the middleware pass them. `browser.cloudflare` reads the headers a Cloudflare Transform Rule sets
|
|
401
|
+
(`verified_bot?`, `bot_category`, `bot_score`) and `CF-IPCountry` (`country`); their names are configurable with
|
|
402
|
+
`Browser::Cloudflare.header_names = {bot_score: "X-My-Score"}`. Chromium's Client Hints (`Sec-CH-UA`,
|
|
403
|
+
`Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`, `Sec-CH-UA-Platform-Version`) refine the platform and the device, and
|
|
404
|
+
`browser.platform.windows11?` needs them. Without `headers:`, every answer is the user agent's, as before. The
|
|
405
|
+
rule to create and the details are in the [guide](https://levups.github.io/agentry/guide.html#cloudflare).
|
|
406
|
+
|
|
407
|
+
`bench/signals.rb` measures what they cost.
|
|
408
|
+
|
|
409
|
+
### Middleware
|
|
410
|
+
|
|
411
|
+
You can use the `Browser::Middleware` to redirect user agents.
|
|
412
|
+
|
|
413
|
+
```ruby
|
|
414
|
+
use Browser::Middleware do
|
|
415
|
+
redirect_to "/upgrade" if browser.ie?
|
|
416
|
+
end
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
If you're using Rails, you can use the route helper methods. Just add something
|
|
420
|
+
like the following to a initializer file (`config/initializers/browser.rb`).
|
|
421
|
+
|
|
422
|
+
```ruby
|
|
423
|
+
Rails.configuration.middleware.use Browser::Middleware do
|
|
424
|
+
redirect_to upgrade_path if browser.ie?
|
|
425
|
+
end
|
|
426
|
+
```
|
|
427
|
+
|
|
428
|
+
If you need access to the `Rack::Request` object (e.g. to exclude a path), you
|
|
429
|
+
can do so with `request`.
|
|
430
|
+
|
|
431
|
+
```ruby
|
|
432
|
+
Rails.configuration.middleware.use Browser::Middleware do
|
|
433
|
+
redirect_to upgrade_path if browser.ie? && request.env["PATH_INFO"] != "/exclude_me"
|
|
434
|
+
end
|
|
435
|
+
```
|
|
436
|
+
|
|
437
|
+
### Restrictions
|
|
438
|
+
|
|
439
|
+
- User agent has a size limit of 2048 bytes. This can be customized through
|
|
440
|
+
`Browser.user_agent_size_limit=(size)`.
|
|
441
|
+
- Accept-Language has a size limit of 2048 bytes. This can be customized through
|
|
442
|
+
`Browser.accept_language_size_limit=(size)`.
|
|
443
|
+
|
|
444
|
+
If size is not respected, then `Browser::Error` is raised.
|
|
445
|
+
|
|
446
|
+
```ruby
|
|
447
|
+
Browser.user_agent_size_limit = 4096
|
|
448
|
+
Browser.accept_language_size_limit = 4096
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
### Cache
|
|
452
|
+
|
|
453
|
+
Each process keeps what it detected for the last 1,000 distinct user agents (browser, platform, device, bot
|
|
454
|
+
answers) and the last 1,000 Accept-Language headers it parsed, about 1 KB per user agent: a repeated user agent
|
|
455
|
+
costs about 2 µs. The cache empties itself when full, and when `Browser.matchers`, `Browser::Bot.matchers`,
|
|
456
|
+
`Browser::Platform.matchers`, `Browser::Device.matchers`, the bot list or its exceptions change. Setting
|
|
457
|
+
`Browser.cache_size` empties it too, for example after renaming an entry of `Browser::Bot.bots` in place.
|
|
458
|
+
|
|
459
|
+
```ruby
|
|
460
|
+
Browser.cache_size = 5_000 # 0 turns it off
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
### Preloading before a fork
|
|
464
|
+
|
|
465
|
+
The lists, the matchers and the bot keyword tries are built on first use: the first request of each process pays
|
|
466
|
+
about 13 ms for them, the next ones 0.2 ms. `Browser.eager_load!` builds them up front, so a server that preloads
|
|
467
|
+
its app before forking (Puma's `preload_app!`, Unicorn, Pitchfork) shares them with every worker. Rails calls it on
|
|
468
|
+
its own when it eager loads, which production does by default. Elsewhere, call it before the fork:
|
|
469
|
+
|
|
470
|
+
```ruby
|
|
471
|
+
Browser.eager_load!
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
## Performance
|
|
475
|
+
|
|
476
|
+
`bench/request.rb` replays what a web request asks (name, version, `bot?`, platform, device, accept-language) over
|
|
477
|
+
2,000 distinct user agents of the corpus and over 20 repeated ones, median of 7. On Ruby 4.0.9 with YJIT, browser
|
|
478
|
+
6.2.0 takes 126 µs and 245 objects per request, Agentry 25 µs and 52 objects on a new user agent, 2 µs and 12
|
|
479
|
+
objects on a repeated one. All four runs, with and without YJIT, are on the
|
|
480
|
+
[benchmarks page](https://levups.github.io/agentry/benchmarks.html).
|
|
481
|
+
|
|
482
|
+
```bash
|
|
483
|
+
bundle exec rake bench UPSTREAM=path/to/upstream # fnando/browser at dd88933
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
It runs `ruby [--yjit] -I lib bench/request.rb --write` for each gem and JIT, which records the run (gem, commit,
|
|
487
|
+
Ruby, YJIT, machine, date) in `docs/benchmarks.json`, then `rake docs:render` rewrites the site's numbers from it.
|
|
488
|
+
|
|
489
|
+
## Development
|
|
490
|
+
|
|
491
|
+
```bash
|
|
492
|
+
bundle install
|
|
493
|
+
bundle exec rake test
|
|
494
|
+
bundle exec rubocop
|
|
495
|
+
```
|
|
496
|
+
|
|
497
|
+
A change that is not meant to change an answer proves it on the corpus of 20,000 user agents, against a plain
|
|
498
|
+
clone of [fnando/browser](https://github.com/fnando/browser) at `dd88933`, as CI does on every pull request:
|
|
499
|
+
|
|
500
|
+
```bash
|
|
501
|
+
ruby -I path/to/upstream/lib script/corpus_snapshot.rb > /tmp/upstream.jsonl
|
|
502
|
+
ruby -I lib script/corpus_snapshot.rb > /tmp/agentry.jsonl
|
|
503
|
+
ruby script/corpus_diff.rb /tmp/upstream.jsonl /tmp/agentry.jsonl
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
The diff passes when the answers that differ are exactly those of `corpus/expected_differences.jsonl`: a change
|
|
507
|
+
meant to change answers rewrites it with `ruby script/corpus_diff.rb --write …` and shows its diff.
|
|
508
|
+
|
|
509
|
+
The weekly bot sync (`.github/workflows/bot-sync.yml`) runs the same steps by hand:
|
|
510
|
+
|
|
511
|
+
```bash
|
|
512
|
+
ruby script/sync_bots.rb # bots.yml and bot_categories.yml
|
|
513
|
+
ruby -I lib script/sync_ip_ranges.rb # bot_ip_ranges.yml
|
|
514
|
+
ruby script/build_data.rb # data/*.json, what the gem reads, from the *.yml
|
|
515
|
+
ruby script/bot_coverage.rb path/to/upstream/lib # docs/bot_coverage.json, and its Markdown table
|
|
516
|
+
ruby script/render_docs.rb # the site's numbers, from docs/*.json and the lists
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
The numbers of `docs/*.html` sit between `<!-- render:… -->` markers and come from `docs/benchmarks.json`,
|
|
520
|
+
`docs/bot_coverage.json`, the bot lists and `corpus/expected_differences.jsonl`: CI fails when a page differs from
|
|
521
|
+
what `bundle exec rake docs:render` writes.
|
|
522
|
+
|
|
523
|
+
To add a bot, add its user agent to `test/ua_bots.yml` and its readable name to `bots.yml` above the synced
|
|
524
|
+
section, keyed by a lowercased string the user agent contains. A search engine goes in `test/ua_search_engines.yml`,
|
|
525
|
+
`test/ua_bots.yml` and `search_engines.yml`.
|
|
526
|
+
|
|
527
|
+
### Releasing
|
|
528
|
+
|
|
529
|
+
Pull requests leave `CHANGELOG.md` alone: a release starts by writing its notes under `## [Unreleased]`, from the
|
|
530
|
+
pull requests merged since the last tag, without committing them. Then, from an up-to-date `main` whose CI is
|
|
531
|
+
green:
|
|
532
|
+
|
|
533
|
+
```sh
|
|
534
|
+
bin/release minor --dry-run # the notes and the changes, nothing written
|
|
535
|
+
bin/release minor # or patch, major, or an exact version such as 1.2.0
|
|
536
|
+
```
|
|
537
|
+
|
|
538
|
+
`main` takes no direct push, so `bin/release` dates the Unreleased section and bumps the version on a branch
|
|
539
|
+
`release/vX.Y.Z`, opens its pull request, which merges itself once CI is green, and waits for it. It then pushes a
|
|
540
|
+
signed tag `vX.Y.Z` on the merge commit (`bin/release --tag X.Y.Z` does it if you stopped waiting). The tag starts
|
|
541
|
+
the Release workflow, which publishes the gem to RubyGems through trusted publishing and creates the GitHub release
|
|
542
|
+
with the version's notes and the `.gem`.
|
|
543
|
+
|
|
544
|
+
## License
|
|
545
|
+
|
|
546
|
+
MIT, by Level UP Solutions, see [LICENSE.md](LICENSE.md). Originally written by Nando Vieira as
|
|
547
|
+
[fnando/browser](https://github.com/fnando/browser) (MIT). The corpus gathers user agents from
|
|
548
|
+
[ua-parser/uap-core](https://github.com/ua-parser/uap-core) (Apache 2.0),
|
|
549
|
+
[crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT) and browser's test fixtures; the bot
|
|
550
|
+
list and its categories come from [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) (MIT)
|
|
551
|
+
and [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) (MIT); the crawler IP ranges are those their
|
|
552
|
+
owners publish for that purpose: see [corpus/SOURCES.md](corpus/SOURCES.md).
|