datacenter-ip-ranges 0.0.0-stage → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +432 -2
- package/bin/check.js +98 -0
- package/data/SOURCES.md +36 -0
- package/data/all.txt +12322 -0
- package/data/categories/cdn.txt +46 -0
- package/data/categories/cloud.txt +6080 -0
- package/data/categories/crawler.txt +1303 -0
- package/data/categories/privacy-relay.txt +13994 -0
- package/data/categories/services.txt +6967 -0
- package/data/categories/tor.txt +697 -0
- package/data/firewall/datacenter.ipset +12327 -0
- package/data/firewall/datacenter.nft +12340 -0
- package/data/firewall/tor.ipset +702 -0
- package/data/firewall/tor.nft +713 -0
- package/data/ipv4.txt +7272 -0
- package/data/ipv6.txt +5050 -0
- package/data/providers/ahrefsbot.txt +81 -0
- package/data/providers/applebot.txt +12 -0
- package/data/providers/aws.txt +4114 -0
- package/data/providers/bingbot.txt +29 -0
- package/data/providers/ccbot.txt +5 -0
- package/data/providers/chatgpt-user.txt +225 -0
- package/data/providers/cloudflare.txt +22 -0
- package/data/providers/digitalocean.txt +241 -0
- package/data/providers/duckduckbot.txt +484 -0
- package/data/providers/fastly.txt +24 -0
- package/data/providers/github.txt +6572 -0
- package/data/providers/google-cloud.txt +495 -0
- package/data/providers/google-services.txt +395 -0
- package/data/providers/google-special-crawlers.txt +36 -0
- package/data/providers/google-user-fetchers-google.txt +87 -0
- package/data/providers/google-user-fetchers.txt +233 -0
- package/data/providers/googlebot.txt +59 -0
- package/data/providers/gptbot.txt +15 -0
- package/data/providers/icloud-private-relay.txt +13994 -0
- package/data/providers/linode.txt +279 -0
- package/data/providers/mistralai-user.txt +4 -0
- package/data/providers/oai-searchbot.txt +36 -0
- package/data/providers/oracle-cloud.txt +809 -0
- package/data/providers/perplexity-user.txt +3 -0
- package/data/providers/perplexitybot.txt +6 -0
- package/data/providers/tor.txt +697 -0
- package/data/providers/vultr.txt +142 -0
- package/data/ranges.csv +29100 -0
- package/data/ranges.json +29101 -0
- package/data/sources.json +379 -0
- package/package.json +61 -4
- package/php/DatacenterIp.php +320 -0
- package/src/bots.js +53 -0
- package/src/build.js +321 -0
- package/src/index.d.ts +55 -0
- package/src/index.js +81 -0
- package/src/ip.js +215 -0
- package/src/matcher.js +146 -0
- package/src/sources.js +221 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Moonetix
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,433 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Datacenter IP ranges
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**Cloud, hosting and CDN IP ranges from the providers' own feeds, rebuilt every day, plus the ranges that search engines and AI companies publish for their crawlers. Lookup and crawler verification helpers for Node.js, PHP and Python, and ready-made files for nginx, nftables and ipset.**
|
|
4
|
+
|
|
5
|
+
[](https://github.com/moonetix/datacenter-ip-ranges/actions/workflows/ci.yml)
|
|
6
|
+
[](https://github.com/moonetix/datacenter-ip-ranges/actions/workflows/update.yml)
|
|
7
|
+
[](LICENSE)
|
|
8
|
+
[](https://www.npmjs.com/package/datacenter-ip-ranges)
|
|
9
|
+
[](https://packagist.org/packages/moonetix/datacenter-ip-ranges)
|
|
10
|
+
[](#nodejs)
|
|
11
|
+
[](#php)
|
|
12
|
+
[](#python)
|
|
13
|
+
[](package.json)
|
|
14
|
+
[](https://moonetix.com)
|
|
15
|
+
[](https://moonetix.github.io/datacenter-ip-ranges/)
|
|
16
|
+
|
|
17
|
+
Maintained by [Moonetix](https://moonetix.com).
|
|
18
|
+
|
|
19
|
+
A sign-up from an AWS address, an offer click from a DigitalOcean droplet, a "user" on a GitHub Actions runner: real people rarely browse from a datacenter. This repo collects the address ranges that AWS, Google Cloud, Oracle Cloud, DigitalOcean, Linode, Vultr, Cloudflare, Fastly and GitHub publish about themselves, merges them into plain CIDR lists, and keeps Tor exits and iCloud Private Relay in separate lists so you can treat them differently.
|
|
20
|
+
|
|
21
|
+
The crawler ranges that Google, Bing, Apple, DuckDuckGo, OpenAI, Perplexity, Mistral AI, Common Crawl and Ahrefs publish are a separate list too, with `verifyBot()` to tell a real Googlebot or GPTBot from a scraper that only copies its user agent.
|
|
22
|
+
|
|
23
|
+
Every range comes from a feed the provider documents. Nothing is scraped from third-party lists, nothing is guessed from ASN names.
|
|
24
|
+
|
|
25
|
+
```sh
|
|
26
|
+
git clone --depth 1 https://github.com/moonetix/datacenter-ip-ranges
|
|
27
|
+
node datacenter-ip-ranges/bin/check.js 3.5.140.1 8.8.8.8 192.168.1.10
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
```text
|
|
31
|
+
3.5.140.1 aws cloud 3.5.128.0/19
|
|
32
|
+
8.8.8.8 google-services services 8.8.8.0/24
|
|
33
|
+
192.168.1.10 not listed
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
To try an address without installing anything, open the [live demo](https://moonetix.github.io/datacenter-ip-ranges/). It runs the same matcher in your browser on the current lists.
|
|
37
|
+
|
|
38
|
+
Only need the list? `curl -fsSL https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/all.txt` gives one CIDR per line.
|
|
39
|
+
|
|
40
|
+
## Why check for datacenter addresses
|
|
41
|
+
|
|
42
|
+
Most people browse from home, mobile or office networks. Traffic from rented servers is usually automated, so a datacenter address is a useful signal wherever automation costs you money:
|
|
43
|
+
|
|
44
|
+
- **Sign-ups and free trials.** One server can create hundreds of accounts and claim a bonus or trial with each.
|
|
45
|
+
- **Paid actions.** Ad networks, affiliate programs and offerwalls reject or reverse conversions from hosting networks, so a reward paid for such traffic is often money you never get back.
|
|
46
|
+
- **Clicks, votes and scraping.** Headless browsers on cloud instances inflate counts and load your servers.
|
|
47
|
+
- **Fake crawlers.** Scrapers send a Googlebot or GPTBot user agent to get past rate limits and robots rules. The crawler lists show which of those requests really come from the crawler's operator.
|
|
48
|
+
- **Cheap to check.** A lookup here takes well under a microsecond in Node.js, so it can run on every request.
|
|
49
|
+
|
|
50
|
+
A match is a signal, not a verdict. See [Limitations](#limitations).
|
|
51
|
+
|
|
52
|
+
## The lists
|
|
53
|
+
|
|
54
|
+
| File | Contents |
|
|
55
|
+
| --- | --- |
|
|
56
|
+
| [`data/all.txt`](data/all.txt) | Every datacenter range: categories `cloud`, `cdn`, `services`. IPv4 first, then IPv6. One CIDR per line. |
|
|
57
|
+
| [`data/ipv4.txt`](data/ipv4.txt), [`data/ipv6.txt`](data/ipv6.txt) | The same, split by family. |
|
|
58
|
+
| [`data/providers/<id>.txt`](data/providers) | One provider, for example `aws.txt`, `cloudflare.txt`, `tor.txt`. |
|
|
59
|
+
| [`data/categories/<category>.txt`](data/categories) | One category: `cloud`, `cdn`, `services`, `privacy-relay`, `tor`, `crawler`. |
|
|
60
|
+
| [`data/firewall/`](data/firewall) | `datacenter.nft`, `datacenter.ipset`, `tor.nft`, `tor.ipset`: the datacenter and Tor lists as nftables sets and ipset restore files. |
|
|
61
|
+
| [`data/ranges.json`](data/ranges.json), [`data/ranges.csv`](data/ranges.csv) | Every provider range with `provider`, `category`, `cidr` and `source` URL. All categories. |
|
|
62
|
+
| [`data/sources.json`](data/sources.json), [`data/SOURCES.md`](data/SOURCES.md) | Per feed: status, range counts, last fetch time, feed URL, terms. |
|
|
63
|
+
|
|
64
|
+
`all.txt`, `ipv4.txt` and `ipv6.txt` do **not** contain Tor exits, iCloud Private Relay or the crawler lists. Tor and Private Relay carry real users and deserve a different policy, and a crawler is something you may want to let in. Use `data/categories/tor.txt`, `data/categories/privacy-relay.txt` and `data/categories/crawler.txt` for them.
|
|
65
|
+
|
|
66
|
+
Ranges are normalized: host bits cleared, IPv6 in RFC 5952 form, overlapping and adjacent blocks merged, private and documentation ranges removed. A merged block can be larger than any single block in the provider's feed; the set of addresses is the same.
|
|
67
|
+
|
|
68
|
+
### Categories
|
|
69
|
+
|
|
70
|
+
| Category | Providers | Typical traffic |
|
|
71
|
+
| --- | --- | --- |
|
|
72
|
+
| `cloud` | AWS, Google Cloud, Oracle Cloud, DigitalOcean, Akamai Cloud (Linode), Vultr | VPS and cloud instances: bots, scrapers, self-hosted proxies and VPNs |
|
|
73
|
+
| `cdn` | Cloudflare, Fastly | Edge networks: requests relayed or sent by code running on the CDN. Cloudflare WARP exits from addresses outside the published ranges, so WARP users are not matched. |
|
|
74
|
+
| `services` | Google services (goog.json minus cloud.json), GitHub (incl. Actions runners) | Crawlers, fetchers, CI runners, automation |
|
|
75
|
+
| `privacy-relay` | iCloud Private Relay | Safari users on iCloud+. Real people, shared addresses. |
|
|
76
|
+
| `tor` | Tor exit nodes | Tor Browser users and automated tools on Tor |
|
|
77
|
+
| `crawler` | Googlebot and the other Google crawlers and fetchers, Bingbot, Applebot, DuckDuckBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, MistralAI-User, CCBot, AhrefsBot | Search engine, AI and SEO crawlers, and fetches made on behalf of a chat user. Some of these addresses also sit in the `services` or cloud lists. See [Verify crawlers](#verify-crawlers). |
|
|
78
|
+
|
|
79
|
+
## Raw files
|
|
80
|
+
|
|
81
|
+
Use the files straight from GitHub. They change only when a provider changes its ranges.
|
|
82
|
+
|
|
83
|
+
```text
|
|
84
|
+
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/all.txt
|
|
85
|
+
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/ipv4.txt
|
|
86
|
+
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/ipv6.txt
|
|
87
|
+
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/providers/aws.txt
|
|
88
|
+
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/categories/tor.txt
|
|
89
|
+
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/ranges.json
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Download once a day, not on every request:
|
|
93
|
+
|
|
94
|
+
```sh
|
|
95
|
+
curl -fsSL -o /etc/dcip/all.txt \
|
|
96
|
+
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/all.txt
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
nginx can tag requests with the `geo` module. Rebuild the map after each download:
|
|
100
|
+
|
|
101
|
+
```sh
|
|
102
|
+
sed 's/$/ 1;/' /etc/dcip/all.txt > /etc/nginx/datacenter.map && nginx -s reload
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
```nginx
|
|
106
|
+
geo $remote_addr $is_datacenter {
|
|
107
|
+
default 0;
|
|
108
|
+
include /etc/nginx/datacenter.map;
|
|
109
|
+
}
|
|
110
|
+
# Pass it to your app, for example: proxy_set_header X-Datacenter $is_datacenter;
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
## Firewalls
|
|
114
|
+
|
|
115
|
+
`data/firewall/` has the datacenter list (the same ranges as `all.txt`) and the Tor exits in two firewall formats. They are rebuilt with the lists, so a daily download keeps them current.
|
|
116
|
+
|
|
117
|
+
nftables: each file defines an IPv4 and an IPv6 interval set to include inside a table.
|
|
118
|
+
|
|
119
|
+
```sh
|
|
120
|
+
curl -fsSL -o /etc/nftables.d/datacenter.nft \
|
|
121
|
+
https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/firewall/datacenter.nft
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
```nft
|
|
125
|
+
table inet filter {
|
|
126
|
+
include "/etc/nftables.d/datacenter.nft"
|
|
127
|
+
chain input {
|
|
128
|
+
type filter hook input priority 0;
|
|
129
|
+
tcp dport 443 ip saddr @datacenter_v4 counter
|
|
130
|
+
tcp dport 443 ip6 saddr @datacenter_v6 counter
|
|
131
|
+
}
|
|
132
|
+
}
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
ipset: each file creates `<name>-v4` and `<name>-v6` hash:net sets and fills them.
|
|
136
|
+
|
|
137
|
+
```sh
|
|
138
|
+
curl -fsSL https://raw.githubusercontent.com/moonetix/datacenter-ip-ranges/main/data/firewall/tor.ipset | ipset restore -exist
|
|
139
|
+
iptables -I INPUT -p tcp --dport 443 -m set --match-set tor-v4 src -j DROP
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
Count before you drop. A firewall rule cannot tell a scraper from a person on a cloud desktop, and it also blocks webhooks and postbacks your providers send from their servers.
|
|
143
|
+
|
|
144
|
+
## Node.js
|
|
145
|
+
|
|
146
|
+
Node 22 or newer, no dependencies, TypeScript declarations included.
|
|
147
|
+
|
|
148
|
+
```sh
|
|
149
|
+
npm install datacenter-ip-ranges
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
The package contains the lists as they were at its release. To follow the daily lists, download the raw files and pass `dataDir` (see below), or install from GitHub with `npm install github:moonetix/datacenter-ip-ranges`.
|
|
153
|
+
|
|
154
|
+
```js
|
|
155
|
+
const { isDatacenterIp, lookup } = require('datacenter-ip-ranges');
|
|
156
|
+
|
|
157
|
+
isDatacenterIp('3.5.140.1'); // true (AWS)
|
|
158
|
+
isDatacenterIp('::ffff:104.16.0.1'); // true (Cloudflare, IPv4-mapped form from a dual-stack socket)
|
|
159
|
+
isDatacenterIp('192.168.1.10'); // false
|
|
160
|
+
isDatacenterIp('not an ip'); // false
|
|
161
|
+
|
|
162
|
+
lookup('172.224.226.1'); // [ { provider: 'icloud-private-relay', category: 'privacy-relay' } ]
|
|
163
|
+
lookup('8.8.8.8'); // [ { provider: 'google-services', category: 'services' } ]
|
|
164
|
+
lookup('198.51.100.7'); // []
|
|
165
|
+
lookup('nope'); // null
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
`isDatacenterIp()` checks `cloud`, `cdn` and `services`. `lookup()` checks every category and returns all lists that contain the address; Private Relay egress often sits inside Cloudflare or Akamai space, so one address can match more than one list.
|
|
169
|
+
|
|
170
|
+
For a custom set, build your own matcher. It loads in about 100 ms and is meant to be built once per process:
|
|
171
|
+
|
|
172
|
+
```js
|
|
173
|
+
const { load } = require('datacenter-ip-ranges');
|
|
174
|
+
|
|
175
|
+
const cloudOnly = load({ categories: ['cloud'] });
|
|
176
|
+
const torOrRelay = load({ categories: ['tor', 'privacy-relay'] });
|
|
177
|
+
const awsOnly = load({ providers: ['aws'] });
|
|
178
|
+
const fromMyCopy = load({ dataDir: '/etc/dcip' }); // a folder with sources.json, providers/
|
|
179
|
+
|
|
180
|
+
cloudOnly.contains(req.socket.remoteAddress);
|
|
181
|
+
torOrRelay.lookup('185.220.101.1');
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
The matcher cuts all ranges into disjoint segments, each with a bit mask of the lists that cover it, and answers with one binary search. Plain IPv4 strings skip the general parser. On a small VPS, one million IPv4 lookups take about 160 ms; IPv6 lookups use `BigInt` and take a few microseconds each.
|
|
185
|
+
|
|
186
|
+
`createMatcher(lists)` from `datacenter-ip-ranges/matcher` builds a matcher from your own CIDR lists, if you want the same engine for an allow list or your own blocklist.
|
|
187
|
+
|
|
188
|
+
## PHP
|
|
189
|
+
|
|
190
|
+
PHP 8.1 or newer, one file, no extensions beyond the standard ones.
|
|
191
|
+
|
|
192
|
+
```sh
|
|
193
|
+
composer require moonetix/datacenter-ip-ranges
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
Composer autoloads the file. Without Composer, clone the repo (or copy `php/` and `data/` side by side) and include it:
|
|
197
|
+
|
|
198
|
+
```php
|
|
199
|
+
require __DIR__ . '/datacenter-ip-ranges/php/DatacenterIp.php';
|
|
200
|
+
|
|
201
|
+
if (is_datacenter_ip($_SERVER['REMOTE_ADDR'])) {
|
|
202
|
+
// Hold the reward for review, ask for verification, or skip the bonus.
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
// Other categories or your own copy of the data:
|
|
206
|
+
$torOrRelay = DatacenterIp::fromDataDir('/etc/dcip', ['tor', 'privacy-relay']);
|
|
207
|
+
$torOrRelay->contains('185.220.101.1');
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
`is_datacenter_ip()` loads the lists once per process. Under PHP-FPM that is once per request, about 25 ms. For busy sites, export once after each download and let OPcache keep the arrays in shared memory:
|
|
211
|
+
|
|
212
|
+
```php
|
|
213
|
+
// After downloading new lists (cron):
|
|
214
|
+
DatacenterIp::fromDataDir('/etc/dcip')->exportPhp('/etc/dcip/datacenter.php');
|
|
215
|
+
|
|
216
|
+
// In the request:
|
|
217
|
+
$dc = DatacenterIp::fromPhp('/etc/dcip/datacenter.php');
|
|
218
|
+
$dc->contains($_SERVER['REMOTE_ADDR']);
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
## Python
|
|
222
|
+
|
|
223
|
+
Python 3.8 or newer, standard library only, one file. Copy [`python/datacenter_ip.py`](python/datacenter_ip.py) next to a copy of `data/`, or pass `data_dir`:
|
|
224
|
+
|
|
225
|
+
```python
|
|
226
|
+
from datacenter_ip import is_datacenter_ip, lookup, verify_bot, load
|
|
227
|
+
|
|
228
|
+
is_datacenter_ip("3.5.140.1") # True (AWS)
|
|
229
|
+
lookup("185.220.101.1") # [{'provider': 'tor', 'category': 'tor'}]
|
|
230
|
+
|
|
231
|
+
tor_or_relay = load("/etc/dcip", categories=("tor", "privacy-relay"))
|
|
232
|
+
tor_or_relay.contains("185.220.101.1")
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
The functions load the lists once per process. From the command line, `python3 python/datacenter_ip.py 3.5.140.1` prints every list that contains the address, with the same exit codes as `bin/check.js`.
|
|
236
|
+
|
|
237
|
+
## Verify crawlers
|
|
238
|
+
|
|
239
|
+
Anyone can put `Googlebot` in a user agent. The operators below publish the addresses their crawlers use, and `verifyBot()` checks a request against them:
|
|
240
|
+
|
|
241
|
+
```js
|
|
242
|
+
const { verifyBot } = require('datacenter-ip-ranges');
|
|
243
|
+
|
|
244
|
+
verifyBot(req.headers['user-agent'], clientIp);
|
|
245
|
+
// null no known crawler claimed
|
|
246
|
+
// { bot: 'google', name: 'Google', verified: true, provider: 'googlebot' }
|
|
247
|
+
// { bot: 'gptbot', name: 'OpenAI GPTBot', verified: false, provider: null } the claim is false
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
```php
|
|
251
|
+
$check = verify_bot($_SERVER['HTTP_USER_AGENT'] ?? '', $clientIp);
|
|
252
|
+
if ($check !== null && !$check['verified']) {
|
|
253
|
+
// Claims to be a crawler, is not one: treat it like any other scraper.
|
|
254
|
+
}
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
```python
|
|
258
|
+
verify_bot(user_agent, client_ip) # same result as a dict
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
`verified: true` means the address is in a list the claimed crawler's operator publishes. A typical policy lets verified crawlers past rate limits and treats false claims like any other scraper. Whether you want a given crawler at all is a separate decision for robots.txt or your own rules.
|
|
262
|
+
|
|
263
|
+
| Bot | User-agent tokens | Lists |
|
|
264
|
+
| --- | --- | --- |
|
|
265
|
+
| Google | `Googlebot`, `Storebot-Google`, `Google-InspectionTool`, `GoogleOther`, `Google-CloudVertexBot`, `AdsBot-Google`, `Mediapartners-Google`, `APIs-Google`, `FeedFetcher-Google`, `Google-Read-Aloud`, `Google-Site-Verification`, `GoogleProducer`, `Google-Safety` | `googlebot`, `google-special-crawlers`, `google-user-fetchers`, `google-user-fetchers-google` |
|
|
266
|
+
| Bing | `bingbot`, `adidxbot`, `BingPreview`, `MicrosoftPreview` | `bingbot` |
|
|
267
|
+
| Applebot | `Applebot` | `applebot` |
|
|
268
|
+
| DuckDuckBot | `DuckDuckBot` | `duckduckbot` |
|
|
269
|
+
| OpenAI GPTBot | `GPTBot` | `gptbot` |
|
|
270
|
+
| OpenAI OAI-SearchBot | `OAI-SearchBot` | `oai-searchbot` |
|
|
271
|
+
| OpenAI ChatGPT-User | `ChatGPT-User` | `chatgpt-user` |
|
|
272
|
+
| PerplexityBot | `PerplexityBot` | `perplexitybot` |
|
|
273
|
+
| Perplexity-User | `Perplexity-User` | `perplexity-user` |
|
|
274
|
+
| MistralAI-User | `MistralAI-User` | `mistralai-user` |
|
|
275
|
+
| Common Crawl CCBot | `CCBot` | `ccbot` |
|
|
276
|
+
| Ahrefs | `AhrefsBot`, `AhrefsSiteAudit` | `ahrefsbot` |
|
|
277
|
+
|
|
278
|
+
Tokens match case-insensitively anywhere in the user agent. `BOTS` (`DatacenterIp::BOTS`, `datacenter_ip.BOTS`) exports the table, and `claimedBot()` returns the entry a user agent claims without checking the address. Crawlers whose operators publish no address list, such as YandexBot or Baiduspider, are not covered; their operators document reverse DNS checks instead.
|
|
279
|
+
|
|
280
|
+
## Command line
|
|
281
|
+
|
|
282
|
+
```sh
|
|
283
|
+
node bin/check.js 3.5.140.1 8.8.8.8 192.168.1.10 172.224.226.1
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
```text
|
|
287
|
+
3.5.140.1 aws cloud 3.5.128.0/19
|
|
288
|
+
8.8.8.8 google-services services 8.8.8.0/24
|
|
289
|
+
192.168.1.10 not listed
|
|
290
|
+
172.224.226.1 icloud-private-relay privacy-relay 172.224.226.0/26
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
Options: `--json` for JSON output, `--only cloud,cdn` to limit categories, `--data <dir>` for another copy of the data, `-` to read addresses from stdin. Exit codes follow `grep`: 0 if any address is listed, 1 if none is, 2 for invalid input. Installed from GitHub, the same command is available as `npx datacenter-ip`.
|
|
294
|
+
|
|
295
|
+
```sh
|
|
296
|
+
# Which of yesterday's sign-up IPs came from a datacenter?
|
|
297
|
+
cut -d, -f3 signups.csv | node bin/check.js --only cloud,cdn,services - | grep -v 'not listed'
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
PHP has the same check: `php php/check.php 3.5.140.1`.
|
|
301
|
+
|
|
302
|
+
## Behind a proxy or CDN
|
|
303
|
+
|
|
304
|
+
Check the address of the visitor, not of your own infrastructure. If your site runs behind Cloudflare, a load balancer or a reverse proxy, the TCP peer is that proxy, and every request matches. Read the client address from the header your proxy sets (`CF-Connecting-IP`, `X-Forwarded-For`, `X-Real-IP`), and trust that header only when the request actually came from your proxy. The `cdn` lists help with the second part:
|
|
305
|
+
|
|
306
|
+
```js
|
|
307
|
+
const cdn = load({ categories: ['cdn'] });
|
|
308
|
+
const peer = req.socket.remoteAddress;
|
|
309
|
+
const client = cdn.contains(peer) ? req.headers['cf-connecting-ip'] : peer;
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
Do not apply these lists to your postback endpoint. Networks deliver postbacks from their servers, which are datacenter addresses by design. Verify the postback signature instead.
|
|
313
|
+
|
|
314
|
+
## Sources and terms
|
|
315
|
+
|
|
316
|
+
Each feed is published by the provider for allow lists, firewalls or geolocation. None of them states a license that forbids redistribution, except Microsoft's Download Center terms, which is why Azure is not in the committed data (see below). Check the terms yourself before you build a commercial product on them.
|
|
317
|
+
|
|
318
|
+
| Provider | Category | Feed | Documentation | Terms |
|
|
319
|
+
| --- | --- | --- | --- | --- |
|
|
320
|
+
| Amazon Web Services | cloud | [ip-ranges.json](https://ip-ranges.amazonaws.com/ip-ranges.json) | [AWS IP address ranges](https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html) | Published for filtering and allow lists; no license stated |
|
|
321
|
+
| Google Cloud | cloud | [cloud.json](https://www.gstatic.com/ipranges/cloud.json) | [Google IP address ranges](https://support.google.com/a/answer/10026322) | Published for allow lists; no license stated |
|
|
322
|
+
| Google services | services | [goog.json](https://www.gstatic.com/ipranges/goog.json) minus cloud.json | [Google IP address ranges](https://support.google.com/a/answer/10026322) | Published for allow lists; no license stated |
|
|
323
|
+
| Oracle Cloud | cloud | [public_ip_ranges.json](https://docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json) | [OCI IP address ranges](https://docs.oracle.com/en-us/iaas/Content/General/Concepts/addressranges.htm) | Oracle documentation; no license stated |
|
|
324
|
+
| DigitalOcean | cloud | [geo/google.csv](https://www.digitalocean.com/geo/google.csv) | Linked from [DigitalOcean platform docs](https://docs.digitalocean.com/platform/) | RFC 8805 geofeed; no license stated |
|
|
325
|
+
| Akamai Cloud (Linode) | cloud | [linode-geofeed.csv](https://ipgeo.akamai.com/linode-geofeed.csv) | Declared in the [ARIN record](https://rdap.arin.net/registry/ip/172.104.0.1) | RFC 8805 geofeed; no license stated |
|
|
326
|
+
| Vultr | cloud | [geofeed.constant.com](https://geofeed.constant.com/) | Declared in the [ARIN record](https://rdap.arin.net/registry/ip/45.32.0.1) | RFC 8805 geofeed of AS20473; no license stated |
|
|
327
|
+
| Cloudflare | cdn | [ips-v4](https://www.cloudflare.com/ips-v4), [ips-v6](https://www.cloudflare.com/ips-v6) | [Cloudflare IP ranges](https://www.cloudflare.com/ips/) | Published for allow lists; no license stated |
|
|
328
|
+
| Fastly | cdn | [public-ip-list](https://api.fastly.com/public-ip-list) | [Fastly API reference](https://www.fastly.com/documentation/reference/api/utils/public-ip-list/) | Public API endpoint; no license stated |
|
|
329
|
+
| GitHub | services | [api.github.com/meta](https://api.github.com/meta) | [Meta API](https://docs.github.com/en/rest/meta/meta), [GitHub's IP addresses](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-githubs-ip-addresses) | [GitHub Terms of Service](https://docs.github.com/en/site-policy/github-terms/github-terms-of-service), API terms |
|
|
330
|
+
| iCloud Private Relay | privacy-relay | [egress-ip-ranges.csv](https://mask-api.icloud.com/egress-ip-ranges.csv) | [Prepare your network for iCloud Private Relay](https://developer.apple.com/support/prepare-your-network-for-icloud-private-relay/) | Published for server operators and geolocation; no license stated |
|
|
331
|
+
| Tor Project | tor | [torbulkexitlist](https://check.torproject.org/torbulkexitlist) | [Tor Metrics](https://metrics.torproject.org/about.html) | Tor Project data is CC0 |
|
|
332
|
+
| Googlebot and other Google common crawlers | crawler | [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json) | [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) | Published for crawler verification; no license stated |
|
|
333
|
+
| Google special-case crawlers (AdsBot, Mediapartners) | crawler | [special-crawlers.json](https://developers.google.com/static/crawling/ipranges/special-crawlers.json) | [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) | Published for crawler verification; no license stated |
|
|
334
|
+
| Google user-triggered fetchers | crawler | [user-triggered-fetchers.json](https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json) | [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) | Published for crawler verification; no license stated |
|
|
335
|
+
| Google user-triggered fetchers (Google products) | crawler | [user-triggered-fetchers-google.json](https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json) | [Documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) | Published for crawler verification; no license stated |
|
|
336
|
+
| Bingbot | crawler | [bingbot.json](https://www.bing.com/toolbox/bingbot.json) | [Documentation](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26) | Published for crawler verification; no license stated |
|
|
337
|
+
| Applebot | crawler | [applebot.json](https://search.developer.apple.com/applebot.json) | [Documentation](https://support.apple.com/en-us/119829) | Published for crawler verification; no license stated |
|
|
338
|
+
| DuckDuckBot | crawler | [duckduckbot.json](https://duckduckgo.com/duckduckbot.json) | [Documentation](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot) | Published for crawler verification; no license stated |
|
|
339
|
+
| OpenAI GPTBot | crawler | [gptbot.json](https://openai.com/gptbot.json) | [Documentation](https://developers.openai.com/api/docs/bots) | Published for crawler verification; no license stated |
|
|
340
|
+
| OpenAI OAI-SearchBot | crawler | [searchbot.json](https://openai.com/searchbot.json) | [Documentation](https://developers.openai.com/api/docs/bots) | Published for crawler verification; no license stated |
|
|
341
|
+
| OpenAI ChatGPT-User | crawler | [chatgpt-user.json](https://openai.com/chatgpt-user.json) | [Documentation](https://developers.openai.com/api/docs/bots) | Published for crawler verification; no license stated |
|
|
342
|
+
| PerplexityBot | crawler | [perplexitybot.json](https://www.perplexity.ai/perplexitybot.json) | [Documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) | Published for crawler verification; no license stated |
|
|
343
|
+
| Perplexity-User | crawler | [perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) | [Documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) | Published for crawler verification; no license stated |
|
|
344
|
+
| MistralAI-User | crawler | [mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json) | [Documentation](https://docs.mistral.ai/robots) | Published for crawler verification; no license stated |
|
|
345
|
+
| Common Crawl CCBot | crawler | [ccbot.json](https://index.commoncrawl.org/ccbot.json) | [Documentation](https://commoncrawl.org/ccbot) | Published for crawler verification; no license stated |
|
|
346
|
+
| AhrefsBot and AhrefsSiteAudit | crawler | [crawler-ip-ranges](https://api.ahrefs.com/v3/public/crawler-ip-ranges) | [Documentation](https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges) | Published for crawler verification; no license stated |
|
|
347
|
+
| Microsoft Azure | cloud | [Service Tags download page](https://www.microsoft.com/en-us/download/details.aspx?id=56519) | [Azure service tags](https://learn.microsoft.com/en-us/azure/virtual-network/service-tags-overview) | Not redistributed, see below |
|
|
348
|
+
|
|
349
|
+
**Microsoft Azure.** The weekly Service Tags file comes from the Microsoft Download Center, and the [Microsoft Terms of Use](https://www.microsoft.com/en-us/legal/terms-of-use) restrict reposting its downloads. The script supports it, but only for your own copy:
|
|
350
|
+
|
|
351
|
+
```sh
|
|
352
|
+
node scripts/update.js --include azure --out /etc/dcip
|
|
353
|
+
```
|
|
354
|
+
|
|
355
|
+
That reads the current file name from the download page and adds the `AzureCloud` tag as `providers/azure.txt` and to `all.txt`. `data/providers/azure.txt` is in `.gitignore`. Without it, Azure coverage is partial: GitHub's Actions runner ranges, which are Azure addresses, are included through the GitHub feed.
|
|
356
|
+
|
|
357
|
+
Providers without an official feed are not included. Hetzner, OVHcloud, Scaleway, Contabo, Alibaba Cloud and many smaller hosts do not publish one, and this repo does not fill the gap with ASN or WHOIS guesses. If your provider publishes a feed, open an issue with the link to its documentation.
|
|
358
|
+
|
|
359
|
+
## Updates
|
|
360
|
+
|
|
361
|
+
[`.github/workflows/update.yml`](.github/workflows/update.yml) runs every day at 03:17 UTC and on demand. It fetches all feeds, rebuilds `data/`, runs the tests, and commits only if a range list changed. A run where only the fetch time moved commits nothing, so the history shows real changes only.
|
|
362
|
+
|
|
363
|
+
```mermaid
|
|
364
|
+
flowchart LR
|
|
365
|
+
F["12 provider<br/>feeds"] --> N["Parse, drop private<br/>and too-broad<br/>prefixes, merge"]
|
|
366
|
+
N --> G{"Failed, empty<br/>or shrank<br/>by half?"}
|
|
367
|
+
G -- no --> D["data/<br/>providers, categories,<br/>all.txt, ranges.json"]
|
|
368
|
+
G -- "yes: keep previous<br/>list, mark stale" --> D
|
|
369
|
+
```
|
|
370
|
+
|
|
371
|
+
Failures are handled per feed. Each request gets 60 seconds and three attempts. If a feed is down, returns nothing, or suddenly returns less than half of its previous ranges, its previous list is kept and the feed is marked `stale` in [`data/sources.json`](data/sources.json) with the reason. Nothing is written if every feed fails. Entries that do not parse, prefixes broader than /8 (IPv4) or /16 (IPv6), and private, loopback, link-local, multicast and documentation ranges are dropped.
|
|
372
|
+
|
|
373
|
+
A feed that is still stale on the next run, or a run that fails outright, opens a single issue labeled `automation` with the feed, the error and the source URL. The issue closes itself once every feed is fresh again.
|
|
374
|
+
|
|
375
|
+
To rebuild by hand:
|
|
376
|
+
|
|
377
|
+
```sh
|
|
378
|
+
node scripts/update.js # writes data/
|
|
379
|
+
node scripts/update.js --out /etc/dcip # writes somewhere else
|
|
380
|
+
GITHUB_TOKEN=... node scripts/update.js # optional: authenticated api.github.com/meta request
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
`--out` and `--include azure` are the only arguments. `GITHUB_TOKEN` is the only environment variable; it is sent to `api.github.com` and nowhere else, and only matters if your IP has used up the unauthenticated API limit.
|
|
384
|
+
|
|
385
|
+
## Troubleshooting
|
|
386
|
+
|
|
387
|
+
| What you see | Cause and fix |
|
|
388
|
+
| --- | --- |
|
|
389
|
+
| Every request matches | You are checking your proxy or CDN, not the visitor. See [Behind a proxy or CDN](#behind-a-proxy-or-cdn). |
|
|
390
|
+
| `invalid IP address` and exit code 2 from `bin/check.js` | The argument is not an IP. Ports (`1.2.3.4:443`) are not accepted; brackets and IPv6 zone IDs are. |
|
|
391
|
+
| `check.js` prints its usage and exits 2 | `--only` got a category that does not exist. Valid: `cloud`, `cdn`, `services`, `privacy-relay`, `tor`. |
|
|
392
|
+
| A feed shows `"status": "stale"` with `fetch or parse failed: ... HTTP 403` | For `github`, the API rate limit. Set `GITHUB_TOKEN`. Other feeds recover on the next run; the previous list stays in place. |
|
|
393
|
+
| `Every feed failed. Nothing was written.` | No outbound HTTPS from the machine, or a proxy blocks it. The message lists the reason per feed. |
|
|
394
|
+
| `no ServiceTags_Public link on the download page` | Microsoft changed the Azure download page. Azure is opt-in; run without `--include azure` and open an issue. |
|
|
395
|
+
|
|
396
|
+
## Limitations
|
|
397
|
+
|
|
398
|
+
- **Residential proxies and most commercial VPNs are not covered.** They route traffic through home and mobile connections or through hosts without public feeds. This list catches the cheap, server-based share of bad traffic. For the rest you need an IP intelligence service, device signals and behavior.
|
|
399
|
+
- **One signal, not a verdict.** People do use cloud desktops, corporate proxies and VPNs hosted on rented servers. Prefer soft actions: hold a reward for review, require a verified phone, skip a sign-up bonus. Block outright only where a human has no reason to come from a server, for example on an offer click or a reward claim.
|
|
400
|
+
- **Tor and Private Relay are real people.** Apple asks servers to treat Private Relay addresses like carrier-grade NAT: many users share one address. They are separate lists for that reason.
|
|
401
|
+
- **Freshness.** Lists are at most a day behind the feeds, and the feeds themselves lag new allocations. IP addresses also move between owners.
|
|
402
|
+
- **Crawler lists cover only operators that publish one.** A crawler that is not in the table is not verified either way, and an operator can start crawling from new addresses before its list catches up.
|
|
403
|
+
- **Provider feeds include their own infrastructure.** Google's services list contains crawlers and fetch proxies; GitHub's contains its web and API servers. That is usually what you want for fraud filtering, but not for every use.
|
|
404
|
+
|
|
405
|
+
## Further reading
|
|
406
|
+
|
|
407
|
+
- [OWASP Automated Threats to Web Applications](https://owasp.org/www-project-automated-threats-to-web-applications/): the abuse patterns this kind of check helps with.
|
|
408
|
+
- Moonetix Academy: [What is IP reputation?](https://moonetix.com/academy/glossary/what-is-ip-reputation) and [Ad fraud and bot traffic](https://moonetix.com/academy/strategy/ad-fraud-and-bot-traffic).
|
|
409
|
+
|
|
410
|
+
## Development
|
|
411
|
+
|
|
412
|
+
```sh
|
|
413
|
+
npm test # Node tests; also runs the PHP and Python tests and the cross-checks if php and python3 are installed
|
|
414
|
+
php php/test.php # PHP tests only
|
|
415
|
+
python3 python/test_datacenter_ip.py # Python tests only
|
|
416
|
+
npm run update # fetch all feeds and rebuild data/
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
The tests cover address parsing (IPv4-mapped, zone IDs, brackets, RFC 5952 output), range arithmetic, the matcher against a brute-force check on random data and range edges, the build step against fake feeds (merging, special ranges, stale feeds, the shrink guard, opt-in sources), the integrity of the committed data, the CLI, the firewall files, crawler verification, and agreement between the Node, PHP and Python helpers on boundary addresses of every list.
|
|
420
|
+
|
|
421
|
+
[CI](.github/workflows/ci.yml) runs the same commands on every push and pull request. To propose a new provider, use the **Add a provider feed** issue form.
|
|
422
|
+
|
|
423
|
+
Layout: `src/ip.js` parsing and range math, `src/matcher.js` lookup engine, `src/sources.js` feed definitions and parsers, `src/bots.js` the crawler table and `verifyBot()`, `src/build.js` fetch and write, `scripts/update.js` and `bin/check.js` command line entry points, `php/` the PHP helper, `python/` the Python helper.
|
|
424
|
+
|
|
425
|
+
## About Moonetix
|
|
426
|
+
|
|
427
|
+
[Moonetix](https://moonetix.com) is an affiliate network. Publishers add its offerwall to apps, games, Discord and Telegram communities and rewards sites, and users earn the publisher's own currency by completing offers. Every click and conversion goes through traffic checks like the ones described in [How Moonetix Checks Traffic Quality](https://moonetix.com/academy/traffic-quality/how-moonetix-checks-traffic).
|
|
428
|
+
|
|
429
|
+
Related open source: [offerwall-postback-handlers](https://github.com/moonetix/offerwall-postback-handlers), [postback-tester](https://github.com/moonetix/postback-tester), [awesome-offerwalls](https://github.com/moonetix/awesome-offerwalls), and more at [github.com/moonetix](https://github.com/moonetix).
|
|
430
|
+
|
|
431
|
+
## License
|
|
432
|
+
|
|
433
|
+
Code: MIT. See [LICENSE](LICENSE). The range data belongs to the providers listed in [Sources and terms](#sources-and-terms) and is subject to their terms.
|
package/bin/check.js
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
'use strict';
|
|
3
|
+
|
|
4
|
+
// Usage: node bin/check.js [--json] [--only categories] [--data dir] <ip> [ip ...]
|
|
5
|
+
// echo 1.2.3.4 | node bin/check.js -
|
|
6
|
+
//
|
|
7
|
+
// Exit codes, like grep: 0 at least one address is listed, 1 none is listed,
|
|
8
|
+
// 2 an argument is not a valid IP address.
|
|
9
|
+
|
|
10
|
+
const fs = require('node:fs');
|
|
11
|
+
const path = require('node:path');
|
|
12
|
+
const { load, readList, DEFAULT_DATA_DIR } = require('../src/index');
|
|
13
|
+
const { CATEGORIES } = require('../src/sources');
|
|
14
|
+
const { parseAddress, parseCidr } = require('../src/ip');
|
|
15
|
+
|
|
16
|
+
function usage(code = 2) {
|
|
17
|
+
(code === 0 ? process.stdout : process.stderr).write(
|
|
18
|
+
'Usage: check.js [--json] [--only categories] [--data dir] <ip> [ip ...]\n' +
|
|
19
|
+
' check.js - read addresses from stdin, one per line\n' +
|
|
20
|
+
' --only cloud,cdn check only these categories (' + Object.keys(CATEGORIES).join(', ') + ')\n' +
|
|
21
|
+
' --json print JSON\n' +
|
|
22
|
+
'Exit code 0 if any address is listed, 1 if none is, 2 on invalid input.\n',
|
|
23
|
+
);
|
|
24
|
+
process.exit(code);
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
function main() {
|
|
28
|
+
const argv = process.argv.slice(2);
|
|
29
|
+
let json = false;
|
|
30
|
+
let dataDir = DEFAULT_DATA_DIR;
|
|
31
|
+
let categories = Object.keys(CATEGORIES);
|
|
32
|
+
const ips = [];
|
|
33
|
+
for (let i = 0; i < argv.length; i++) {
|
|
34
|
+
const a = argv[i];
|
|
35
|
+
if (a === '--json') json = true;
|
|
36
|
+
else if (a === '--data') dataDir = path.resolve(argv[++i] || '');
|
|
37
|
+
else if (a === '--only') {
|
|
38
|
+
categories = String(argv[++i] || '').split(',').filter(Boolean);
|
|
39
|
+
if (!categories.length || categories.some((c) => !CATEGORIES[c])) usage();
|
|
40
|
+
} else if (a === '-h' || a === '--help') usage(0);
|
|
41
|
+
else if (a === '-') ips.push(...fs.readFileSync(0, 'utf8').split(/\s+/).filter(Boolean));
|
|
42
|
+
else ips.push(a);
|
|
43
|
+
}
|
|
44
|
+
if (ips.length === 0) usage();
|
|
45
|
+
|
|
46
|
+
const matcher = load({ dataDir, categories });
|
|
47
|
+
const lists = new Map();
|
|
48
|
+
// The matcher knows which providers match. For the report we also show the CIDR block.
|
|
49
|
+
function blockFor(provider, ip) {
|
|
50
|
+
if (!lists.has(provider)) {
|
|
51
|
+
const cidrs = readList(path.join(dataDir, 'providers', `${provider}.txt`));
|
|
52
|
+
lists.set(provider, cidrs.map((text) => ({ text, ...parseCidr(text) })));
|
|
53
|
+
}
|
|
54
|
+
const a = parseAddress(ip);
|
|
55
|
+
const hit = lists.get(provider).find((r) => r.family === a.family && a.value >= r.start && a.value <= r.end);
|
|
56
|
+
return hit ? hit.text : '';
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
let anyListed = false;
|
|
60
|
+
let anyInvalid = false;
|
|
61
|
+
const results = [];
|
|
62
|
+
for (const ip of ips) {
|
|
63
|
+
const matches = matcher.lookup(ip);
|
|
64
|
+
if (matches === null) {
|
|
65
|
+
anyInvalid = true;
|
|
66
|
+
results.push({ ip, valid: false, listed: false, matches: [] });
|
|
67
|
+
continue;
|
|
68
|
+
}
|
|
69
|
+
if (matches.length) anyListed = true;
|
|
70
|
+
results.push({
|
|
71
|
+
ip,
|
|
72
|
+
valid: true,
|
|
73
|
+
listed: matches.length > 0,
|
|
74
|
+
matches: matches.map((m) => ({ ...m, cidr: blockFor(m.provider, ip) })),
|
|
75
|
+
});
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
if (json) {
|
|
79
|
+
process.stdout.write(JSON.stringify(results.length === 1 ? results[0] : results, null, 2) + '\n');
|
|
80
|
+
} else {
|
|
81
|
+
const width = Math.max(...ips.map((s) => s.length)) + 2;
|
|
82
|
+
for (const r of results) {
|
|
83
|
+
if (!r.valid) {
|
|
84
|
+
process.stdout.write(`${r.ip.padEnd(width)}invalid IP address\n`);
|
|
85
|
+
} else if (!r.listed) {
|
|
86
|
+
process.stdout.write(`${r.ip.padEnd(width)}not listed\n`);
|
|
87
|
+
} else {
|
|
88
|
+
for (const [i, m] of r.matches.entries()) {
|
|
89
|
+
const label = i === 0 ? r.ip.padEnd(width) : ''.padEnd(width);
|
|
90
|
+
process.stdout.write(`${label}${m.provider.padEnd(22)}${m.category.padEnd(15)}${m.cidr}\n`);
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
process.exitCode = anyInvalid ? 2 : anyListed ? 0 : 1;
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
main();
|
package/data/SOURCES.md
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Sources
|
|
2
|
+
|
|
3
|
+
Last run 2026-11-17T08:51:50Z by `scripts/update.js`. This file is rewritten on every run.
|
|
4
|
+
|
|
5
|
+
Datacenter lists (`all.txt`, `ipv4.txt`, `ipv6.txt`): 7272 IPv4 and 5050 IPv6 CIDR blocks.
|
|
6
|
+
|
|
7
|
+
| Provider | Category | IPv4 | IPv6 | Status | Fetched (UTC) | Feed |
|
|
8
|
+
| --- | --- | ---: | ---: | --- | --- | --- |
|
|
9
|
+
| Amazon Web Services | cloud | 1751 | 2363 | ok | 2026-11-17T08:51:50Z | [ip-ranges.amazonaws.com/ip-ranges.json](https://ip-ranges.amazonaws.com/ip-ranges.json) |
|
|
10
|
+
| Google Cloud | cloud | 467 | 28 | ok | 2026-11-17T08:51:50Z | [www.gstatic.com/ipranges/cloud.json](https://www.gstatic.com/ipranges/cloud.json) |
|
|
11
|
+
| Google services (excluding Google Cloud) | services | 297 | 98 | ok | 2026-11-17T08:51:50Z | [www.gstatic.com/ipranges/goog.json](https://www.gstatic.com/ipranges/goog.json) |
|
|
12
|
+
| Oracle Cloud Infrastructure | cloud | 809 | 0 | ok | 2026-11-17T08:51:50Z | [docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json](https://docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json) |
|
|
13
|
+
| DigitalOcean | cloud | 187 | 54 | ok | 2026-11-17T08:51:50Z | [www.digitalocean.com/geo/google.csv](https://www.digitalocean.com/geo/google.csv) |
|
|
14
|
+
| Akamai Cloud (Linode) | cloud | 240 | 39 | ok | 2026-11-17T08:51:50Z | [ipgeo.akamai.com/linode-geofeed.csv](https://ipgeo.akamai.com/linode-geofeed.csv) |
|
|
15
|
+
| Vultr | cloud | 121 | 21 | ok | 2026-11-17T08:51:50Z | [geofeed.constant.com/](https://geofeed.constant.com/) |
|
|
16
|
+
| Microsoft Azure | cloud | - | - | not included (opt-in) | - | [www.microsoft.com/en-us/download/details.aspx](https://www.microsoft.com/en-us/download/details.aspx?id=56519) |
|
|
17
|
+
| Cloudflare | cdn | 15 | 7 | ok | 2026-11-17T08:51:50Z | [www.cloudflare.com/ips-v4](https://www.cloudflare.com/ips-v4), [www.cloudflare.com/ips-v6](https://www.cloudflare.com/ips-v6) |
|
|
18
|
+
| Fastly | cdn | 21 | 3 | ok | 2026-11-17T08:51:50Z | [api.fastly.com/public-ip-list](https://api.fastly.com/public-ip-list) |
|
|
19
|
+
| GitHub (incl. Actions runners) | services | 4025 | 2547 | ok | 2026-11-17T08:51:50Z | [api.github.com/meta](https://api.github.com/meta) |
|
|
20
|
+
| iCloud Private Relay egress | privacy-relay | 3414 | 10580 | ok | 2026-11-17T08:51:50Z | [mask-api.icloud.com/egress-ip-ranges.csv](https://mask-api.icloud.com/egress-ip-ranges.csv) |
|
|
21
|
+
| Tor exit nodes | tor | 697 | 0 | ok | 2026-11-17T08:51:50Z | [check.torproject.org/torbulkexitlist](https://check.torproject.org/torbulkexitlist) |
|
|
22
|
+
| Googlebot and other Google common crawlers | crawler | 37 | 22 | ok | 2026-11-17T08:51:50Z | [developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json) |
|
|
23
|
+
| Google special-case crawlers (AdsBot, Mediapartners) | crawler | 15 | 21 | ok | 2026-11-17T08:51:50Z | [developers.google.com/static/crawling/ipranges/special-crawlers.json](https://developers.google.com/static/crawling/ipranges/special-crawlers.json) |
|
|
24
|
+
| Google user-triggered fetchers | crawler | 105 | 128 | ok | 2026-11-17T08:51:50Z | [developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json](https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json) |
|
|
25
|
+
| Google user-triggered fetchers (Google products) | crawler | 39 | 48 | ok | 2026-11-17T08:51:50Z | [developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json](https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json) |
|
|
26
|
+
| Bingbot | crawler | 29 | 0 | ok | 2026-11-17T08:51:50Z | [www.bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json) |
|
|
27
|
+
| Applebot | crawler | 12 | 0 | ok | 2026-11-17T08:51:50Z | [search.developer.apple.com/applebot.json](https://search.developer.apple.com/applebot.json) |
|
|
28
|
+
| DuckDuckBot | crawler | 484 | 0 | ok | 2026-11-17T08:51:50Z | [duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json) |
|
|
29
|
+
| OpenAI GPTBot | crawler | 15 | 0 | ok | 2026-11-17T08:51:50Z | [openai.com/gptbot.json](https://openai.com/gptbot.json) |
|
|
30
|
+
| OpenAI OAI-SearchBot | crawler | 36 | 0 | ok | 2026-11-17T08:51:50Z | [openai.com/searchbot.json](https://openai.com/searchbot.json) |
|
|
31
|
+
| OpenAI ChatGPT-User | crawler | 225 | 0 | ok | 2026-11-17T08:51:50Z | [openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json) |
|
|
32
|
+
| PerplexityBot | crawler | 6 | 0 | ok | 2026-11-17T08:51:50Z | [www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json) |
|
|
33
|
+
| Perplexity-User | crawler | 3 | 0 | ok | 2026-11-17T08:51:50Z | [www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) |
|
|
34
|
+
| MistralAI-User | crawler | 4 | 0 | ok | 2026-11-17T08:51:50Z | [mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json) |
|
|
35
|
+
| Common Crawl CCBot | crawler | 4 | 1 | ok | 2026-11-17T08:51:50Z | [index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json) |
|
|
36
|
+
| AhrefsBot and AhrefsSiteAudit | crawler | 81 | 0 | ok | 2026-11-17T08:51:50Z | [api.ahrefs.com/v3/public/crawler-ip-ranges](https://api.ahrefs.com/v3/public/crawler-ip-ranges) |
|