cdns 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- cdns-0.1.0/.gitignore +6 -0
- cdns-0.1.0/PKG-INFO +161 -0
- cdns-0.1.0/README.md +148 -0
- cdns-0.1.0/dist/cdns.json +3258 -0
- cdns-0.1.0/pyproject.toml +61 -0
- cdns-0.1.0/python/cdns/__init__.py +12 -0
- cdns-0.1.0/python/cdns/cdn.py +19 -0
- cdns-0.1.0/python/cdns/domains.py +24 -0
- cdns-0.1.0/python/cdns/headers.py +62 -0
cdns-0.1.0/.gitignore
ADDED
cdns-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,161 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: cdns
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: List of CDNs with identification and detection tools.
|
|
5
|
+
Keywords: cdn,detection,identification,ip,ranges
|
|
6
|
+
Requires-Python: >=3.12
|
|
7
|
+
Requires-Dist: pyyaml~=6.0.0
|
|
8
|
+
Provides-Extra: ranges
|
|
9
|
+
Requires-Dist: beautifulsoup4>=4.15.0; extra == 'ranges'
|
|
10
|
+
Requires-Dist: clickhouse-connect>=1.3.0; extra == 'ranges'
|
|
11
|
+
Requires-Dist: requests~=2.34.2; extra == 'ranges'
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
|
|
14
|
+
# CDN Mapping
|
|
15
|
+
|
|
16
|
+
Provides a JS library [`@ds-ukassel/cdns`](https://www.npmjs.com/package/@ds-ukassel/cdns) to identify CDNs.
|
|
17
|
+
All CDNs are listed in [`cdns.yaml`](cdns.yaml) in this repository, and [`dist/cdns.json`](dist/cdns.json) in the package.
|
|
18
|
+
|
|
19
|
+
## YAML Format
|
|
20
|
+
|
|
21
|
+
```yaml
|
|
22
|
+
- name: AwesomeCDN
|
|
23
|
+
homepage: https://www.awesomecdn.com
|
|
24
|
+
type: commercial
|
|
25
|
+
domains: # See "Domain patterns" below
|
|
26
|
+
- 'awesomecdn.com' # 'awesomecdn.com' is a full match of the domain (e.g. 'awesomecdn.com')
|
|
27
|
+
- '**awesomecdn**' # 'awesomecdn' is a substring of the domain (e.g. 'a.awessomecdn.com')
|
|
28
|
+
- '*.cdn.awesome.com' # '*' is a wildcard for a single domain level (e.g. 'a.cdn.awesome.com' or 'b.cdn.awesome.com')
|
|
29
|
+
- '**.awesomecdn.*' # '**' is a wildcard for one or more domain levels (e.g. 'a.b.awesomecdn.de' or 'c.d.e.awesomecdn.com')
|
|
30
|
+
ranges:
|
|
31
|
+
- range_type: 'ipv4'
|
|
32
|
+
url: 'https://api.awesomecdn.com/ips-v4/'
|
|
33
|
+
- range_type: 'ipv6'
|
|
34
|
+
url: 'https://api.awesomecdn.com/ips-v6/'
|
|
35
|
+
- range_type: 'combined'
|
|
36
|
+
url: 'https://api.awesomecdn.com/ips/'
|
|
37
|
+
- range_type: 'ipv4'
|
|
38
|
+
ranges:
|
|
39
|
+
- '111.111.111.111/32'
|
|
40
|
+
last_updated: 1970-01-01
|
|
41
|
+
asns:
|
|
42
|
+
- 12345
|
|
43
|
+
- 67890
|
|
44
|
+
headers: # See "Header Patterns" below
|
|
45
|
+
- X-AwesomeCDN: # 'X-AwesomeCDN' header must be present
|
|
46
|
+
- Server: AwesomeCDN # 'Server' header must contain 'AwesomeCDN'
|
|
47
|
+
- X-Cache: HIT # 'X-Cache' header must contain 'HIT'
|
|
48
|
+
X-Whatever: # AND 'X-Whatever' header must be present
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
## Domain Wildcard Syntax
|
|
52
|
+
When implementing the wildcard syntax for domains, the following rules apply:
|
|
53
|
+
- `*` matches a single domain level
|
|
54
|
+
- `*.example.com` matches `abc.example.com`
|
|
55
|
+
- `*.example.com` does **not** match `abc.def.example.com`
|
|
56
|
+
- `**` matches one or more domain levels or any prefix
|
|
57
|
+
- `**.example.com` matches `a.example.com` and `b.c.example.com`
|
|
58
|
+
- `**example.com` matches `example.com`, `a.example.com`, and `b.c.example.com`
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
`*` and `**` can appear anywhere in the domain string, including the beginning, middle, or end.
|
|
62
|
+
|
|
63
|
+
### Regex Equivalents
|
|
64
|
+
- `*` is equivalent to `[^.]+` (matches any character except dots, one or more times)
|
|
65
|
+
- `**` is equivalent to `.*` (matches any character, including dots, zero or more times)
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
- All special characters are escaped, e.g. `.` turns into `\.`.
|
|
69
|
+
- The regex receives the prefix `^` and suffix `$`.
|
|
70
|
+
|
|
71
|
+
#### Examples
|
|
72
|
+
|
|
73
|
+
- `**example.com` -> `^.*example\.com$`
|
|
74
|
+
- `*.example.com` -> `^[^.]+\.example\.com$`
|
|
75
|
+
|
|
76
|
+
## Header Patterns
|
|
77
|
+
|
|
78
|
+
Header patterns have a few rules:
|
|
79
|
+
- `Header:` (JSON: `{"Header": null}`) means the header must be present, but can have any value.
|
|
80
|
+
- `Header: foo` (JSON: `{"Header": "foo"}`) means the header must be present and equal the value.
|
|
81
|
+
- Multiple patterns in the same entry, e.g.
|
|
82
|
+
```yaml
|
|
83
|
+
Header-1:
|
|
84
|
+
Header-2: bar
|
|
85
|
+
```
|
|
86
|
+
or JSON: `{"Header-1": null, "Header-2": "bar"}`
|
|
87
|
+
Means ALL header patterns must match.
|
|
88
|
+
- Header keys are case-insensitive, but values are case-sensitive.
|
|
89
|
+
|
|
90
|
+
The values can use `*` as a wildcard for any number of characters, e.g. `foo*` matches `foo`, `foobar`, and `foo123`.
|
|
91
|
+
Any amount of `*` can be used at different positions.
|
|
92
|
+
|
|
93
|
+
To use a literal `*`, escape it with a backslash: `\*`.
|
|
94
|
+
|
|
95
|
+
## Types
|
|
96
|
+
|
|
97
|
+
Each CDN can have a designated `type` field.
|
|
98
|
+
The type can either be `commercial`, `private`, `public_assets` or `other`.
|
|
99
|
+
|
|
100
|
+
- `commercial`: CDNs used by different customers (e.g. Akamai, Fastly, ...)
|
|
101
|
+
- `private`: CDNs used internally by a single company (e.g. Meta, Instagram, ...)
|
|
102
|
+
- `public_assets`: CDNs providing public assets not uploaded by customers (e.g. BootstrapCDN)
|
|
103
|
+
- `other`: CDNs with an entirely different model
|
|
104
|
+
|
|
105
|
+
The field is optional as this information isn't always available/clear.
|
|
106
|
+
|
|
107
|
+
## Python API
|
|
108
|
+
|
|
109
|
+
```python
|
|
110
|
+
def compile_domains(domains: list[str]) -> Pattern:
|
|
111
|
+
...
|
|
112
|
+
def compile_domain(domain: str) -> Pattern:
|
|
113
|
+
...
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
([source](python/cdns/domains.py)):
|
|
117
|
+
Compile domain patterns to a RegExp according to the [Domain Pattern rules](#regex-equivalents).
|
|
118
|
+
|
|
119
|
+
```python
|
|
120
|
+
def matches(headers: Dict[str, str], matcher: Dict[str, Optional[str]]) -> bool:
|
|
121
|
+
...
|
|
122
|
+
def match_headers(headers: Dict[str, str], matchers: List[Dict[str, Optional[str]]]) -> Dict[str, str]:
|
|
123
|
+
...
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
([source](python/cdns/headers.py)):
|
|
127
|
+
Match response headers against the patterns according to the [Header Pattern rules](#header-patterns).
|
|
128
|
+
|
|
129
|
+
## TS API
|
|
130
|
+
|
|
131
|
+
```ts
|
|
132
|
+
function compileDomains(domains: string[]): RegExp
|
|
133
|
+
function compileDomain(domain: string): RegExp
|
|
134
|
+
```
|
|
135
|
+
([source](src/domains.ts)):
|
|
136
|
+
Compile domain patterns to a RegExp according to the [Domain Pattern rules](#regex-equivalents).
|
|
137
|
+
|
|
138
|
+
```ts
|
|
139
|
+
function matchHeaders(headers: Record<string, string>, patterns: Record<string, string | null>[]
|
|
140
|
+
```
|
|
141
|
+
([source](src/headers.ts)):
|
|
142
|
+
Match response headers against the patterns according to the [Header Pattern rules](#header-patterns).
|
|
143
|
+
|
|
144
|
+
## References
|
|
145
|
+
- [TurboBytes CDN Finder](https://github.com/turbobytes/cdnfinder) ([MIT](https://github.com/turbobytes/cdnfinder/blob/master/LICENSE))
|
|
146
|
+
- [CNAME Chain](https://github.com/turbobytes/cdnfinder/blob/master/assets/cnamechain.json)
|
|
147
|
+
- [CDNProbe](https://github.com/njuzmy/CDNProbe) ([Apache 2.0](https://github.com/njuzmy/CDNProbe/blob/main/LICENSE))
|
|
148
|
+
- [cname_cache](https://github.com/njuzmy/CDNProbe/blob/main/assets/static/cdn/cname_cache.json): Parts of the substring list were used to extend our own list
|
|
149
|
+
- [cdnlist](https://github.com/njuzmy/CDNProbe/blob/main/assets/static/cdn/cdnlist.txt): Some CDNs from the cdn list were used to extend our own list
|
|
150
|
+
- [CDN Handbook](https://www.cdnhandbook.com/)
|
|
151
|
+
- Information about some CDNs
|
|
152
|
+
- [IPInfo](https://ipinfo.io/)
|
|
153
|
+
- Information about ASNs used by CDNs
|
|
154
|
+
- [CDN-CNAME-Domain](https://github.com/u9sky/cdn-cname-domain/blob/main/cdn.yml)
|
|
155
|
+
- [cdn-ip-database](https://github.com/mansourjabin/cdn-ip-database) ([MIT](https://github.com/mansourjabin/cdn-ip-database/blob/main/LICENSE))
|
|
156
|
+
- [sources.json](https://github.com/mansourjabin/cdn-ip-database/blob/main/data/sources.json)
|
|
157
|
+
- [Third Party Web](https://github.com/patrickhulce/third-party-web) ([MIT](https://github.com/patrickhulce/third-party-web/blob/master/LICENSE))
|
|
158
|
+
- [entities.json](https://github.com/patrickhulce/third-party-web/blob/master/data/entities.js)
|
|
159
|
+
- [ProjectDiscovery cdncheck](https://github.com/projectdiscovery/cdncheck) ([MIT](https://github.com/projectdiscovery/cdncheck/blob/main/LICENSE.md))
|
|
160
|
+
- [provider.yaml](https://github.com/projectdiscovery/cdncheck/blob/main/cmd/generate-index/provider.yaml)
|
|
161
|
+
- [Netify](https://www.netify.ai/resources/cdn)
|
cdns-0.1.0/README.md
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
# CDN Mapping
|
|
2
|
+
|
|
3
|
+
Provides a JS library [`@ds-ukassel/cdns`](https://www.npmjs.com/package/@ds-ukassel/cdns) to identify CDNs.
|
|
4
|
+
All CDNs are listed in [`cdns.yaml`](cdns.yaml) in this repository, and [`dist/cdns.json`](dist/cdns.json) in the package.
|
|
5
|
+
|
|
6
|
+
## YAML Format
|
|
7
|
+
|
|
8
|
+
```yaml
|
|
9
|
+
- name: AwesomeCDN
|
|
10
|
+
homepage: https://www.awesomecdn.com
|
|
11
|
+
type: commercial
|
|
12
|
+
domains: # See "Domain patterns" below
|
|
13
|
+
- 'awesomecdn.com' # 'awesomecdn.com' is a full match of the domain (e.g. 'awesomecdn.com')
|
|
14
|
+
- '**awesomecdn**' # 'awesomecdn' is a substring of the domain (e.g. 'a.awessomecdn.com')
|
|
15
|
+
- '*.cdn.awesome.com' # '*' is a wildcard for a single domain level (e.g. 'a.cdn.awesome.com' or 'b.cdn.awesome.com')
|
|
16
|
+
- '**.awesomecdn.*' # '**' is a wildcard for one or more domain levels (e.g. 'a.b.awesomecdn.de' or 'c.d.e.awesomecdn.com')
|
|
17
|
+
ranges:
|
|
18
|
+
- range_type: 'ipv4'
|
|
19
|
+
url: 'https://api.awesomecdn.com/ips-v4/'
|
|
20
|
+
- range_type: 'ipv6'
|
|
21
|
+
url: 'https://api.awesomecdn.com/ips-v6/'
|
|
22
|
+
- range_type: 'combined'
|
|
23
|
+
url: 'https://api.awesomecdn.com/ips/'
|
|
24
|
+
- range_type: 'ipv4'
|
|
25
|
+
ranges:
|
|
26
|
+
- '111.111.111.111/32'
|
|
27
|
+
last_updated: 1970-01-01
|
|
28
|
+
asns:
|
|
29
|
+
- 12345
|
|
30
|
+
- 67890
|
|
31
|
+
headers: # See "Header Patterns" below
|
|
32
|
+
- X-AwesomeCDN: # 'X-AwesomeCDN' header must be present
|
|
33
|
+
- Server: AwesomeCDN # 'Server' header must contain 'AwesomeCDN'
|
|
34
|
+
- X-Cache: HIT # 'X-Cache' header must contain 'HIT'
|
|
35
|
+
X-Whatever: # AND 'X-Whatever' header must be present
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Domain Wildcard Syntax
|
|
39
|
+
When implementing the wildcard syntax for domains, the following rules apply:
|
|
40
|
+
- `*` matches a single domain level
|
|
41
|
+
- `*.example.com` matches `abc.example.com`
|
|
42
|
+
- `*.example.com` does **not** match `abc.def.example.com`
|
|
43
|
+
- `**` matches one or more domain levels or any prefix
|
|
44
|
+
- `**.example.com` matches `a.example.com` and `b.c.example.com`
|
|
45
|
+
- `**example.com` matches `example.com`, `a.example.com`, and `b.c.example.com`
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
`*` and `**` can appear anywhere in the domain string, including the beginning, middle, or end.
|
|
49
|
+
|
|
50
|
+
### Regex Equivalents
|
|
51
|
+
- `*` is equivalent to `[^.]+` (matches any character except dots, one or more times)
|
|
52
|
+
- `**` is equivalent to `.*` (matches any character, including dots, zero or more times)
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
- All special characters are escaped, e.g. `.` turns into `\.`.
|
|
56
|
+
- The regex receives the prefix `^` and suffix `$`.
|
|
57
|
+
|
|
58
|
+
#### Examples
|
|
59
|
+
|
|
60
|
+
- `**example.com` -> `^.*example\.com$`
|
|
61
|
+
- `*.example.com` -> `^[^.]+\.example\.com$`
|
|
62
|
+
|
|
63
|
+
## Header Patterns
|
|
64
|
+
|
|
65
|
+
Header patterns have a few rules:
|
|
66
|
+
- `Header:` (JSON: `{"Header": null}`) means the header must be present, but can have any value.
|
|
67
|
+
- `Header: foo` (JSON: `{"Header": "foo"}`) means the header must be present and equal the value.
|
|
68
|
+
- Multiple patterns in the same entry, e.g.
|
|
69
|
+
```yaml
|
|
70
|
+
Header-1:
|
|
71
|
+
Header-2: bar
|
|
72
|
+
```
|
|
73
|
+
or JSON: `{"Header-1": null, "Header-2": "bar"}`
|
|
74
|
+
Means ALL header patterns must match.
|
|
75
|
+
- Header keys are case-insensitive, but values are case-sensitive.
|
|
76
|
+
|
|
77
|
+
The values can use `*` as a wildcard for any number of characters, e.g. `foo*` matches `foo`, `foobar`, and `foo123`.
|
|
78
|
+
Any amount of `*` can be used at different positions.
|
|
79
|
+
|
|
80
|
+
To use a literal `*`, escape it with a backslash: `\*`.
|
|
81
|
+
|
|
82
|
+
## Types
|
|
83
|
+
|
|
84
|
+
Each CDN can have a designated `type` field.
|
|
85
|
+
The type can either be `commercial`, `private`, `public_assets` or `other`.
|
|
86
|
+
|
|
87
|
+
- `commercial`: CDNs used by different customers (e.g. Akamai, Fastly, ...)
|
|
88
|
+
- `private`: CDNs used internally by a single company (e.g. Meta, Instagram, ...)
|
|
89
|
+
- `public_assets`: CDNs providing public assets not uploaded by customers (e.g. BootstrapCDN)
|
|
90
|
+
- `other`: CDNs with an entirely different model
|
|
91
|
+
|
|
92
|
+
The field is optional as this information isn't always available/clear.
|
|
93
|
+
|
|
94
|
+
## Python API
|
|
95
|
+
|
|
96
|
+
```python
|
|
97
|
+
def compile_domains(domains: list[str]) -> Pattern:
|
|
98
|
+
...
|
|
99
|
+
def compile_domain(domain: str) -> Pattern:
|
|
100
|
+
...
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
([source](python/cdns/domains.py)):
|
|
104
|
+
Compile domain patterns to a RegExp according to the [Domain Pattern rules](#regex-equivalents).
|
|
105
|
+
|
|
106
|
+
```python
|
|
107
|
+
def matches(headers: Dict[str, str], matcher: Dict[str, Optional[str]]) -> bool:
|
|
108
|
+
...
|
|
109
|
+
def match_headers(headers: Dict[str, str], matchers: List[Dict[str, Optional[str]]]) -> Dict[str, str]:
|
|
110
|
+
...
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
([source](python/cdns/headers.py)):
|
|
114
|
+
Match response headers against the patterns according to the [Header Pattern rules](#header-patterns).
|
|
115
|
+
|
|
116
|
+
## TS API
|
|
117
|
+
|
|
118
|
+
```ts
|
|
119
|
+
function compileDomains(domains: string[]): RegExp
|
|
120
|
+
function compileDomain(domain: string): RegExp
|
|
121
|
+
```
|
|
122
|
+
([source](src/domains.ts)):
|
|
123
|
+
Compile domain patterns to a RegExp according to the [Domain Pattern rules](#regex-equivalents).
|
|
124
|
+
|
|
125
|
+
```ts
|
|
126
|
+
function matchHeaders(headers: Record<string, string>, patterns: Record<string, string | null>[]
|
|
127
|
+
```
|
|
128
|
+
([source](src/headers.ts)):
|
|
129
|
+
Match response headers against the patterns according to the [Header Pattern rules](#header-patterns).
|
|
130
|
+
|
|
131
|
+
## References
|
|
132
|
+
- [TurboBytes CDN Finder](https://github.com/turbobytes/cdnfinder) ([MIT](https://github.com/turbobytes/cdnfinder/blob/master/LICENSE))
|
|
133
|
+
- [CNAME Chain](https://github.com/turbobytes/cdnfinder/blob/master/assets/cnamechain.json)
|
|
134
|
+
- [CDNProbe](https://github.com/njuzmy/CDNProbe) ([Apache 2.0](https://github.com/njuzmy/CDNProbe/blob/main/LICENSE))
|
|
135
|
+
- [cname_cache](https://github.com/njuzmy/CDNProbe/blob/main/assets/static/cdn/cname_cache.json): Parts of the substring list were used to extend our own list
|
|
136
|
+
- [cdnlist](https://github.com/njuzmy/CDNProbe/blob/main/assets/static/cdn/cdnlist.txt): Some CDNs from the cdn list were used to extend our own list
|
|
137
|
+
- [CDN Handbook](https://www.cdnhandbook.com/)
|
|
138
|
+
- Information about some CDNs
|
|
139
|
+
- [IPInfo](https://ipinfo.io/)
|
|
140
|
+
- Information about ASNs used by CDNs
|
|
141
|
+
- [CDN-CNAME-Domain](https://github.com/u9sky/cdn-cname-domain/blob/main/cdn.yml)
|
|
142
|
+
- [cdn-ip-database](https://github.com/mansourjabin/cdn-ip-database) ([MIT](https://github.com/mansourjabin/cdn-ip-database/blob/main/LICENSE))
|
|
143
|
+
- [sources.json](https://github.com/mansourjabin/cdn-ip-database/blob/main/data/sources.json)
|
|
144
|
+
- [Third Party Web](https://github.com/patrickhulce/third-party-web) ([MIT](https://github.com/patrickhulce/third-party-web/blob/master/LICENSE))
|
|
145
|
+
- [entities.json](https://github.com/patrickhulce/third-party-web/blob/master/data/entities.js)
|
|
146
|
+
- [ProjectDiscovery cdncheck](https://github.com/projectdiscovery/cdncheck) ([MIT](https://github.com/projectdiscovery/cdncheck/blob/main/LICENSE.md))
|
|
147
|
+
- [provider.yaml](https://github.com/projectdiscovery/cdncheck/blob/main/cmd/generate-index/provider.yaml)
|
|
148
|
+
- [Netify](https://www.netify.ai/resources/cdn)
|