@adguard/hostlist-compiler 2.0.0 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/README.md +37 -3
- package/package.json +3 -3
- package/src/index.d.ts +1 -0
- package/src/index.js +1 -0
- package/src/schemas/configuration.schema.json +98 -0
- package/src/transformations/ip-normalize.js +182 -0
- package/src/transformations/transform.js +6 -4
- package/src/transformations/validate-allow-ip-and-public-suffix.js +33 -0
- package/src/transformations/validate-allow-ip.js +16 -3
- package/src/transformations/validate.js +118 -53
- package/src/utils.js +137 -0
- package/test/configuration.test.js +6 -0
- package/test/transformations/ip-alignment.test.js +65 -0
- package/test/transformations/ip-normalize.test.js +331 -0
- package/test/transformations/transform.test.js +6 -0
- package/test/transformations/validate-allow-ip-and-public-suffix.test.js +167 -0
- package/test/transformations/validate-allow-ip.test.js +46 -18
- package/test/transformations/validate-allow-public-suffix.test.js +20 -0
- package/test/transformations/validate.test.js +57 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [2.1.0] - 2026-04-16
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- `ValidateAllowIpAndPublicSuffix` validation that keeps both IP address rules and rules matching whole public suffixes (e.g. `||185.149.120.173^`, `.org^`). [#126]
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
`ValidateAllowIp` now normalizes incomplete IP rules to the safe format `||ip^`. [#127]
|
|
17
|
+
- 4-octet IPs without proper separators (e.g. `1.2.3.4`, `1.2.3.4^`, `|1.2.3.4^`) are normalized to `||1.2.3.4^`
|
|
18
|
+
- 3-octet subnet wildcards (e.g. `192.168.1.`, `192.168.1.*`) are allowed with `||` prefix
|
|
19
|
+
- 3-octet patterns with `^` are rejected (e.g. `192.168.1^`, `||192.168.1^`)
|
|
20
|
+
- 3-octet patterns without trailing dot/wildcard are rejected (e.g. `192.168.1`, `||192.168.1`)
|
|
21
|
+
- 1-2 octet patterns are rejected (too wide, use regex instead)
|
|
22
|
+
- Normalization (`ip-normalize.js`) is now purely a normalization step: it only rewrites valid patterns to canonical form and passes everything else through unchanged. Rejection of invalid patterns is the responsibility of the validator (`validate.js`), eliminating double-rejection.
|
|
23
|
+
|
|
24
|
+
### Fixed
|
|
25
|
+
|
|
26
|
+
- `ValidateAllowPublicSuffix` was letting through terminated ICANN TLDs (e.g. `xn--jlq61u9w7b`) because `tldts` v5.x had stale Public Suffix List data. Updated `tldts` from v5 to v7 which has the current PSL. [#128]
|
|
27
|
+
- Hardened `validHostname()` to reject hostnames that do not contain any alphanumeric character (e.g. `..`), covering a behavioral change in `tldts` v7.
|
|
28
|
+
|
|
29
|
+
[2.1.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.0.0...v2.1.0
|
|
30
|
+
[#126]: https://github.com/AdguardTeam/HostlistCompiler/issues/126
|
|
31
|
+
[#127]: https://github.com/AdguardTeam/HostlistCompiler/issues/127
|
|
32
|
+
[#128]: https://github.com/AdguardTeam/HostlistCompiler/issues/128
|
|
33
|
+
|
|
8
34
|
## [2.0.0] - 2026-04-02
|
|
9
35
|
|
|
10
36
|
### Added
|
package/README.md
CHANGED
|
@@ -18,6 +18,7 @@ This is a simple tool that makes it easier to compile a [hosts blocklist](https:
|
|
|
18
18
|
- [Validate](#validate)
|
|
19
19
|
- [ValidateAllowIp](#validate-allow-ip)
|
|
20
20
|
- [ValidateAllowPublicSuffix](#validate-allow-public-suffix)
|
|
21
|
+
- [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix)
|
|
21
22
|
- [Deduplicate](#deduplicate)
|
|
22
23
|
- [RemoveEmptyLines](#removeemptylines)
|
|
23
24
|
- [InsertFinalNewLine](#insertfinalnewline)
|
|
@@ -315,6 +316,7 @@ Here is the full list of transformations that are available:
|
|
|
315
316
|
1. `Validate`
|
|
316
317
|
1. `ValidateAllowIp`
|
|
317
318
|
1. `ValidateAllowPublicSuffix`
|
|
319
|
+
1. `ValidateAllowIpAndPublicSuffix`
|
|
318
320
|
1. `Deduplicate`
|
|
319
321
|
1. `RemoveEmptyLines`
|
|
320
322
|
1. `InsertFinalNewLine`
|
|
@@ -442,17 +444,41 @@ So here's what it does:
|
|
|
442
444
|
- Discards rules with unsupported modifiers. [Click here](https://github.com/AdguardTeam/AdGuardHome/wiki/Hosts-Blocklists#-adblock-style-syntax) to learn more about which modifiers are supported.
|
|
443
445
|
- Discards rules that are too short.
|
|
444
446
|
- Discards IP addresses. If you need to keep IP addresses, use [ValidateAllowIp](#validate-allow-ip) instead.
|
|
447
|
+
|
|
448
|
+
#### <a name="rejected-ip-patterns"></a>Rejected IP Patterns
|
|
449
|
+
|
|
450
|
+
The following IP patterns are rejected by all validation transformations (`Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`, `ValidateAllowIpAndPublicSuffix`) as they are either unsafe or ambiguous:
|
|
451
|
+
|
|
452
|
+
- `||192.168.1^` — 3-octet with `^` - does not work
|
|
453
|
+
- `192.168.1` — Ambiguous: would match `192.168.11`, `192.168.111`, etc.
|
|
454
|
+
- `1.2.` or `1.2.*` — Too wide (1-2 octets), use regex instead
|
|
455
|
+
|
|
445
456
|
- Removes rules that block entire top-level domains (TLDs) like `||*.org^`, unless they have specific limiting modifiers such as `$denyallow`, `$badfilter`, or `$client`.
|
|
446
457
|
Examples:
|
|
447
458
|
- `||*.org^` - this rule will be removed
|
|
448
459
|
- `||*.org^$denyallow=example.com` - this rule will be kept because it has a limiting modifier
|
|
449
|
-
If such rules must be saved, use [ValidateAllowPublicSuffix](#validate-allow-public-suffix).
|
|
460
|
+
If such rules must be saved, use [ValidateAllowPublicSuffix](#validate-allow-public-suffix) or [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix).
|
|
450
461
|
|
|
451
462
|
If there are comments preceding the invalid rule, they will be removed as well.
|
|
452
463
|
|
|
453
464
|
### <a name="validate-allow-ip"></a> ValidateAllowIp
|
|
454
465
|
|
|
455
|
-
This transformation
|
|
466
|
+
This transformation extends [Validate](#validate) to allow IP addresses in the lists. It also **normalizes IP rules** in Adblock-style to the safe format `||ip^`.
|
|
467
|
+
|
|
468
|
+
#### IP Rule Normalization
|
|
469
|
+
|
|
470
|
+
- `1.2.3.4` → `||1.2.3.4^` (add both separators)
|
|
471
|
+
- `1.2.3.4^` → `||1.2.3.4^` (add left anchor)
|
|
472
|
+
- `|1.2.3.4` → `||1.2.3.4^` (replace `|` with `||`, add `^`)
|
|
473
|
+
- `|1.2.3.4^` → `||1.2.3.4^` (replace `|` with `||`)
|
|
474
|
+
- `||1.2.3.4` → `||1.2.3.4^` (add right separator)
|
|
475
|
+
- `||1.2.3.4^` → (no change, already canonical)
|
|
476
|
+
- `192.168.1.` → `||192.168.1.` (3-octet subnet wildcard)
|
|
477
|
+
- `192.168.1.*` → `||192.168.1.*` (3-octet subnet wildcard)
|
|
478
|
+
|
|
479
|
+
Modifiers like `$important`, `$client`, `$denyallow`, `$badfilter` are preserved during normalization.
|
|
480
|
+
|
|
481
|
+
> **Note:** Invalid IP patterns are rejected solely by the base [Validate](#validate) logic — normalization only converts valid patterns to canonical form and passes everything else through unchanged. See [Rejected IP Patterns](#rejected-ip-patterns) for details.
|
|
456
482
|
|
|
457
483
|
### <a name="validate-allow-public-suffix"></a> ValidateAllowPublicSuffix
|
|
458
484
|
|
|
@@ -460,10 +486,18 @@ This transformation exactly repeats the behavior of [Validate](#validate), but l
|
|
|
460
486
|
|
|
461
487
|
It still filters out invalid syntax rules and unsupported modifiers, but does not reject public-suffix rules unless the rule itself is malformed.
|
|
462
488
|
|
|
463
|
-
> **Note:** Combining any `Validate`, `ValidateAllowIp`, and `
|
|
489
|
+
> **Note:** Combining any `Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`, and `ValidateAllowIpAndPublicSuffix` in one transformation list is not allowed and will result in an error. Each runs its own validator on the already-filtered output of the previous one, so allow-modes become silently ineffective.
|
|
464
490
|
|
|
465
491
|
> **Important:** Validation transformations also cannot be used at both source-level and top-level simultaneously. For example, if a source uses `ValidateAllowPublicSuffix` and the top-level configuration uses `Validate`, the compiler will throw an error. This is because the top-level `Validate` would override the source-level validation, making `ValidateAllowPublicSuffix` ineffective. Use validation transformations at only one level.
|
|
466
492
|
|
|
493
|
+
### <a name="validate-allow-ip-and-public-suffix"></a> ValidateAllowIpAndPublicSuffix
|
|
494
|
+
|
|
495
|
+
This transformation combines the behavior of [ValidateAllowIp](#validate-allow-ip) and [ValidateAllowPublicSuffix](#validate-allow-public-suffix). It allows both IP addresses and public suffix rules in the list.
|
|
496
|
+
|
|
497
|
+
Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format `||ip^` before validation. See [IP Rule Normalization](#ip-rule-normalization) for details.
|
|
498
|
+
|
|
499
|
+
Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`).
|
|
500
|
+
|
|
467
501
|
### <a name="deduplicate"></a> Deduplicate
|
|
468
502
|
|
|
469
503
|
This transformation simply removes the duplicates from the specified source.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@adguard/hostlist-compiler",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.1.0",
|
|
4
4
|
"description": "A simple tool that compiles hosts blocklists from multiple sources",
|
|
5
5
|
"main": "src/index.js",
|
|
6
6
|
"types": "src/index.d.ts",
|
|
@@ -31,11 +31,11 @@
|
|
|
31
31
|
"@adguard/filters-downloader": "^2.3.1",
|
|
32
32
|
"ajv": "^6.12.0",
|
|
33
33
|
"ajv-errors": "^1.0.1",
|
|
34
|
-
"axios": "^
|
|
34
|
+
"axios": "^1.15.0",
|
|
35
35
|
"better-ajv-errors": "^0.6.7",
|
|
36
36
|
"consola": "^2.11.3",
|
|
37
37
|
"lodash": "^4.17.15",
|
|
38
|
-
"tldts": "
|
|
38
|
+
"tldts": "7.0.28",
|
|
39
39
|
"yargs": "^15.3.0"
|
|
40
40
|
}
|
|
41
41
|
}
|
package/src/index.d.ts
CHANGED
package/src/index.js
CHANGED
|
@@ -74,6 +74,7 @@ function checkCrossLevelValidationConflicts(configuration) {
|
|
|
74
74
|
TRANSFORMATIONS.Validate,
|
|
75
75
|
TRANSFORMATIONS.ValidateAllowIp,
|
|
76
76
|
TRANSFORMATIONS.ValidateAllowPublicSuffix,
|
|
77
|
+
TRANSFORMATIONS.ValidateAllowIpAndPublicSuffix,
|
|
77
78
|
];
|
|
78
79
|
|
|
79
80
|
const topLevelTransformations = configuration.transformations || [];
|
|
@@ -85,6 +85,54 @@
|
|
|
85
85
|
}
|
|
86
86
|
]
|
|
87
87
|
}
|
|
88
|
+
},
|
|
89
|
+
{
|
|
90
|
+
"not": {
|
|
91
|
+
"allOf": [
|
|
92
|
+
{
|
|
93
|
+
"contains": {
|
|
94
|
+
"const": "Validate"
|
|
95
|
+
}
|
|
96
|
+
},
|
|
97
|
+
{
|
|
98
|
+
"contains": {
|
|
99
|
+
"const": "ValidateAllowIpAndPublicSuffix"
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
]
|
|
103
|
+
}
|
|
104
|
+
},
|
|
105
|
+
{
|
|
106
|
+
"not": {
|
|
107
|
+
"allOf": [
|
|
108
|
+
{
|
|
109
|
+
"contains": {
|
|
110
|
+
"const": "ValidateAllowIp"
|
|
111
|
+
}
|
|
112
|
+
},
|
|
113
|
+
{
|
|
114
|
+
"contains": {
|
|
115
|
+
"const": "ValidateAllowIpAndPublicSuffix"
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
]
|
|
119
|
+
}
|
|
120
|
+
},
|
|
121
|
+
{
|
|
122
|
+
"not": {
|
|
123
|
+
"allOf": [
|
|
124
|
+
{
|
|
125
|
+
"contains": {
|
|
126
|
+
"const": "ValidateAllowPublicSuffix"
|
|
127
|
+
}
|
|
128
|
+
},
|
|
129
|
+
{
|
|
130
|
+
"contains": {
|
|
131
|
+
"const": "ValidateAllowIpAndPublicSuffix"
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
]
|
|
135
|
+
}
|
|
88
136
|
}
|
|
89
137
|
],
|
|
90
138
|
"items": {
|
|
@@ -96,6 +144,7 @@
|
|
|
96
144
|
"Validate",
|
|
97
145
|
"ValidateAllowIp",
|
|
98
146
|
"ValidateAllowPublicSuffix",
|
|
147
|
+
"ValidateAllowIpAndPublicSuffix",
|
|
99
148
|
"Deduplicate",
|
|
100
149
|
"InvertAllow",
|
|
101
150
|
"RemoveEmptyLines",
|
|
@@ -213,6 +262,54 @@
|
|
|
213
262
|
}
|
|
214
263
|
]
|
|
215
264
|
}
|
|
265
|
+
},
|
|
266
|
+
{
|
|
267
|
+
"not": {
|
|
268
|
+
"allOf": [
|
|
269
|
+
{
|
|
270
|
+
"contains": {
|
|
271
|
+
"const": "Validate"
|
|
272
|
+
}
|
|
273
|
+
},
|
|
274
|
+
{
|
|
275
|
+
"contains": {
|
|
276
|
+
"const": "ValidateAllowIpAndPublicSuffix"
|
|
277
|
+
}
|
|
278
|
+
}
|
|
279
|
+
]
|
|
280
|
+
}
|
|
281
|
+
},
|
|
282
|
+
{
|
|
283
|
+
"not": {
|
|
284
|
+
"allOf": [
|
|
285
|
+
{
|
|
286
|
+
"contains": {
|
|
287
|
+
"const": "ValidateAllowIp"
|
|
288
|
+
}
|
|
289
|
+
},
|
|
290
|
+
{
|
|
291
|
+
"contains": {
|
|
292
|
+
"const": "ValidateAllowIpAndPublicSuffix"
|
|
293
|
+
}
|
|
294
|
+
}
|
|
295
|
+
]
|
|
296
|
+
}
|
|
297
|
+
},
|
|
298
|
+
{
|
|
299
|
+
"not": {
|
|
300
|
+
"allOf": [
|
|
301
|
+
{
|
|
302
|
+
"contains": {
|
|
303
|
+
"const": "ValidateAllowPublicSuffix"
|
|
304
|
+
}
|
|
305
|
+
},
|
|
306
|
+
{
|
|
307
|
+
"contains": {
|
|
308
|
+
"const": "ValidateAllowIpAndPublicSuffix"
|
|
309
|
+
}
|
|
310
|
+
}
|
|
311
|
+
]
|
|
312
|
+
}
|
|
216
313
|
}
|
|
217
314
|
],
|
|
218
315
|
"items": {
|
|
@@ -224,6 +321,7 @@
|
|
|
224
321
|
"Validate",
|
|
225
322
|
"ValidateAllowIp",
|
|
226
323
|
"ValidateAllowPublicSuffix",
|
|
324
|
+
"ValidateAllowIpAndPublicSuffix",
|
|
227
325
|
"Deduplicate",
|
|
228
326
|
"InvertAllow",
|
|
229
327
|
"RemoveEmptyLines",
|
|
@@ -0,0 +1,182 @@
|
|
|
1
|
+
const consola = require('consola');
|
|
2
|
+
const ruleUtils = require('../rule');
|
|
3
|
+
const { classifyIpPattern, parseIpPattern } = require('../utils');
|
|
4
|
+
|
|
5
|
+
// Prefix for canonical IP/domain patterns
|
|
6
|
+
const IP_PREFIX = '||';
|
|
7
|
+
|
|
8
|
+
/**
|
|
9
|
+
* Action constants for IP rule processing results.
|
|
10
|
+
* Used by check helpers, processIpRule, and normalizeIpRules.
|
|
11
|
+
*/
|
|
12
|
+
const ACTION = Object.freeze({
|
|
13
|
+
/**
|
|
14
|
+
* Pattern is valid and already in canonical form — no change needed.
|
|
15
|
+
*/
|
|
16
|
+
KEEP: 'keep',
|
|
17
|
+
|
|
18
|
+
/**
|
|
19
|
+
* Pattern needs rewriting to canonical form.
|
|
20
|
+
*/
|
|
21
|
+
NORMALIZE: 'normalize',
|
|
22
|
+
|
|
23
|
+
/**
|
|
24
|
+
* Pattern is a valid subnet already in canonical form (||prefix present).
|
|
25
|
+
*/
|
|
26
|
+
ALLOW: 'allow',
|
|
27
|
+
});
|
|
28
|
+
|
|
29
|
+
/**
|
|
30
|
+
* Determines if a pattern is a full 4-octet IP address that needs normalization.
|
|
31
|
+
*
|
|
32
|
+
* @param {string} pattern - The adblock pattern.
|
|
33
|
+
* @returns {string|null} The normalized pattern (||ip^) or null if not applicable.
|
|
34
|
+
*/
|
|
35
|
+
function normalizeFullIp(pattern) {
|
|
36
|
+
const c = classifyIpPattern(pattern);
|
|
37
|
+
if (!c || !c.isFullIp) {
|
|
38
|
+
return null;
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
const ip = c.octets.join('.');
|
|
42
|
+
|
|
43
|
+
// Already in correct format ||ip^ (||ip^| is not a non-canonical variant)
|
|
44
|
+
if (c.prefix === IP_PREFIX && c.hasCaret && !c.hasCaretPipe) {
|
|
45
|
+
return null;
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
// Normalize to ||ip^
|
|
49
|
+
return `||${ip}^`;
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
/**
|
|
53
|
+
* Determines if a pattern is a 3-octet subnet that should be allowed.
|
|
54
|
+
*
|
|
55
|
+
* Valid 3-octet patterns:
|
|
56
|
+
* - 192.168.1. (trailing dot)
|
|
57
|
+
* - 192.168.1.* (trailing wildcard)
|
|
58
|
+
* - ||192.168.1. (with prefix)
|
|
59
|
+
* - ||192.168.1.* (with prefix)
|
|
60
|
+
*
|
|
61
|
+
* Invalid 3-octet patterns (returned as null — caller treats as pass-through):
|
|
62
|
+
* - 192.168.1 (no trailing dot/wildcard - ambiguous)
|
|
63
|
+
* - ||192.168.1^ (with caret - doesn't work in AdGuard Home)
|
|
64
|
+
*
|
|
65
|
+
* @param {string} pattern - The adblock pattern.
|
|
66
|
+
* @returns {object|null} Result with action and normalized pattern, or null if not applicable.
|
|
67
|
+
* - action: ACTION.ALLOW, ACTION.NORMALIZE
|
|
68
|
+
* - normalized: the normalized pattern (if action is ACTION.NORMALIZE)
|
|
69
|
+
* - null means the pattern is not a 3-octet IP subnet, or is an invalid 3-octet pattern;
|
|
70
|
+
* caller should pass it through unchanged.
|
|
71
|
+
*/
|
|
72
|
+
function check3OctetSubnet(pattern) {
|
|
73
|
+
const c = classifyIpPattern(pattern);
|
|
74
|
+
if (!c || c.octetCount !== 3) {
|
|
75
|
+
return null;
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
const ip = c.octets.join('.');
|
|
79
|
+
|
|
80
|
+
// 3 octets with ^ - doesn't work in AdGuard Home; pass through for validator
|
|
81
|
+
if (c.hasCaret) {
|
|
82
|
+
return null;
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
// 3 octets without trailing dot/wildcard - ambiguous; pass through for validator
|
|
86
|
+
if (!c.hasTrailingDot && !c.hasTrailingWildcard) {
|
|
87
|
+
return null;
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
// Valid 3-octet subnet pattern - normalize to add || if missing
|
|
91
|
+
if (c.prefix === IP_PREFIX) {
|
|
92
|
+
return {
|
|
93
|
+
action: ACTION.ALLOW,
|
|
94
|
+
};
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
// Normalize: add || prefix
|
|
98
|
+
const suffix = c.hasTrailingWildcard ? '.*' : '.';
|
|
99
|
+
return {
|
|
100
|
+
action: ACTION.NORMALIZE,
|
|
101
|
+
normalized: `||${ip}${suffix}`,
|
|
102
|
+
};
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
/**
|
|
106
|
+
* Normalizes an IP rule if needed, or keeps it unchanged.
|
|
107
|
+
* Invalid patterns are passed through for the validator to reject.
|
|
108
|
+
*
|
|
109
|
+
* @param {string} ruleText - The full rule text (may include modifiers and @@ prefix).
|
|
110
|
+
* @returns {object} Result object.
|
|
111
|
+
* - action: ACTION.KEEP (no change), ACTION.NORMALIZE
|
|
112
|
+
* - normalized: the normalized rule (if action is ACTION.NORMALIZE)
|
|
113
|
+
*/
|
|
114
|
+
function processIpRule(ruleText) {
|
|
115
|
+
// Skip comments and empty lines
|
|
116
|
+
if (!ruleText || ruleText.startsWith('!') || ruleText.startsWith('#') || ruleText.trim() === '') {
|
|
117
|
+
return { action: ACTION.KEEP };
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
const ruleProps = ruleUtils.loadAdblockRuleProperties(ruleText);
|
|
121
|
+
const { pattern } = ruleProps;
|
|
122
|
+
|
|
123
|
+
// Check for 3-octet subnet patterns (normalize or keep)
|
|
124
|
+
const subnet = check3OctetSubnet(pattern);
|
|
125
|
+
if (subnet?.action === ACTION.NORMALIZE) {
|
|
126
|
+
ruleProps.pattern = subnet.normalized;
|
|
127
|
+
return {
|
|
128
|
+
action: ACTION.NORMALIZE,
|
|
129
|
+
normalized: ruleUtils.adblockRuleToString(ruleProps),
|
|
130
|
+
};
|
|
131
|
+
}
|
|
132
|
+
if (subnet?.action === ACTION.ALLOW) {
|
|
133
|
+
// Already in canonical form — no change needed
|
|
134
|
+
return { action: ACTION.KEEP };
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
// Check for full 4-octet IP normalization
|
|
138
|
+
const normalized = normalizeFullIp(pattern);
|
|
139
|
+
if (normalized) {
|
|
140
|
+
ruleProps.pattern = normalized;
|
|
141
|
+
return {
|
|
142
|
+
action: ACTION.NORMALIZE,
|
|
143
|
+
normalized: ruleUtils.adblockRuleToString(ruleProps),
|
|
144
|
+
};
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
return { action: ACTION.KEEP };
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
/**
|
|
151
|
+
* Processes a list of rules, normalizing IP rules and passing invalid ones through unchanged.
|
|
152
|
+
*
|
|
153
|
+
* @param {string[]} rules - Array of rules.
|
|
154
|
+
* @returns {string[]} Processed rules with normalization applied.
|
|
155
|
+
*/
|
|
156
|
+
function normalizeIpRules(rules) {
|
|
157
|
+
const result = [];
|
|
158
|
+
|
|
159
|
+
// eslint-disable-next-line no-restricted-syntax
|
|
160
|
+
for (const rule of rules) {
|
|
161
|
+
const processed = processIpRule(rule);
|
|
162
|
+
|
|
163
|
+
if (processed.action === ACTION.NORMALIZE) {
|
|
164
|
+
consola.info(`Normalized IP rule: ${rule} → ${processed.normalized}`);
|
|
165
|
+
result.push(processed.normalized);
|
|
166
|
+
} else {
|
|
167
|
+
// KEEP — pass through unchanged; validator will reject invalid patterns
|
|
168
|
+
result.push(rule);
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
return result;
|
|
173
|
+
}
|
|
174
|
+
|
|
175
|
+
module.exports = {
|
|
176
|
+
ACTION,
|
|
177
|
+
parseIpPattern,
|
|
178
|
+
normalizeFullIp,
|
|
179
|
+
check3OctetSubnet,
|
|
180
|
+
processIpRule,
|
|
181
|
+
normalizeIpRules,
|
|
182
|
+
};
|
|
@@ -3,6 +3,7 @@ const removeModifiers = require('./remove-modifiers');
|
|
|
3
3
|
const { validate } = require('./validate');
|
|
4
4
|
const { validateAllowIp } = require('./validate-allow-ip');
|
|
5
5
|
const { validateAllowPublicSuffix } = require('./validate-allow-public-suffix');
|
|
6
|
+
const { validateAllowIpAndPublicSuffix } = require('./validate-allow-ip-and-public-suffix');
|
|
6
7
|
const exclude = require('./exclude');
|
|
7
8
|
const include = require('./include');
|
|
8
9
|
const deduplicate = require('./deduplicate');
|
|
@@ -23,6 +24,7 @@ const TRANSFORMATIONS = Object.freeze({
|
|
|
23
24
|
Validate: 'Validate',
|
|
24
25
|
ValidateAllowIp: 'ValidateAllowIp',
|
|
25
26
|
ValidateAllowPublicSuffix: 'ValidateAllowPublicSuffix',
|
|
27
|
+
ValidateAllowIpAndPublicSuffix: 'ValidateAllowIpAndPublicSuffix',
|
|
26
28
|
Deduplicate: 'Deduplicate',
|
|
27
29
|
InvertAllow: 'InvertAllow',
|
|
28
30
|
RemoveEmptyLines: 'RemoveEmptyLines',
|
|
@@ -35,6 +37,7 @@ const VALIDATION_TRANSFORMATIONS = [
|
|
|
35
37
|
TRANSFORMATIONS.Validate,
|
|
36
38
|
TRANSFORMATIONS.ValidateAllowIp,
|
|
37
39
|
TRANSFORMATIONS.ValidateAllowPublicSuffix,
|
|
40
|
+
TRANSFORMATIONS.ValidateAllowIpAndPublicSuffix,
|
|
38
41
|
];
|
|
39
42
|
|
|
40
43
|
/**
|
|
@@ -42,10 +45,6 @@ const VALIDATION_TRANSFORMATIONS = [
|
|
|
42
45
|
* Combining them causes silent data loss: each validator runs on the already-filtered
|
|
43
46
|
* output of the previous one, so allow-modes (AllowIp, AllowPublicSuffix) become ineffective.
|
|
44
47
|
*
|
|
45
|
-
* TODO: When a combined mode for ValidateAllowIp + ValidateAllowPublicSuffix is implemented
|
|
46
|
-
* (see https://github.com/AdguardTeam/HostlistCompiler/issues/126),
|
|
47
|
-
* update this check to allow that specific pair and run Validator(true, true) instead.
|
|
48
|
-
*
|
|
49
48
|
* @param {Array<string>} transformations - configured transformations.
|
|
50
49
|
*/
|
|
51
50
|
function checkIncompatibleValidationTransformations(transformations) {
|
|
@@ -115,6 +114,9 @@ async function transform(rules, configuration, transformations) {
|
|
|
115
114
|
if (transformations.indexOf(TRANSFORMATIONS.ValidateAllowPublicSuffix) !== -1) {
|
|
116
115
|
transformed = validateAllowPublicSuffix(transformed);
|
|
117
116
|
}
|
|
117
|
+
if (transformations.indexOf(TRANSFORMATIONS.ValidateAllowIpAndPublicSuffix) !== -1) {
|
|
118
|
+
transformed = validateAllowIpAndPublicSuffix(transformed);
|
|
119
|
+
}
|
|
118
120
|
if (transformations.indexOf(TRANSFORMATIONS.Deduplicate) !== -1) {
|
|
119
121
|
transformed = deduplicate(transformed);
|
|
120
122
|
}
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
const { Validator } = require('./validate');
|
|
2
|
+
const { normalizeIpRules } = require('./ip-normalize');
|
|
3
|
+
|
|
4
|
+
/**
|
|
5
|
+
* Validates a list of rules, allowing both IP addresses and public suffix matching.
|
|
6
|
+
*
|
|
7
|
+
* This function first normalizes incomplete IP rules to the safe format ||ip^,
|
|
8
|
+
* then creates an instance of the `Validator` class, enabling both IP address
|
|
9
|
+
* validation and public suffix matching, and returns a filtered list of valid rules.
|
|
10
|
+
*
|
|
11
|
+
* IP normalization:
|
|
12
|
+
* - 4-octet IPs without proper separators are normalized to ||ip^
|
|
13
|
+
* - 3-octet subnet patterns (192.168.1., 192.168.1.*) are allowed with || prefix
|
|
14
|
+
* - 3-octet patterns without trailing dot/wildcard are rejected (ambiguous)
|
|
15
|
+
* - 3-octet patterns with ^ are rejected (don't work in AdGuard Home)
|
|
16
|
+
* - 1-2 octet patterns are rejected (too wide, use regex instead)
|
|
17
|
+
*
|
|
18
|
+
* @param {Array<string>} rules - The array of rules to validate.
|
|
19
|
+
* @returns {Array<string>} - The filtered array of valid rules, allowing both
|
|
20
|
+
* IP addresses and public suffixes.
|
|
21
|
+
*/
|
|
22
|
+
function validateAllowIpAndPublicSuffix(rules) {
|
|
23
|
+
// First, normalize IP rules (convert incomplete patterns to ||ip^)
|
|
24
|
+
const normalizedRules = normalizeIpRules(rules);
|
|
25
|
+
|
|
26
|
+
// Then validate with both IP addresses and public suffixes allowed
|
|
27
|
+
const validator = new Validator(true, true);
|
|
28
|
+
return validator.validate(normalizedRules);
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
module.exports = {
|
|
32
|
+
validateAllowIpAndPublicSuffix,
|
|
33
|
+
};
|
|
@@ -1,20 +1,33 @@
|
|
|
1
1
|
const { Validator } = require('./validate');
|
|
2
|
+
const { normalizeIpRules } = require('./ip-normalize');
|
|
2
3
|
|
|
3
4
|
/**
|
|
4
5
|
* Validates a list of rules and allows IP validation.
|
|
5
6
|
*
|
|
6
|
-
* This function
|
|
7
|
+
* This function first normalizes incomplete IP rules to the safe format ||ip^,
|
|
8
|
+
* then creates an instance of the `Validator` class, enabling IP address validation,
|
|
7
9
|
* and returns a filtered list of valid rules.
|
|
8
10
|
*
|
|
11
|
+
* IP normalization:
|
|
12
|
+
* - 4-octet IPs without proper separators are normalized to ||ip^
|
|
13
|
+
* - 3-octet subnet patterns (192.168.1., 192.168.1.*) are allowed with || prefix
|
|
14
|
+
* - 3-octet patterns without trailing dot/wildcard are rejected (ambiguous)
|
|
15
|
+
* - 3-octet patterns with ^ are rejected (don't work in AdGuard Home)
|
|
16
|
+
* - 1-2 octet patterns are rejected (too wide, use regex instead)
|
|
17
|
+
*
|
|
9
18
|
* @param {Array<string>} rules - The array of rules to validate.
|
|
10
19
|
* @returns {Array<string>} - The filtered array of valid rules, allowing IP addresses.
|
|
11
20
|
*/
|
|
12
21
|
function validateAllowIp(rules) {
|
|
13
|
-
//
|
|
22
|
+
// First, normalize IP rules (convert incomplete patterns to ||ip^)
|
|
23
|
+
const normalizedRules = normalizeIpRules(rules);
|
|
24
|
+
|
|
25
|
+
// Then validate with IP addresses allowed
|
|
14
26
|
// tld is not allowed by default here
|
|
15
27
|
const validator = new Validator(true, false);
|
|
16
|
-
return validator.validate(
|
|
28
|
+
return validator.validate(normalizedRules);
|
|
17
29
|
}
|
|
30
|
+
|
|
18
31
|
module.exports = {
|
|
19
32
|
validateAllowIp,
|
|
20
33
|
};
|