@adguard/hostlist-compiler 2.1.1 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. package/.github/workflows/prepare-release.yml +5 -0
  2. package/.markdownlint.json +29 -0
  3. package/.markdownlintignore +1 -0
  4. package/AGENTS.md +310 -98
  5. package/CHANGELOG.md +62 -8
  6. package/DEPLOYMENT.md +175 -0
  7. package/DEVELOPMENT.md +171 -72
  8. package/README.md +238 -143
  9. package/examples/china/README.md +2 -2
  10. package/examples/energized/README.md +4 -2
  11. package/examples/sdn/README.md +2 -2
  12. package/examples/whitelist/README.md +2 -2
  13. package/package.json +24 -18
  14. package/src/cli.js +53 -31
  15. package/src/compile-source.js +4 -3
  16. package/src/configuration.js +132 -2
  17. package/src/download.js +27 -0
  18. package/src/filter.js +2 -2
  19. package/src/index.d.ts +2 -0
  20. package/src/index.js +16 -50
  21. package/src/{transformations/ip-normalize.js → ip-normalize.js} +2 -2
  22. package/src/schemas/configuration.schema.js +137 -0
  23. package/src/source-types.js +17 -0
  24. package/src/transformations/enum.js +49 -0
  25. package/src/transformations/include.js +1 -1
  26. package/src/transformations/transform.js +40 -74
  27. package/src/transformations/validate-allow-ip-and-public-suffix.js +1 -1
  28. package/src/transformations/validate-allow-ip.js +1 -1
  29. package/src/utils.js +1284 -0
  30. package/src/validation-conflicts.js +106 -0
  31. package/test/cli-error-output.test.js +305 -0
  32. package/test/cli.test.js +47 -0
  33. package/test/compile-source.test.js +167 -7
  34. package/test/configuration.test.js +438 -0
  35. package/test/download.test.js +260 -0
  36. package/test/helpers/http-mock.js +94 -0
  37. package/test/index.test.js +497 -4
  38. package/test/{transformations/ip-normalize.test.js → ip-normalize.test.js} +1 -1
  39. package/test/resolve-include.test.js +8 -8
  40. package/test/transformations/covert-to-ascii.test.js +1 -1
  41. package/test/transformations/exclude.test.js +2 -2
  42. package/test/transformations/include.test.js +9 -3
  43. package/test/transformations/ip-alignment.test.js +1 -1
  44. package/test/transformations/transform.test.js +66 -1
  45. package/test/utils.test.js +1559 -0
  46. package/test/validation-conflicts.test.js +230 -0
  47. package/src/schemas/configuration.schema.json +0 -369
package/README.md CHANGED
@@ -1,84 +1,130 @@
1
- # Hostlist compiler
1
+ # Hostlist Compiler
2
2
 
3
- [![NPM](https://nodei.co/npm/@adguard/hostlist-compiler.png?compact=true)](https://www.npmjs.com/package/@adguard/hostlist-compiler/)
3
+ [![NPM][npm-badge]][npm]
4
4
 
5
- This is a simple tool that makes it easier to compile a [hosts blocklist](https://adguard-dns.io/kb/general/dns-filtering-syntax/) compatible with AdGuard Home or any other AdGuard product with **DNS filtering**.
5
+ ## Description
6
+
7
+ Hostlist Compiler is a Node.js CLI tool and library for filter list maintainers and developers. It
8
+ compiles DNS hosts blocklists from multiple sources — `/etc/hosts` files and adblock-style filter
9
+ lists — into a single filter list compatible with AdGuard Home and other AdGuard products with DNS
10
+ filtering.
11
+
12
+ Compiling a blocklist by hand is error-prone: sources use different syntaxes, rules must be
13
+ deduplicated, dangerous or unsupported rules have to be removed, and lists need to be filtered
14
+ through exclusion and inclusion rules. Hostlist Compiler automates this pipeline — it downloads
15
+ or reads the configured sources, converts `/etc/hosts` rules to AdGuard syntax, resolves
16
+ `!#include` directives, applies a fixed-order pipeline of transformations, and writes the
17
+ resulting filter list.
6
18
 
7
19
  > **Note on repositories:** Active development happens in the private
8
- > [AdGuardSoftwareLimited/filters-hostlist-compiler](https://github.com/AdGuardSoftwareLimited/filters-hostlist-compiler)
9
- > repository; this public [AdguardTeam/HostlistCompiler](https://github.com/AdguardTeam/HostlistCompiler)
10
- > repository is a read-only mirror that is updated automatically from it.
20
+ > [AdGuardSoftwareLimited/filters-hostlist-compiler][private-repo] repository;
21
+ > this public [AdguardTeam/HostlistCompiler][public-mirror] repository is a
22
+ > read-only mirror that is updated automatically from it.
11
23
 
24
+ ## Table of Contents
25
+
26
+ - [Installation](#installation)
27
+ - [Quick Start](#quick-start)
12
28
  - [Usage](#usage)
13
- - [Configuration](#configuration)
14
- - [Command-line](#command-line)
15
- - [API](#api)
29
+ - [Configuration](#configuration)
16
30
  - [Transformations](#transformations)
17
- - [ConvertToAscii](#convert-to-ascii)
18
- - [TrimLines](#trimlines)
19
- - [RemoveComments](#remove-comments)
20
- - [Compress](#compress)
21
- - [RemoveModifiers](#remove-modifiers)
22
- - [InvertAllow](#invertallow)
23
- - [Validate](#validate)
24
- - [ValidateAllowIp](#validate-allow-ip)
25
- - [ValidateAllowPublicSuffix](#validate-allow-public-suffix)
26
- - [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix)
27
- - [Deduplicate](#deduplicate)
28
- - [RemoveEmptyLines](#removeemptylines)
29
- - [InsertFinalNewLine](#insertfinalnewline)
31
+ - [API](#api)
30
32
  - [Documentation](#documentation)
31
33
 
32
- ## <a name="usage"></a> Usage
34
+ ---
35
+
36
+ ## Installation
33
37
 
34
- First of all, install the `hostlist-compiler`:
38
+ Requires Node.js 22.22.2 or newer. Node.js 23.x is not supported by
39
+ `@adguard/filters-downloader` 3.x.
40
+
41
+ Install the package globally to get the `hostlist-compiler` command:
35
42
 
36
43
  ```bash
37
44
  npm i -g @adguard/hostlist-compiler
38
45
  ```
39
46
 
40
- After that you have two options.
47
+ Or install it as a dependency to use the compiler as a library:
48
+
49
+ ```bash
50
+ npm i @adguard/hostlist-compiler
51
+ ```
52
+
53
+ yarn is also supported:
54
+
55
+ ```bash
56
+ yarn add @adguard/hostlist-compiler
57
+ ```
58
+
59
+ ## Quick Start
41
60
 
42
- **Quick hosts conversion**
61
+ Convert a `/etc/hosts`-syntax blocklist to an AdGuard-compatible filter list in one command:
43
62
 
44
- Convert and compress a `/etc/hosts`-syntax blocklist to [AdGuard syntax](https://adguard-dns.io/kb/general/dns-filtering-syntax/).
63
+ ```bash
64
+ hostlist-compiler -i hosts.txt -o output.txt
65
+ ```
45
66
 
67
+ Or compile a configurable blocklist from multiple sources:
68
+
69
+ ```bash
70
+ hostlist-compiler -c configuration.json -o output.txt
46
71
  ```
72
+
73
+ ## Usage
74
+
75
+ ### Command-Line Options
76
+
77
+ | Flag | Description |
78
+ | --------------------------- | -------------------------------------------------------------------------------------------------- |
79
+ | `--config, -c <path>` | Path to the compiler configuration file |
80
+ | `--input, -i <url-or-path>` | URL (or path to a file) to convert to an AdGuard-syntax blocklist. Can be specified multiple times |
81
+ | `--input-type, -t <type>` | Type of the input file: `hosts` or `adblock` |
82
+ | `--output, -o <path>` | Path to the output file (required) |
83
+ | `--verbose, -v` | Run with verbose logging |
84
+ | `--version` | Show the version number |
85
+ | `-h, --help` | Show help |
86
+
87
+ The compiler exits with code `0` on success and `1` on failure; errors are printed to stderr.
88
+
89
+ ### Proxy
90
+
91
+ To download sources through a proxy, set `NODE_USE_ENV_PROXY=1` together with
92
+ the `HTTP_PROXY` (also used for `https://` when `HTTPS_PROXY` is unset),
93
+ `HTTPS_PROXY`, and `NO_PROXY` environment variables. The Node.js native
94
+ `fetch` ignores these variables unless `NODE_USE_ENV_PROXY=1` is set.
95
+
96
+ ### Quick Hosts Conversion
97
+
98
+ Convert and compress one or more `/etc/hosts`-syntax blocklists to
99
+ [AdGuard syntax](https://adguard-dns.io/kb/general/dns-filtering-syntax/):
100
+
101
+ ```bash
47
102
  hostlist-compiler -i hosts.txt -i hosts2.txt -o output.txt
48
103
  ```
49
104
 
50
- **Build a configurable blocklist from multiple sources**
105
+ The quick conversion applies a default set of transformations to convert
106
+ `/etc/hosts` rules to AdGuard syntax: `RemoveComments`, `Deduplicate`,
107
+ `Compress`, `Validate`, `TrimLines`, `InsertFinalNewLine`, applied in the
108
+ order described in the [Transformations](#transformations) section. To
109
+ control the pipeline explicitly, use a [configuration file](#configuration)
110
+ instead.
51
111
 
52
- Prepare the list configuration (read more about that [below](#configuration)) and run the compiler:
112
+ ### Build a Configurable Blocklist from Multiple Sources
113
+
114
+ Prepare the list configuration (read more about that
115
+ [below](#configuration)) and run the compiler:
53
116
 
54
117
  ```bash
55
118
  hostlist-compiler -c configuration.json -o output.txt
56
119
  ```
57
120
 
58
- **All command line options**
121
+ A configurable blocklist can also be built from a remote URL without a configuration file:
59
122
 
60
- ```
61
- Usage: hostlist-compiler [options]
62
-
63
- Options:
64
- --config, -c Path to the compiler configuration file [string]
65
- --input, -i URL (or path to a file) to convert to an AdGuard-syntax
66
- blocklist. Can be specified multiple times. [array]
67
- --input-type, -t Type of the input file (/etc/hosts, adguard) [string]
68
- --output, -o Path to the output file [string] [required]
69
- --verbose, -v Run with verbose logging [boolean]
70
- --version Show version number [boolean]
71
- -h, --help Show help [boolean]
72
-
73
- Examples:
74
- hostlist-compiler -c config.json -o compile a blocklist and write the
75
- output.txt output to output.txt
76
- hostlist-compiler -i compile a blocklist from the URL and
77
- https://example.org/hosts.txt -o write the output to output.txt
78
- output.txt
123
+ ```bash
124
+ hostlist-compiler -i https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -o output.txt
79
125
  ```
80
126
 
81
- ### <a name="configuration"></a> Configuration
127
+ ## Configuration
82
128
 
83
129
  Configuration defines your filter list sources, and the transformations that are applied to the sources.
84
130
 
@@ -88,7 +134,7 @@ Here is an example of this configuration:
88
134
  {
89
135
  "name": "List name",
90
136
  "description": "List description",
91
- "homepage": "https://example.org/",
137
+ "homepage": "https://github.com/AdguardTeam/AdguardSDNSFilter",
92
138
  "license": "GPLv3",
93
139
  "version": "1.0.0.0",
94
140
  "sources": [
@@ -104,8 +150,8 @@ Here is an example of this configuration:
104
150
  },
105
151
  {
106
152
  "name": "Remote rules",
107
- "source": "https://example.org/rules",
108
- "type": "hosts",
153
+ "source": "https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt",
154
+ "type": "adblock",
109
155
  "exclusions": ["excluded rule 1"]
110
156
  }
111
157
  ],
@@ -123,19 +169,31 @@ Here is an example of this configuration:
123
169
  - `license` - (optional) Filter list license.
124
170
  - `version` - (optional) Filter list version.
125
171
  - `sources` - (mandatory) array of the list sources.
126
- - `.source` - (mandatory) path or URL of the source. It can be a traditional filter list or a hosts file.
127
- - `.name` - (optional) name of the source.
128
- - `.type` - (optional) type of the source. It could be `adblock` for Adblock-style lists or `hosts` for /etc/hosts style lists. If not specified, `adblock` is assumed.
129
- - `.transformations` - (optional) a list of transformations to apply to the source rules. By default, **no transformations** are applied. Learn more about possible transformations [here](#transformations).
130
- - `.exclusions` - (optional) a list of rules (or wildcards) to exclude from the source.
131
- - `.exclusions_sources` - (optional) a list of files with exclusions.
132
- - `.inclusions` - (optional) a list of wildcards to include from the source. All rules that don't match these wildcards won't be included.
133
- - `.inclusions_sources` - (optional) a list of files with inclusions.
134
- - `transformations` - (optional) a list of transformations to apply to the final list of rules. By default, **no transformations** are applied. Learn more about possible transformations [here](#transformations).
135
- - `exclusions` - (optional) a list of rules (or wildcards) to exclude from the source.
172
+ - `.source` - (mandatory) path or URL of the source. It can be a
173
+ traditional filter list or a hosts file.
174
+ - `.name` - (optional) name of the source.
175
+ - `.type` - (optional) type of the source. It can be `adblock` for
176
+ Adblock-style lists or `hosts` for /etc/hosts style lists. The value is
177
+ validated against these two options; rule parsing auto-detects the
178
+ format, so the type does not affect parsing.
179
+ - `.transformations` - (optional) a list of transformations to apply to
180
+ the source rules. By default, **no transformations** are applied. Learn
181
+ more about the possible [transformations](#transformations).
182
+ - `.exclusions` - (optional) a list of rules (or wildcards) to exclude
183
+ from the source.
184
+ - `.exclusions_sources` - (optional) a list of files with exclusions.
185
+ - `.inclusions` - (optional) a list of wildcards to include from the
186
+ source. All rules that don't match these wildcards won't be included.
187
+ - `.inclusions_sources` - (optional) a list of files with inclusions.
188
+ - `transformations` - (optional) a list of transformations to apply to the
189
+ final list of rules. By default, **no transformations** are applied. Learn
190
+ more about the possible [transformations](#transformations).
191
+ - `exclusions` - (optional) a list of rules (or wildcards) to exclude from
192
+ the final list.
136
193
  - `exclusions_sources` - (optional) a list of files with exclusions.
137
- - `.inclusions` - (optional) a list of wildcards to include from the source. All rules that don't match these wildcards won't be included.
138
- - `.inclusions_sources` - (optional) a list of files with inclusions.
194
+ - `inclusions` - (optional) a list of wildcards to include in the final
195
+ list. All rules that don't match these wildcards won't be included.
196
+ - `inclusions_sources` - (optional) a list of files with inclusions.
139
197
 
140
198
  Here is an example of a minimal configuration:
141
199
 
@@ -150,13 +208,15 @@ Here is an example of a minimal configuration:
150
208
  }
151
209
  ```
152
210
 
153
- **Exclusion and inclusion rules**
211
+ ### Exclusion and Inclusion Rules
154
212
 
155
- Please note, that exclusion or inclusion rules may be a plain string, wildcard, or a regular expression.
213
+ Please note, that exclusion or inclusion rules may be a plain string,
214
+ wildcard, or a regular expression.
156
215
 
157
216
  - `plainstring` - every rule that contains `plainstring` will match the rule
158
217
  - `*.plainstring` - every rule that matches this wildcard will match the rule
159
- - `/regex/` - every rule that matches this regular expression, will match the rule. By default, regular expressions are case-insensitive.
218
+ - `/regex/` - every rule that matches this regular expression, will match the
219
+ rule. By default, regular expressions are case-insensitive.
160
220
  - `! comment` - comments will be ignored.
161
221
 
162
222
  > [!IMPORTANT]
@@ -168,7 +228,7 @@ Here is an example:
168
228
 
169
229
  Rules in HOSTS syntax: `/hosts.txt`
170
230
 
171
- ```txt
231
+ ```text
172
232
  0.0.0.0 ads.example.com
173
233
  0.0.0.0 tracking.example1.com
174
234
  0.0.0.0 example.com
@@ -176,7 +236,7 @@ Rules in HOSTS syntax: `/hosts.txt`
176
236
 
177
237
  Exclusion rules in adblock syntax: `/exclusions.txt`
178
238
 
179
- ```txt
239
+ ```text
180
240
  ||example.com^
181
241
  ```
182
242
 
@@ -201,45 +261,29 @@ Configuration of the final list:
201
261
 
202
262
  Final filter output of `/hosts.txt` after applying the `Compress` transformation and exclusions:
203
263
 
204
- ```txt
264
+ ```text
205
265
  ||ads.example.com^
206
266
  ||tracking.example1.com^
207
267
  ```
208
268
 
209
269
  The last rule now `||example.com^` will correctly match the rule from the exclusion list and will be excluded.
210
270
 
211
- ### <a name="command-line"></a> Command-line
271
+ ## API
212
272
 
213
- Command-line arguments.
273
+ The library exports a single function: `compile(configuration)`. It compiles
274
+ the filter list and returns a `Promise` that resolves to an array of rule
275
+ strings. TypeScript declarations are included with the package.
214
276
 
215
- ```
216
- Usage: hostlist-compiler [options]
217
-
218
- Options:
219
- --version Show version number [boolean]
220
- --config, -c Path to the compiler configuration file [string] [required]
221
- --output, -o Path to the output file [string] [required]
222
- --verbose, -v Run with verbose logging [boolean]
223
- -h, --help Show help [boolean]
224
-
225
- Examples:
226
- hostlist-compiler -c config.json -o compile a blocklist and write the
227
- output.txt output to output.txt
228
- ```
229
-
230
- ### <a name="api"></a> API
231
-
232
- Install: `npm i @adguard/hostlist-compiler` or `yarn add @adguard/hostlist-compiler`
233
-
234
- #### JavaScript example:
277
+ ### JavaScript Example
235
278
 
236
279
  ```javascript
237
- const compile = require("@adguard/hostlist-compiler");
280
+ const compile = require('@adguard/hostlist-compiler');
281
+ const { writeFileSync } = require('fs');
238
282
 
239
283
  ;(async () => {
240
284
  // Compile filters
241
285
  const result = await compile({
242
- name: 'Your Hostlist',
286
+ name: 'My hostlist',
243
287
  sources: [
244
288
  {
245
289
  type: 'adblock',
@@ -251,11 +295,11 @@ const compile = require("@adguard/hostlist-compiler");
251
295
  });
252
296
 
253
297
  // Write to file
254
- writeFileSync('your-hostlist.txt', result.join('\n'));
298
+ writeFileSync('output.txt', result.join('\n'));
255
299
  })();
256
300
  ```
257
301
 
258
- #### TypeScript example:
302
+ ### TypeScript Example
259
303
 
260
304
  ```typescript
261
305
  import compile from '@adguard/hostlist-compiler';
@@ -264,7 +308,7 @@ import { writeFileSync } from 'fs';
264
308
  ;(async () => {
265
309
  // Compile filters
266
310
  const result = await compile({
267
- name: 'Your Hostlist',
311
+ name: 'My hostlist',
268
312
  sources: [
269
313
  {
270
314
  type: 'adblock',
@@ -276,7 +320,7 @@ import { writeFileSync } from 'fs';
276
320
  });
277
321
 
278
322
  // Write to file
279
- writeFileSync('your-hostlist.txt', result.join('\n'));
323
+ writeFileSync('output.txt', result.join('\n'));
280
324
  })();
281
325
  ```
282
326
 
@@ -289,7 +333,7 @@ import { writeFileSync } from 'fs';
289
333
  ;(async () => {
290
334
  // Configuration
291
335
  const config: HostlistCompilerConfiguration = {
292
- name: 'Your Hostlist',
336
+ name: 'My hostlist',
293
337
  sources: [
294
338
  {
295
339
  type: 'adblock',
@@ -304,11 +348,11 @@ import { writeFileSync } from 'fs';
304
348
  const result = await HostlistCompiler(config);
305
349
 
306
350
  // Write to file
307
- writeFileSync('your-hostlist.txt', result.join('\n'));
351
+ writeFileSync('output.txt', result.join('\n'));
308
352
  })();
309
353
  ```
310
354
 
311
- ## <a name="transformations"></a> Transformations
355
+ ## Transformations
312
356
 
313
357
  Here is the full list of transformations that are available:
314
358
 
@@ -336,7 +380,7 @@ This transformation converts all non-ASCII characters to their ASCII equivalents
336
380
 
337
381
  Original list:
338
382
 
339
- ```
383
+ ```text
340
384
  ||*.рус^
341
385
  ||*.कॉम^
342
386
  ||*.セール^
@@ -344,7 +388,7 @@ Original list:
344
388
 
345
389
  Here's what we will have after applying this transformation:
346
390
 
347
- ```
391
+ ```text
348
392
  ||*.xn--p1acf^
349
393
  ||*.xn--11b4c3d^
350
394
  ||*.xn--1qqw23a^
@@ -358,16 +402,16 @@ This is a very simple transformation that removes leading and trailing spaces/ta
358
402
 
359
403
  Original list:
360
404
 
361
- ```
405
+ ```text
362
406
  rule1
363
407
  rule2
364
408
  rule3
365
- rule4
409
+ rule4
366
410
  ```
367
411
 
368
412
  Here's what we will have after applying this transformation:
369
413
 
370
- ```
414
+ ```text
371
415
  rule1
372
416
  rule2
373
417
  rule3
@@ -385,12 +429,20 @@ This is a very simple transformation that simply removes comments (e.g. all rule
385
429
 
386
430
  Here's what it does:
387
431
 
388
- 1. It converts all rules to adblock-style rules. For instance, `0.0.0.0 example.org` will be converted to `||example.org^`.
389
- 2. It discards the rules that are now redundant because of other existing rules. For instance, `||example.org` blocks `example.org` and all it's subdomains, therefore additional rules for the subdomains are now redundant.
432
+ 1. It converts all rules to adblock-style rules. For instance,
433
+ `0.0.0.0 example.org` will be converted to `||example.org^`.
434
+ 2. It discards the rules that are now redundant because of other existing
435
+ rules. For instance, `||example.org` blocks `example.org` and all it's
436
+ subdomains, therefore additional rules for the subdomains are now
437
+ redundant.
390
438
 
391
439
  ### <a name="remove-modifiers"></a> RemoveModifiers
392
440
 
393
- By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will ignore rules with unsupported modifiers, and all of the modifiers listed here are unsupported. However, the rules with these modifiers are likely to be okay for DNS-level blocking, that's why you might want to remove them when importing rules from a traditional filter list.
441
+ By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will
442
+ ignore rules with unsupported modifiers, and all of the modifiers listed here
443
+ are unsupported. However, the rules with these modifiers are likely to be okay
444
+ for DNS-level blocking, that's why you might want to remove them when importing
445
+ rules from a traditional filter list.
394
446
 
395
447
  Here is the list of modifiers that will be removed:
396
448
 
@@ -406,7 +458,10 @@ Here is the list of modifiers that will be removed:
406
458
 
407
459
  ### <a name="invertallow"></a> InvertAllow
408
460
 
409
- This transformation converts blocking rules to "allow" rules. Note, that it does nothing to /etc/hosts rules (unless they were previously converted to adblock-style syntax by a different transformation, for example [Compress](#compress)).
461
+ This transformation converts blocking rules to "allow" rules. Note, that it
462
+ does nothing to /etc/hosts rules (unless they were previously converted to
463
+ adblock-style syntax by a different transformation, for example
464
+ [Compress](#compress)).
410
465
 
411
466
  There are two important notes about this transformation:
412
467
 
@@ -417,7 +472,7 @@ There are two important notes about this transformation:
417
472
 
418
473
  Original list:
419
474
 
420
- ```
475
+ ```text
421
476
  ! comment 1
422
477
  rule1
423
478
 
@@ -428,7 +483,7 @@ rule1
428
483
 
429
484
  Here's what we will have after applying this transformation:
430
485
 
431
- ```
486
+ ```text
432
487
  ! comment 1
433
488
  @@rule1
434
489
 
@@ -439,36 +494,47 @@ Here's what we will have after applying this transformation:
439
494
 
440
495
  ### <a name="validate"></a> Validate
441
496
 
442
- This transformation is really crucial if you're using a filter list for a traditional ad blocker as a source.
497
+ This transformation is really crucial if you're using a filter list for a
498
+ traditional ad blocker as a source.
443
499
 
444
500
  It removes dangerous or incompatible rules from the list.
445
501
 
446
502
  So here's what it does:
447
503
 
448
- - Discards domain-specific rules (e.g. `||example.org^$domain=example.com`). You don't want to have domain-specific rules working globally.
449
- - Discards rules with unsupported modifiers. [Click here](https://github.com/AdguardTeam/AdGuardHome/wiki/Hosts-Blocklists#-adblock-style-syntax) to learn more about which modifiers are supported.
504
+ - Discards domain-specific rules (e.g. `||example.org^$domain=example.com`).
505
+ You don't want to have domain-specific rules working globally.
506
+ - Discards rules with unsupported modifiers. Learn more about the
507
+ [supported modifiers](https://github.com/AdguardTeam/AdGuardHome/wiki/Hosts-Blocklists#-adblock-style-syntax).
450
508
  - Discards rules that are too short.
451
509
  - Discards IP addresses. If you need to keep IP addresses, use [ValidateAllowIp](#validate-allow-ip) instead.
452
510
 
453
511
  #### <a name="rejected-ip-patterns"></a>Rejected IP Patterns
454
512
 
455
- The following IP patterns are rejected by all validation transformations (`Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`, `ValidateAllowIpAndPublicSuffix`) as they are either unsafe or ambiguous:
513
+ The following IP patterns are rejected by all validation transformations
514
+ (`Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`,
515
+ `ValidateAllowIpAndPublicSuffix`) as they are either unsafe or ambiguous:
456
516
 
457
- - `||192.168.1^` — 3-octet with `^` - does not work
458
- - `192.168.1` — Ambiguous: would match `192.168.11`, `192.168.111`, etc.
459
- - `1.2.` or `1.2.*` — Too wide (1-2 octets), use regex instead
517
+ - `||192.168.1^` — 3-octet with `^` - does not work
518
+ - `192.168.1` — Ambiguous: would match `192.168.11`, `192.168.111`, etc.
519
+ - `1.2.` or `1.2.*` — Too wide (1-2 octets), use regex instead
460
520
 
461
- - Removes rules that block entire top-level domains (TLDs) like `||*.org^`, unless they have specific limiting modifiers such as `$denyallow`, `$badfilter`, or `$client`.
462
- Examples:
463
- - `||*.org^` - this rule will be removed
464
- - `||*.org^$denyallow=example.com` - this rule will be kept because it has a limiting modifier
465
- If such rules must be saved, use [ValidateAllowPublicSuffix](#validate-allow-public-suffix) or [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix).
521
+ - Removes rules that block entire top-level domains (TLDs) like `||*.org^`,
522
+ unless they have specific limiting modifiers such as `$denyallow`,
523
+ `$badfilter`, or `$client`. Examples:
524
+ - `||*.org^` - this rule will be removed
525
+ - `||*.org^$denyallow=example.com` - this rule will be kept because it
526
+ has a limiting modifier
527
+ If such rules must be saved, use
528
+ [ValidateAllowPublicSuffix](#validate-allow-public-suffix) or
529
+ [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix).
466
530
 
467
531
  If there are comments preceding the invalid rule, they will be removed as well.
468
532
 
469
533
  ### <a name="validate-allow-ip"></a> ValidateAllowIp
470
534
 
471
- This transformation extends [Validate](#validate) to allow IP addresses in the lists. It also **normalizes IP rules** in Adblock-style to the safe format `||ip^`.
535
+ This transformation extends [Validate](#validate) to allow IP addresses in
536
+ the lists. It also **normalizes IP rules** in Adblock-style to the safe format
537
+ `||ip^`.
472
538
 
473
539
  #### IP Rule Normalization
474
540
 
@@ -483,25 +549,47 @@ This transformation extends [Validate](#validate) to allow IP addresses in the l
483
549
 
484
550
  Modifiers like `$important`, `$client`, `$denyallow`, `$badfilter` are preserved during normalization.
485
551
 
486
- > **Note:** Invalid IP patterns are rejected solely by the base [Validate](#validate) logic — normalization only converts valid patterns to canonical form and passes everything else through unchanged. See [Rejected IP Patterns](#rejected-ip-patterns) for details.
552
+ > **Note:** Invalid IP patterns are rejected solely by the base
553
+ > [Validate](#validate) logic — normalization only converts valid patterns to
554
+ > canonical form and passes everything else through unchanged. See
555
+ > [Rejected IP Patterns](#rejected-ip-patterns) for details.
487
556
 
488
557
  ### <a name="validate-allow-public-suffix"></a> ValidateAllowPublicSuffix
489
558
 
490
- This transformation exactly repeats the behavior of [Validate](#validate), but leaves rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`) in the list.
559
+ This transformation exactly repeats the behavior of [Validate](#validate),
560
+ but leaves rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`)
561
+ in the list.
491
562
 
492
- It still filters out invalid syntax rules and unsupported modifiers, but does not reject public-suffix rules unless the rule itself is malformed.
563
+ It still filters out invalid syntax rules and unsupported modifiers, but does
564
+ not reject public-suffix rules unless the rule itself is malformed.
493
565
 
494
- > **Note:** Combining any `Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`, and `ValidateAllowIpAndPublicSuffix` in one transformation list is not allowed and will result in an error. Each runs its own validator on the already-filtered output of the previous one, so allow-modes become silently ineffective.
566
+ > **Note:** Combining any `Validate`, `ValidateAllowIp`,
567
+ > `ValidateAllowPublicSuffix`, and `ValidateAllowIpAndPublicSuffix` in one
568
+ > transformation list is not allowed and will result in an error. Each runs
569
+ > its own validator on the already-filtered output of the previous one, so
570
+ > allow-modes become silently ineffective.
495
571
 
496
- > **Important:** Validation transformations also cannot be used at both source-level and top-level simultaneously. For example, if a source uses `ValidateAllowPublicSuffix` and the top-level configuration uses `Validate`, the compiler will throw an error. This is because the top-level `Validate` would override the source-level validation, making `ValidateAllowPublicSuffix` ineffective. Use validation transformations at only one level.
572
+ > **Important:** Validation transformations also cannot be used at both
573
+ > source-level and top-level simultaneously. For example, if a source uses
574
+ > `ValidateAllowPublicSuffix` and the top-level configuration uses `Validate`,
575
+ > the compiler will throw an error. This is because the top-level `Validate`
576
+ > would override the source-level validation, making
577
+ > `ValidateAllowPublicSuffix` ineffective. Use validation transformations at
578
+ > only one level.
497
579
 
498
580
  ### <a name="validate-allow-ip-and-public-suffix"></a> ValidateAllowIpAndPublicSuffix
499
581
 
500
- This transformation combines the behavior of [ValidateAllowIp](#validate-allow-ip) and [ValidateAllowPublicSuffix](#validate-allow-public-suffix). It allows both IP addresses and public suffix rules in the list.
582
+ This transformation combines the behavior of
583
+ [ValidateAllowIp](#validate-allow-ip) and
584
+ [ValidateAllowPublicSuffix](#validate-allow-public-suffix). It allows both IP
585
+ addresses and public suffix rules in the list.
501
586
 
502
- Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format `||ip^` before validation. See [IP Rule Normalization](#ip-rule-normalization) for details.
587
+ Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format
588
+ `||ip^` before validation. See [IP Rule Normalization](#ip-rule-normalization)
589
+ for details.
503
590
 
504
- Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`).
591
+ Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public
592
+ suffixes (e.g. `||hl.cn^`, `||org^`).
505
593
 
506
594
  ### <a name="deduplicate"></a> Deduplicate
507
595
 
@@ -510,11 +598,12 @@ This transformation simply removes the duplicates from the specified source.
510
598
  There are two important notes about this transformation:
511
599
 
512
600
  1. It keeps the original rules order.
513
- 2. It ignores comments. However, if the comments precede the rule that is being removed, the comments will be also removed.
601
+ 2. It ignores comments. However, if the comments precede the rule that is
602
+ being removed, the comments will be also removed.
514
603
 
515
604
  For instance:
516
605
 
517
- ```
606
+ ```text
518
607
  ! rule1 comment 1
519
608
  rule1
520
609
  ! rule1 comment 2
@@ -523,7 +612,7 @@ rule1
523
612
 
524
613
  Here's what will be left after the transformation:
525
614
 
526
- ```
615
+ ```text
527
616
  ! rule1 comment 2
528
617
  rule1
529
618
  ```
@@ -536,7 +625,7 @@ This is a very simple transformation that removes empty lines.
536
625
 
537
626
  Original list:
538
627
 
539
- ```
628
+ ```text
540
629
  rule1
541
630
 
542
631
  rule2
@@ -547,7 +636,7 @@ rule3
547
636
 
548
637
  Here's what we will have after applying this transformation:
549
638
 
550
- ```
639
+ ```text
551
640
  rule1
552
641
  rule2
553
642
  rule3
@@ -561,7 +650,7 @@ This is a very simple transformation that inserts a final newline.
561
650
 
562
651
  Original list:
563
652
 
564
- ```
653
+ ```text
565
654
  rule1
566
655
  rule2
567
656
  rule3
@@ -569,7 +658,7 @@ rule3
569
658
 
570
659
  Here's what we will have after applying this transformation:
571
660
 
572
- ```
661
+ ```text
573
662
  rule1
574
663
  rule2
575
664
  rule3
@@ -578,8 +667,14 @@ rule3
578
667
 
579
668
  `RemoveEmptyLines` doesn't delete this empty row due to the execution order.
580
669
 
581
- ## <a name="documentation"></a> Documentation
670
+ ## Documentation
582
671
 
583
- - [Development guide](DEVELOPMENT.md) — how to set up the environment, build, test, and contribute
672
+ - [Development](DEVELOPMENT.md) — how to set up the environment, build, test, and contribute
673
+ - [Deployment and configuration](DEPLOYMENT.md) — release pipeline, CI/CD, and Docker build
584
674
  - [LLM agent rules](AGENTS.md) — code guidelines and project conventions
585
675
  - [Changelog](CHANGELOG.md) — release history
676
+
677
+ [npm-badge]: https://nodei.co/npm/@adguard/hostlist-compiler.png?compact=true
678
+ [npm]: https://www.npmjs.com/package/@adguard/hostlist-compiler/
679
+ [private-repo]: https://github.com/AdGuardSoftwareLimited/filters-hostlist-compiler
680
+ [public-mirror]: https://github.com/AdguardTeam/HostlistCompiler