@adguard/hostlist-compiler 2.1.0 → 3.0.0-beta.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (58) hide show
  1. package/.dockerignore +29 -0
  2. package/.github/workflows/ci.yml +22 -0
  3. package/.github/workflows/mirror.yml +24 -0
  4. package/.github/workflows/prepare-release.yml +26 -0
  5. package/.github/workflows/publish-release.yml +44 -0
  6. package/.markdownlint.json +29 -0
  7. package/.markdownlintignore +1 -0
  8. package/AGENTS.md +314 -103
  9. package/CHANGELOG.md +76 -7
  10. package/DEPLOYMENT.md +171 -0
  11. package/DEVELOPMENT.md +174 -76
  12. package/README.md +240 -140
  13. package/examples/china/README.md +2 -2
  14. package/examples/energized/README.md +4 -2
  15. package/examples/sdn/README.md +2 -2
  16. package/examples/whitelist/README.md +2 -2
  17. package/package.json +29 -24
  18. package/src/cli.js +59 -33
  19. package/src/compile-source.js +4 -3
  20. package/src/configuration.js +132 -2
  21. package/src/download.js +27 -0
  22. package/src/filter.js +2 -2
  23. package/src/index.d.ts +2 -0
  24. package/src/index.js +21 -51
  25. package/src/{transformations/ip-normalize.js → ip-normalize.js} +2 -2
  26. package/src/schemas/configuration.schema.js +137 -0
  27. package/src/source-types.js +17 -0
  28. package/src/transformations/enum.js +49 -0
  29. package/src/transformations/include.js +1 -1
  30. package/src/transformations/transform.js +40 -74
  31. package/src/transformations/validate-allow-ip-and-public-suffix.js +1 -1
  32. package/src/transformations/validate-allow-ip.js +1 -1
  33. package/src/utils.js +1284 -0
  34. package/src/validation-conflicts.js +106 -0
  35. package/test/cli-error-output.test.js +305 -0
  36. package/test/cli.test.js +47 -0
  37. package/test/compile-source.test.js +167 -7
  38. package/test/configuration.test.js +438 -0
  39. package/test/download.test.js +260 -0
  40. package/test/helpers/http-mock.js +94 -0
  41. package/test/index.test.js +497 -4
  42. package/test/{transformations/ip-normalize.test.js → ip-normalize.test.js} +1 -1
  43. package/test/resolve-include.test.js +8 -8
  44. package/test/transformations/covert-to-ascii.test.js +1 -1
  45. package/test/transformations/exclude.test.js +2 -2
  46. package/test/transformations/include.test.js +9 -3
  47. package/test/transformations/ip-alignment.test.js +1 -1
  48. package/test/transformations/transform.test.js +66 -1
  49. package/test/utils.test.js +1559 -0
  50. package/test/validation-conflicts.test.js +230 -0
  51. package/bamboo-specs/bamboo.yaml +0 -14
  52. package/bamboo-specs/build.yaml +0 -92
  53. package/bamboo-specs/deploy.yaml +0 -38
  54. package/bamboo-specs/increment.yaml +0 -54
  55. package/bamboo-specs/permissions.yaml +0 -16
  56. package/bamboo-specs/test.yaml +0 -56
  57. package/src/schemas/configuration.schema.json +0 -369
  58. package/tools/build-txt.js +0 -19
package/README.md CHANGED
@@ -1,79 +1,130 @@
1
- # Hostlist compiler
1
+ # Hostlist Compiler
2
2
 
3
- [![NPM](https://nodei.co/npm/@adguard/hostlist-compiler.png?compact=true)](https://www.npmjs.com/package/@adguard/hostlist-compiler/)
3
+ [![NPM][npm-badge]][npm]
4
4
 
5
- This is a simple tool that makes it easier to compile a [hosts blocklist](https://adguard-dns.io/kb/general/dns-filtering-syntax/) compatible with AdGuard Home or any other AdGuard product with **DNS filtering**.
5
+ ## Description
6
6
 
7
+ Hostlist Compiler is a Node.js CLI tool and library for filter list maintainers and developers. It
8
+ compiles DNS hosts blocklists from multiple sources — `/etc/hosts` files and adblock-style filter
9
+ lists — into a single filter list compatible with AdGuard Home and other AdGuard products with DNS
10
+ filtering.
11
+
12
+ Compiling a blocklist by hand is error-prone: sources use different syntaxes, rules must be
13
+ deduplicated, dangerous or unsupported rules have to be removed, and lists need to be filtered
14
+ through exclusion and inclusion rules. Hostlist Compiler automates this pipeline — it downloads
15
+ or reads the configured sources, converts `/etc/hosts` rules to AdGuard syntax, resolves
16
+ `!#include` directives, applies a fixed-order pipeline of transformations, and writes the
17
+ resulting filter list.
18
+
19
+ > **Note on repositories:** Active development happens in the private
20
+ > [AdGuardSoftwareLimited/filters-hostlist-compiler][private-repo] repository;
21
+ > this public [AdguardTeam/HostlistCompiler][public-mirror] repository is a
22
+ > read-only mirror that is updated automatically from it.
23
+
24
+ ## Table of Contents
25
+
26
+ - [Installation](#installation)
27
+ - [Quick Start](#quick-start)
7
28
  - [Usage](#usage)
8
- - [Configuration](#configuration)
9
- - [Command-line](#command-line)
10
- - [API](#api)
29
+ - [Configuration](#configuration)
11
30
  - [Transformations](#transformations)
12
- - [ConvertToAscii](#convert-to-ascii)
13
- - [TrimLines](#trimlines)
14
- - [RemoveComments](#remove-comments)
15
- - [Compress](#compress)
16
- - [RemoveModifiers](#remove-modifiers)
17
- - [InvertAllow](#invertallow)
18
- - [Validate](#validate)
19
- - [ValidateAllowIp](#validate-allow-ip)
20
- - [ValidateAllowPublicSuffix](#validate-allow-public-suffix)
21
- - [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix)
22
- - [Deduplicate](#deduplicate)
23
- - [RemoveEmptyLines](#removeemptylines)
24
- - [InsertFinalNewLine](#insertfinalnewline)
31
+ - [API](#api)
25
32
  - [Documentation](#documentation)
26
33
 
27
- ## <a name="usage"></a> Usage
34
+ ---
28
35
 
29
- First of all, install the `hostlist-compiler`:
36
+ ## Installation
37
+
38
+ Requires Node.js 22.22.2 or newer. Node.js 23.x is not supported by
39
+ `@adguard/filters-downloader` 3.x.
40
+
41
+ Install the package globally to get the `hostlist-compiler` command:
30
42
 
31
43
  ```bash
32
44
  npm i -g @adguard/hostlist-compiler
33
45
  ```
34
46
 
35
- After that you have two options.
47
+ Or install it as a dependency to use the compiler as a library:
48
+
49
+ ```bash
50
+ npm i @adguard/hostlist-compiler
51
+ ```
52
+
53
+ yarn is also supported:
54
+
55
+ ```bash
56
+ yarn add @adguard/hostlist-compiler
57
+ ```
58
+
59
+ ## Quick Start
60
+
61
+ Convert a `/etc/hosts`-syntax blocklist to an AdGuard-compatible filter list in one command:
36
62
 
37
- **Quick hosts conversion**
63
+ ```bash
64
+ hostlist-compiler -i hosts.txt -o output.txt
65
+ ```
38
66
 
39
- Convert and compress a `/etc/hosts`-syntax blocklist to [AdGuard syntax](https://adguard-dns.io/kb/general/dns-filtering-syntax/).
67
+ Or compile a configurable blocklist from multiple sources:
40
68
 
69
+ ```bash
70
+ hostlist-compiler -c configuration.json -o output.txt
41
71
  ```
72
+
73
+ ## Usage
74
+
75
+ ### Command-Line Options
76
+
77
+ | Flag | Description |
78
+ | --------------------------- | -------------------------------------------------------------------------------------------------- |
79
+ | `--config, -c <path>` | Path to the compiler configuration file |
80
+ | `--input, -i <url-or-path>` | URL (or path to a file) to convert to an AdGuard-syntax blocklist. Can be specified multiple times |
81
+ | `--input-type, -t <type>` | Type of the input file: `hosts` or `adblock` |
82
+ | `--output, -o <path>` | Path to the output file (required) |
83
+ | `--verbose, -v` | Run with verbose logging |
84
+ | `--version` | Show the version number |
85
+ | `-h, --help` | Show help |
86
+
87
+ The compiler exits with code `0` on success and `1` on failure; errors are printed to stderr.
88
+
89
+ ### Proxy
90
+
91
+ To download sources through a proxy, set `NODE_USE_ENV_PROXY=1` together with
92
+ the `HTTP_PROXY` (also used for `https://` when `HTTPS_PROXY` is unset),
93
+ `HTTPS_PROXY`, and `NO_PROXY` environment variables. The Node.js native
94
+ `fetch` ignores these variables unless `NODE_USE_ENV_PROXY=1` is set.
95
+
96
+ ### Quick Hosts Conversion
97
+
98
+ Convert and compress one or more `/etc/hosts`-syntax blocklists to
99
+ [AdGuard syntax](https://adguard-dns.io/kb/general/dns-filtering-syntax/):
100
+
101
+ ```bash
42
102
  hostlist-compiler -i hosts.txt -i hosts2.txt -o output.txt
43
103
  ```
44
104
 
45
- **Build a configurable blocklist from multiple sources**
105
+ The quick conversion applies a default set of transformations to convert
106
+ `/etc/hosts` rules to AdGuard syntax: `RemoveComments`, `Deduplicate`,
107
+ `Compress`, `Validate`, `TrimLines`, `InsertFinalNewLine`, applied in the
108
+ order described in the [Transformations](#transformations) section. To
109
+ control the pipeline explicitly, use a [configuration file](#configuration)
110
+ instead.
111
+
112
+ ### Build a Configurable Blocklist from Multiple Sources
46
113
 
47
- Prepare the list configuration (read more about that [below](#configuration)) and run the compiler:
114
+ Prepare the list configuration (read more about that
115
+ [below](#configuration)) and run the compiler:
48
116
 
49
117
  ```bash
50
118
  hostlist-compiler -c configuration.json -o output.txt
51
119
  ```
52
120
 
53
- **All command line options**
121
+ A configurable blocklist can also be built from a remote URL without a configuration file:
54
122
 
55
- ```
56
- Usage: hostlist-compiler [options]
57
-
58
- Options:
59
- --config, -c Path to the compiler configuration file [string]
60
- --input, -i URL (or path to a file) to convert to an AdGuard-syntax
61
- blocklist. Can be specified multiple times. [array]
62
- --input-type, -t Type of the input file (/etc/hosts, adguard) [string]
63
- --output, -o Path to the output file [string] [required]
64
- --verbose, -v Run with verbose logging [boolean]
65
- --version Show version number [boolean]
66
- -h, --help Show help [boolean]
67
-
68
- Examples:
69
- hostlist-compiler -c config.json -o compile a blocklist and write the
70
- output.txt output to output.txt
71
- hostlist-compiler -i compile a blocklist from the URL and
72
- https://example.org/hosts.txt -o write the output to output.txt
73
- output.txt
123
+ ```bash
124
+ hostlist-compiler -i https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -o output.txt
74
125
  ```
75
126
 
76
- ### <a name="configuration"></a> Configuration
127
+ ## Configuration
77
128
 
78
129
  Configuration defines your filter list sources, and the transformations that are applied to the sources.
79
130
 
@@ -83,7 +134,7 @@ Here is an example of this configuration:
83
134
  {
84
135
  "name": "List name",
85
136
  "description": "List description",
86
- "homepage": "https://example.org/",
137
+ "homepage": "https://github.com/AdguardTeam/AdguardSDNSFilter",
87
138
  "license": "GPLv3",
88
139
  "version": "1.0.0.0",
89
140
  "sources": [
@@ -99,8 +150,8 @@ Here is an example of this configuration:
99
150
  },
100
151
  {
101
152
  "name": "Remote rules",
102
- "source": "https://example.org/rules",
103
- "type": "hosts",
153
+ "source": "https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt",
154
+ "type": "adblock",
104
155
  "exclusions": ["excluded rule 1"]
105
156
  }
106
157
  ],
@@ -118,19 +169,31 @@ Here is an example of this configuration:
118
169
  - `license` - (optional) Filter list license.
119
170
  - `version` - (optional) Filter list version.
120
171
  - `sources` - (mandatory) array of the list sources.
121
- - `.source` - (mandatory) path or URL of the source. It can be a traditional filter list or a hosts file.
122
- - `.name` - (optional) name of the source.
123
- - `.type` - (optional) type of the source. It could be `adblock` for Adblock-style lists or `hosts` for /etc/hosts style lists. If not specified, `adblock` is assumed.
124
- - `.transformations` - (optional) a list of transformations to apply to the source rules. By default, **no transformations** are applied. Learn more about possible transformations [here](#transformations).
125
- - `.exclusions` - (optional) a list of rules (or wildcards) to exclude from the source.
126
- - `.exclusions_sources` - (optional) a list of files with exclusions.
127
- - `.inclusions` - (optional) a list of wildcards to include from the source. All rules that don't match these wildcards won't be included.
128
- - `.inclusions_sources` - (optional) a list of files with inclusions.
129
- - `transformations` - (optional) a list of transformations to apply to the final list of rules. By default, **no transformations** are applied. Learn more about possible transformations [here](#transformations).
130
- - `exclusions` - (optional) a list of rules (or wildcards) to exclude from the source.
172
+ - `.source` - (mandatory) path or URL of the source. It can be a
173
+ traditional filter list or a hosts file.
174
+ - `.name` - (optional) name of the source.
175
+ - `.type` - (optional) type of the source. It can be `adblock` for
176
+ Adblock-style lists or `hosts` for /etc/hosts style lists. The value is
177
+ validated against these two options; rule parsing auto-detects the
178
+ format, so the type does not affect parsing.
179
+ - `.transformations` - (optional) a list of transformations to apply to
180
+ the source rules. By default, **no transformations** are applied. Learn
181
+ more about the possible [transformations](#transformations).
182
+ - `.exclusions` - (optional) a list of rules (or wildcards) to exclude
183
+ from the source.
184
+ - `.exclusions_sources` - (optional) a list of files with exclusions.
185
+ - `.inclusions` - (optional) a list of wildcards to include from the
186
+ source. All rules that don't match these wildcards won't be included.
187
+ - `.inclusions_sources` - (optional) a list of files with inclusions.
188
+ - `transformations` - (optional) a list of transformations to apply to the
189
+ final list of rules. By default, **no transformations** are applied. Learn
190
+ more about the possible [transformations](#transformations).
191
+ - `exclusions` - (optional) a list of rules (or wildcards) to exclude from
192
+ the final list.
131
193
  - `exclusions_sources` - (optional) a list of files with exclusions.
132
- - `.inclusions` - (optional) a list of wildcards to include from the source. All rules that don't match these wildcards won't be included.
133
- - `.inclusions_sources` - (optional) a list of files with inclusions.
194
+ - `inclusions` - (optional) a list of wildcards to include in the final
195
+ list. All rules that don't match these wildcards won't be included.
196
+ - `inclusions_sources` - (optional) a list of files with inclusions.
134
197
 
135
198
  Here is an example of a minimal configuration:
136
199
 
@@ -145,13 +208,15 @@ Here is an example of a minimal configuration:
145
208
  }
146
209
  ```
147
210
 
148
- **Exclusion and inclusion rules**
211
+ ### Exclusion and Inclusion Rules
149
212
 
150
- Please note, that exclusion or inclusion rules may be a plain string, wildcard, or a regular expression.
213
+ Please note, that exclusion or inclusion rules may be a plain string,
214
+ wildcard, or a regular expression.
151
215
 
152
216
  - `plainstring` - every rule that contains `plainstring` will match the rule
153
217
  - `*.plainstring` - every rule that matches this wildcard will match the rule
154
- - `/regex/` - every rule that matches this regular expression, will match the rule. By default, regular expressions are case-insensitive.
218
+ - `/regex/` - every rule that matches this regular expression, will match the
219
+ rule. By default, regular expressions are case-insensitive.
155
220
  - `! comment` - comments will be ignored.
156
221
 
157
222
  > [!IMPORTANT]
@@ -163,7 +228,7 @@ Here is an example:
163
228
 
164
229
  Rules in HOSTS syntax: `/hosts.txt`
165
230
 
166
- ```txt
231
+ ```text
167
232
  0.0.0.0 ads.example.com
168
233
  0.0.0.0 tracking.example1.com
169
234
  0.0.0.0 example.com
@@ -171,7 +236,7 @@ Rules in HOSTS syntax: `/hosts.txt`
171
236
 
172
237
  Exclusion rules in adblock syntax: `/exclusions.txt`
173
238
 
174
- ```txt
239
+ ```text
175
240
  ||example.com^
176
241
  ```
177
242
 
@@ -196,45 +261,29 @@ Configuration of the final list:
196
261
 
197
262
  Final filter output of `/hosts.txt` after applying the `Compress` transformation and exclusions:
198
263
 
199
- ```txt
264
+ ```text
200
265
  ||ads.example.com^
201
266
  ||tracking.example1.com^
202
267
  ```
203
268
 
204
269
  The last rule now `||example.com^` will correctly match the rule from the exclusion list and will be excluded.
205
270
 
206
- ### <a name="command-line"></a> Command-line
207
-
208
- Command-line arguments.
209
-
210
- ```
211
- Usage: hostlist-compiler [options]
212
-
213
- Options:
214
- --version Show version number [boolean]
215
- --config, -c Path to the compiler configuration file [string] [required]
216
- --output, -o Path to the output file [string] [required]
217
- --verbose, -v Run with verbose logging [boolean]
218
- -h, --help Show help [boolean]
219
-
220
- Examples:
221
- hostlist-compiler -c config.json -o compile a blocklist and write the
222
- output.txt output to output.txt
223
- ```
224
-
225
- ### <a name="api"></a> API
271
+ ## API
226
272
 
227
- Install: `npm i @adguard/hostlist-compiler` or `yarn add @adguard/hostlist-compiler`
273
+ The library exports a single function: `compile(configuration)`. It compiles
274
+ the filter list and returns a `Promise` that resolves to an array of rule
275
+ strings. TypeScript declarations are included with the package.
228
276
 
229
- #### JavaScript example:
277
+ ### JavaScript Example
230
278
 
231
279
  ```javascript
232
- const compile = require("@adguard/hostlist-compiler");
280
+ const compile = require('@adguard/hostlist-compiler');
281
+ const { writeFileSync } = require('fs');
233
282
 
234
283
  ;(async () => {
235
284
  // Compile filters
236
285
  const result = await compile({
237
- name: 'Your Hostlist',
286
+ name: 'My hostlist',
238
287
  sources: [
239
288
  {
240
289
  type: 'adblock',
@@ -246,11 +295,11 @@ const compile = require("@adguard/hostlist-compiler");
246
295
  });
247
296
 
248
297
  // Write to file
249
- writeFileSync('your-hostlist.txt', result.join('\n'));
298
+ writeFileSync('output.txt', result.join('\n'));
250
299
  })();
251
300
  ```
252
301
 
253
- #### TypeScript example:
302
+ ### TypeScript Example
254
303
 
255
304
  ```typescript
256
305
  import compile from '@adguard/hostlist-compiler';
@@ -259,7 +308,7 @@ import { writeFileSync } from 'fs';
259
308
  ;(async () => {
260
309
  // Compile filters
261
310
  const result = await compile({
262
- name: 'Your Hostlist',
311
+ name: 'My hostlist',
263
312
  sources: [
264
313
  {
265
314
  type: 'adblock',
@@ -271,7 +320,7 @@ import { writeFileSync } from 'fs';
271
320
  });
272
321
 
273
322
  // Write to file
274
- writeFileSync('your-hostlist.txt', result.join('\n'));
323
+ writeFileSync('output.txt', result.join('\n'));
275
324
  })();
276
325
  ```
277
326
 
@@ -284,7 +333,7 @@ import { writeFileSync } from 'fs';
284
333
  ;(async () => {
285
334
  // Configuration
286
335
  const config: HostlistCompilerConfiguration = {
287
- name: 'Your Hostlist',
336
+ name: 'My hostlist',
288
337
  sources: [
289
338
  {
290
339
  type: 'adblock',
@@ -299,11 +348,11 @@ import { writeFileSync } from 'fs';
299
348
  const result = await HostlistCompiler(config);
300
349
 
301
350
  // Write to file
302
- writeFileSync('your-hostlist.txt', result.join('\n'));
351
+ writeFileSync('output.txt', result.join('\n'));
303
352
  })();
304
353
  ```
305
354
 
306
- ## <a name="transformations"></a> Transformations
355
+ ## Transformations
307
356
 
308
357
  Here is the full list of transformations that are available:
309
358
 
@@ -331,7 +380,7 @@ This transformation converts all non-ASCII characters to their ASCII equivalents
331
380
 
332
381
  Original list:
333
382
 
334
- ```
383
+ ```text
335
384
  ||*.рус^
336
385
  ||*.कॉम^
337
386
  ||*.セール^
@@ -339,7 +388,7 @@ Original list:
339
388
 
340
389
  Here's what we will have after applying this transformation:
341
390
 
342
- ```
391
+ ```text
343
392
  ||*.xn--p1acf^
344
393
  ||*.xn--11b4c3d^
345
394
  ||*.xn--1qqw23a^
@@ -353,16 +402,16 @@ This is a very simple transformation that removes leading and trailing spaces/ta
353
402
 
354
403
  Original list:
355
404
 
356
- ```
405
+ ```text
357
406
  rule1
358
407
  rule2
359
408
  rule3
360
- rule4
409
+ rule4
361
410
  ```
362
411
 
363
412
  Here's what we will have after applying this transformation:
364
413
 
365
- ```
414
+ ```text
366
415
  rule1
367
416
  rule2
368
417
  rule3
@@ -380,12 +429,20 @@ This is a very simple transformation that simply removes comments (e.g. all rule
380
429
 
381
430
  Here's what it does:
382
431
 
383
- 1. It converts all rules to adblock-style rules. For instance, `0.0.0.0 example.org` will be converted to `||example.org^`.
384
- 2. It discards the rules that are now redundant because of other existing rules. For instance, `||example.org` blocks `example.org` and all it's subdomains, therefore additional rules for the subdomains are now redundant.
432
+ 1. It converts all rules to adblock-style rules. For instance,
433
+ `0.0.0.0 example.org` will be converted to `||example.org^`.
434
+ 2. It discards the rules that are now redundant because of other existing
435
+ rules. For instance, `||example.org` blocks `example.org` and all it's
436
+ subdomains, therefore additional rules for the subdomains are now
437
+ redundant.
385
438
 
386
439
  ### <a name="remove-modifiers"></a> RemoveModifiers
387
440
 
388
- By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will ignore rules with unsupported modifiers, and all of the modifiers listed here are unsupported. However, the rules with these modifiers are likely to be okay for DNS-level blocking, that's why you might want to remove them when importing rules from a traditional filter list.
441
+ By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will
442
+ ignore rules with unsupported modifiers, and all of the modifiers listed here
443
+ are unsupported. However, the rules with these modifiers are likely to be okay
444
+ for DNS-level blocking, that's why you might want to remove them when importing
445
+ rules from a traditional filter list.
389
446
 
390
447
  Here is the list of modifiers that will be removed:
391
448
 
@@ -401,7 +458,10 @@ Here is the list of modifiers that will be removed:
401
458
 
402
459
  ### <a name="invertallow"></a> InvertAllow
403
460
 
404
- This transformation converts blocking rules to "allow" rules. Note, that it does nothing to /etc/hosts rules (unless they were previously converted to adblock-style syntax by a different transformation, for example [Compress](#compress)).
461
+ This transformation converts blocking rules to "allow" rules. Note, that it
462
+ does nothing to /etc/hosts rules (unless they were previously converted to
463
+ adblock-style syntax by a different transformation, for example
464
+ [Compress](#compress)).
405
465
 
406
466
  There are two important notes about this transformation:
407
467
 
@@ -412,7 +472,7 @@ There are two important notes about this transformation:
412
472
 
413
473
  Original list:
414
474
 
415
- ```
475
+ ```text
416
476
  ! comment 1
417
477
  rule1
418
478
 
@@ -423,7 +483,7 @@ rule1
423
483
 
424
484
  Here's what we will have after applying this transformation:
425
485
 
426
- ```
486
+ ```text
427
487
  ! comment 1
428
488
  @@rule1
429
489
 
@@ -434,36 +494,47 @@ Here's what we will have after applying this transformation:
434
494
 
435
495
  ### <a name="validate"></a> Validate
436
496
 
437
- This transformation is really crucial if you're using a filter list for a traditional ad blocker as a source.
497
+ This transformation is really crucial if you're using a filter list for a
498
+ traditional ad blocker as a source.
438
499
 
439
500
  It removes dangerous or incompatible rules from the list.
440
501
 
441
502
  So here's what it does:
442
503
 
443
- - Discards domain-specific rules (e.g. `||example.org^$domain=example.com`). You don't want to have domain-specific rules working globally.
444
- - Discards rules with unsupported modifiers. [Click here](https://github.com/AdguardTeam/AdGuardHome/wiki/Hosts-Blocklists#-adblock-style-syntax) to learn more about which modifiers are supported.
504
+ - Discards domain-specific rules (e.g. `||example.org^$domain=example.com`).
505
+ You don't want to have domain-specific rules working globally.
506
+ - Discards rules with unsupported modifiers. Learn more about the
507
+ [supported modifiers](https://github.com/AdguardTeam/AdGuardHome/wiki/Hosts-Blocklists#-adblock-style-syntax).
445
508
  - Discards rules that are too short.
446
509
  - Discards IP addresses. If you need to keep IP addresses, use [ValidateAllowIp](#validate-allow-ip) instead.
447
510
 
448
511
  #### <a name="rejected-ip-patterns"></a>Rejected IP Patterns
449
512
 
450
- The following IP patterns are rejected by all validation transformations (`Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`, `ValidateAllowIpAndPublicSuffix`) as they are either unsafe or ambiguous:
513
+ The following IP patterns are rejected by all validation transformations
514
+ (`Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`,
515
+ `ValidateAllowIpAndPublicSuffix`) as they are either unsafe or ambiguous:
451
516
 
452
- - `||192.168.1^` — 3-octet with `^` - does not work
453
- - `192.168.1` — Ambiguous: would match `192.168.11`, `192.168.111`, etc.
454
- - `1.2.` or `1.2.*` — Too wide (1-2 octets), use regex instead
517
+ - `||192.168.1^` — 3-octet with `^` - does not work
518
+ - `192.168.1` — Ambiguous: would match `192.168.11`, `192.168.111`, etc.
519
+ - `1.2.` or `1.2.*` — Too wide (1-2 octets), use regex instead
455
520
 
456
- - Removes rules that block entire top-level domains (TLDs) like `||*.org^`, unless they have specific limiting modifiers such as `$denyallow`, `$badfilter`, or `$client`.
457
- Examples:
458
- - `||*.org^` - this rule will be removed
459
- - `||*.org^$denyallow=example.com` - this rule will be kept because it has a limiting modifier
460
- If such rules must be saved, use [ValidateAllowPublicSuffix](#validate-allow-public-suffix) or [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix).
521
+ - Removes rules that block entire top-level domains (TLDs) like `||*.org^`,
522
+ unless they have specific limiting modifiers such as `$denyallow`,
523
+ `$badfilter`, or `$client`. Examples:
524
+ - `||*.org^` - this rule will be removed
525
+ - `||*.org^$denyallow=example.com` - this rule will be kept because it
526
+ has a limiting modifier
527
+ If such rules must be saved, use
528
+ [ValidateAllowPublicSuffix](#validate-allow-public-suffix) or
529
+ [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix).
461
530
 
462
531
  If there are comments preceding the invalid rule, they will be removed as well.
463
532
 
464
533
  ### <a name="validate-allow-ip"></a> ValidateAllowIp
465
534
 
466
- This transformation extends [Validate](#validate) to allow IP addresses in the lists. It also **normalizes IP rules** in Adblock-style to the safe format `||ip^`.
535
+ This transformation extends [Validate](#validate) to allow IP addresses in
536
+ the lists. It also **normalizes IP rules** in Adblock-style to the safe format
537
+ `||ip^`.
467
538
 
468
539
  #### IP Rule Normalization
469
540
 
@@ -478,25 +549,47 @@ This transformation extends [Validate](#validate) to allow IP addresses in the l
478
549
 
479
550
  Modifiers like `$important`, `$client`, `$denyallow`, `$badfilter` are preserved during normalization.
480
551
 
481
- > **Note:** Invalid IP patterns are rejected solely by the base [Validate](#validate) logic — normalization only converts valid patterns to canonical form and passes everything else through unchanged. See [Rejected IP Patterns](#rejected-ip-patterns) for details.
552
+ > **Note:** Invalid IP patterns are rejected solely by the base
553
+ > [Validate](#validate) logic — normalization only converts valid patterns to
554
+ > canonical form and passes everything else through unchanged. See
555
+ > [Rejected IP Patterns](#rejected-ip-patterns) for details.
482
556
 
483
557
  ### <a name="validate-allow-public-suffix"></a> ValidateAllowPublicSuffix
484
558
 
485
- This transformation exactly repeats the behavior of [Validate](#validate), but leaves rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`) in the list.
559
+ This transformation exactly repeats the behavior of [Validate](#validate),
560
+ but leaves rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`)
561
+ in the list.
486
562
 
487
- It still filters out invalid syntax rules and unsupported modifiers, but does not reject public-suffix rules unless the rule itself is malformed.
563
+ It still filters out invalid syntax rules and unsupported modifiers, but does
564
+ not reject public-suffix rules unless the rule itself is malformed.
488
565
 
489
- > **Note:** Combining any `Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`, and `ValidateAllowIpAndPublicSuffix` in one transformation list is not allowed and will result in an error. Each runs its own validator on the already-filtered output of the previous one, so allow-modes become silently ineffective.
566
+ > **Note:** Combining any `Validate`, `ValidateAllowIp`,
567
+ > `ValidateAllowPublicSuffix`, and `ValidateAllowIpAndPublicSuffix` in one
568
+ > transformation list is not allowed and will result in an error. Each runs
569
+ > its own validator on the already-filtered output of the previous one, so
570
+ > allow-modes become silently ineffective.
490
571
 
491
- > **Important:** Validation transformations also cannot be used at both source-level and top-level simultaneously. For example, if a source uses `ValidateAllowPublicSuffix` and the top-level configuration uses `Validate`, the compiler will throw an error. This is because the top-level `Validate` would override the source-level validation, making `ValidateAllowPublicSuffix` ineffective. Use validation transformations at only one level.
572
+ > **Important:** Validation transformations also cannot be used at both
573
+ > source-level and top-level simultaneously. For example, if a source uses
574
+ > `ValidateAllowPublicSuffix` and the top-level configuration uses `Validate`,
575
+ > the compiler will throw an error. This is because the top-level `Validate`
576
+ > would override the source-level validation, making
577
+ > `ValidateAllowPublicSuffix` ineffective. Use validation transformations at
578
+ > only one level.
492
579
 
493
580
  ### <a name="validate-allow-ip-and-public-suffix"></a> ValidateAllowIpAndPublicSuffix
494
581
 
495
- This transformation combines the behavior of [ValidateAllowIp](#validate-allow-ip) and [ValidateAllowPublicSuffix](#validate-allow-public-suffix). It allows both IP addresses and public suffix rules in the list.
582
+ This transformation combines the behavior of
583
+ [ValidateAllowIp](#validate-allow-ip) and
584
+ [ValidateAllowPublicSuffix](#validate-allow-public-suffix). It allows both IP
585
+ addresses and public suffix rules in the list.
496
586
 
497
- Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format `||ip^` before validation. See [IP Rule Normalization](#ip-rule-normalization) for details.
587
+ Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format
588
+ `||ip^` before validation. See [IP Rule Normalization](#ip-rule-normalization)
589
+ for details.
498
590
 
499
- Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`).
591
+ Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public
592
+ suffixes (e.g. `||hl.cn^`, `||org^`).
500
593
 
501
594
  ### <a name="deduplicate"></a> Deduplicate
502
595
 
@@ -505,11 +598,12 @@ This transformation simply removes the duplicates from the specified source.
505
598
  There are two important notes about this transformation:
506
599
 
507
600
  1. It keeps the original rules order.
508
- 2. It ignores comments. However, if the comments precede the rule that is being removed, the comments will be also removed.
601
+ 2. It ignores comments. However, if the comments precede the rule that is
602
+ being removed, the comments will be also removed.
509
603
 
510
604
  For instance:
511
605
 
512
- ```
606
+ ```text
513
607
  ! rule1 comment 1
514
608
  rule1
515
609
  ! rule1 comment 2
@@ -518,7 +612,7 @@ rule1
518
612
 
519
613
  Here's what will be left after the transformation:
520
614
 
521
- ```
615
+ ```text
522
616
  ! rule1 comment 2
523
617
  rule1
524
618
  ```
@@ -531,7 +625,7 @@ This is a very simple transformation that removes empty lines.
531
625
 
532
626
  Original list:
533
627
 
534
- ```
628
+ ```text
535
629
  rule1
536
630
 
537
631
  rule2
@@ -542,7 +636,7 @@ rule3
542
636
 
543
637
  Here's what we will have after applying this transformation:
544
638
 
545
- ```
639
+ ```text
546
640
  rule1
547
641
  rule2
548
642
  rule3
@@ -556,7 +650,7 @@ This is a very simple transformation that inserts a final newline.
556
650
 
557
651
  Original list:
558
652
 
559
- ```
653
+ ```text
560
654
  rule1
561
655
  rule2
562
656
  rule3
@@ -564,7 +658,7 @@ rule3
564
658
 
565
659
  Here's what we will have after applying this transformation:
566
660
 
567
- ```
661
+ ```text
568
662
  rule1
569
663
  rule2
570
664
  rule3
@@ -573,8 +667,14 @@ rule3
573
667
 
574
668
  `RemoveEmptyLines` doesn't delete this empty row due to the execution order.
575
669
 
576
- ## <a name="documentation"></a> Documentation
670
+ ## Documentation
577
671
 
578
- - [Development guide](DEVELOPMENT.md) — how to set up the environment, build, test, and contribute
672
+ - [Development](DEVELOPMENT.md) — how to set up the environment, build, test, and contribute
673
+ - [Deployment and configuration](DEPLOYMENT.md) — release pipeline, CI/CD, and Docker build
579
674
  - [LLM agent rules](AGENTS.md) — code guidelines and project conventions
580
675
  - [Changelog](CHANGELOG.md) — release history
676
+
677
+ [npm-badge]: https://nodei.co/npm/@adguard/hostlist-compiler.png?compact=true
678
+ [npm]: https://www.npmjs.com/package/@adguard/hostlist-compiler/
679
+ [private-repo]: https://github.com/AdGuardSoftwareLimited/filters-hostlist-compiler
680
+ [public-mirror]: https://github.com/AdguardTeam/HostlistCompiler