@adguard/hostlist-compiler 2.1.1 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/prepare-release.yml +5 -0
- package/.markdownlint.json +29 -0
- package/.markdownlintignore +1 -0
- package/AGENTS.md +310 -98
- package/CHANGELOG.md +62 -8
- package/DEPLOYMENT.md +175 -0
- package/DEVELOPMENT.md +171 -72
- package/README.md +238 -143
- package/examples/china/README.md +2 -2
- package/examples/energized/README.md +4 -2
- package/examples/sdn/README.md +2 -2
- package/examples/whitelist/README.md +2 -2
- package/package.json +24 -18
- package/src/cli.js +53 -31
- package/src/compile-source.js +4 -3
- package/src/configuration.js +132 -2
- package/src/download.js +27 -0
- package/src/filter.js +2 -2
- package/src/index.d.ts +2 -0
- package/src/index.js +16 -50
- package/src/{transformations/ip-normalize.js → ip-normalize.js} +2 -2
- package/src/schemas/configuration.schema.js +137 -0
- package/src/source-types.js +17 -0
- package/src/transformations/enum.js +49 -0
- package/src/transformations/include.js +1 -1
- package/src/transformations/transform.js +40 -74
- package/src/transformations/validate-allow-ip-and-public-suffix.js +1 -1
- package/src/transformations/validate-allow-ip.js +1 -1
- package/src/utils.js +1284 -0
- package/src/validation-conflicts.js +106 -0
- package/test/cli-error-output.test.js +305 -0
- package/test/cli.test.js +47 -0
- package/test/compile-source.test.js +167 -7
- package/test/configuration.test.js +438 -0
- package/test/download.test.js +260 -0
- package/test/helpers/http-mock.js +94 -0
- package/test/index.test.js +497 -4
- package/test/{transformations/ip-normalize.test.js → ip-normalize.test.js} +1 -1
- package/test/resolve-include.test.js +8 -8
- package/test/transformations/covert-to-ascii.test.js +1 -1
- package/test/transformations/exclude.test.js +2 -2
- package/test/transformations/include.test.js +9 -3
- package/test/transformations/ip-alignment.test.js +1 -1
- package/test/transformations/transform.test.js +66 -1
- package/test/utils.test.js +1559 -0
- package/test/validation-conflicts.test.js +230 -0
- package/src/schemas/configuration.schema.json +0 -369
package/README.md
CHANGED
|
@@ -1,84 +1,130 @@
|
|
|
1
|
-
# Hostlist
|
|
1
|
+
# Hostlist Compiler
|
|
2
2
|
|
|
3
|
-
[![NPM]
|
|
3
|
+
[![NPM][npm-badge]][npm]
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## Description
|
|
6
|
+
|
|
7
|
+
Hostlist Compiler is a Node.js CLI tool and library for filter list maintainers and developers. It
|
|
8
|
+
compiles DNS hosts blocklists from multiple sources — `/etc/hosts` files and adblock-style filter
|
|
9
|
+
lists — into a single filter list compatible with AdGuard Home and other AdGuard products with DNS
|
|
10
|
+
filtering.
|
|
11
|
+
|
|
12
|
+
Compiling a blocklist by hand is error-prone: sources use different syntaxes, rules must be
|
|
13
|
+
deduplicated, dangerous or unsupported rules have to be removed, and lists need to be filtered
|
|
14
|
+
through exclusion and inclusion rules. Hostlist Compiler automates this pipeline — it downloads
|
|
15
|
+
or reads the configured sources, converts `/etc/hosts` rules to AdGuard syntax, resolves
|
|
16
|
+
`!#include` directives, applies a fixed-order pipeline of transformations, and writes the
|
|
17
|
+
resulting filter list.
|
|
6
18
|
|
|
7
19
|
> **Note on repositories:** Active development happens in the private
|
|
8
|
-
> [AdGuardSoftwareLimited/filters-hostlist-compiler]
|
|
9
|
-
>
|
|
10
|
-
>
|
|
20
|
+
> [AdGuardSoftwareLimited/filters-hostlist-compiler][private-repo] repository;
|
|
21
|
+
> this public [AdguardTeam/HostlistCompiler][public-mirror] repository is a
|
|
22
|
+
> read-only mirror that is updated automatically from it.
|
|
11
23
|
|
|
24
|
+
## Table of Contents
|
|
25
|
+
|
|
26
|
+
- [Installation](#installation)
|
|
27
|
+
- [Quick Start](#quick-start)
|
|
12
28
|
- [Usage](#usage)
|
|
13
|
-
|
|
14
|
-
- [Command-line](#command-line)
|
|
15
|
-
- [API](#api)
|
|
29
|
+
- [Configuration](#configuration)
|
|
16
30
|
- [Transformations](#transformations)
|
|
17
|
-
|
|
18
|
-
- [TrimLines](#trimlines)
|
|
19
|
-
- [RemoveComments](#remove-comments)
|
|
20
|
-
- [Compress](#compress)
|
|
21
|
-
- [RemoveModifiers](#remove-modifiers)
|
|
22
|
-
- [InvertAllow](#invertallow)
|
|
23
|
-
- [Validate](#validate)
|
|
24
|
-
- [ValidateAllowIp](#validate-allow-ip)
|
|
25
|
-
- [ValidateAllowPublicSuffix](#validate-allow-public-suffix)
|
|
26
|
-
- [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix)
|
|
27
|
-
- [Deduplicate](#deduplicate)
|
|
28
|
-
- [RemoveEmptyLines](#removeemptylines)
|
|
29
|
-
- [InsertFinalNewLine](#insertfinalnewline)
|
|
31
|
+
- [API](#api)
|
|
30
32
|
- [Documentation](#documentation)
|
|
31
33
|
|
|
32
|
-
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Installation
|
|
33
37
|
|
|
34
|
-
|
|
38
|
+
Requires Node.js 22.22.2 or newer. Node.js 23.x is not supported by
|
|
39
|
+
`@adguard/filters-downloader` 3.x.
|
|
40
|
+
|
|
41
|
+
Install the package globally to get the `hostlist-compiler` command:
|
|
35
42
|
|
|
36
43
|
```bash
|
|
37
44
|
npm i -g @adguard/hostlist-compiler
|
|
38
45
|
```
|
|
39
46
|
|
|
40
|
-
|
|
47
|
+
Or install it as a dependency to use the compiler as a library:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npm i @adguard/hostlist-compiler
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
yarn is also supported:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
yarn add @adguard/hostlist-compiler
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
## Quick Start
|
|
41
60
|
|
|
42
|
-
|
|
61
|
+
Convert a `/etc/hosts`-syntax blocklist to an AdGuard-compatible filter list in one command:
|
|
43
62
|
|
|
44
|
-
|
|
63
|
+
```bash
|
|
64
|
+
hostlist-compiler -i hosts.txt -o output.txt
|
|
65
|
+
```
|
|
45
66
|
|
|
67
|
+
Or compile a configurable blocklist from multiple sources:
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
hostlist-compiler -c configuration.json -o output.txt
|
|
46
71
|
```
|
|
72
|
+
|
|
73
|
+
## Usage
|
|
74
|
+
|
|
75
|
+
### Command-Line Options
|
|
76
|
+
|
|
77
|
+
| Flag | Description |
|
|
78
|
+
| --------------------------- | -------------------------------------------------------------------------------------------------- |
|
|
79
|
+
| `--config, -c <path>` | Path to the compiler configuration file |
|
|
80
|
+
| `--input, -i <url-or-path>` | URL (or path to a file) to convert to an AdGuard-syntax blocklist. Can be specified multiple times |
|
|
81
|
+
| `--input-type, -t <type>` | Type of the input file: `hosts` or `adblock` |
|
|
82
|
+
| `--output, -o <path>` | Path to the output file (required) |
|
|
83
|
+
| `--verbose, -v` | Run with verbose logging |
|
|
84
|
+
| `--version` | Show the version number |
|
|
85
|
+
| `-h, --help` | Show help |
|
|
86
|
+
|
|
87
|
+
The compiler exits with code `0` on success and `1` on failure; errors are printed to stderr.
|
|
88
|
+
|
|
89
|
+
### Proxy
|
|
90
|
+
|
|
91
|
+
To download sources through a proxy, set `NODE_USE_ENV_PROXY=1` together with
|
|
92
|
+
the `HTTP_PROXY` (also used for `https://` when `HTTPS_PROXY` is unset),
|
|
93
|
+
`HTTPS_PROXY`, and `NO_PROXY` environment variables. The Node.js native
|
|
94
|
+
`fetch` ignores these variables unless `NODE_USE_ENV_PROXY=1` is set.
|
|
95
|
+
|
|
96
|
+
### Quick Hosts Conversion
|
|
97
|
+
|
|
98
|
+
Convert and compress one or more `/etc/hosts`-syntax blocklists to
|
|
99
|
+
[AdGuard syntax](https://adguard-dns.io/kb/general/dns-filtering-syntax/):
|
|
100
|
+
|
|
101
|
+
```bash
|
|
47
102
|
hostlist-compiler -i hosts.txt -i hosts2.txt -o output.txt
|
|
48
103
|
```
|
|
49
104
|
|
|
50
|
-
|
|
105
|
+
The quick conversion applies a default set of transformations to convert
|
|
106
|
+
`/etc/hosts` rules to AdGuard syntax: `RemoveComments`, `Deduplicate`,
|
|
107
|
+
`Compress`, `Validate`, `TrimLines`, `InsertFinalNewLine`, applied in the
|
|
108
|
+
order described in the [Transformations](#transformations) section. To
|
|
109
|
+
control the pipeline explicitly, use a [configuration file](#configuration)
|
|
110
|
+
instead.
|
|
51
111
|
|
|
52
|
-
|
|
112
|
+
### Build a Configurable Blocklist from Multiple Sources
|
|
113
|
+
|
|
114
|
+
Prepare the list configuration (read more about that
|
|
115
|
+
[below](#configuration)) and run the compiler:
|
|
53
116
|
|
|
54
117
|
```bash
|
|
55
118
|
hostlist-compiler -c configuration.json -o output.txt
|
|
56
119
|
```
|
|
57
120
|
|
|
58
|
-
|
|
121
|
+
A configurable blocklist can also be built from a remote URL without a configuration file:
|
|
59
122
|
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
Options:
|
|
64
|
-
--config, -c Path to the compiler configuration file [string]
|
|
65
|
-
--input, -i URL (or path to a file) to convert to an AdGuard-syntax
|
|
66
|
-
blocklist. Can be specified multiple times. [array]
|
|
67
|
-
--input-type, -t Type of the input file (/etc/hosts, adguard) [string]
|
|
68
|
-
--output, -o Path to the output file [string] [required]
|
|
69
|
-
--verbose, -v Run with verbose logging [boolean]
|
|
70
|
-
--version Show version number [boolean]
|
|
71
|
-
-h, --help Show help [boolean]
|
|
72
|
-
|
|
73
|
-
Examples:
|
|
74
|
-
hostlist-compiler -c config.json -o compile a blocklist and write the
|
|
75
|
-
output.txt output to output.txt
|
|
76
|
-
hostlist-compiler -i compile a blocklist from the URL and
|
|
77
|
-
https://example.org/hosts.txt -o write the output to output.txt
|
|
78
|
-
output.txt
|
|
123
|
+
```bash
|
|
124
|
+
hostlist-compiler -i https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -o output.txt
|
|
79
125
|
```
|
|
80
126
|
|
|
81
|
-
|
|
127
|
+
## Configuration
|
|
82
128
|
|
|
83
129
|
Configuration defines your filter list sources, and the transformations that are applied to the sources.
|
|
84
130
|
|
|
@@ -88,7 +134,7 @@ Here is an example of this configuration:
|
|
|
88
134
|
{
|
|
89
135
|
"name": "List name",
|
|
90
136
|
"description": "List description",
|
|
91
|
-
"homepage": "https://
|
|
137
|
+
"homepage": "https://github.com/AdguardTeam/AdguardSDNSFilter",
|
|
92
138
|
"license": "GPLv3",
|
|
93
139
|
"version": "1.0.0.0",
|
|
94
140
|
"sources": [
|
|
@@ -104,8 +150,8 @@ Here is an example of this configuration:
|
|
|
104
150
|
},
|
|
105
151
|
{
|
|
106
152
|
"name": "Remote rules",
|
|
107
|
-
"source": "https://
|
|
108
|
-
"type": "
|
|
153
|
+
"source": "https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt",
|
|
154
|
+
"type": "adblock",
|
|
109
155
|
"exclusions": ["excluded rule 1"]
|
|
110
156
|
}
|
|
111
157
|
],
|
|
@@ -123,19 +169,31 @@ Here is an example of this configuration:
|
|
|
123
169
|
- `license` - (optional) Filter list license.
|
|
124
170
|
- `version` - (optional) Filter list version.
|
|
125
171
|
- `sources` - (mandatory) array of the list sources.
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
172
|
+
- `.source` - (mandatory) path or URL of the source. It can be a
|
|
173
|
+
traditional filter list or a hosts file.
|
|
174
|
+
- `.name` - (optional) name of the source.
|
|
175
|
+
- `.type` - (optional) type of the source. It can be `adblock` for
|
|
176
|
+
Adblock-style lists or `hosts` for /etc/hosts style lists. The value is
|
|
177
|
+
validated against these two options; rule parsing auto-detects the
|
|
178
|
+
format, so the type does not affect parsing.
|
|
179
|
+
- `.transformations` - (optional) a list of transformations to apply to
|
|
180
|
+
the source rules. By default, **no transformations** are applied. Learn
|
|
181
|
+
more about the possible [transformations](#transformations).
|
|
182
|
+
- `.exclusions` - (optional) a list of rules (or wildcards) to exclude
|
|
183
|
+
from the source.
|
|
184
|
+
- `.exclusions_sources` - (optional) a list of files with exclusions.
|
|
185
|
+
- `.inclusions` - (optional) a list of wildcards to include from the
|
|
186
|
+
source. All rules that don't match these wildcards won't be included.
|
|
187
|
+
- `.inclusions_sources` - (optional) a list of files with inclusions.
|
|
188
|
+
- `transformations` - (optional) a list of transformations to apply to the
|
|
189
|
+
final list of rules. By default, **no transformations** are applied. Learn
|
|
190
|
+
more about the possible [transformations](#transformations).
|
|
191
|
+
- `exclusions` - (optional) a list of rules (or wildcards) to exclude from
|
|
192
|
+
the final list.
|
|
136
193
|
- `exclusions_sources` - (optional) a list of files with exclusions.
|
|
137
|
-
-
|
|
138
|
-
|
|
194
|
+
- `inclusions` - (optional) a list of wildcards to include in the final
|
|
195
|
+
list. All rules that don't match these wildcards won't be included.
|
|
196
|
+
- `inclusions_sources` - (optional) a list of files with inclusions.
|
|
139
197
|
|
|
140
198
|
Here is an example of a minimal configuration:
|
|
141
199
|
|
|
@@ -150,13 +208,15 @@ Here is an example of a minimal configuration:
|
|
|
150
208
|
}
|
|
151
209
|
```
|
|
152
210
|
|
|
153
|
-
|
|
211
|
+
### Exclusion and Inclusion Rules
|
|
154
212
|
|
|
155
|
-
Please note, that exclusion or inclusion rules may be a plain string,
|
|
213
|
+
Please note, that exclusion or inclusion rules may be a plain string,
|
|
214
|
+
wildcard, or a regular expression.
|
|
156
215
|
|
|
157
216
|
- `plainstring` - every rule that contains `plainstring` will match the rule
|
|
158
217
|
- `*.plainstring` - every rule that matches this wildcard will match the rule
|
|
159
|
-
- `/regex/` - every rule that matches this regular expression, will match the
|
|
218
|
+
- `/regex/` - every rule that matches this regular expression, will match the
|
|
219
|
+
rule. By default, regular expressions are case-insensitive.
|
|
160
220
|
- `! comment` - comments will be ignored.
|
|
161
221
|
|
|
162
222
|
> [!IMPORTANT]
|
|
@@ -168,7 +228,7 @@ Here is an example:
|
|
|
168
228
|
|
|
169
229
|
Rules in HOSTS syntax: `/hosts.txt`
|
|
170
230
|
|
|
171
|
-
```
|
|
231
|
+
```text
|
|
172
232
|
0.0.0.0 ads.example.com
|
|
173
233
|
0.0.0.0 tracking.example1.com
|
|
174
234
|
0.0.0.0 example.com
|
|
@@ -176,7 +236,7 @@ Rules in HOSTS syntax: `/hosts.txt`
|
|
|
176
236
|
|
|
177
237
|
Exclusion rules in adblock syntax: `/exclusions.txt`
|
|
178
238
|
|
|
179
|
-
```
|
|
239
|
+
```text
|
|
180
240
|
||example.com^
|
|
181
241
|
```
|
|
182
242
|
|
|
@@ -201,45 +261,29 @@ Configuration of the final list:
|
|
|
201
261
|
|
|
202
262
|
Final filter output of `/hosts.txt` after applying the `Compress` transformation and exclusions:
|
|
203
263
|
|
|
204
|
-
```
|
|
264
|
+
```text
|
|
205
265
|
||ads.example.com^
|
|
206
266
|
||tracking.example1.com^
|
|
207
267
|
```
|
|
208
268
|
|
|
209
269
|
The last rule now `||example.com^` will correctly match the rule from the exclusion list and will be excluded.
|
|
210
270
|
|
|
211
|
-
|
|
271
|
+
## API
|
|
212
272
|
|
|
213
|
-
|
|
273
|
+
The library exports a single function: `compile(configuration)`. It compiles
|
|
274
|
+
the filter list and returns a `Promise` that resolves to an array of rule
|
|
275
|
+
strings. TypeScript declarations are included with the package.
|
|
214
276
|
|
|
215
|
-
|
|
216
|
-
Usage: hostlist-compiler [options]
|
|
217
|
-
|
|
218
|
-
Options:
|
|
219
|
-
--version Show version number [boolean]
|
|
220
|
-
--config, -c Path to the compiler configuration file [string] [required]
|
|
221
|
-
--output, -o Path to the output file [string] [required]
|
|
222
|
-
--verbose, -v Run with verbose logging [boolean]
|
|
223
|
-
-h, --help Show help [boolean]
|
|
224
|
-
|
|
225
|
-
Examples:
|
|
226
|
-
hostlist-compiler -c config.json -o compile a blocklist and write the
|
|
227
|
-
output.txt output to output.txt
|
|
228
|
-
```
|
|
229
|
-
|
|
230
|
-
### <a name="api"></a> API
|
|
231
|
-
|
|
232
|
-
Install: `npm i @adguard/hostlist-compiler` or `yarn add @adguard/hostlist-compiler`
|
|
233
|
-
|
|
234
|
-
#### JavaScript example:
|
|
277
|
+
### JavaScript Example
|
|
235
278
|
|
|
236
279
|
```javascript
|
|
237
|
-
const compile = require(
|
|
280
|
+
const compile = require('@adguard/hostlist-compiler');
|
|
281
|
+
const { writeFileSync } = require('fs');
|
|
238
282
|
|
|
239
283
|
;(async () => {
|
|
240
284
|
// Compile filters
|
|
241
285
|
const result = await compile({
|
|
242
|
-
name: '
|
|
286
|
+
name: 'My hostlist',
|
|
243
287
|
sources: [
|
|
244
288
|
{
|
|
245
289
|
type: 'adblock',
|
|
@@ -251,11 +295,11 @@ const compile = require("@adguard/hostlist-compiler");
|
|
|
251
295
|
});
|
|
252
296
|
|
|
253
297
|
// Write to file
|
|
254
|
-
writeFileSync('
|
|
298
|
+
writeFileSync('output.txt', result.join('\n'));
|
|
255
299
|
})();
|
|
256
300
|
```
|
|
257
301
|
|
|
258
|
-
|
|
302
|
+
### TypeScript Example
|
|
259
303
|
|
|
260
304
|
```typescript
|
|
261
305
|
import compile from '@adguard/hostlist-compiler';
|
|
@@ -264,7 +308,7 @@ import { writeFileSync } from 'fs';
|
|
|
264
308
|
;(async () => {
|
|
265
309
|
// Compile filters
|
|
266
310
|
const result = await compile({
|
|
267
|
-
name: '
|
|
311
|
+
name: 'My hostlist',
|
|
268
312
|
sources: [
|
|
269
313
|
{
|
|
270
314
|
type: 'adblock',
|
|
@@ -276,7 +320,7 @@ import { writeFileSync } from 'fs';
|
|
|
276
320
|
});
|
|
277
321
|
|
|
278
322
|
// Write to file
|
|
279
|
-
writeFileSync('
|
|
323
|
+
writeFileSync('output.txt', result.join('\n'));
|
|
280
324
|
})();
|
|
281
325
|
```
|
|
282
326
|
|
|
@@ -289,7 +333,7 @@ import { writeFileSync } from 'fs';
|
|
|
289
333
|
;(async () => {
|
|
290
334
|
// Configuration
|
|
291
335
|
const config: HostlistCompilerConfiguration = {
|
|
292
|
-
name: '
|
|
336
|
+
name: 'My hostlist',
|
|
293
337
|
sources: [
|
|
294
338
|
{
|
|
295
339
|
type: 'adblock',
|
|
@@ -304,11 +348,11 @@ import { writeFileSync } from 'fs';
|
|
|
304
348
|
const result = await HostlistCompiler(config);
|
|
305
349
|
|
|
306
350
|
// Write to file
|
|
307
|
-
writeFileSync('
|
|
351
|
+
writeFileSync('output.txt', result.join('\n'));
|
|
308
352
|
})();
|
|
309
353
|
```
|
|
310
354
|
|
|
311
|
-
##
|
|
355
|
+
## Transformations
|
|
312
356
|
|
|
313
357
|
Here is the full list of transformations that are available:
|
|
314
358
|
|
|
@@ -336,7 +380,7 @@ This transformation converts all non-ASCII characters to their ASCII equivalents
|
|
|
336
380
|
|
|
337
381
|
Original list:
|
|
338
382
|
|
|
339
|
-
```
|
|
383
|
+
```text
|
|
340
384
|
||*.рус^
|
|
341
385
|
||*.कॉम^
|
|
342
386
|
||*.セール^
|
|
@@ -344,7 +388,7 @@ Original list:
|
|
|
344
388
|
|
|
345
389
|
Here's what we will have after applying this transformation:
|
|
346
390
|
|
|
347
|
-
```
|
|
391
|
+
```text
|
|
348
392
|
||*.xn--p1acf^
|
|
349
393
|
||*.xn--11b4c3d^
|
|
350
394
|
||*.xn--1qqw23a^
|
|
@@ -358,16 +402,16 @@ This is a very simple transformation that removes leading and trailing spaces/ta
|
|
|
358
402
|
|
|
359
403
|
Original list:
|
|
360
404
|
|
|
361
|
-
```
|
|
405
|
+
```text
|
|
362
406
|
rule1
|
|
363
407
|
rule2
|
|
364
408
|
rule3
|
|
365
|
-
|
|
409
|
+
rule4
|
|
366
410
|
```
|
|
367
411
|
|
|
368
412
|
Here's what we will have after applying this transformation:
|
|
369
413
|
|
|
370
|
-
```
|
|
414
|
+
```text
|
|
371
415
|
rule1
|
|
372
416
|
rule2
|
|
373
417
|
rule3
|
|
@@ -385,12 +429,20 @@ This is a very simple transformation that simply removes comments (e.g. all rule
|
|
|
385
429
|
|
|
386
430
|
Here's what it does:
|
|
387
431
|
|
|
388
|
-
1. It converts all rules to adblock-style rules. For instance,
|
|
389
|
-
|
|
432
|
+
1. It converts all rules to adblock-style rules. For instance,
|
|
433
|
+
`0.0.0.0 example.org` will be converted to `||example.org^`.
|
|
434
|
+
2. It discards the rules that are now redundant because of other existing
|
|
435
|
+
rules. For instance, `||example.org` blocks `example.org` and all it's
|
|
436
|
+
subdomains, therefore additional rules for the subdomains are now
|
|
437
|
+
redundant.
|
|
390
438
|
|
|
391
439
|
### <a name="remove-modifiers"></a> RemoveModifiers
|
|
392
440
|
|
|
393
|
-
By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will
|
|
441
|
+
By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will
|
|
442
|
+
ignore rules with unsupported modifiers, and all of the modifiers listed here
|
|
443
|
+
are unsupported. However, the rules with these modifiers are likely to be okay
|
|
444
|
+
for DNS-level blocking, that's why you might want to remove them when importing
|
|
445
|
+
rules from a traditional filter list.
|
|
394
446
|
|
|
395
447
|
Here is the list of modifiers that will be removed:
|
|
396
448
|
|
|
@@ -406,7 +458,10 @@ Here is the list of modifiers that will be removed:
|
|
|
406
458
|
|
|
407
459
|
### <a name="invertallow"></a> InvertAllow
|
|
408
460
|
|
|
409
|
-
This transformation converts blocking rules to "allow" rules. Note, that it
|
|
461
|
+
This transformation converts blocking rules to "allow" rules. Note, that it
|
|
462
|
+
does nothing to /etc/hosts rules (unless they were previously converted to
|
|
463
|
+
adblock-style syntax by a different transformation, for example
|
|
464
|
+
[Compress](#compress)).
|
|
410
465
|
|
|
411
466
|
There are two important notes about this transformation:
|
|
412
467
|
|
|
@@ -417,7 +472,7 @@ There are two important notes about this transformation:
|
|
|
417
472
|
|
|
418
473
|
Original list:
|
|
419
474
|
|
|
420
|
-
```
|
|
475
|
+
```text
|
|
421
476
|
! comment 1
|
|
422
477
|
rule1
|
|
423
478
|
|
|
@@ -428,7 +483,7 @@ rule1
|
|
|
428
483
|
|
|
429
484
|
Here's what we will have after applying this transformation:
|
|
430
485
|
|
|
431
|
-
```
|
|
486
|
+
```text
|
|
432
487
|
! comment 1
|
|
433
488
|
@@rule1
|
|
434
489
|
|
|
@@ -439,36 +494,47 @@ Here's what we will have after applying this transformation:
|
|
|
439
494
|
|
|
440
495
|
### <a name="validate"></a> Validate
|
|
441
496
|
|
|
442
|
-
This transformation is really crucial if you're using a filter list for a
|
|
497
|
+
This transformation is really crucial if you're using a filter list for a
|
|
498
|
+
traditional ad blocker as a source.
|
|
443
499
|
|
|
444
500
|
It removes dangerous or incompatible rules from the list.
|
|
445
501
|
|
|
446
502
|
So here's what it does:
|
|
447
503
|
|
|
448
|
-
- Discards domain-specific rules (e.g. `||example.org^$domain=example.com`).
|
|
449
|
-
|
|
504
|
+
- Discards domain-specific rules (e.g. `||example.org^$domain=example.com`).
|
|
505
|
+
You don't want to have domain-specific rules working globally.
|
|
506
|
+
- Discards rules with unsupported modifiers. Learn more about the
|
|
507
|
+
[supported modifiers](https://github.com/AdguardTeam/AdGuardHome/wiki/Hosts-Blocklists#-adblock-style-syntax).
|
|
450
508
|
- Discards rules that are too short.
|
|
451
509
|
- Discards IP addresses. If you need to keep IP addresses, use [ValidateAllowIp](#validate-allow-ip) instead.
|
|
452
510
|
|
|
453
511
|
#### <a name="rejected-ip-patterns"></a>Rejected IP Patterns
|
|
454
512
|
|
|
455
|
-
|
|
513
|
+
The following IP patterns are rejected by all validation transformations
|
|
514
|
+
(`Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`,
|
|
515
|
+
`ValidateAllowIpAndPublicSuffix`) as they are either unsafe or ambiguous:
|
|
456
516
|
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
517
|
+
- `||192.168.1^` — 3-octet with `^` - does not work
|
|
518
|
+
- `192.168.1` — Ambiguous: would match `192.168.11`, `192.168.111`, etc.
|
|
519
|
+
- `1.2.` or `1.2.*` — Too wide (1-2 octets), use regex instead
|
|
460
520
|
|
|
461
|
-
- Removes rules that block entire top-level domains (TLDs) like `||*.org^`,
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
521
|
+
- Removes rules that block entire top-level domains (TLDs) like `||*.org^`,
|
|
522
|
+
unless they have specific limiting modifiers such as `$denyallow`,
|
|
523
|
+
`$badfilter`, or `$client`. Examples:
|
|
524
|
+
- `||*.org^` - this rule will be removed
|
|
525
|
+
- `||*.org^$denyallow=example.com` - this rule will be kept because it
|
|
526
|
+
has a limiting modifier
|
|
527
|
+
If such rules must be saved, use
|
|
528
|
+
[ValidateAllowPublicSuffix](#validate-allow-public-suffix) or
|
|
529
|
+
[ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix).
|
|
466
530
|
|
|
467
531
|
If there are comments preceding the invalid rule, they will be removed as well.
|
|
468
532
|
|
|
469
533
|
### <a name="validate-allow-ip"></a> ValidateAllowIp
|
|
470
534
|
|
|
471
|
-
This transformation extends [Validate](#validate) to allow IP addresses in
|
|
535
|
+
This transformation extends [Validate](#validate) to allow IP addresses in
|
|
536
|
+
the lists. It also **normalizes IP rules** in Adblock-style to the safe format
|
|
537
|
+
`||ip^`.
|
|
472
538
|
|
|
473
539
|
#### IP Rule Normalization
|
|
474
540
|
|
|
@@ -483,25 +549,47 @@ This transformation extends [Validate](#validate) to allow IP addresses in the l
|
|
|
483
549
|
|
|
484
550
|
Modifiers like `$important`, `$client`, `$denyallow`, `$badfilter` are preserved during normalization.
|
|
485
551
|
|
|
486
|
-
> **Note:** Invalid IP patterns are rejected solely by the base
|
|
552
|
+
> **Note:** Invalid IP patterns are rejected solely by the base
|
|
553
|
+
> [Validate](#validate) logic — normalization only converts valid patterns to
|
|
554
|
+
> canonical form and passes everything else through unchanged. See
|
|
555
|
+
> [Rejected IP Patterns](#rejected-ip-patterns) for details.
|
|
487
556
|
|
|
488
557
|
### <a name="validate-allow-public-suffix"></a> ValidateAllowPublicSuffix
|
|
489
558
|
|
|
490
|
-
This transformation exactly repeats the behavior of [Validate](#validate),
|
|
559
|
+
This transformation exactly repeats the behavior of [Validate](#validate),
|
|
560
|
+
but leaves rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`)
|
|
561
|
+
in the list.
|
|
491
562
|
|
|
492
|
-
It still filters out invalid syntax rules and unsupported modifiers, but does
|
|
563
|
+
It still filters out invalid syntax rules and unsupported modifiers, but does
|
|
564
|
+
not reject public-suffix rules unless the rule itself is malformed.
|
|
493
565
|
|
|
494
|
-
> **Note:** Combining any `Validate`, `ValidateAllowIp`,
|
|
566
|
+
> **Note:** Combining any `Validate`, `ValidateAllowIp`,
|
|
567
|
+
> `ValidateAllowPublicSuffix`, and `ValidateAllowIpAndPublicSuffix` in one
|
|
568
|
+
> transformation list is not allowed and will result in an error. Each runs
|
|
569
|
+
> its own validator on the already-filtered output of the previous one, so
|
|
570
|
+
> allow-modes become silently ineffective.
|
|
495
571
|
|
|
496
|
-
> **Important:** Validation transformations also cannot be used at both
|
|
572
|
+
> **Important:** Validation transformations also cannot be used at both
|
|
573
|
+
> source-level and top-level simultaneously. For example, if a source uses
|
|
574
|
+
> `ValidateAllowPublicSuffix` and the top-level configuration uses `Validate`,
|
|
575
|
+
> the compiler will throw an error. This is because the top-level `Validate`
|
|
576
|
+
> would override the source-level validation, making
|
|
577
|
+
> `ValidateAllowPublicSuffix` ineffective. Use validation transformations at
|
|
578
|
+
> only one level.
|
|
497
579
|
|
|
498
580
|
### <a name="validate-allow-ip-and-public-suffix"></a> ValidateAllowIpAndPublicSuffix
|
|
499
581
|
|
|
500
|
-
This transformation combines the behavior of
|
|
582
|
+
This transformation combines the behavior of
|
|
583
|
+
[ValidateAllowIp](#validate-allow-ip) and
|
|
584
|
+
[ValidateAllowPublicSuffix](#validate-allow-public-suffix). It allows both IP
|
|
585
|
+
addresses and public suffix rules in the list.
|
|
501
586
|
|
|
502
|
-
Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format
|
|
587
|
+
Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format
|
|
588
|
+
`||ip^` before validation. See [IP Rule Normalization](#ip-rule-normalization)
|
|
589
|
+
for details.
|
|
503
590
|
|
|
504
|
-
Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public
|
|
591
|
+
Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public
|
|
592
|
+
suffixes (e.g. `||hl.cn^`, `||org^`).
|
|
505
593
|
|
|
506
594
|
### <a name="deduplicate"></a> Deduplicate
|
|
507
595
|
|
|
@@ -510,11 +598,12 @@ This transformation simply removes the duplicates from the specified source.
|
|
|
510
598
|
There are two important notes about this transformation:
|
|
511
599
|
|
|
512
600
|
1. It keeps the original rules order.
|
|
513
|
-
2. It ignores comments. However, if the comments precede the rule that is
|
|
601
|
+
2. It ignores comments. However, if the comments precede the rule that is
|
|
602
|
+
being removed, the comments will be also removed.
|
|
514
603
|
|
|
515
604
|
For instance:
|
|
516
605
|
|
|
517
|
-
```
|
|
606
|
+
```text
|
|
518
607
|
! rule1 comment 1
|
|
519
608
|
rule1
|
|
520
609
|
! rule1 comment 2
|
|
@@ -523,7 +612,7 @@ rule1
|
|
|
523
612
|
|
|
524
613
|
Here's what will be left after the transformation:
|
|
525
614
|
|
|
526
|
-
```
|
|
615
|
+
```text
|
|
527
616
|
! rule1 comment 2
|
|
528
617
|
rule1
|
|
529
618
|
```
|
|
@@ -536,7 +625,7 @@ This is a very simple transformation that removes empty lines.
|
|
|
536
625
|
|
|
537
626
|
Original list:
|
|
538
627
|
|
|
539
|
-
```
|
|
628
|
+
```text
|
|
540
629
|
rule1
|
|
541
630
|
|
|
542
631
|
rule2
|
|
@@ -547,7 +636,7 @@ rule3
|
|
|
547
636
|
|
|
548
637
|
Here's what we will have after applying this transformation:
|
|
549
638
|
|
|
550
|
-
```
|
|
639
|
+
```text
|
|
551
640
|
rule1
|
|
552
641
|
rule2
|
|
553
642
|
rule3
|
|
@@ -561,7 +650,7 @@ This is a very simple transformation that inserts a final newline.
|
|
|
561
650
|
|
|
562
651
|
Original list:
|
|
563
652
|
|
|
564
|
-
```
|
|
653
|
+
```text
|
|
565
654
|
rule1
|
|
566
655
|
rule2
|
|
567
656
|
rule3
|
|
@@ -569,7 +658,7 @@ rule3
|
|
|
569
658
|
|
|
570
659
|
Here's what we will have after applying this transformation:
|
|
571
660
|
|
|
572
|
-
```
|
|
661
|
+
```text
|
|
573
662
|
rule1
|
|
574
663
|
rule2
|
|
575
664
|
rule3
|
|
@@ -578,8 +667,14 @@ rule3
|
|
|
578
667
|
|
|
579
668
|
`RemoveEmptyLines` doesn't delete this empty row due to the execution order.
|
|
580
669
|
|
|
581
|
-
##
|
|
670
|
+
## Documentation
|
|
582
671
|
|
|
583
|
-
- [Development
|
|
672
|
+
- [Development](DEVELOPMENT.md) — how to set up the environment, build, test, and contribute
|
|
673
|
+
- [Deployment and configuration](DEPLOYMENT.md) — release pipeline, CI/CD, and Docker build
|
|
584
674
|
- [LLM agent rules](AGENTS.md) — code guidelines and project conventions
|
|
585
675
|
- [Changelog](CHANGELOG.md) — release history
|
|
676
|
+
|
|
677
|
+
[npm-badge]: https://nodei.co/npm/@adguard/hostlist-compiler.png?compact=true
|
|
678
|
+
[npm]: https://www.npmjs.com/package/@adguard/hostlist-compiler/
|
|
679
|
+
[private-repo]: https://github.com/AdGuardSoftwareLimited/filters-hostlist-compiler
|
|
680
|
+
[public-mirror]: https://github.com/AdguardTeam/HostlistCompiler
|