@adguard/hostlist-compiler 2.1.0 → 3.0.0-beta.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.dockerignore +29 -0
- package/.github/workflows/ci.yml +22 -0
- package/.github/workflows/mirror.yml +24 -0
- package/.github/workflows/prepare-release.yml +26 -0
- package/.github/workflows/publish-release.yml +44 -0
- package/.markdownlint.json +29 -0
- package/.markdownlintignore +1 -0
- package/AGENTS.md +314 -103
- package/CHANGELOG.md +76 -7
- package/DEPLOYMENT.md +171 -0
- package/DEVELOPMENT.md +174 -76
- package/README.md +240 -140
- package/examples/china/README.md +2 -2
- package/examples/energized/README.md +4 -2
- package/examples/sdn/README.md +2 -2
- package/examples/whitelist/README.md +2 -2
- package/package.json +29 -24
- package/src/cli.js +59 -33
- package/src/compile-source.js +4 -3
- package/src/configuration.js +132 -2
- package/src/download.js +27 -0
- package/src/filter.js +2 -2
- package/src/index.d.ts +2 -0
- package/src/index.js +21 -51
- package/src/{transformations/ip-normalize.js → ip-normalize.js} +2 -2
- package/src/schemas/configuration.schema.js +137 -0
- package/src/source-types.js +17 -0
- package/src/transformations/enum.js +49 -0
- package/src/transformations/include.js +1 -1
- package/src/transformations/transform.js +40 -74
- package/src/transformations/validate-allow-ip-and-public-suffix.js +1 -1
- package/src/transformations/validate-allow-ip.js +1 -1
- package/src/utils.js +1284 -0
- package/src/validation-conflicts.js +106 -0
- package/test/cli-error-output.test.js +305 -0
- package/test/cli.test.js +47 -0
- package/test/compile-source.test.js +167 -7
- package/test/configuration.test.js +438 -0
- package/test/download.test.js +260 -0
- package/test/helpers/http-mock.js +94 -0
- package/test/index.test.js +497 -4
- package/test/{transformations/ip-normalize.test.js → ip-normalize.test.js} +1 -1
- package/test/resolve-include.test.js +8 -8
- package/test/transformations/covert-to-ascii.test.js +1 -1
- package/test/transformations/exclude.test.js +2 -2
- package/test/transformations/include.test.js +9 -3
- package/test/transformations/ip-alignment.test.js +1 -1
- package/test/transformations/transform.test.js +66 -1
- package/test/utils.test.js +1559 -0
- package/test/validation-conflicts.test.js +230 -0
- package/bamboo-specs/bamboo.yaml +0 -14
- package/bamboo-specs/build.yaml +0 -92
- package/bamboo-specs/deploy.yaml +0 -38
- package/bamboo-specs/increment.yaml +0 -54
- package/bamboo-specs/permissions.yaml +0 -16
- package/bamboo-specs/test.yaml +0 -56
- package/src/schemas/configuration.schema.json +0 -369
- package/tools/build-txt.js +0 -19
package/README.md
CHANGED
|
@@ -1,79 +1,130 @@
|
|
|
1
|
-
# Hostlist
|
|
1
|
+
# Hostlist Compiler
|
|
2
2
|
|
|
3
|
-
[![NPM]
|
|
3
|
+
[![NPM][npm-badge]][npm]
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## Description
|
|
6
6
|
|
|
7
|
+
Hostlist Compiler is a Node.js CLI tool and library for filter list maintainers and developers. It
|
|
8
|
+
compiles DNS hosts blocklists from multiple sources — `/etc/hosts` files and adblock-style filter
|
|
9
|
+
lists — into a single filter list compatible with AdGuard Home and other AdGuard products with DNS
|
|
10
|
+
filtering.
|
|
11
|
+
|
|
12
|
+
Compiling a blocklist by hand is error-prone: sources use different syntaxes, rules must be
|
|
13
|
+
deduplicated, dangerous or unsupported rules have to be removed, and lists need to be filtered
|
|
14
|
+
through exclusion and inclusion rules. Hostlist Compiler automates this pipeline — it downloads
|
|
15
|
+
or reads the configured sources, converts `/etc/hosts` rules to AdGuard syntax, resolves
|
|
16
|
+
`!#include` directives, applies a fixed-order pipeline of transformations, and writes the
|
|
17
|
+
resulting filter list.
|
|
18
|
+
|
|
19
|
+
> **Note on repositories:** Active development happens in the private
|
|
20
|
+
> [AdGuardSoftwareLimited/filters-hostlist-compiler][private-repo] repository;
|
|
21
|
+
> this public [AdguardTeam/HostlistCompiler][public-mirror] repository is a
|
|
22
|
+
> read-only mirror that is updated automatically from it.
|
|
23
|
+
|
|
24
|
+
## Table of Contents
|
|
25
|
+
|
|
26
|
+
- [Installation](#installation)
|
|
27
|
+
- [Quick Start](#quick-start)
|
|
7
28
|
- [Usage](#usage)
|
|
8
|
-
|
|
9
|
-
- [Command-line](#command-line)
|
|
10
|
-
- [API](#api)
|
|
29
|
+
- [Configuration](#configuration)
|
|
11
30
|
- [Transformations](#transformations)
|
|
12
|
-
|
|
13
|
-
- [TrimLines](#trimlines)
|
|
14
|
-
- [RemoveComments](#remove-comments)
|
|
15
|
-
- [Compress](#compress)
|
|
16
|
-
- [RemoveModifiers](#remove-modifiers)
|
|
17
|
-
- [InvertAllow](#invertallow)
|
|
18
|
-
- [Validate](#validate)
|
|
19
|
-
- [ValidateAllowIp](#validate-allow-ip)
|
|
20
|
-
- [ValidateAllowPublicSuffix](#validate-allow-public-suffix)
|
|
21
|
-
- [ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix)
|
|
22
|
-
- [Deduplicate](#deduplicate)
|
|
23
|
-
- [RemoveEmptyLines](#removeemptylines)
|
|
24
|
-
- [InsertFinalNewLine](#insertfinalnewline)
|
|
31
|
+
- [API](#api)
|
|
25
32
|
- [Documentation](#documentation)
|
|
26
33
|
|
|
27
|
-
|
|
34
|
+
---
|
|
28
35
|
|
|
29
|
-
|
|
36
|
+
## Installation
|
|
37
|
+
|
|
38
|
+
Requires Node.js 22.22.2 or newer. Node.js 23.x is not supported by
|
|
39
|
+
`@adguard/filters-downloader` 3.x.
|
|
40
|
+
|
|
41
|
+
Install the package globally to get the `hostlist-compiler` command:
|
|
30
42
|
|
|
31
43
|
```bash
|
|
32
44
|
npm i -g @adguard/hostlist-compiler
|
|
33
45
|
```
|
|
34
46
|
|
|
35
|
-
|
|
47
|
+
Or install it as a dependency to use the compiler as a library:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npm i @adguard/hostlist-compiler
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
yarn is also supported:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
yarn add @adguard/hostlist-compiler
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
## Quick Start
|
|
60
|
+
|
|
61
|
+
Convert a `/etc/hosts`-syntax blocklist to an AdGuard-compatible filter list in one command:
|
|
36
62
|
|
|
37
|
-
|
|
63
|
+
```bash
|
|
64
|
+
hostlist-compiler -i hosts.txt -o output.txt
|
|
65
|
+
```
|
|
38
66
|
|
|
39
|
-
|
|
67
|
+
Or compile a configurable blocklist from multiple sources:
|
|
40
68
|
|
|
69
|
+
```bash
|
|
70
|
+
hostlist-compiler -c configuration.json -o output.txt
|
|
41
71
|
```
|
|
72
|
+
|
|
73
|
+
## Usage
|
|
74
|
+
|
|
75
|
+
### Command-Line Options
|
|
76
|
+
|
|
77
|
+
| Flag | Description |
|
|
78
|
+
| --------------------------- | -------------------------------------------------------------------------------------------------- |
|
|
79
|
+
| `--config, -c <path>` | Path to the compiler configuration file |
|
|
80
|
+
| `--input, -i <url-or-path>` | URL (or path to a file) to convert to an AdGuard-syntax blocklist. Can be specified multiple times |
|
|
81
|
+
| `--input-type, -t <type>` | Type of the input file: `hosts` or `adblock` |
|
|
82
|
+
| `--output, -o <path>` | Path to the output file (required) |
|
|
83
|
+
| `--verbose, -v` | Run with verbose logging |
|
|
84
|
+
| `--version` | Show the version number |
|
|
85
|
+
| `-h, --help` | Show help |
|
|
86
|
+
|
|
87
|
+
The compiler exits with code `0` on success and `1` on failure; errors are printed to stderr.
|
|
88
|
+
|
|
89
|
+
### Proxy
|
|
90
|
+
|
|
91
|
+
To download sources through a proxy, set `NODE_USE_ENV_PROXY=1` together with
|
|
92
|
+
the `HTTP_PROXY` (also used for `https://` when `HTTPS_PROXY` is unset),
|
|
93
|
+
`HTTPS_PROXY`, and `NO_PROXY` environment variables. The Node.js native
|
|
94
|
+
`fetch` ignores these variables unless `NODE_USE_ENV_PROXY=1` is set.
|
|
95
|
+
|
|
96
|
+
### Quick Hosts Conversion
|
|
97
|
+
|
|
98
|
+
Convert and compress one or more `/etc/hosts`-syntax blocklists to
|
|
99
|
+
[AdGuard syntax](https://adguard-dns.io/kb/general/dns-filtering-syntax/):
|
|
100
|
+
|
|
101
|
+
```bash
|
|
42
102
|
hostlist-compiler -i hosts.txt -i hosts2.txt -o output.txt
|
|
43
103
|
```
|
|
44
104
|
|
|
45
|
-
|
|
105
|
+
The quick conversion applies a default set of transformations to convert
|
|
106
|
+
`/etc/hosts` rules to AdGuard syntax: `RemoveComments`, `Deduplicate`,
|
|
107
|
+
`Compress`, `Validate`, `TrimLines`, `InsertFinalNewLine`, applied in the
|
|
108
|
+
order described in the [Transformations](#transformations) section. To
|
|
109
|
+
control the pipeline explicitly, use a [configuration file](#configuration)
|
|
110
|
+
instead.
|
|
111
|
+
|
|
112
|
+
### Build a Configurable Blocklist from Multiple Sources
|
|
46
113
|
|
|
47
|
-
Prepare the list configuration (read more about that
|
|
114
|
+
Prepare the list configuration (read more about that
|
|
115
|
+
[below](#configuration)) and run the compiler:
|
|
48
116
|
|
|
49
117
|
```bash
|
|
50
118
|
hostlist-compiler -c configuration.json -o output.txt
|
|
51
119
|
```
|
|
52
120
|
|
|
53
|
-
|
|
121
|
+
A configurable blocklist can also be built from a remote URL without a configuration file:
|
|
54
122
|
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
Options:
|
|
59
|
-
--config, -c Path to the compiler configuration file [string]
|
|
60
|
-
--input, -i URL (or path to a file) to convert to an AdGuard-syntax
|
|
61
|
-
blocklist. Can be specified multiple times. [array]
|
|
62
|
-
--input-type, -t Type of the input file (/etc/hosts, adguard) [string]
|
|
63
|
-
--output, -o Path to the output file [string] [required]
|
|
64
|
-
--verbose, -v Run with verbose logging [boolean]
|
|
65
|
-
--version Show version number [boolean]
|
|
66
|
-
-h, --help Show help [boolean]
|
|
67
|
-
|
|
68
|
-
Examples:
|
|
69
|
-
hostlist-compiler -c config.json -o compile a blocklist and write the
|
|
70
|
-
output.txt output to output.txt
|
|
71
|
-
hostlist-compiler -i compile a blocklist from the URL and
|
|
72
|
-
https://example.org/hosts.txt -o write the output to output.txt
|
|
73
|
-
output.txt
|
|
123
|
+
```bash
|
|
124
|
+
hostlist-compiler -i https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -o output.txt
|
|
74
125
|
```
|
|
75
126
|
|
|
76
|
-
|
|
127
|
+
## Configuration
|
|
77
128
|
|
|
78
129
|
Configuration defines your filter list sources, and the transformations that are applied to the sources.
|
|
79
130
|
|
|
@@ -83,7 +134,7 @@ Here is an example of this configuration:
|
|
|
83
134
|
{
|
|
84
135
|
"name": "List name",
|
|
85
136
|
"description": "List description",
|
|
86
|
-
"homepage": "https://
|
|
137
|
+
"homepage": "https://github.com/AdguardTeam/AdguardSDNSFilter",
|
|
87
138
|
"license": "GPLv3",
|
|
88
139
|
"version": "1.0.0.0",
|
|
89
140
|
"sources": [
|
|
@@ -99,8 +150,8 @@ Here is an example of this configuration:
|
|
|
99
150
|
},
|
|
100
151
|
{
|
|
101
152
|
"name": "Remote rules",
|
|
102
|
-
"source": "https://
|
|
103
|
-
"type": "
|
|
153
|
+
"source": "https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt",
|
|
154
|
+
"type": "adblock",
|
|
104
155
|
"exclusions": ["excluded rule 1"]
|
|
105
156
|
}
|
|
106
157
|
],
|
|
@@ -118,19 +169,31 @@ Here is an example of this configuration:
|
|
|
118
169
|
- `license` - (optional) Filter list license.
|
|
119
170
|
- `version` - (optional) Filter list version.
|
|
120
171
|
- `sources` - (mandatory) array of the list sources.
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
172
|
+
- `.source` - (mandatory) path or URL of the source. It can be a
|
|
173
|
+
traditional filter list or a hosts file.
|
|
174
|
+
- `.name` - (optional) name of the source.
|
|
175
|
+
- `.type` - (optional) type of the source. It can be `adblock` for
|
|
176
|
+
Adblock-style lists or `hosts` for /etc/hosts style lists. The value is
|
|
177
|
+
validated against these two options; rule parsing auto-detects the
|
|
178
|
+
format, so the type does not affect parsing.
|
|
179
|
+
- `.transformations` - (optional) a list of transformations to apply to
|
|
180
|
+
the source rules. By default, **no transformations** are applied. Learn
|
|
181
|
+
more about the possible [transformations](#transformations).
|
|
182
|
+
- `.exclusions` - (optional) a list of rules (or wildcards) to exclude
|
|
183
|
+
from the source.
|
|
184
|
+
- `.exclusions_sources` - (optional) a list of files with exclusions.
|
|
185
|
+
- `.inclusions` - (optional) a list of wildcards to include from the
|
|
186
|
+
source. All rules that don't match these wildcards won't be included.
|
|
187
|
+
- `.inclusions_sources` - (optional) a list of files with inclusions.
|
|
188
|
+
- `transformations` - (optional) a list of transformations to apply to the
|
|
189
|
+
final list of rules. By default, **no transformations** are applied. Learn
|
|
190
|
+
more about the possible [transformations](#transformations).
|
|
191
|
+
- `exclusions` - (optional) a list of rules (or wildcards) to exclude from
|
|
192
|
+
the final list.
|
|
131
193
|
- `exclusions_sources` - (optional) a list of files with exclusions.
|
|
132
|
-
-
|
|
133
|
-
|
|
194
|
+
- `inclusions` - (optional) a list of wildcards to include in the final
|
|
195
|
+
list. All rules that don't match these wildcards won't be included.
|
|
196
|
+
- `inclusions_sources` - (optional) a list of files with inclusions.
|
|
134
197
|
|
|
135
198
|
Here is an example of a minimal configuration:
|
|
136
199
|
|
|
@@ -145,13 +208,15 @@ Here is an example of a minimal configuration:
|
|
|
145
208
|
}
|
|
146
209
|
```
|
|
147
210
|
|
|
148
|
-
|
|
211
|
+
### Exclusion and Inclusion Rules
|
|
149
212
|
|
|
150
|
-
Please note, that exclusion or inclusion rules may be a plain string,
|
|
213
|
+
Please note, that exclusion or inclusion rules may be a plain string,
|
|
214
|
+
wildcard, or a regular expression.
|
|
151
215
|
|
|
152
216
|
- `plainstring` - every rule that contains `plainstring` will match the rule
|
|
153
217
|
- `*.plainstring` - every rule that matches this wildcard will match the rule
|
|
154
|
-
- `/regex/` - every rule that matches this regular expression, will match the
|
|
218
|
+
- `/regex/` - every rule that matches this regular expression, will match the
|
|
219
|
+
rule. By default, regular expressions are case-insensitive.
|
|
155
220
|
- `! comment` - comments will be ignored.
|
|
156
221
|
|
|
157
222
|
> [!IMPORTANT]
|
|
@@ -163,7 +228,7 @@ Here is an example:
|
|
|
163
228
|
|
|
164
229
|
Rules in HOSTS syntax: `/hosts.txt`
|
|
165
230
|
|
|
166
|
-
```
|
|
231
|
+
```text
|
|
167
232
|
0.0.0.0 ads.example.com
|
|
168
233
|
0.0.0.0 tracking.example1.com
|
|
169
234
|
0.0.0.0 example.com
|
|
@@ -171,7 +236,7 @@ Rules in HOSTS syntax: `/hosts.txt`
|
|
|
171
236
|
|
|
172
237
|
Exclusion rules in adblock syntax: `/exclusions.txt`
|
|
173
238
|
|
|
174
|
-
```
|
|
239
|
+
```text
|
|
175
240
|
||example.com^
|
|
176
241
|
```
|
|
177
242
|
|
|
@@ -196,45 +261,29 @@ Configuration of the final list:
|
|
|
196
261
|
|
|
197
262
|
Final filter output of `/hosts.txt` after applying the `Compress` transformation and exclusions:
|
|
198
263
|
|
|
199
|
-
```
|
|
264
|
+
```text
|
|
200
265
|
||ads.example.com^
|
|
201
266
|
||tracking.example1.com^
|
|
202
267
|
```
|
|
203
268
|
|
|
204
269
|
The last rule now `||example.com^` will correctly match the rule from the exclusion list and will be excluded.
|
|
205
270
|
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
Command-line arguments.
|
|
209
|
-
|
|
210
|
-
```
|
|
211
|
-
Usage: hostlist-compiler [options]
|
|
212
|
-
|
|
213
|
-
Options:
|
|
214
|
-
--version Show version number [boolean]
|
|
215
|
-
--config, -c Path to the compiler configuration file [string] [required]
|
|
216
|
-
--output, -o Path to the output file [string] [required]
|
|
217
|
-
--verbose, -v Run with verbose logging [boolean]
|
|
218
|
-
-h, --help Show help [boolean]
|
|
219
|
-
|
|
220
|
-
Examples:
|
|
221
|
-
hostlist-compiler -c config.json -o compile a blocklist and write the
|
|
222
|
-
output.txt output to output.txt
|
|
223
|
-
```
|
|
224
|
-
|
|
225
|
-
### <a name="api"></a> API
|
|
271
|
+
## API
|
|
226
272
|
|
|
227
|
-
|
|
273
|
+
The library exports a single function: `compile(configuration)`. It compiles
|
|
274
|
+
the filter list and returns a `Promise` that resolves to an array of rule
|
|
275
|
+
strings. TypeScript declarations are included with the package.
|
|
228
276
|
|
|
229
|
-
|
|
277
|
+
### JavaScript Example
|
|
230
278
|
|
|
231
279
|
```javascript
|
|
232
|
-
const compile = require(
|
|
280
|
+
const compile = require('@adguard/hostlist-compiler');
|
|
281
|
+
const { writeFileSync } = require('fs');
|
|
233
282
|
|
|
234
283
|
;(async () => {
|
|
235
284
|
// Compile filters
|
|
236
285
|
const result = await compile({
|
|
237
|
-
name: '
|
|
286
|
+
name: 'My hostlist',
|
|
238
287
|
sources: [
|
|
239
288
|
{
|
|
240
289
|
type: 'adblock',
|
|
@@ -246,11 +295,11 @@ const compile = require("@adguard/hostlist-compiler");
|
|
|
246
295
|
});
|
|
247
296
|
|
|
248
297
|
// Write to file
|
|
249
|
-
writeFileSync('
|
|
298
|
+
writeFileSync('output.txt', result.join('\n'));
|
|
250
299
|
})();
|
|
251
300
|
```
|
|
252
301
|
|
|
253
|
-
|
|
302
|
+
### TypeScript Example
|
|
254
303
|
|
|
255
304
|
```typescript
|
|
256
305
|
import compile from '@adguard/hostlist-compiler';
|
|
@@ -259,7 +308,7 @@ import { writeFileSync } from 'fs';
|
|
|
259
308
|
;(async () => {
|
|
260
309
|
// Compile filters
|
|
261
310
|
const result = await compile({
|
|
262
|
-
name: '
|
|
311
|
+
name: 'My hostlist',
|
|
263
312
|
sources: [
|
|
264
313
|
{
|
|
265
314
|
type: 'adblock',
|
|
@@ -271,7 +320,7 @@ import { writeFileSync } from 'fs';
|
|
|
271
320
|
});
|
|
272
321
|
|
|
273
322
|
// Write to file
|
|
274
|
-
writeFileSync('
|
|
323
|
+
writeFileSync('output.txt', result.join('\n'));
|
|
275
324
|
})();
|
|
276
325
|
```
|
|
277
326
|
|
|
@@ -284,7 +333,7 @@ import { writeFileSync } from 'fs';
|
|
|
284
333
|
;(async () => {
|
|
285
334
|
// Configuration
|
|
286
335
|
const config: HostlistCompilerConfiguration = {
|
|
287
|
-
name: '
|
|
336
|
+
name: 'My hostlist',
|
|
288
337
|
sources: [
|
|
289
338
|
{
|
|
290
339
|
type: 'adblock',
|
|
@@ -299,11 +348,11 @@ import { writeFileSync } from 'fs';
|
|
|
299
348
|
const result = await HostlistCompiler(config);
|
|
300
349
|
|
|
301
350
|
// Write to file
|
|
302
|
-
writeFileSync('
|
|
351
|
+
writeFileSync('output.txt', result.join('\n'));
|
|
303
352
|
})();
|
|
304
353
|
```
|
|
305
354
|
|
|
306
|
-
##
|
|
355
|
+
## Transformations
|
|
307
356
|
|
|
308
357
|
Here is the full list of transformations that are available:
|
|
309
358
|
|
|
@@ -331,7 +380,7 @@ This transformation converts all non-ASCII characters to their ASCII equivalents
|
|
|
331
380
|
|
|
332
381
|
Original list:
|
|
333
382
|
|
|
334
|
-
```
|
|
383
|
+
```text
|
|
335
384
|
||*.рус^
|
|
336
385
|
||*.कॉम^
|
|
337
386
|
||*.セール^
|
|
@@ -339,7 +388,7 @@ Original list:
|
|
|
339
388
|
|
|
340
389
|
Here's what we will have after applying this transformation:
|
|
341
390
|
|
|
342
|
-
```
|
|
391
|
+
```text
|
|
343
392
|
||*.xn--p1acf^
|
|
344
393
|
||*.xn--11b4c3d^
|
|
345
394
|
||*.xn--1qqw23a^
|
|
@@ -353,16 +402,16 @@ This is a very simple transformation that removes leading and trailing spaces/ta
|
|
|
353
402
|
|
|
354
403
|
Original list:
|
|
355
404
|
|
|
356
|
-
```
|
|
405
|
+
```text
|
|
357
406
|
rule1
|
|
358
407
|
rule2
|
|
359
408
|
rule3
|
|
360
|
-
|
|
409
|
+
rule4
|
|
361
410
|
```
|
|
362
411
|
|
|
363
412
|
Here's what we will have after applying this transformation:
|
|
364
413
|
|
|
365
|
-
```
|
|
414
|
+
```text
|
|
366
415
|
rule1
|
|
367
416
|
rule2
|
|
368
417
|
rule3
|
|
@@ -380,12 +429,20 @@ This is a very simple transformation that simply removes comments (e.g. all rule
|
|
|
380
429
|
|
|
381
430
|
Here's what it does:
|
|
382
431
|
|
|
383
|
-
1. It converts all rules to adblock-style rules. For instance,
|
|
384
|
-
|
|
432
|
+
1. It converts all rules to adblock-style rules. For instance,
|
|
433
|
+
`0.0.0.0 example.org` will be converted to `||example.org^`.
|
|
434
|
+
2. It discards the rules that are now redundant because of other existing
|
|
435
|
+
rules. For instance, `||example.org` blocks `example.org` and all it's
|
|
436
|
+
subdomains, therefore additional rules for the subdomains are now
|
|
437
|
+
redundant.
|
|
385
438
|
|
|
386
439
|
### <a name="remove-modifiers"></a> RemoveModifiers
|
|
387
440
|
|
|
388
|
-
By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will
|
|
441
|
+
By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will
|
|
442
|
+
ignore rules with unsupported modifiers, and all of the modifiers listed here
|
|
443
|
+
are unsupported. However, the rules with these modifiers are likely to be okay
|
|
444
|
+
for DNS-level blocking, that's why you might want to remove them when importing
|
|
445
|
+
rules from a traditional filter list.
|
|
389
446
|
|
|
390
447
|
Here is the list of modifiers that will be removed:
|
|
391
448
|
|
|
@@ -401,7 +458,10 @@ Here is the list of modifiers that will be removed:
|
|
|
401
458
|
|
|
402
459
|
### <a name="invertallow"></a> InvertAllow
|
|
403
460
|
|
|
404
|
-
This transformation converts blocking rules to "allow" rules. Note, that it
|
|
461
|
+
This transformation converts blocking rules to "allow" rules. Note, that it
|
|
462
|
+
does nothing to /etc/hosts rules (unless they were previously converted to
|
|
463
|
+
adblock-style syntax by a different transformation, for example
|
|
464
|
+
[Compress](#compress)).
|
|
405
465
|
|
|
406
466
|
There are two important notes about this transformation:
|
|
407
467
|
|
|
@@ -412,7 +472,7 @@ There are two important notes about this transformation:
|
|
|
412
472
|
|
|
413
473
|
Original list:
|
|
414
474
|
|
|
415
|
-
```
|
|
475
|
+
```text
|
|
416
476
|
! comment 1
|
|
417
477
|
rule1
|
|
418
478
|
|
|
@@ -423,7 +483,7 @@ rule1
|
|
|
423
483
|
|
|
424
484
|
Here's what we will have after applying this transformation:
|
|
425
485
|
|
|
426
|
-
```
|
|
486
|
+
```text
|
|
427
487
|
! comment 1
|
|
428
488
|
@@rule1
|
|
429
489
|
|
|
@@ -434,36 +494,47 @@ Here's what we will have after applying this transformation:
|
|
|
434
494
|
|
|
435
495
|
### <a name="validate"></a> Validate
|
|
436
496
|
|
|
437
|
-
This transformation is really crucial if you're using a filter list for a
|
|
497
|
+
This transformation is really crucial if you're using a filter list for a
|
|
498
|
+
traditional ad blocker as a source.
|
|
438
499
|
|
|
439
500
|
It removes dangerous or incompatible rules from the list.
|
|
440
501
|
|
|
441
502
|
So here's what it does:
|
|
442
503
|
|
|
443
|
-
- Discards domain-specific rules (e.g. `||example.org^$domain=example.com`).
|
|
444
|
-
|
|
504
|
+
- Discards domain-specific rules (e.g. `||example.org^$domain=example.com`).
|
|
505
|
+
You don't want to have domain-specific rules working globally.
|
|
506
|
+
- Discards rules with unsupported modifiers. Learn more about the
|
|
507
|
+
[supported modifiers](https://github.com/AdguardTeam/AdGuardHome/wiki/Hosts-Blocklists#-adblock-style-syntax).
|
|
445
508
|
- Discards rules that are too short.
|
|
446
509
|
- Discards IP addresses. If you need to keep IP addresses, use [ValidateAllowIp](#validate-allow-ip) instead.
|
|
447
510
|
|
|
448
511
|
#### <a name="rejected-ip-patterns"></a>Rejected IP Patterns
|
|
449
512
|
|
|
450
|
-
|
|
513
|
+
The following IP patterns are rejected by all validation transformations
|
|
514
|
+
(`Validate`, `ValidateAllowIp`, `ValidateAllowPublicSuffix`,
|
|
515
|
+
`ValidateAllowIpAndPublicSuffix`) as they are either unsafe or ambiguous:
|
|
451
516
|
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
517
|
+
- `||192.168.1^` — 3-octet with `^` - does not work
|
|
518
|
+
- `192.168.1` — Ambiguous: would match `192.168.11`, `192.168.111`, etc.
|
|
519
|
+
- `1.2.` or `1.2.*` — Too wide (1-2 octets), use regex instead
|
|
455
520
|
|
|
456
|
-
- Removes rules that block entire top-level domains (TLDs) like `||*.org^`,
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
521
|
+
- Removes rules that block entire top-level domains (TLDs) like `||*.org^`,
|
|
522
|
+
unless they have specific limiting modifiers such as `$denyallow`,
|
|
523
|
+
`$badfilter`, or `$client`. Examples:
|
|
524
|
+
- `||*.org^` - this rule will be removed
|
|
525
|
+
- `||*.org^$denyallow=example.com` - this rule will be kept because it
|
|
526
|
+
has a limiting modifier
|
|
527
|
+
If such rules must be saved, use
|
|
528
|
+
[ValidateAllowPublicSuffix](#validate-allow-public-suffix) or
|
|
529
|
+
[ValidateAllowIpAndPublicSuffix](#validate-allow-ip-and-public-suffix).
|
|
461
530
|
|
|
462
531
|
If there are comments preceding the invalid rule, they will be removed as well.
|
|
463
532
|
|
|
464
533
|
### <a name="validate-allow-ip"></a> ValidateAllowIp
|
|
465
534
|
|
|
466
|
-
This transformation extends [Validate](#validate) to allow IP addresses in
|
|
535
|
+
This transformation extends [Validate](#validate) to allow IP addresses in
|
|
536
|
+
the lists. It also **normalizes IP rules** in Adblock-style to the safe format
|
|
537
|
+
`||ip^`.
|
|
467
538
|
|
|
468
539
|
#### IP Rule Normalization
|
|
469
540
|
|
|
@@ -478,25 +549,47 @@ This transformation extends [Validate](#validate) to allow IP addresses in the l
|
|
|
478
549
|
|
|
479
550
|
Modifiers like `$important`, `$client`, `$denyallow`, `$badfilter` are preserved during normalization.
|
|
480
551
|
|
|
481
|
-
> **Note:** Invalid IP patterns are rejected solely by the base
|
|
552
|
+
> **Note:** Invalid IP patterns are rejected solely by the base
|
|
553
|
+
> [Validate](#validate) logic — normalization only converts valid patterns to
|
|
554
|
+
> canonical form and passes everything else through unchanged. See
|
|
555
|
+
> [Rejected IP Patterns](#rejected-ip-patterns) for details.
|
|
482
556
|
|
|
483
557
|
### <a name="validate-allow-public-suffix"></a> ValidateAllowPublicSuffix
|
|
484
558
|
|
|
485
|
-
This transformation exactly repeats the behavior of [Validate](#validate),
|
|
559
|
+
This transformation exactly repeats the behavior of [Validate](#validate),
|
|
560
|
+
but leaves rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`)
|
|
561
|
+
in the list.
|
|
486
562
|
|
|
487
|
-
It still filters out invalid syntax rules and unsupported modifiers, but does
|
|
563
|
+
It still filters out invalid syntax rules and unsupported modifiers, but does
|
|
564
|
+
not reject public-suffix rules unless the rule itself is malformed.
|
|
488
565
|
|
|
489
|
-
> **Note:** Combining any `Validate`, `ValidateAllowIp`,
|
|
566
|
+
> **Note:** Combining any `Validate`, `ValidateAllowIp`,
|
|
567
|
+
> `ValidateAllowPublicSuffix`, and `ValidateAllowIpAndPublicSuffix` in one
|
|
568
|
+
> transformation list is not allowed and will result in an error. Each runs
|
|
569
|
+
> its own validator on the already-filtered output of the previous one, so
|
|
570
|
+
> allow-modes become silently ineffective.
|
|
490
571
|
|
|
491
|
-
> **Important:** Validation transformations also cannot be used at both
|
|
572
|
+
> **Important:** Validation transformations also cannot be used at both
|
|
573
|
+
> source-level and top-level simultaneously. For example, if a source uses
|
|
574
|
+
> `ValidateAllowPublicSuffix` and the top-level configuration uses `Validate`,
|
|
575
|
+
> the compiler will throw an error. This is because the top-level `Validate`
|
|
576
|
+
> would override the source-level validation, making
|
|
577
|
+
> `ValidateAllowPublicSuffix` ineffective. Use validation transformations at
|
|
578
|
+
> only one level.
|
|
492
579
|
|
|
493
580
|
### <a name="validate-allow-ip-and-public-suffix"></a> ValidateAllowIpAndPublicSuffix
|
|
494
581
|
|
|
495
|
-
This transformation combines the behavior of
|
|
582
|
+
This transformation combines the behavior of
|
|
583
|
+
[ValidateAllowIp](#validate-allow-ip) and
|
|
584
|
+
[ValidateAllowPublicSuffix](#validate-allow-public-suffix). It allows both IP
|
|
585
|
+
addresses and public suffix rules in the list.
|
|
496
586
|
|
|
497
|
-
Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format
|
|
587
|
+
Like `ValidateAllowIp`, it normalizes incomplete IP rules to the safe format
|
|
588
|
+
`||ip^` before validation. See [IP Rule Normalization](#ip-rule-normalization)
|
|
589
|
+
for details.
|
|
498
590
|
|
|
499
|
-
Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public
|
|
591
|
+
Like `ValidateAllowPublicSuffix`, it keeps rules that match whole public
|
|
592
|
+
suffixes (e.g. `||hl.cn^`, `||org^`).
|
|
500
593
|
|
|
501
594
|
### <a name="deduplicate"></a> Deduplicate
|
|
502
595
|
|
|
@@ -505,11 +598,12 @@ This transformation simply removes the duplicates from the specified source.
|
|
|
505
598
|
There are two important notes about this transformation:
|
|
506
599
|
|
|
507
600
|
1. It keeps the original rules order.
|
|
508
|
-
2. It ignores comments. However, if the comments precede the rule that is
|
|
601
|
+
2. It ignores comments. However, if the comments precede the rule that is
|
|
602
|
+
being removed, the comments will be also removed.
|
|
509
603
|
|
|
510
604
|
For instance:
|
|
511
605
|
|
|
512
|
-
```
|
|
606
|
+
```text
|
|
513
607
|
! rule1 comment 1
|
|
514
608
|
rule1
|
|
515
609
|
! rule1 comment 2
|
|
@@ -518,7 +612,7 @@ rule1
|
|
|
518
612
|
|
|
519
613
|
Here's what will be left after the transformation:
|
|
520
614
|
|
|
521
|
-
```
|
|
615
|
+
```text
|
|
522
616
|
! rule1 comment 2
|
|
523
617
|
rule1
|
|
524
618
|
```
|
|
@@ -531,7 +625,7 @@ This is a very simple transformation that removes empty lines.
|
|
|
531
625
|
|
|
532
626
|
Original list:
|
|
533
627
|
|
|
534
|
-
```
|
|
628
|
+
```text
|
|
535
629
|
rule1
|
|
536
630
|
|
|
537
631
|
rule2
|
|
@@ -542,7 +636,7 @@ rule3
|
|
|
542
636
|
|
|
543
637
|
Here's what we will have after applying this transformation:
|
|
544
638
|
|
|
545
|
-
```
|
|
639
|
+
```text
|
|
546
640
|
rule1
|
|
547
641
|
rule2
|
|
548
642
|
rule3
|
|
@@ -556,7 +650,7 @@ This is a very simple transformation that inserts a final newline.
|
|
|
556
650
|
|
|
557
651
|
Original list:
|
|
558
652
|
|
|
559
|
-
```
|
|
653
|
+
```text
|
|
560
654
|
rule1
|
|
561
655
|
rule2
|
|
562
656
|
rule3
|
|
@@ -564,7 +658,7 @@ rule3
|
|
|
564
658
|
|
|
565
659
|
Here's what we will have after applying this transformation:
|
|
566
660
|
|
|
567
|
-
```
|
|
661
|
+
```text
|
|
568
662
|
rule1
|
|
569
663
|
rule2
|
|
570
664
|
rule3
|
|
@@ -573,8 +667,14 @@ rule3
|
|
|
573
667
|
|
|
574
668
|
`RemoveEmptyLines` doesn't delete this empty row due to the execution order.
|
|
575
669
|
|
|
576
|
-
##
|
|
670
|
+
## Documentation
|
|
577
671
|
|
|
578
|
-
- [Development
|
|
672
|
+
- [Development](DEVELOPMENT.md) — how to set up the environment, build, test, and contribute
|
|
673
|
+
- [Deployment and configuration](DEPLOYMENT.md) — release pipeline, CI/CD, and Docker build
|
|
579
674
|
- [LLM agent rules](AGENTS.md) — code guidelines and project conventions
|
|
580
675
|
- [Changelog](CHANGELOG.md) — release history
|
|
676
|
+
|
|
677
|
+
[npm-badge]: https://nodei.co/npm/@adguard/hostlist-compiler.png?compact=true
|
|
678
|
+
[npm]: https://www.npmjs.com/package/@adguard/hostlist-compiler/
|
|
679
|
+
[private-repo]: https://github.com/AdGuardSoftwareLimited/filters-hostlist-compiler
|
|
680
|
+
[public-mirror]: https://github.com/AdguardTeam/HostlistCompiler
|