@adguard/hostlist-compiler 2.1.1 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/prepare-release.yml +5 -0
- package/.markdownlint.json +29 -0
- package/.markdownlintignore +1 -0
- package/AGENTS.md +310 -98
- package/CHANGELOG.md +62 -8
- package/DEPLOYMENT.md +175 -0
- package/DEVELOPMENT.md +171 -72
- package/README.md +238 -143
- package/examples/china/README.md +2 -2
- package/examples/energized/README.md +4 -2
- package/examples/sdn/README.md +2 -2
- package/examples/whitelist/README.md +2 -2
- package/package.json +24 -18
- package/src/cli.js +53 -31
- package/src/compile-source.js +4 -3
- package/src/configuration.js +132 -2
- package/src/download.js +27 -0
- package/src/filter.js +2 -2
- package/src/index.d.ts +2 -0
- package/src/index.js +16 -50
- package/src/{transformations/ip-normalize.js → ip-normalize.js} +2 -2
- package/src/schemas/configuration.schema.js +137 -0
- package/src/source-types.js +17 -0
- package/src/transformations/enum.js +49 -0
- package/src/transformations/include.js +1 -1
- package/src/transformations/transform.js +40 -74
- package/src/transformations/validate-allow-ip-and-public-suffix.js +1 -1
- package/src/transformations/validate-allow-ip.js +1 -1
- package/src/utils.js +1284 -0
- package/src/validation-conflicts.js +106 -0
- package/test/cli-error-output.test.js +305 -0
- package/test/cli.test.js +47 -0
- package/test/compile-source.test.js +167 -7
- package/test/configuration.test.js +438 -0
- package/test/download.test.js +260 -0
- package/test/helpers/http-mock.js +94 -0
- package/test/index.test.js +497 -4
- package/test/{transformations/ip-normalize.test.js → ip-normalize.test.js} +1 -1
- package/test/resolve-include.test.js +8 -8
- package/test/transformations/covert-to-ascii.test.js +1 -1
- package/test/transformations/exclude.test.js +2 -2
- package/test/transformations/include.test.js +9 -3
- package/test/transformations/ip-alignment.test.js +1 -1
- package/test/transformations/transform.test.js +66 -1
- package/test/utils.test.js +1559 -0
- package/test/validation-conflicts.test.js +230 -0
- package/src/schemas/configuration.schema.json +0 -369
|
@@ -1,5 +1,10 @@
|
|
|
1
1
|
name: Prepare release
|
|
2
2
|
|
|
3
|
+
# The run list shows the requested tag together with the workflow name.
|
|
4
|
+
# `github.workflow` repeats the `name` value from one place, so a rename of the
|
|
5
|
+
# workflow also updates the run name.
|
|
6
|
+
run-name: '${{ github.workflow }} ${{ inputs.tag }}'
|
|
7
|
+
|
|
3
8
|
description: >-
|
|
4
9
|
Opens a release PR that finalizes CHANGELOG.md for the given tag.
|
|
5
10
|
After merge, publish-release.yml takes over.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
{
|
|
2
|
+
"ul-indent": {
|
|
3
|
+
"indent": 4
|
|
4
|
+
},
|
|
5
|
+
"line-length": {
|
|
6
|
+
"strict": true,
|
|
7
|
+
"line_length": 120,
|
|
8
|
+
"tables": false
|
|
9
|
+
},
|
|
10
|
+
"no-multiple-blanks": {
|
|
11
|
+
"maximum": 2
|
|
12
|
+
},
|
|
13
|
+
"no-inline-html": {
|
|
14
|
+
"allowed_elements": [
|
|
15
|
+
"a"
|
|
16
|
+
]
|
|
17
|
+
},
|
|
18
|
+
"no-duplicate-heading": {
|
|
19
|
+
"siblings_only": true
|
|
20
|
+
},
|
|
21
|
+
"no-blanks-blockquote": false,
|
|
22
|
+
"no-bare-urls": false,
|
|
23
|
+
"table-column-style": {
|
|
24
|
+
"style": "aligned"
|
|
25
|
+
},
|
|
26
|
+
"ul-style": {
|
|
27
|
+
"style": "dash"
|
|
28
|
+
}
|
|
29
|
+
}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
node_modules/
|
package/AGENTS.md
CHANGED
|
@@ -2,12 +2,29 @@
|
|
|
2
2
|
|
|
3
3
|
Guidelines for AI agents working on this project.
|
|
4
4
|
|
|
5
|
+
## Table of Contents
|
|
6
|
+
|
|
7
|
+
- [Project Overview](#project-overview)
|
|
8
|
+
- [Technical Context](#technical-context)
|
|
9
|
+
- [Project Structure](#project-structure)
|
|
10
|
+
- [Build And Test Commands](#build-and-test-commands)
|
|
11
|
+
- [Contribution Instructions](#contribution-instructions)
|
|
12
|
+
- [Code Guidelines](#code-guidelines)
|
|
13
|
+
- [System Design](#system-design)
|
|
14
|
+
- [Architecture](#architecture)
|
|
15
|
+
- [Code Quality](#code-quality)
|
|
16
|
+
- [Testing](#testing)
|
|
17
|
+
- [Dependency Management](#dependency-management)
|
|
18
|
+
- [Configuration & Documentation](#configuration--documentation)
|
|
19
|
+
- [Markdown Formatting](#markdown-formatting)
|
|
20
|
+
- [Other](#other)
|
|
21
|
+
|
|
5
22
|
## Project Overview
|
|
6
23
|
|
|
7
24
|
**Hostlist Compiler** (`@adguard/hostlist-compiler`) is a Node.js CLI tool and
|
|
8
|
-
library
|
|
9
|
-
|
|
10
|
-
|
|
25
|
+
library for building DNS filter lists; see [README.md](README.md) for the
|
|
26
|
+
user-facing description. The sections below summarize what agents need to know
|
|
27
|
+
when working on this codebase.
|
|
11
28
|
|
|
12
29
|
It supports multiple input formats (`/etc/hosts`, adblock-style), configurable
|
|
13
30
|
transformations (compress, deduplicate, validate, remove comments, etc.),
|
|
@@ -17,49 +34,63 @@ inclusion/exclusion rules, and `!#include` directive resolution via
|
|
|
17
34
|
## Technical Context
|
|
18
35
|
|
|
19
36
|
- **Language**: JavaScript (Node.js, CommonJS modules)
|
|
20
|
-
- **Node
|
|
37
|
+
- **Node / pnpm Versions**: see the `engines` field in `package.json` (the
|
|
38
|
+
single source of truth)
|
|
21
39
|
- **Primary Dependencies**:
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
- **Linting**: ESLint with `airbnb-base` config, `@babel/eslint-parser`
|
|
40
|
+
- `@adguard/filters-downloader` — resolves `!#include` directives and
|
|
41
|
+
downloads remote sources
|
|
42
|
+
- `ajv` / `better-ajv-errors` — JSON schema validation for configuration
|
|
43
|
+
- `lodash` — utility functions
|
|
44
|
+
- `tldts` — TLD parsing for domain validation
|
|
45
|
+
- `yargs` — CLI argument parsing
|
|
46
|
+
- `consola` — logging
|
|
47
|
+
- **Testing**: Jest (`jest --runInBand --detectOpenHandles`, silent mode)
|
|
48
|
+
- **Linting**: ESLint with `airbnb-base` config and `@babel/eslint-parser`;
|
|
49
|
+
markdownlint for Markdown files
|
|
33
50
|
- **Storage**: None (stateless CLI tool, reads remote/local files, writes output
|
|
34
51
|
files)
|
|
35
52
|
- **Target Platform**: Node.js (CLI and library; published to npm)
|
|
36
|
-
- **Project Type**: Single package (CLI + library)
|
|
53
|
+
- **Project Type**: Single package (CLI tool + library)
|
|
37
54
|
- **License**: GPL-3.0
|
|
38
55
|
|
|
39
56
|
## Project Structure
|
|
40
57
|
|
|
41
58
|
```text
|
|
42
59
|
├── src/ # Main source code
|
|
43
|
-
│ ├── cli.js # CLI entry point (hashbang, yargs
|
|
44
|
-
│ ├── index.js # Library entry point (
|
|
45
|
-
│ ├── index.d.ts # TypeScript type declarations
|
|
46
|
-
│ ├── compile-source.js # Compiles
|
|
47
|
-
│ ├── configuration.js # Configuration validation (AJV)
|
|
48
|
-
│ ├──
|
|
60
|
+
│ ├── cli.js # CLI entry point (hashbang, yargs, config loading)
|
|
61
|
+
│ ├── index.js # Library entry point: compile(configuration)
|
|
62
|
+
│ ├── index.d.ts # TypeScript type declarations for consumers
|
|
63
|
+
│ ├── compile-source.js # Compiles a single source
|
|
64
|
+
│ ├── configuration.js # Configuration creation and validation (AJV + schema)
|
|
65
|
+
│ ├── download.js # Central download layer (wraps FiltersDownloader)
|
|
66
|
+
│ ├── filter.js # Include/exclude wildcard filtering
|
|
49
67
|
│ ├── rule.js # Rule parsing (adblock, /etc/hosts)
|
|
50
|
-
│ ├──
|
|
51
|
-
│ ├──
|
|
52
|
-
│
|
|
68
|
+
│ ├── ip-normalize.js # Internal helper: IP rule normalization (used by transformations)
|
|
69
|
+
│ ├── source-types.js # Source-type vocabulary (SOURCE_TYPES) for the schema and the CLI
|
|
70
|
+
│ ├── validation-conflicts.js # Validation-conflict rules (runtime checks + generated schema fragments)
|
|
71
|
+
│ ├── utils.js # String, IP, wildcard, and credential-redaction utilities
|
|
72
|
+
│ ├── schemas/ # Configuration schema (transformation names + conflict rules)
|
|
73
|
+
│ └── transformations/ # 15 transformations + fixed-order pipeline
|
|
74
|
+
│ ├── enum.js # TRANSFORMATIONS enum + VALIDATION_TRANSFORMATIONS (single source of truth)
|
|
75
|
+
│ ├── transform.js # Pipeline orchestrator
|
|
76
|
+
│ ├── exclude.js # Exclusion rules (async, ungated)
|
|
77
|
+
│ ├── include.js # Inclusion rules (async, ungated)
|
|
78
|
+
│ └── # ... (13 simple transforms: compress, validate, deduplicate, etc.)
|
|
53
79
|
├── test/ # Jest test suites mirroring src/
|
|
80
|
+
│ ├── helpers/ # Shared test helpers (HTTP mocking)
|
|
54
81
|
│ ├── resources/ # Test fixture files
|
|
55
82
|
│ └── transformations/ # Transformation-specific tests
|
|
56
|
-
├── examples/ # Example configurations (sdn, energized,
|
|
83
|
+
├── examples/ # Example configurations (sdn, energized, china, whitelist)
|
|
57
84
|
├── .github/workflows/ # GitHub Actions CI (ci, mirror, prepare-release, publish-release)
|
|
58
85
|
├── Dockerfile # Multi-stage build: test-output / build-output targets
|
|
59
86
|
├── package.json # Package manifest and scripts
|
|
60
87
|
├── jest.config.js # Jest configuration
|
|
61
|
-
├── .eslintrc.js # ESLint configuration
|
|
88
|
+
├── .eslintrc.js # ESLint configuration (airbnb-base)
|
|
89
|
+
├── .markdownlint.json # Markdownlint configuration
|
|
90
|
+
├── .markdownlintignore # Files excluded from Markdown linting
|
|
62
91
|
├── CHANGELOG.md # Release history
|
|
92
|
+
├── DEPLOYMENT.md # Deployment, CI/CD and release documentation
|
|
93
|
+
├── DEVELOPMENT.md # Environment setup guide (setup, not agent rules)
|
|
63
94
|
├── README.md # User documentation
|
|
64
95
|
└── LICENSE # GPL-3.0
|
|
65
96
|
```
|
|
@@ -73,88 +104,160 @@ pnpm lint
|
|
|
73
104
|
pnpm test
|
|
74
105
|
```
|
|
75
106
|
|
|
76
|
-
| Action
|
|
77
|
-
|
|
78
|
-
| Install
|
|
79
|
-
| Lint | `pnpm lint` |
|
|
80
|
-
|
|
|
81
|
-
|
|
|
82
|
-
|
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
You MUST follow these rules for EVERY task you perform:
|
|
87
|
-
|
|
88
|
-
1. **Run the linter**: `pnpm lint`. Fix all warnings and errors before
|
|
89
|
-
proceeding. The project uses ESLint with `airbnb-base`.
|
|
90
|
-
2. **Run the tests**: `pnpm test`. All tests must pass. If you changed
|
|
91
|
-
behavior, verify that existing tests still reflect correct expectations.
|
|
92
|
-
3. **Check the outcome**. If either command fails, fix the issues and re-run
|
|
93
|
-
until both pass. A task is not complete until all checks are green.
|
|
94
|
-
|
|
95
|
-
## Mandatory Task Actions
|
|
96
|
-
|
|
97
|
-
For every task you perform, you MUST:
|
|
98
|
-
|
|
99
|
-
- **Update or add unit tests** for any changed or new code. Mirror the source
|
|
100
|
-
file structure under `test/`.
|
|
101
|
-
- **Run the full verification sequence** (`pnpm lint` and `pnpm test`) and
|
|
102
|
-
confirm both pass.
|
|
103
|
-
- **Update `AGENTS.md`** if you change the project structure, add new build
|
|
104
|
-
commands, or discover a recurring agent mistake that should become a rule.
|
|
105
|
-
- **Update `CHANGELOG.md`** when making user-facing changes.
|
|
106
|
-
- **Update the JSON schema** (`src/schemas/configuration.schema.json`) when
|
|
107
|
-
adding or changing configuration options.
|
|
108
|
-
- **Update TypeScript declarations** (`src/index.d.ts`) when changing the
|
|
109
|
-
public API.
|
|
110
|
-
- **Verify your code follows the Code Guidelines** in this file after
|
|
111
|
-
completing the task.
|
|
112
|
-
- If a prompt asks you to refactor or improve existing code, check whether
|
|
113
|
-
the insight can be phrased as a code guideline. If so, add it to the
|
|
114
|
-
relevant Code Guidelines section below.
|
|
107
|
+
| Action | Command |
|
|
108
|
+
| ------------- | ---------------------------------------------------- |
|
|
109
|
+
| Install | `pnpm install` |
|
|
110
|
+
| Lint (all) | `pnpm lint` (runs `lint:code` and `lint:md`) |
|
|
111
|
+
| Lint code | `pnpm lint:code` (ESLint, `airbnb-base` config) |
|
|
112
|
+
| Lint markdown | `pnpm lint:md` (markdownlint) |
|
|
113
|
+
| Test | `pnpm test` (`jest --runInBand --detectOpenHandles`) |
|
|
114
|
+
|
|
115
|
+
For running the CLI locally and bumping the version, see
|
|
116
|
+
[DEVELOPMENT.md](DEVELOPMENT.md).
|
|
115
117
|
|
|
116
118
|
## Contribution Instructions
|
|
117
119
|
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
120
|
+
- You MUST verify your work with the linter and the Markdown formatter. The
|
|
121
|
+
project has no type checker (plain JavaScript); ESLint and markdownlint are
|
|
122
|
+
the gates (see [Build and Test Commands](#build-and-test-commands)); use
|
|
123
|
+
`pnpm exec eslint . --fix` and `pnpm exec markdownlint . --fix` to
|
|
124
|
+
auto-fix issues.
|
|
125
|
+
- You MUST update or add unit tests for any changed or new code, mirroring the
|
|
126
|
+
source file structure under `test/` (`<module>.test.js`).
|
|
127
|
+
- You MUST run `pnpm test` and confirm all tests pass — a task is not complete
|
|
128
|
+
until all checks are green.
|
|
129
|
+
- When you change the project structure or add new build commands, update the
|
|
130
|
+
Project Structure and Build And Test Commands sections in `AGENTS.md`.
|
|
131
|
+
- If a prompt asks you to refactor or improve existing code, check whether the
|
|
132
|
+
insight can be phrased as a code guideline; if so, add it to the relevant
|
|
133
|
+
Code Guidelines section in `AGENTS.md`.
|
|
134
|
+
- After completing a task, verify that the code you wrote follows the Code
|
|
135
|
+
Guidelines in this file.
|
|
136
|
+
- Even when the task is to write or update a document (e.g. a plan or any
|
|
137
|
+
Markdown file) rather than code, run `pnpm lint:md` to verify and fix
|
|
138
|
+
Markdown formatting.
|
|
139
|
+
- Install dependencies with `pnpm install` before working.
|
|
140
|
+
- Do not commit generated files, `node_modules`, or editor configs.
|
|
141
|
+
- Do not create git commits or PRs unless explicitly asked.
|
|
142
|
+
- Keep documentation in sync as described in
|
|
143
|
+
[Configuration & Documentation](#configuration--documentation) below.
|
|
144
|
+
- **Use ticket-prefixed commit messages**: Commit messages MUST start with the
|
|
145
|
+
ticket number (`AG-XXX`) so they auto-link with the task tracker, followed
|
|
146
|
+
by a short description in the present tense (e.g. `AG-1234 Fix login
|
|
147
|
+
redirect`). Automated commits made by CI (e.g. the CHANGELOG finalization
|
|
148
|
+
in the release-* PRs) use a [Conventional Commits] prefix such as `docs:`
|
|
149
|
+
instead.
|
|
150
|
+
|
|
151
|
+
[Conventional Commits]: https://www.conventionalcommits.org/en/v1.0.0/
|
|
125
152
|
|
|
126
153
|
## Code Guidelines
|
|
127
154
|
|
|
155
|
+
### System Design
|
|
156
|
+
|
|
157
|
+
The package is both a CLI tool and a library, so both sets of rules apply:
|
|
158
|
+
|
|
159
|
+
- The CLI runs and exits — no long-lived state, no background daemons. It
|
|
160
|
+
exits with code 0 on success and code 1 on failure; error handling is
|
|
161
|
+
specified in [Code Quality](#code-quality).
|
|
162
|
+
- Write the compiled filter list to the required `-o` output file; use
|
|
163
|
+
`consola` for diagnostics (`-v` / `--verbose` enables debug and trace).
|
|
164
|
+
- Be composable — support quick `hosts`-file conversion via `-i` without a
|
|
165
|
+
configuration file, and full mode via `-c <config>`; sources can be local
|
|
166
|
+
files or remote URLs.
|
|
167
|
+
- Fail fast with clear messages — validate the configuration with AJV before
|
|
168
|
+
compiling; throw errors that include the invalid value so the user can fix
|
|
169
|
+
it.
|
|
170
|
+
- Keep startup time fast — no heavy initialization on the CLI code path.
|
|
171
|
+
- As a library, `compile(configuration)` must not mutate global state
|
|
172
|
+
(environment variables, process listeners); the only I/O it performs is the
|
|
173
|
+
downloads explicitly requested by the configuration's sources.
|
|
174
|
+
- Export a stable public API (`compile`, configuration/source types) with
|
|
175
|
+
complete TypeScript declarations in `src/index.d.ts`; internal functions are
|
|
176
|
+
not exported.
|
|
177
|
+
- Handle errors by throwing descriptive `Error` / `TypeError` instances and
|
|
178
|
+
let the caller decide how to recover; see the error-handling rules in
|
|
179
|
+
[Code Quality](#code-quality).
|
|
180
|
+
|
|
128
181
|
### Architecture
|
|
129
182
|
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
183
|
+
The easiest way to achieve these principles is **layered architecture**.
|
|
184
|
+
This project's layers, from top to bottom:
|
|
185
|
+
|
|
186
|
+
```text
|
|
187
|
+
CLI entry (src/cli.js)
|
|
188
|
+
↓
|
|
189
|
+
Library entry / orchestration (src/index.js)
|
|
190
|
+
↓
|
|
191
|
+
Configuration creation and validation (src/configuration.js, src/schemas/configuration.schema.js)
|
|
192
|
+
Per-source compilation (src/compile-source.js)
|
|
193
|
+
↓
|
|
194
|
+
Transformation pipeline (src/transformations/transform.js)
|
|
195
|
+
↓
|
|
196
|
+
Transformations (src/transformations/*.js)
|
|
197
|
+
↓
|
|
198
|
+
Rule parsing (src/rule.js) Filtering (src/filter.js) Utilities (src/utils.js)
|
|
199
|
+
↓
|
|
200
|
+
Central download layer (src/download.js) (used by compile-source.js and filter.js)
|
|
201
|
+
↓
|
|
202
|
+
@adguard/filters-downloader (external package, downloads sources and includes)
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Each layer may call the layers below it; no layer depends on a layer above it.
|
|
206
|
+
`utils.js` and `download.js` are the leaf modules (they depend only on external
|
|
207
|
+
packages — `lodash` and `@adguard/filters-downloader` respectively); there are
|
|
208
|
+
no circular dependencies.
|
|
209
|
+
|
|
210
|
+
Universal design principles the codebase should follow:
|
|
211
|
+
|
|
212
|
+
- **Separation of Concerns** — parsing, filtering, transformations, and
|
|
213
|
+
configuration validation live in separate modules.
|
|
214
|
+
- **Single Responsibility Principle** — each transformation module does exactly
|
|
215
|
+
one transformation; each module has one reason to change.
|
|
216
|
+
- **Dependency Direction** — dependencies point downward, from entry points to
|
|
217
|
+
utilities; lower layers never import entry points.
|
|
218
|
+
- **Explicit Boundaries** — transformations communicate through
|
|
219
|
+
`(rules, ...) => rules` signatures; the `TRANSFORMATIONS` enum is frozen.
|
|
220
|
+
- **Data Flow Clarity** — rules flow through a single fixed-order pipeline,
|
|
221
|
+
making the data path predictable and traceable.
|
|
222
|
+
- **Minimize Coupling, Maximize Cohesion** — modules are self-contained and
|
|
223
|
+
interact through narrow interfaces.
|
|
224
|
+
- **Make Invalid States Impossible** — configuration is validated against a
|
|
225
|
+
JSON schema; incompatible transformation combinations are rejected at
|
|
226
|
+
runtime and in the schema. Less critical at compile time: the project is
|
|
227
|
+
plain JavaScript, so runtime validation carries this burden.
|
|
228
|
+
- **Single Source of Truth for Transformation Names** — the `TRANSFORMATIONS`
|
|
229
|
+
enum in `src/transformations/enum.js` is the single source of truth for
|
|
230
|
+
transformation *names*: the configuration schema's accepted names and the
|
|
231
|
+
validation-conflict rules are derived from it, so renaming a transformation
|
|
232
|
+
in the enum updates those automatically. Registering a *new* transformation
|
|
233
|
+
is still a multi-step process — add the enum entry (`enum.js`), register the
|
|
234
|
+
implementation in the pipeline (`src/transformations/transform.js`), and
|
|
235
|
+
update the exported `Transformation` type (`src/index.d.ts`) and the order
|
|
236
|
+
list (`README.md`) by hand; a test asserts the pipeline covers every enum
|
|
237
|
+
entry. The steps are documented in DEVELOPMENT.md.
|
|
238
|
+
- **Single Source of Truth for Conflict Rules** — validation-conflict rules
|
|
239
|
+
(which validation transformations conflict, and at which levels) live in
|
|
240
|
+
`src/validation-conflicts.js`; the schema conflict rules and the runtime
|
|
241
|
+
checks are generated from that module, so they cannot drift apart. The list
|
|
242
|
+
of validation transformations (`VALIDATION_TRANSFORMATIONS`) is defined next
|
|
243
|
+
to the enum in `src/transformations/enum.js`, so the conflict-rule side of
|
|
244
|
+
adding a validator is a single-file change.
|
|
245
|
+
- **Observability Built-in** — all logging goes through `consola`; see the
|
|
246
|
+
logging and error-handling rules in [Code Quality](#code-quality).
|
|
247
|
+
- **Keep It Boring** — plain CommonJS modules, no framework magic, simple
|
|
248
|
+
synchronous or async transformation functions.
|
|
138
249
|
|
|
139
250
|
### Code Quality
|
|
140
251
|
|
|
141
|
-
- **ESLint airbnb-base** rules apply — run `pnpm lint` to check
|
|
142
|
-
`.eslintrc.js`
|
|
252
|
+
- **ESLint airbnb-base** rules apply — run `pnpm lint:code` to check; the
|
|
253
|
+
project-specific overrides are configured in `.eslintrc.js` and not
|
|
254
|
+
restated here.
|
|
143
255
|
- Use `lodash` helpers (e.g., `_.startsWith`, `_.isEmpty`, `_.trim`) where they
|
|
144
256
|
are already used — stay consistent with the existing style.
|
|
145
257
|
- Prefer `for...of` loops over `Array.forEach` for async iteration (the
|
|
146
258
|
codebase disables `no-restricted-syntax` where needed).
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
- **Framework**: Jest, running in Node environment with `--runInBand`.
|
|
151
|
-
- **Mocking**: `mock-fs` for filesystem mocks, `nock` for HTTP request
|
|
152
|
-
interception.
|
|
153
|
-
- Test files follow the naming convention `<module>.test.js` under `test/`.
|
|
154
|
-
- Each transformation has its own test file under `test/transformations/`.
|
|
155
|
-
|
|
156
|
-
### Logging and Error Handling
|
|
157
|
-
|
|
259
|
+
- Use JSDoc comments for public functions (parameters and return values), e.g.
|
|
260
|
+
`transform()` in `src/transformations/transform.js`.
|
|
158
261
|
- Use `consola` for all logging (`consola.info`, `consola.debug`,
|
|
159
262
|
`consola.error`). Do not use `console.log` / `console.error` directly.
|
|
160
263
|
- Throw descriptive `Error` (or `TypeError`) instances with messages that
|
|
@@ -162,13 +265,122 @@ For every task you perform, you MUST:
|
|
|
162
265
|
and rule problems.
|
|
163
266
|
- Propagate errors up to the CLI entry point (`src/cli.js`), which catches
|
|
164
267
|
them, logs via `consola.error`, and exits with code 1.
|
|
268
|
+
- Credential-bearing values MUST pass through the `src/utils.js` sanitizers
|
|
269
|
+
before they reach a header, a log, or an error message. Sanitize the object
|
|
270
|
+
graph and the rendered form: `JSON.stringify`, `better-ajv-errors` output,
|
|
271
|
+
and `cause`/`errors` chains. Use `sanitizeConfigurationForLog()` for a
|
|
272
|
+
configuration dump or a validation error, `stripKnownUrlCredentials()` for a
|
|
273
|
+
value that must be a URL, and `stripUrlValueCredentials()` for free text. Do
|
|
274
|
+
not re-scan an already rendered document: a path-less URL can make the text
|
|
275
|
+
up to a later `@` look like userinfo.
|
|
276
|
+
- The free-text credential parser is a heuristic: it keeps text that reads as
|
|
277
|
+
prose. It can keep a credential that is lexically identical to prose, and it
|
|
278
|
+
can remove prose after a single-label intranet host. Do not add a one-off
|
|
279
|
+
rule for each reported shape: add a row to the host × separator family in
|
|
280
|
+
`test/utils.test.js` and document the shape in the `stripUrlCredentials()`
|
|
281
|
+
JSDoc instead.
|
|
282
|
+
|
|
283
|
+
### Testing
|
|
284
|
+
|
|
285
|
+
- **Framework**: Jest, running in Node environment with `--runInBand
|
|
286
|
+
--detectOpenHandles` and `silent: true`.
|
|
287
|
+
- **Mocking**: `mock-fs` for filesystem mocks, `undici` MockAgent for HTTP
|
|
288
|
+
request interception.
|
|
289
|
+
- Test files follow the naming convention `<module>.test.js` under `test/`,
|
|
290
|
+
mirroring the structure of `src/`.
|
|
291
|
+
- Each transformation has its own test file under `test/transformations/`.
|
|
292
|
+
- Add or update tests for every changed or new code path; all tests must pass
|
|
293
|
+
before a task is complete.
|
|
294
|
+
|
|
295
|
+
### Dependency Management
|
|
296
|
+
|
|
297
|
+
- **Pin all dependency versions explicitly** — all entries in `dependencies`
|
|
298
|
+
and `devDependencies` in `package.json` use exact versions (no `^`/`~`
|
|
299
|
+
ranges). When adding or updating a dependency, pin it to the version
|
|
300
|
+
resolved in `pnpm-lock.yaml`; never lower a version below what the lockfile
|
|
301
|
+
currently resolves to.
|
|
302
|
+
- **Do not commit the `packageManager` field** — pnpm 10 auto-adds it to
|
|
303
|
+
`package.json` when pnpm commands are run. The `engines.pnpm` field is the
|
|
304
|
+
single source of truth for the pnpm version; revert `packageManager` before
|
|
305
|
+
committing if pnpm added it.
|
|
306
|
+
- **Prefer vanilla solutions** — use the language's standard library and
|
|
307
|
+
built-in APIs when they adequately solve the problem. Only add a dependency
|
|
308
|
+
when it provides significant value over a vanilla implementation.
|
|
309
|
+
- **Reputable sources only** — dependencies MUST come from well-established,
|
|
310
|
+
actively maintained projects (npm download counts, repository activity,
|
|
311
|
+
known maintainers).
|
|
312
|
+
- **Avoid unpopular libraries** — do NOT add niche or obscure packages with
|
|
313
|
+
limited community adoption; they pose security risks and may become
|
|
314
|
+
unmaintained.
|
|
315
|
+
- **Minimize dependency count** — each new dependency increases attack
|
|
316
|
+
surface, bundle size, and maintenance burden. Justify every addition.
|
|
317
|
+
- **Use the latest stable version** — when adding a new dependency, check the
|
|
318
|
+
package registry for the latest stable release and use it; do not copy
|
|
319
|
+
outdated version numbers from memory or other projects.
|
|
320
|
+
- **Acknowledge the library-side trade-off** — exact pins favor reproducibility
|
|
321
|
+
but hurt library consumers: duplicates of the same dependency may be
|
|
322
|
+
installed alongside their own ranges, and transitive patches only arrive with
|
|
323
|
+
a new release. This trade-off is deliberate; do not loosen pins without
|
|
324
|
+
discussing the impact.
|
|
325
|
+
|
|
326
|
+
**Rationale**: Fewer, well-vetted dependencies reduce security
|
|
327
|
+
vulnerabilities, supply chain risks, and long-term maintenance costs.
|
|
328
|
+
|
|
329
|
+
### Configuration & Documentation
|
|
330
|
+
|
|
331
|
+
- The compiler is controlled by a JSON configuration object validated against
|
|
332
|
+
`src/schemas/configuration.schema.js` (draft-07) via AJV; unknown keys are
|
|
333
|
+
rejected (`additionalProperties: false`).
|
|
334
|
+
- Runtime behavior is selected via CLI flags (`-c` config file, `-i` quick
|
|
335
|
+
hosts conversion, `-o` output file, `-v` verbose logging). The compiler
|
|
336
|
+
defines no environment variables of its own and reads no `.env` files, but
|
|
337
|
+
the native `fetch` honors the standard proxy variables (`HTTP_PROXY`,
|
|
338
|
+
`HTTPS_PROXY`, `NO_PROXY`) when `NODE_USE_ENV_PROXY=1` is set.
|
|
339
|
+
- Example configurations live in `examples/` (`sdn`, `energized`, `china`,
|
|
340
|
+
`whitelist`) and are used for manual testing.
|
|
341
|
+
- Keep documentation in sync with code: update `README.md` when adding or
|
|
342
|
+
reordering transformations (the transformation order in `README.md` MUST
|
|
343
|
+
match `src/transformations/transform.js`), the default quick-conversion
|
|
344
|
+
pipeline documented in the `README.md` "Quick Hosts Conversion" section MUST
|
|
345
|
+
match `DEFAULT_TRANSFORMATIONS` in `src/transformations/transform.js`,
|
|
346
|
+
update `CHANGELOG.md` for user-facing changes, update
|
|
347
|
+
`src/schemas/configuration.schema.js` when adding configuration options
|
|
348
|
+
(transformation names come from `src/transformations/enum.js`, source types
|
|
349
|
+
from `src/source-types.js`), update `src/index.d.ts` when changing the
|
|
350
|
+
public API, and update `DEPLOYMENT.md` when changing the Dockerfile, CI
|
|
351
|
+
workflows, or deployment setup.
|
|
352
|
+
- The repository stores no secrets or credentials of its own and commits all
|
|
353
|
+
configuration files. A configuration can still contain credentialed source
|
|
354
|
+
URLs. The compiler sends them to the downloader as Basic auth. It MUST
|
|
355
|
+
redact them before any log, header, or error output (see the sanitizer rule
|
|
356
|
+
in [Code Quality](#code-quality)).
|
|
357
|
+
|
|
358
|
+
### Markdown Formatting
|
|
359
|
+
|
|
360
|
+
All Markdown files MUST pass markdownlint; the enforced rules live in
|
|
361
|
+
`.markdownlint.json` (plus markdownlint defaults) and are not restated here.
|
|
362
|
+
The following conventions go beyond what the linter enforces:
|
|
363
|
+
|
|
364
|
+
- **Line length**: Don't wrap lines artificially short — keep them close to
|
|
365
|
+
the length limit configured in the linter where possible.
|
|
366
|
+
- **Continuation lines**: When a list item wraps to the next line, align the
|
|
367
|
+
continuation with the first character of the item text, not the list
|
|
368
|
+
marker. This applies to all list types (ordered and unordered).
|
|
369
|
+
- **Emphasis**: Use asterisks (`*`) for emphasis (`*italic*`, `**bold**`) —
|
|
370
|
+
the linter only enforces a consistent style, not a specific one.
|
|
371
|
+
|
|
372
|
+
**Rationale**: Uniform Markdown formatting improves readability for both
|
|
373
|
+
humans and AI agents that consume project documentation.
|
|
165
374
|
|
|
166
375
|
### Other
|
|
167
376
|
|
|
168
|
-
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
377
|
+
- Do not restate rules that are configured in linter configs (`.eslintrc.js`,
|
|
378
|
+
`.markdownlint.json`) — reference the config files instead; they are the
|
|
379
|
+
source of truth.
|
|
380
|
+
- The transformation pipeline always runs in a fixed order regardless of the
|
|
381
|
+
order specified in configuration (see
|
|
382
|
+
[Configuration & Documentation](#configuration--documentation) for keeping
|
|
383
|
+
`README.md` in sync).
|
|
172
384
|
- Any recurring agent error is a defect in this document. If you had to be
|
|
173
385
|
corrected twice on the same issue, add a rule here to prevent it in the
|
|
174
386
|
future.
|
package/CHANGELOG.md
CHANGED
|
@@ -19,13 +19,53 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
19
19
|
|
|
20
20
|
### Security
|
|
21
21
|
|
|
22
|
+
## [3.0.0] - 2026-11-17
|
|
23
|
+
|
|
24
|
+
## [3.0.0-beta.0] - 2026-11-17
|
|
25
|
+
|
|
26
|
+
### Changed
|
|
27
|
+
|
|
28
|
+
- Updated [@adguard/filters-downloader] to v3.1.0. This version removes
|
|
29
|
+
`axios` and downloads filters with native `fetch` in Node.js. It requires
|
|
30
|
+
Node.js 22.22.2 or newer, and Node.js 23.x is not supported. Environment
|
|
31
|
+
proxy variables now require `NODE_USE_ENV_PROXY=1`. A relative `!#include`
|
|
32
|
+
path that leaves the directory of the containing file is now ignored
|
|
33
|
+
instead of failing the filter update. Credentials from malformed source
|
|
34
|
+
URLs no longer appear in error messages, stacks, causes, or other error
|
|
35
|
+
properties. `stripUrlCredentials` is now exported from the package root and
|
|
36
|
+
the browser entry point. [#130]
|
|
37
|
+
- Remote bodies that look like JSON (e.g. `[]`, `123`, `{"a":1}`) are now
|
|
38
|
+
treated as plain text instead of failing with
|
|
39
|
+
`filter.trim is not a function`. A JSON file served as `text/plain` now
|
|
40
|
+
adds its content to the list. [#130]
|
|
41
|
+
- The internal configuration schema moved from `src/schemas/configuration.schema.json`
|
|
42
|
+
to `src/schemas/configuration.schema.js`; the documented API is unchanged.
|
|
43
|
+
|
|
44
|
+
### Fixed
|
|
45
|
+
|
|
46
|
+
- CLI to print the original network error (`cause` chain) when a compile fails.
|
|
47
|
+
|
|
48
|
+
### Security
|
|
49
|
+
|
|
50
|
+
- Credentials in URL configuration fields (`source`, `homepage`, and the
|
|
51
|
+
source lists) no longer appear in the compiled list, the logs, or error
|
|
52
|
+
messages, including validation, conflict, and download errors. Free-text
|
|
53
|
+
fields use a prose-aware parser, so a credential that is lexically identical
|
|
54
|
+
to prose can still survive.
|
|
55
|
+
- Updated lodash to 4.17.21 to fix the ReDoS in `_.trim`
|
|
56
|
+
(GHSA-29mw-wpgm-hmr9).
|
|
57
|
+
- The password in a filter URL no longer appears in
|
|
58
|
+
`@adguard/filters-downloader` error messages. [#130]
|
|
59
|
+
|
|
22
60
|
## [2.1.1] - 2026-08-21
|
|
23
61
|
|
|
24
62
|
## [2.1.0] - 2026-04-16
|
|
25
63
|
|
|
26
64
|
### Added
|
|
27
65
|
|
|
28
|
-
- `ValidateAllowIpAndPublicSuffix` validation that keeps both IP address rules
|
|
66
|
+
- `ValidateAllowIpAndPublicSuffix` validation that keeps both IP address rules
|
|
67
|
+
and rules matching whole public suffixes (e.g. `||185.149.120.173^`, `.org^`).
|
|
68
|
+
[#126]
|
|
29
69
|
|
|
30
70
|
### Changed
|
|
31
71
|
|
|
@@ -35,30 +75,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
35
75
|
- 3-octet patterns with `^` are rejected (e.g. `192.168.1^`, `||192.168.1^`)
|
|
36
76
|
- 3-octet patterns without trailing dot/wildcard are rejected (e.g. `192.168.1`, `||192.168.1`)
|
|
37
77
|
- 1-2 octet patterns are rejected (too wide, use regex instead)
|
|
38
|
-
- Normalization (`ip-normalize.js`) is now purely a normalization step:
|
|
78
|
+
- Normalization (`ip-normalize.js`) is now purely a normalization step:
|
|
79
|
+
it only rewrites valid patterns to canonical form and passes everything
|
|
80
|
+
else through unchanged. Rejection of invalid patterns is the responsibility
|
|
81
|
+
of the validator (`validate.js`), eliminating double-rejection.
|
|
39
82
|
|
|
40
83
|
### Fixed
|
|
41
84
|
|
|
42
|
-
- `ValidateAllowPublicSuffix` was letting through terminated ICANN TLDs
|
|
43
|
-
|
|
85
|
+
- `ValidateAllowPublicSuffix` was letting through terminated ICANN TLDs
|
|
86
|
+
(e.g. `xn--jlq61u9w7b`) because `tldts` v5.x had stale Public Suffix List data.
|
|
87
|
+
Updated `tldts` from v5 to v7 which has the current PSL. [#128]
|
|
88
|
+
- Hardened `validHostname()` to reject hostnames that do not contain any
|
|
89
|
+
alphanumeric character (e.g. `..`), covering a behavioral change in `tldts` v7.
|
|
44
90
|
|
|
45
|
-
[Unreleased]: https://github.com/AdguardTeam/HostlistCompiler/compare/
|
|
91
|
+
[Unreleased]: https://github.com/AdguardTeam/HostlistCompiler/compare/v3.0.0...HEAD
|
|
92
|
+
[3.0.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v3.0.0-beta.0...v3.0.0
|
|
93
|
+
[3.0.0-beta.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.1.1...v3.0.0-beta.0
|
|
46
94
|
[2.1.1]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.1.0...v2.1.1
|
|
47
95
|
[2.1.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.0.0...v2.1.0
|
|
48
96
|
[#126]: https://github.com/AdguardTeam/HostlistCompiler/issues/126
|
|
49
97
|
[#127]: https://github.com/AdguardTeam/HostlistCompiler/issues/127
|
|
50
98
|
[#128]: https://github.com/AdguardTeam/HostlistCompiler/issues/128
|
|
99
|
+
[#130]: https://github.com/AdguardTeam/HostlistCompiler/issues/130
|
|
51
100
|
|
|
52
101
|
## [2.0.0] - 2026-04-02
|
|
53
102
|
|
|
54
103
|
### Added
|
|
55
104
|
|
|
56
|
-
- `ValidateAllowPublicSuffix` transformation option that allows rules matching
|
|
105
|
+
- `ValidateAllowPublicSuffix` transformation option that allows rules matching
|
|
106
|
+
whole public suffixes (e.g. `||hl.cn^`, `||org^`). [#124]
|
|
57
107
|
|
|
58
108
|
### Changed
|
|
59
109
|
|
|
60
|
-
- `Validate` now rejects rules matching whole public suffixes consistently
|
|
61
|
-
|
|
110
|
+
- `Validate` now rejects rules matching whole public suffixes consistently
|
|
111
|
+
across syntactic variants such as `||org^`, `||*.org^`, `.org^`, and `*.org^`.
|
|
112
|
+
Use `ValidateAllowPublicSuffix` if you need to keep such rules.
|
|
113
|
+
- The compiler now warns when incompatible validation transformations are
|
|
114
|
+
combined at runtime, and configuration validation rejects combining `Validate`,
|
|
115
|
+
`ValidateAllowIp`, and `ValidateAllowPublicSuffix` in the same transformations list.
|
|
62
116
|
|
|
63
117
|
[2.0.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v1.0.39...v2.0.0
|
|
64
118
|
[#124]: https://github.com/AdguardTeam/HostlistCompiler/issues/124
|