@adguard/hostlist-compiler 2.1.1 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (47) hide show
  1. package/.github/workflows/prepare-release.yml +5 -0
  2. package/.markdownlint.json +29 -0
  3. package/.markdownlintignore +1 -0
  4. package/AGENTS.md +310 -98
  5. package/CHANGELOG.md +62 -8
  6. package/DEPLOYMENT.md +175 -0
  7. package/DEVELOPMENT.md +171 -72
  8. package/README.md +238 -143
  9. package/examples/china/README.md +2 -2
  10. package/examples/energized/README.md +4 -2
  11. package/examples/sdn/README.md +2 -2
  12. package/examples/whitelist/README.md +2 -2
  13. package/package.json +24 -18
  14. package/src/cli.js +53 -31
  15. package/src/compile-source.js +4 -3
  16. package/src/configuration.js +132 -2
  17. package/src/download.js +27 -0
  18. package/src/filter.js +2 -2
  19. package/src/index.d.ts +2 -0
  20. package/src/index.js +16 -50
  21. package/src/{transformations/ip-normalize.js → ip-normalize.js} +2 -2
  22. package/src/schemas/configuration.schema.js +137 -0
  23. package/src/source-types.js +17 -0
  24. package/src/transformations/enum.js +49 -0
  25. package/src/transformations/include.js +1 -1
  26. package/src/transformations/transform.js +40 -74
  27. package/src/transformations/validate-allow-ip-and-public-suffix.js +1 -1
  28. package/src/transformations/validate-allow-ip.js +1 -1
  29. package/src/utils.js +1284 -0
  30. package/src/validation-conflicts.js +106 -0
  31. package/test/cli-error-output.test.js +305 -0
  32. package/test/cli.test.js +47 -0
  33. package/test/compile-source.test.js +167 -7
  34. package/test/configuration.test.js +438 -0
  35. package/test/download.test.js +260 -0
  36. package/test/helpers/http-mock.js +94 -0
  37. package/test/index.test.js +497 -4
  38. package/test/{transformations/ip-normalize.test.js → ip-normalize.test.js} +1 -1
  39. package/test/resolve-include.test.js +8 -8
  40. package/test/transformations/covert-to-ascii.test.js +1 -1
  41. package/test/transformations/exclude.test.js +2 -2
  42. package/test/transformations/include.test.js +9 -3
  43. package/test/transformations/ip-alignment.test.js +1 -1
  44. package/test/transformations/transform.test.js +66 -1
  45. package/test/utils.test.js +1559 -0
  46. package/test/validation-conflicts.test.js +230 -0
  47. package/src/schemas/configuration.schema.json +0 -369
@@ -1,5 +1,10 @@
1
1
  name: Prepare release
2
2
 
3
+ # The run list shows the requested tag together with the workflow name.
4
+ # `github.workflow` repeats the `name` value from one place, so a rename of the
5
+ # workflow also updates the run name.
6
+ run-name: '${{ github.workflow }} ${{ inputs.tag }}'
7
+
3
8
  description: >-
4
9
  Opens a release PR that finalizes CHANGELOG.md for the given tag.
5
10
  After merge, publish-release.yml takes over.
@@ -0,0 +1,29 @@
1
+ {
2
+ "ul-indent": {
3
+ "indent": 4
4
+ },
5
+ "line-length": {
6
+ "strict": true,
7
+ "line_length": 120,
8
+ "tables": false
9
+ },
10
+ "no-multiple-blanks": {
11
+ "maximum": 2
12
+ },
13
+ "no-inline-html": {
14
+ "allowed_elements": [
15
+ "a"
16
+ ]
17
+ },
18
+ "no-duplicate-heading": {
19
+ "siblings_only": true
20
+ },
21
+ "no-blanks-blockquote": false,
22
+ "no-bare-urls": false,
23
+ "table-column-style": {
24
+ "style": "aligned"
25
+ },
26
+ "ul-style": {
27
+ "style": "dash"
28
+ }
29
+ }
@@ -0,0 +1 @@
1
+ node_modules/
package/AGENTS.md CHANGED
@@ -2,12 +2,29 @@
2
2
 
3
3
  Guidelines for AI agents working on this project.
4
4
 
5
+ ## Table of Contents
6
+
7
+ - [Project Overview](#project-overview)
8
+ - [Technical Context](#technical-context)
9
+ - [Project Structure](#project-structure)
10
+ - [Build And Test Commands](#build-and-test-commands)
11
+ - [Contribution Instructions](#contribution-instructions)
12
+ - [Code Guidelines](#code-guidelines)
13
+ - [System Design](#system-design)
14
+ - [Architecture](#architecture)
15
+ - [Code Quality](#code-quality)
16
+ - [Testing](#testing)
17
+ - [Dependency Management](#dependency-management)
18
+ - [Configuration & Documentation](#configuration--documentation)
19
+ - [Markdown Formatting](#markdown-formatting)
20
+ - [Other](#other)
21
+
5
22
  ## Project Overview
6
23
 
7
24
  **Hostlist Compiler** (`@adguard/hostlist-compiler`) is a Node.js CLI tool and
8
- library that compiles DNS hosts blocklists from multiple sources into a single
9
- filter list compatible with AdGuard Home and other AdGuard products with DNS
10
- filtering.
25
+ library for building DNS filter lists; see [README.md](README.md) for the
26
+ user-facing description. The sections below summarize what agents need to know
27
+ when working on this codebase.
11
28
 
12
29
  It supports multiple input formats (`/etc/hosts`, adblock-style), configurable
13
30
  transformations (compress, deduplicate, validate, remove comments, etc.),
@@ -17,49 +34,63 @@ inclusion/exclusion rules, and `!#include` directive resolution via
17
34
  ## Technical Context
18
35
 
19
36
  - **Language**: JavaScript (Node.js, CommonJS modules)
20
- - **Node Version**: Compatible with current LTS
37
+ - **Node / pnpm Versions**: see the `engines` field in `package.json` (the
38
+ single source of truth)
21
39
  - **Primary Dependencies**:
22
- - `@adguard/filters-downloader` — resolves `!#include` directives and
23
- downloads remote sources
24
- - `ajv` / `ajv-errors` / `better-ajv-errors` — JSON schema validation for
25
- configuration
26
- - `axios` — HTTP requests for remote filter lists
27
- - `lodash` — utility functions
28
- - `tldts` — TLD parsing for domain validation
29
- - `yargs` — CLI argument parsing
30
- - `consola` — logging
31
- - **Testing**: Jest (`jest --runInBand --detectOpenHandles`)
32
- - **Linting**: ESLint with `airbnb-base` config, `@babel/eslint-parser`
40
+ - `@adguard/filters-downloader` — resolves `!#include` directives and
41
+ downloads remote sources
42
+ - `ajv` / `better-ajv-errors` — JSON schema validation for configuration
43
+ - `lodash` — utility functions
44
+ - `tldts` — TLD parsing for domain validation
45
+ - `yargs` — CLI argument parsing
46
+ - `consola` — logging
47
+ - **Testing**: Jest (`jest --runInBand --detectOpenHandles`, silent mode)
48
+ - **Linting**: ESLint with `airbnb-base` config and `@babel/eslint-parser`;
49
+ markdownlint for Markdown files
33
50
  - **Storage**: None (stateless CLI tool, reads remote/local files, writes output
34
51
  files)
35
52
  - **Target Platform**: Node.js (CLI and library; published to npm)
36
- - **Project Type**: Single package (CLI + library)
53
+ - **Project Type**: Single package (CLI tool + library)
37
54
  - **License**: GPL-3.0
38
55
 
39
56
  ## Project Structure
40
57
 
41
58
  ```text
42
59
  ├── src/ # Main source code
43
- │ ├── cli.js # CLI entry point (hashbang, yargs setup)
44
- │ ├── index.js # Library entry point (compile function)
45
- │ ├── index.d.ts # TypeScript type declarations
46
- │ ├── compile-source.js # Compiles individual sources
47
- │ ├── configuration.js # Configuration validation (AJV)
48
- │ ├── filter.js # Filtering utilities
60
+ │ ├── cli.js # CLI entry point (hashbang, yargs, config loading)
61
+ │ ├── index.js # Library entry point: compile(configuration)
62
+ │ ├── index.d.ts # TypeScript type declarations for consumers
63
+ │ ├── compile-source.js # Compiles a single source
64
+ │ ├── configuration.js # Configuration creation and validation (AJV + schema)
65
+ │ ├── download.js # Central download layer (wraps FiltersDownloader)
66
+ │ ├── filter.js # Include/exclude wildcard filtering
49
67
  │ ├── rule.js # Rule parsing (adblock, /etc/hosts)
50
- │ ├── utils.js # String and path utilities
51
- │ ├── schemas/ # JSON schemas for configuration validation
52
- │ └── transformations/ # Transformation modules (14 files)
68
+ │ ├── ip-normalize.js # Internal helper: IP rule normalization (used by transformations)
69
+ │ ├── source-types.js # Source-type vocabulary (SOURCE_TYPES) for the schema and the CLI
70
+ │ ├── validation-conflicts.js # Validation-conflict rules (runtime checks + generated schema fragments)
71
+ │ ├── utils.js # String, IP, wildcard, and credential-redaction utilities
72
+ │ ├── schemas/ # Configuration schema (transformation names + conflict rules)
73
+ │ └── transformations/ # 15 transformations + fixed-order pipeline
74
+ │ ├── enum.js # TRANSFORMATIONS enum + VALIDATION_TRANSFORMATIONS (single source of truth)
75
+ │ ├── transform.js # Pipeline orchestrator
76
+ │ ├── exclude.js # Exclusion rules (async, ungated)
77
+ │ ├── include.js # Inclusion rules (async, ungated)
78
+ │ └── # ... (13 simple transforms: compress, validate, deduplicate, etc.)
53
79
  ├── test/ # Jest test suites mirroring src/
80
+ │ ├── helpers/ # Shared test helpers (HTTP mocking)
54
81
  │ ├── resources/ # Test fixture files
55
82
  │ └── transformations/ # Transformation-specific tests
56
- ├── examples/ # Example configurations (sdn, energized, etc.)
83
+ ├── examples/ # Example configurations (sdn, energized, china, whitelist)
57
84
  ├── .github/workflows/ # GitHub Actions CI (ci, mirror, prepare-release, publish-release)
58
85
  ├── Dockerfile # Multi-stage build: test-output / build-output targets
59
86
  ├── package.json # Package manifest and scripts
60
87
  ├── jest.config.js # Jest configuration
61
- ├── .eslintrc.js # ESLint configuration
88
+ ├── .eslintrc.js # ESLint configuration (airbnb-base)
89
+ ├── .markdownlint.json # Markdownlint configuration
90
+ ├── .markdownlintignore # Files excluded from Markdown linting
62
91
  ├── CHANGELOG.md # Release history
92
+ ├── DEPLOYMENT.md # Deployment, CI/CD and release documentation
93
+ ├── DEVELOPMENT.md # Environment setup guide (setup, not agent rules)
63
94
  ├── README.md # User documentation
64
95
  └── LICENSE # GPL-3.0
65
96
  ```
@@ -73,88 +104,160 @@ pnpm lint
73
104
  pnpm test
74
105
  ```
75
106
 
76
- | Action | Command |
77
- |---------|---------|
78
- | Install | `pnpm install` |
79
- | Lint | `pnpm lint` |
80
- | Test | `pnpm test` |
81
- | Run CLI (example) | `node src/cli.js -c examples/sdn/configuration.json -o filter.txt` |
82
- | Version bump | `pnpm increment` |
83
-
84
- ## Verification
85
-
86
- You MUST follow these rules for EVERY task you perform:
87
-
88
- 1. **Run the linter**: `pnpm lint`. Fix all warnings and errors before
89
- proceeding. The project uses ESLint with `airbnb-base`.
90
- 2. **Run the tests**: `pnpm test`. All tests must pass. If you changed
91
- behavior, verify that existing tests still reflect correct expectations.
92
- 3. **Check the outcome**. If either command fails, fix the issues and re-run
93
- until both pass. A task is not complete until all checks are green.
94
-
95
- ## Mandatory Task Actions
96
-
97
- For every task you perform, you MUST:
98
-
99
- - **Update or add unit tests** for any changed or new code. Mirror the source
100
- file structure under `test/`.
101
- - **Run the full verification sequence** (`pnpm lint` and `pnpm test`) and
102
- confirm both pass.
103
- - **Update `AGENTS.md`** if you change the project structure, add new build
104
- commands, or discover a recurring agent mistake that should become a rule.
105
- - **Update `CHANGELOG.md`** when making user-facing changes.
106
- - **Update the JSON schema** (`src/schemas/configuration.schema.json`) when
107
- adding or changing configuration options.
108
- - **Update TypeScript declarations** (`src/index.d.ts`) when changing the
109
- public API.
110
- - **Verify your code follows the Code Guidelines** in this file after
111
- completing the task.
112
- - If a prompt asks you to refactor or improve existing code, check whether
113
- the insight can be phrased as a code guideline. If so, add it to the
114
- relevant Code Guidelines section below.
107
+ | Action | Command |
108
+ | ------------- | ---------------------------------------------------- |
109
+ | Install | `pnpm install` |
110
+ | Lint (all) | `pnpm lint` (runs `lint:code` and `lint:md`) |
111
+ | Lint code | `pnpm lint:code` (ESLint, `airbnb-base` config) |
112
+ | Lint markdown | `pnpm lint:md` (markdownlint) |
113
+ | Test | `pnpm test` (`jest --runInBand --detectOpenHandles`) |
114
+
115
+ For running the CLI locally and bumping the version, see
116
+ [DEVELOPMENT.md](DEVELOPMENT.md).
115
117
 
116
118
  ## Contribution Instructions
117
119
 
118
- 1. **Install** dependencies with `pnpm install`.
119
- 2. **Lint** before committing: `pnpm lint`. Fix all warnings and errors.
120
- 3. **Test** before committing: `pnpm test`. All tests must pass. Add tests for
121
- new functionality — mirror the source file structure under `test/`.
122
- 4. **Do not commit** generated files, `node_modules`, or editor configs.
123
- 5. **Do not create git commits or PRs** unless explicitly asked.
124
- 6. Keep the `CHANGELOG.md` up to date when making user-facing changes.
120
+ - You MUST verify your work with the linter and the Markdown formatter. The
121
+ project has no type checker (plain JavaScript); ESLint and markdownlint are
122
+ the gates (see [Build and Test Commands](#build-and-test-commands)); use
123
+ `pnpm exec eslint . --fix` and `pnpm exec markdownlint . --fix` to
124
+ auto-fix issues.
125
+ - You MUST update or add unit tests for any changed or new code, mirroring the
126
+ source file structure under `test/` (`<module>.test.js`).
127
+ - You MUST run `pnpm test` and confirm all tests pass — a task is not complete
128
+ until all checks are green.
129
+ - When you change the project structure or add new build commands, update the
130
+ Project Structure and Build And Test Commands sections in `AGENTS.md`.
131
+ - If a prompt asks you to refactor or improve existing code, check whether the
132
+ insight can be phrased as a code guideline; if so, add it to the relevant
133
+ Code Guidelines section in `AGENTS.md`.
134
+ - After completing a task, verify that the code you wrote follows the Code
135
+ Guidelines in this file.
136
+ - Even when the task is to write or update a document (e.g. a plan or any
137
+ Markdown file) rather than code, run `pnpm lint:md` to verify and fix
138
+ Markdown formatting.
139
+ - Install dependencies with `pnpm install` before working.
140
+ - Do not commit generated files, `node_modules`, or editor configs.
141
+ - Do not create git commits or PRs unless explicitly asked.
142
+ - Keep documentation in sync as described in
143
+ [Configuration & Documentation](#configuration--documentation) below.
144
+ - **Use ticket-prefixed commit messages**: Commit messages MUST start with the
145
+ ticket number (`AG-XXX`) so they auto-link with the task tracker, followed
146
+ by a short description in the present tense (e.g. `AG-1234 Fix login
147
+ redirect`). Automated commits made by CI (e.g. the CHANGELOG finalization
148
+ in the release-* PRs) use a [Conventional Commits] prefix such as `docs:`
149
+ instead.
150
+
151
+ [Conventional Commits]: https://www.conventionalcommits.org/en/v1.0.0/
125
152
 
126
153
  ## Code Guidelines
127
154
 
155
+ ### System Design
156
+
157
+ The package is both a CLI tool and a library, so both sets of rules apply:
158
+
159
+ - The CLI runs and exits — no long-lived state, no background daemons. It
160
+ exits with code 0 on success and code 1 on failure; error handling is
161
+ specified in [Code Quality](#code-quality).
162
+ - Write the compiled filter list to the required `-o` output file; use
163
+ `consola` for diagnostics (`-v` / `--verbose` enables debug and trace).
164
+ - Be composable — support quick `hosts`-file conversion via `-i` without a
165
+ configuration file, and full mode via `-c <config>`; sources can be local
166
+ files or remote URLs.
167
+ - Fail fast with clear messages — validate the configuration with AJV before
168
+ compiling; throw errors that include the invalid value so the user can fix
169
+ it.
170
+ - Keep startup time fast — no heavy initialization on the CLI code path.
171
+ - As a library, `compile(configuration)` must not mutate global state
172
+ (environment variables, process listeners); the only I/O it performs is the
173
+ downloads explicitly requested by the configuration's sources.
174
+ - Export a stable public API (`compile`, configuration/source types) with
175
+ complete TypeScript declarations in `src/index.d.ts`; internal functions are
176
+ not exported.
177
+ - Handle errors by throwing descriptive `Error` / `TypeError` instances and
178
+ let the caller decide how to recover; see the error-handling rules in
179
+ [Code Quality](#code-quality).
180
+
128
181
  ### Architecture
129
182
 
130
- - **CommonJS modules** — use `require`/`module.exports` throughout; do not
131
- introduce ES module syntax.
132
- - **Transformation pipeline** — each transformation is a standalone module in
133
- `src/transformations/` exporting a single async function. Transformations are
134
- orchestrated by `src/transformations/transform.js` and always run in a fixed
135
- order regardless of the order specified in configuration.
136
- - **Configuration-driven** — the compiler is controlled by a JSON configuration
137
- object validated against `src/schemas/configuration.schema.json` via AJV.
183
+ The easiest way to achieve these principles is **layered architecture**.
184
+ This project's layers, from top to bottom:
185
+
186
+ ```text
187
+ CLI entry (src/cli.js)
188
+ ↓
189
+ Library entry / orchestration (src/index.js)
190
+ ↓
191
+ Configuration creation and validation (src/configuration.js, src/schemas/configuration.schema.js)
192
+ Per-source compilation (src/compile-source.js)
193
+ ↓
194
+ Transformation pipeline (src/transformations/transform.js)
195
+ ↓
196
+ Transformations (src/transformations/*.js)
197
+ ↓
198
+ Rule parsing (src/rule.js) Filtering (src/filter.js) Utilities (src/utils.js)
199
+ ↓
200
+ Central download layer (src/download.js) (used by compile-source.js and filter.js)
201
+ ↓
202
+ @adguard/filters-downloader (external package, downloads sources and includes)
203
+ ```
204
+
205
+ Each layer may call the layers below it; no layer depends on a layer above it.
206
+ `utils.js` and `download.js` are the leaf modules (they depend only on external
207
+ packages — `lodash` and `@adguard/filters-downloader` respectively); there are
208
+ no circular dependencies.
209
+
210
+ Universal design principles the codebase should follow:
211
+
212
+ - **Separation of Concerns** — parsing, filtering, transformations, and
213
+ configuration validation live in separate modules.
214
+ - **Single Responsibility Principle** — each transformation module does exactly
215
+ one transformation; each module has one reason to change.
216
+ - **Dependency Direction** — dependencies point downward, from entry points to
217
+ utilities; lower layers never import entry points.
218
+ - **Explicit Boundaries** — transformations communicate through
219
+ `(rules, ...) => rules` signatures; the `TRANSFORMATIONS` enum is frozen.
220
+ - **Data Flow Clarity** — rules flow through a single fixed-order pipeline,
221
+ making the data path predictable and traceable.
222
+ - **Minimize Coupling, Maximize Cohesion** — modules are self-contained and
223
+ interact through narrow interfaces.
224
+ - **Make Invalid States Impossible** — configuration is validated against a
225
+ JSON schema; incompatible transformation combinations are rejected at
226
+ runtime and in the schema. Less critical at compile time: the project is
227
+ plain JavaScript, so runtime validation carries this burden.
228
+ - **Single Source of Truth for Transformation Names** — the `TRANSFORMATIONS`
229
+ enum in `src/transformations/enum.js` is the single source of truth for
230
+ transformation *names*: the configuration schema's accepted names and the
231
+ validation-conflict rules are derived from it, so renaming a transformation
232
+ in the enum updates those automatically. Registering a *new* transformation
233
+ is still a multi-step process — add the enum entry (`enum.js`), register the
234
+ implementation in the pipeline (`src/transformations/transform.js`), and
235
+ update the exported `Transformation` type (`src/index.d.ts`) and the order
236
+ list (`README.md`) by hand; a test asserts the pipeline covers every enum
237
+ entry. The steps are documented in DEVELOPMENT.md.
238
+ - **Single Source of Truth for Conflict Rules** — validation-conflict rules
239
+ (which validation transformations conflict, and at which levels) live in
240
+ `src/validation-conflicts.js`; the schema conflict rules and the runtime
241
+ checks are generated from that module, so they cannot drift apart. The list
242
+ of validation transformations (`VALIDATION_TRANSFORMATIONS`) is defined next
243
+ to the enum in `src/transformations/enum.js`, so the conflict-rule side of
244
+ adding a validator is a single-file change.
245
+ - **Observability Built-in** — all logging goes through `consola`; see the
246
+ logging and error-handling rules in [Code Quality](#code-quality).
247
+ - **Keep It Boring** — plain CommonJS modules, no framework magic, simple
248
+ synchronous or async transformation functions.
138
249
 
139
250
  ### Code Quality
140
251
 
141
- - **ESLint airbnb-base** rules apply — run `pnpm lint` to check. See
142
- `.eslintrc.js` for project-specific overrides.
252
+ - **ESLint airbnb-base** rules apply — run `pnpm lint:code` to check; the
253
+ project-specific overrides are configured in `.eslintrc.js` and not
254
+ restated here.
143
255
  - Use `lodash` helpers (e.g., `_.startsWith`, `_.isEmpty`, `_.trim`) where they
144
256
  are already used — stay consistent with the existing style.
145
257
  - Prefer `for...of` loops over `Array.forEach` for async iteration (the
146
258
  codebase disables `no-restricted-syntax` where needed).
147
-
148
- ### Testing
149
-
150
- - **Framework**: Jest, running in Node environment with `--runInBand`.
151
- - **Mocking**: `mock-fs` for filesystem mocks, `nock` for HTTP request
152
- interception.
153
- - Test files follow the naming convention `<module>.test.js` under `test/`.
154
- - Each transformation has its own test file under `test/transformations/`.
155
-
156
- ### Logging and Error Handling
157
-
259
+ - Use JSDoc comments for public functions (parameters and return values), e.g.
260
+ `transform()` in `src/transformations/transform.js`.
158
261
  - Use `consola` for all logging (`consola.info`, `consola.debug`,
159
262
  `consola.error`). Do not use `console.log` / `console.error` directly.
160
263
  - Throw descriptive `Error` (or `TypeError`) instances with messages that
@@ -162,13 +265,122 @@ For every task you perform, you MUST:
162
265
  and rule problems.
163
266
  - Propagate errors up to the CLI entry point (`src/cli.js`), which catches
164
267
  them, logs via `consola.error`, and exits with code 1.
268
+ - Credential-bearing values MUST pass through the `src/utils.js` sanitizers
269
+ before they reach a header, a log, or an error message. Sanitize the object
270
+ graph and the rendered form: `JSON.stringify`, `better-ajv-errors` output,
271
+ and `cause`/`errors` chains. Use `sanitizeConfigurationForLog()` for a
272
+ configuration dump or a validation error, `stripKnownUrlCredentials()` for a
273
+ value that must be a URL, and `stripUrlValueCredentials()` for free text. Do
274
+ not re-scan an already rendered document: a path-less URL can make the text
275
+ up to a later `@` look like userinfo.
276
+ - The free-text credential parser is a heuristic: it keeps text that reads as
277
+ prose. It can keep a credential that is lexically identical to prose, and it
278
+ can remove prose after a single-label intranet host. Do not add a one-off
279
+ rule for each reported shape: add a row to the host × separator family in
280
+ `test/utils.test.js` and document the shape in the `stripUrlCredentials()`
281
+ JSDoc instead.
282
+
283
+ ### Testing
284
+
285
+ - **Framework**: Jest, running in Node environment with `--runInBand
286
+ --detectOpenHandles` and `silent: true`.
287
+ - **Mocking**: `mock-fs` for filesystem mocks, `undici` MockAgent for HTTP
288
+ request interception.
289
+ - Test files follow the naming convention `<module>.test.js` under `test/`,
290
+ mirroring the structure of `src/`.
291
+ - Each transformation has its own test file under `test/transformations/`.
292
+ - Add or update tests for every changed or new code path; all tests must pass
293
+ before a task is complete.
294
+
295
+ ### Dependency Management
296
+
297
+ - **Pin all dependency versions explicitly** — all entries in `dependencies`
298
+ and `devDependencies` in `package.json` use exact versions (no `^`/`~`
299
+ ranges). When adding or updating a dependency, pin it to the version
300
+ resolved in `pnpm-lock.yaml`; never lower a version below what the lockfile
301
+ currently resolves to.
302
+ - **Do not commit the `packageManager` field** — pnpm 10 auto-adds it to
303
+ `package.json` when pnpm commands are run. The `engines.pnpm` field is the
304
+ single source of truth for the pnpm version; revert `packageManager` before
305
+ committing if pnpm added it.
306
+ - **Prefer vanilla solutions** — use the language's standard library and
307
+ built-in APIs when they adequately solve the problem. Only add a dependency
308
+ when it provides significant value over a vanilla implementation.
309
+ - **Reputable sources only** — dependencies MUST come from well-established,
310
+ actively maintained projects (npm download counts, repository activity,
311
+ known maintainers).
312
+ - **Avoid unpopular libraries** — do NOT add niche or obscure packages with
313
+ limited community adoption; they pose security risks and may become
314
+ unmaintained.
315
+ - **Minimize dependency count** — each new dependency increases attack
316
+ surface, bundle size, and maintenance burden. Justify every addition.
317
+ - **Use the latest stable version** — when adding a new dependency, check the
318
+ package registry for the latest stable release and use it; do not copy
319
+ outdated version numbers from memory or other projects.
320
+ - **Acknowledge the library-side trade-off** — exact pins favor reproducibility
321
+ but hurt library consumers: duplicates of the same dependency may be
322
+ installed alongside their own ranges, and transitive patches only arrive with
323
+ a new release. This trade-off is deliberate; do not loosen pins without
324
+ discussing the impact.
325
+
326
+ **Rationale**: Fewer, well-vetted dependencies reduce security
327
+ vulnerabilities, supply chain risks, and long-term maintenance costs.
328
+
329
+ ### Configuration & Documentation
330
+
331
+ - The compiler is controlled by a JSON configuration object validated against
332
+ `src/schemas/configuration.schema.js` (draft-07) via AJV; unknown keys are
333
+ rejected (`additionalProperties: false`).
334
+ - Runtime behavior is selected via CLI flags (`-c` config file, `-i` quick
335
+ hosts conversion, `-o` output file, `-v` verbose logging). The compiler
336
+ defines no environment variables of its own and reads no `.env` files, but
337
+ the native `fetch` honors the standard proxy variables (`HTTP_PROXY`,
338
+ `HTTPS_PROXY`, `NO_PROXY`) when `NODE_USE_ENV_PROXY=1` is set.
339
+ - Example configurations live in `examples/` (`sdn`, `energized`, `china`,
340
+ `whitelist`) and are used for manual testing.
341
+ - Keep documentation in sync with code: update `README.md` when adding or
342
+ reordering transformations (the transformation order in `README.md` MUST
343
+ match `src/transformations/transform.js`), the default quick-conversion
344
+ pipeline documented in the `README.md` "Quick Hosts Conversion" section MUST
345
+ match `DEFAULT_TRANSFORMATIONS` in `src/transformations/transform.js`,
346
+ update `CHANGELOG.md` for user-facing changes, update
347
+ `src/schemas/configuration.schema.js` when adding configuration options
348
+ (transformation names come from `src/transformations/enum.js`, source types
349
+ from `src/source-types.js`), update `src/index.d.ts` when changing the
350
+ public API, and update `DEPLOYMENT.md` when changing the Dockerfile, CI
351
+ workflows, or deployment setup.
352
+ - The repository stores no secrets or credentials of its own and commits all
353
+ configuration files. A configuration can still contain credentialed source
354
+ URLs. The compiler sends them to the downloader as Basic auth. It MUST
355
+ redact them before any log, header, or error output (see the sanitizer rule
356
+ in [Code Quality](#code-quality)).
357
+
358
+ ### Markdown Formatting
359
+
360
+ All Markdown files MUST pass markdownlint; the enforced rules live in
361
+ `.markdownlint.json` (plus markdownlint defaults) and are not restated here.
362
+ The following conventions go beyond what the linter enforces:
363
+
364
+ - **Line length**: Don't wrap lines artificially short — keep them close to
365
+ the length limit configured in the linter where possible.
366
+ - **Continuation lines**: When a list item wraps to the next line, align the
367
+ continuation with the first character of the item text, not the list
368
+ marker. This applies to all list types (ordered and unordered).
369
+ - **Emphasis**: Use asterisks (`*`) for emphasis (`*italic*`, `**bold**`) —
370
+ the linter only enforces a consistent style, not a specific one.
371
+
372
+ **Rationale**: Uniform Markdown formatting improves readability for both
373
+ humans and AI agents that consume project documentation.
165
374
 
166
375
  ### Other
167
376
 
168
- - The `src/index.d.ts` file provides TypeScript type declarations for external
169
- consumers. Keep it in sync when changing the public API.
170
- - The JSON schema in `src/schemas/configuration.schema.json` must be updated
171
- when adding new configuration options or transformations.
377
+ - Do not restate rules that are configured in linter configs (`.eslintrc.js`,
378
+ `.markdownlint.json`) — reference the config files instead; they are the
379
+ source of truth.
380
+ - The transformation pipeline always runs in a fixed order regardless of the
381
+ order specified in configuration (see
382
+ [Configuration & Documentation](#configuration--documentation) for keeping
383
+ `README.md` in sync).
172
384
  - Any recurring agent error is a defect in this document. If you had to be
173
385
  corrected twice on the same issue, add a rule here to prevent it in the
174
386
  future.
package/CHANGELOG.md CHANGED
@@ -19,13 +19,53 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
19
19
 
20
20
  ### Security
21
21
 
22
+ ## [3.0.0] - 2026-11-17
23
+
24
+ ## [3.0.0-beta.0] - 2026-11-17
25
+
26
+ ### Changed
27
+
28
+ - Updated [@adguard/filters-downloader] to v3.1.0. This version removes
29
+ `axios` and downloads filters with native `fetch` in Node.js. It requires
30
+ Node.js 22.22.2 or newer, and Node.js 23.x is not supported. Environment
31
+ proxy variables now require `NODE_USE_ENV_PROXY=1`. A relative `!#include`
32
+ path that leaves the directory of the containing file is now ignored
33
+ instead of failing the filter update. Credentials from malformed source
34
+ URLs no longer appear in error messages, stacks, causes, or other error
35
+ properties. `stripUrlCredentials` is now exported from the package root and
36
+ the browser entry point. [#130]
37
+ - Remote bodies that look like JSON (e.g. `[]`, `123`, `{"a":1}`) are now
38
+ treated as plain text instead of failing with
39
+ `filter.trim is not a function`. A JSON file served as `text/plain` now
40
+ adds its content to the list. [#130]
41
+ - The internal configuration schema moved from `src/schemas/configuration.schema.json`
42
+ to `src/schemas/configuration.schema.js`; the documented API is unchanged.
43
+
44
+ ### Fixed
45
+
46
+ - CLI to print the original network error (`cause` chain) when a compile fails.
47
+
48
+ ### Security
49
+
50
+ - Credentials in URL configuration fields (`source`, `homepage`, and the
51
+ source lists) no longer appear in the compiled list, the logs, or error
52
+ messages, including validation, conflict, and download errors. Free-text
53
+ fields use a prose-aware parser, so a credential that is lexically identical
54
+ to prose can still survive.
55
+ - Updated lodash to 4.17.21 to fix the ReDoS in `_.trim`
56
+ (GHSA-29mw-wpgm-hmr9).
57
+ - The password in a filter URL no longer appears in
58
+ `@adguard/filters-downloader` error messages. [#130]
59
+
22
60
  ## [2.1.1] - 2026-08-21
23
61
 
24
62
  ## [2.1.0] - 2026-04-16
25
63
 
26
64
  ### Added
27
65
 
28
- - `ValidateAllowIpAndPublicSuffix` validation that keeps both IP address rules and rules matching whole public suffixes (e.g. `||185.149.120.173^`, `.org^`). [#126]
66
+ - `ValidateAllowIpAndPublicSuffix` validation that keeps both IP address rules
67
+ and rules matching whole public suffixes (e.g. `||185.149.120.173^`, `.org^`).
68
+ [#126]
29
69
 
30
70
  ### Changed
31
71
 
@@ -35,30 +75,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
35
75
  - 3-octet patterns with `^` are rejected (e.g. `192.168.1^`, `||192.168.1^`)
36
76
  - 3-octet patterns without trailing dot/wildcard are rejected (e.g. `192.168.1`, `||192.168.1`)
37
77
  - 1-2 octet patterns are rejected (too wide, use regex instead)
38
- - Normalization (`ip-normalize.js`) is now purely a normalization step: it only rewrites valid patterns to canonical form and passes everything else through unchanged. Rejection of invalid patterns is the responsibility of the validator (`validate.js`), eliminating double-rejection.
78
+ - Normalization (`ip-normalize.js`) is now purely a normalization step:
79
+ it only rewrites valid patterns to canonical form and passes everything
80
+ else through unchanged. Rejection of invalid patterns is the responsibility
81
+ of the validator (`validate.js`), eliminating double-rejection.
39
82
 
40
83
  ### Fixed
41
84
 
42
- - `ValidateAllowPublicSuffix` was letting through terminated ICANN TLDs (e.g. `xn--jlq61u9w7b`) because `tldts` v5.x had stale Public Suffix List data. Updated `tldts` from v5 to v7 which has the current PSL. [#128]
43
- - Hardened `validHostname()` to reject hostnames that do not contain any alphanumeric character (e.g. `..`), covering a behavioral change in `tldts` v7.
85
+ - `ValidateAllowPublicSuffix` was letting through terminated ICANN TLDs
86
+ (e.g. `xn--jlq61u9w7b`) because `tldts` v5.x had stale Public Suffix List data.
87
+ Updated `tldts` from v5 to v7 which has the current PSL. [#128]
88
+ - Hardened `validHostname()` to reject hostnames that do not contain any
89
+ alphanumeric character (e.g. `..`), covering a behavioral change in `tldts` v7.
44
90
 
45
- [Unreleased]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.1.1...HEAD
91
+ [Unreleased]: https://github.com/AdguardTeam/HostlistCompiler/compare/v3.0.0...HEAD
92
+ [3.0.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v3.0.0-beta.0...v3.0.0
93
+ [3.0.0-beta.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.1.1...v3.0.0-beta.0
46
94
  [2.1.1]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.1.0...v2.1.1
47
95
  [2.1.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.0.0...v2.1.0
48
96
  [#126]: https://github.com/AdguardTeam/HostlistCompiler/issues/126
49
97
  [#127]: https://github.com/AdguardTeam/HostlistCompiler/issues/127
50
98
  [#128]: https://github.com/AdguardTeam/HostlistCompiler/issues/128
99
+ [#130]: https://github.com/AdguardTeam/HostlistCompiler/issues/130
51
100
 
52
101
  ## [2.0.0] - 2026-04-02
53
102
 
54
103
  ### Added
55
104
 
56
- - `ValidateAllowPublicSuffix` transformation option that allows rules matching whole public suffixes (e.g. `||hl.cn^`, `||org^`). [#124]
105
+ - `ValidateAllowPublicSuffix` transformation option that allows rules matching
106
+ whole public suffixes (e.g. `||hl.cn^`, `||org^`). [#124]
57
107
 
58
108
  ### Changed
59
109
 
60
- - `Validate` now rejects rules matching whole public suffixes consistently across syntactic variants such as `||org^`, `||*.org^`, `.org^`, and `*.org^`. Use `ValidateAllowPublicSuffix` if you need to keep such rules.
61
- - The compiler now warns when incompatible validation transformations are combined at runtime, and configuration validation rejects combining `Validate`, `ValidateAllowIp`, and `ValidateAllowPublicSuffix` in the same transformations list.
110
+ - `Validate` now rejects rules matching whole public suffixes consistently
111
+ across syntactic variants such as `||org^`, `||*.org^`, `.org^`, and `*.org^`.
112
+ Use `ValidateAllowPublicSuffix` if you need to keep such rules.
113
+ - The compiler now warns when incompatible validation transformations are
114
+ combined at runtime, and configuration validation rejects combining `Validate`,
115
+ `ValidateAllowIp`, and `ValidateAllowPublicSuffix` in the same transformations list.
62
116
 
63
117
  [2.0.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v1.0.39...v2.0.0
64
118
  [#124]: https://github.com/AdguardTeam/HostlistCompiler/issues/124