@adguard/hostlist-compiler 1.0.39 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md ADDED
@@ -0,0 +1,175 @@
1
+ # AGENTS.md
2
+
3
+ Guidelines for AI agents working on this project.
4
+
5
+ ## Project Overview
6
+
7
+ **Hostlist Compiler** (`@adguard/hostlist-compiler`) is a Node.js CLI tool and
8
+ library that compiles DNS hosts blocklists from multiple sources into a single
9
+ filter list compatible with AdGuard Home and other AdGuard products with DNS
10
+ filtering.
11
+
12
+ It supports multiple input formats (`/etc/hosts`, adblock-style), configurable
13
+ transformations (compress, deduplicate, validate, remove comments, etc.),
14
+ inclusion/exclusion rules, and `!#include` directive resolution via
15
+ `@adguard/filters-downloader`.
16
+
17
+ ## Technical Context
18
+
19
+ - **Language**: JavaScript (Node.js, CommonJS modules)
20
+ - **Node Version**: Compatible with current LTS
21
+ - **Primary Dependencies**:
22
+ - `@adguard/filters-downloader` — resolves `!#include` directives and
23
+ downloads remote sources
24
+ - `ajv` / `ajv-errors` / `better-ajv-errors` — JSON schema validation for
25
+ configuration
26
+ - `axios` — HTTP requests for remote filter lists
27
+ - `lodash` — utility functions
28
+ - `tldts` — TLD parsing for domain validation
29
+ - `yargs` — CLI argument parsing
30
+ - `consola` — logging
31
+ - **Testing**: Jest (`jest --runInBand --detectOpenHandles`)
32
+ - **Linting**: ESLint with `airbnb-base` config, `@babel/eslint-parser`
33
+ - **Storage**: None (stateless CLI tool, reads remote/local files, writes output
34
+ files)
35
+ - **Target Platform**: Node.js (CLI and library; published to npm)
36
+ - **Project Type**: Single package (CLI + library)
37
+ - **License**: GPL-3.0
38
+
39
+ ## Project Structure
40
+
41
+ ```text
42
+ ├── src/ # Main source code
43
+ │ ├── cli.js # CLI entry point (hashbang, yargs setup)
44
+ │ ├── index.js # Library entry point (compile function)
45
+ │ ├── index.d.ts # TypeScript type declarations
46
+ │ ├── compile-source.js # Compiles individual sources
47
+ │ ├── configuration.js # Configuration validation (AJV)
48
+ │ ├── filter.js # Filtering utilities
49
+ │ ├── rule.js # Rule parsing (adblock, /etc/hosts)
50
+ │ ├── utils.js # String and path utilities
51
+ │ ├── schemas/ # JSON schemas for configuration validation
52
+ │ └── transformations/ # Transformation modules (14 files)
53
+ ├── test/ # Jest test suites mirroring src/
54
+ │ ├── resources/ # Test fixture files
55
+ │ └── transformations/ # Transformation-specific tests
56
+ ├── tools/ # Build helper scripts
57
+ │ └── build-txt.js # Generates transformations docs
58
+ ├── examples/ # Example configurations (sdn, energized, etc.)
59
+ ├── bamboo-specs/ # CI/CD pipeline definitions (Bamboo)
60
+ ├── package.json # Package manifest and scripts
61
+ ├── jest.config.js # Jest configuration
62
+ ├── .eslintrc.js # ESLint configuration
63
+ ├── CHANGELOG.md # Release history
64
+ ├── README.md # User documentation
65
+ └── LICENSE # GPL-3.0
66
+ ```
67
+
68
+ ## Build and Test Commands
69
+
70
+ **Full verification sequence** (run all of these before considering a task done):
71
+
72
+ ```bash
73
+ yarn lint
74
+ yarn test
75
+ ```
76
+
77
+ | Action | Command |
78
+ |---------|---------|
79
+ | Install | `yarn install` |
80
+ | Lint | `yarn lint` |
81
+ | Test | `yarn test` |
82
+ | Run CLI (example) | `node src/cli.js -c examples/sdn/configuration.json -o filter.txt` |
83
+ | Version bump | `yarn increment` |
84
+
85
+ ## Verification
86
+
87
+ You MUST follow these rules for EVERY task you perform:
88
+
89
+ 1. **Run the linter**: `yarn lint`. Fix all warnings and errors before
90
+ proceeding. The project uses ESLint with `airbnb-base`.
91
+ 2. **Run the tests**: `yarn test`. All tests must pass. If you changed
92
+ behavior, verify that existing tests still reflect correct expectations.
93
+ 3. **Check the outcome**. If either command fails, fix the issues and re-run
94
+ until both pass. A task is not complete until all checks are green.
95
+
96
+ ## Mandatory Task Actions
97
+
98
+ For every task you perform, you MUST:
99
+
100
+ - **Update or add unit tests** for any changed or new code. Mirror the source
101
+ file structure under `test/`.
102
+ - **Run the full verification sequence** (`yarn lint` and `yarn test`) and
103
+ confirm both pass.
104
+ - **Update `AGENTS.md`** if you change the project structure, add new build
105
+ commands, or discover a recurring agent mistake that should become a rule.
106
+ - **Update `CHANGELOG.md`** when making user-facing changes.
107
+ - **Update the JSON schema** (`src/schemas/configuration.schema.json`) when
108
+ adding or changing configuration options.
109
+ - **Update TypeScript declarations** (`src/index.d.ts`) when changing the
110
+ public API.
111
+ - **Verify your code follows the Code Guidelines** in this file after
112
+ completing the task.
113
+ - If a prompt asks you to refactor or improve existing code, check whether
114
+ the insight can be phrased as a code guideline. If so, add it to the
115
+ relevant Code Guidelines section below.
116
+
117
+ ## Contribution Instructions
118
+
119
+ 1. **Install** dependencies with `yarn install`.
120
+ 2. **Lint** before committing: `yarn lint`. Fix all warnings and errors.
121
+ 3. **Test** before committing: `yarn test`. All tests must pass. Add tests for
122
+ new functionality — mirror the source file structure under `test/`.
123
+ 4. **Do not commit** generated files, `node_modules`, or editor configs.
124
+ 5. **Do not create git commits or PRs** unless explicitly asked.
125
+ 6. Keep the `CHANGELOG.md` up to date when making user-facing changes.
126
+
127
+ ## Code Guidelines
128
+
129
+ ### Architecture
130
+
131
+ - **CommonJS modules** — use `require`/`module.exports` throughout; do not
132
+ introduce ES module syntax.
133
+ - **Transformation pipeline** — each transformation is a standalone module in
134
+ `src/transformations/` exporting a single async function. Transformations are
135
+ orchestrated by `src/transformations/transform.js` and always run in a fixed
136
+ order regardless of the order specified in configuration.
137
+ - **Configuration-driven** — the compiler is controlled by a JSON configuration
138
+ object validated against `src/schemas/configuration.schema.json` via AJV.
139
+
140
+ ### Code Quality
141
+
142
+ - **ESLint airbnb-base** rules apply — run `yarn lint` to check. See
143
+ `.eslintrc.js` for project-specific overrides.
144
+ - Use `lodash` helpers (e.g., `_.startsWith`, `_.isEmpty`, `_.trim`) where they
145
+ are already used — stay consistent with the existing style.
146
+ - Prefer `for...of` loops over `Array.forEach` for async iteration (the
147
+ codebase disables `no-restricted-syntax` where needed).
148
+
149
+ ### Testing
150
+
151
+ - **Framework**: Jest, running in Node environment with `--runInBand`.
152
+ - **Mocking**: `mock-fs` for filesystem mocks, `nock` for HTTP request
153
+ interception.
154
+ - Test files follow the naming convention `<module>.test.js` under `test/`.
155
+ - Each transformation has its own test file under `test/transformations/`.
156
+
157
+ ### Logging and Error Handling
158
+
159
+ - Use `consola` for all logging (`consola.info`, `consola.debug`,
160
+ `consola.error`). Do not use `console.log` / `console.error` directly.
161
+ - Throw descriptive `Error` (or `TypeError`) instances with messages that
162
+ include the invalid input value — this helps users diagnose configuration
163
+ and rule problems.
164
+ - Propagate errors up to the CLI entry point (`src/cli.js`), which catches
165
+ them, logs via `consola.error`, and exits with code 1.
166
+
167
+ ### Other
168
+
169
+ - The `src/index.d.ts` file provides TypeScript type declarations for external
170
+ consumers. Keep it in sync when changing the public API.
171
+ - The JSON schema in `src/schemas/configuration.schema.json` must be updated
172
+ when adding new configuration options or transformations.
173
+ - Any recurring agent error is a defect in this document. If you had to be
174
+ corrected twice on the same issue, add a rule here to prevent it in the
175
+ future.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,47 @@ All notable changes to this project will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [2.1.0] - 2026-04-16
9
+
10
+ ### Added
11
+
12
+ - `ValidateAllowIpAndPublicSuffix` validation that keeps both IP address rules and rules matching whole public suffixes (e.g. `||185.149.120.173^`, `.org^`). [#126]
13
+
14
+ ### Changed
15
+
16
+ `ValidateAllowIp` now normalizes incomplete IP rules to the safe format `||ip^`. [#127]
17
+ - 4-octet IPs without proper separators (e.g. `1.2.3.4`, `1.2.3.4^`, `|1.2.3.4^`) are normalized to `||1.2.3.4^`
18
+ - 3-octet subnet wildcards (e.g. `192.168.1.`, `192.168.1.*`) are allowed with `||` prefix
19
+ - 3-octet patterns with `^` are rejected (e.g. `192.168.1^`, `||192.168.1^`)
20
+ - 3-octet patterns without trailing dot/wildcard are rejected (e.g. `192.168.1`, `||192.168.1`)
21
+ - 1-2 octet patterns are rejected (too wide, use regex instead)
22
+ - Normalization (`ip-normalize.js`) is now purely a normalization step: it only rewrites valid patterns to canonical form and passes everything else through unchanged. Rejection of invalid patterns is the responsibility of the validator (`validate.js`), eliminating double-rejection.
23
+
24
+ ### Fixed
25
+
26
+ - `ValidateAllowPublicSuffix` was letting through terminated ICANN TLDs (e.g. `xn--jlq61u9w7b`) because `tldts` v5.x had stale Public Suffix List data. Updated `tldts` from v5 to v7 which has the current PSL. [#128]
27
+ - Hardened `validHostname()` to reject hostnames that do not contain any alphanumeric character (e.g. `..`), covering a behavioral change in `tldts` v7.
28
+
29
+ [2.1.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v2.0.0...v2.1.0
30
+ [#126]: https://github.com/AdguardTeam/HostlistCompiler/issues/126
31
+ [#127]: https://github.com/AdguardTeam/HostlistCompiler/issues/127
32
+ [#128]: https://github.com/AdguardTeam/HostlistCompiler/issues/128
33
+
34
+ ## [2.0.0] - 2026-04-02
35
+
36
+ ### Added
37
+
38
+ - `ValidateAllowPublicSuffix` transformation option that allows rules matching whole public suffixes (e.g. `||hl.cn^`, `||org^`). [#124]
39
+
40
+ ### Changed
41
+
42
+ - `Validate` now rejects rules matching whole public suffixes consistently across syntactic variants such as `||org^`, `||*.org^`, `.org^`, and `*.org^`. Use `ValidateAllowPublicSuffix` if you need to keep such rules.
43
+ - The compiler now warns when incompatible validation transformations are combined at runtime, and configuration validation rejects combining `Validate`, `ValidateAllowIp`, and `ValidateAllowPublicSuffix` in the same transformations list.
44
+
45
+ [2.0.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v1.0.39...v2.0.0
46
+ [#124]: https://github.com/AdguardTeam/HostlistCompiler/issues/124
47
+
48
+
8
49
  ## [1.0.39] - 2025-03-13
9
50
 
10
51
  ### Changed
package/DEVELOPMENT.md ADDED
@@ -0,0 +1,172 @@
1
+ # Development Guide
2
+
3
+ This document explains how to set up the development environment, run the
4
+ project locally, and contribute code to Hostlist Compiler.
5
+
6
+ For code guidelines and architectural decisions, see [AGENTS.md](AGENTS.md).
7
+
8
+ ## Prerequisites
9
+
10
+ - **Node.js** — current LTS version (20.x or later)
11
+ - **Yarn** — classic (1.x)
12
+ - **Git**
13
+
14
+ ## Getting Started
15
+
16
+ 1. Clone the repository:
17
+
18
+ ```bash
19
+ git clone https://github.com/AdguardTeam/HostlistCompiler.git
20
+ cd HostlistCompiler
21
+ ```
22
+
23
+ 2. Install dependencies:
24
+
25
+ ```bash
26
+ yarn install
27
+ ```
28
+
29
+ 3. Verify the setup by running linter and tests:
30
+
31
+ ```bash
32
+ yarn lint
33
+ yarn test
34
+ ```
35
+
36
+ ## Available Commands
37
+
38
+ | Command | Description |
39
+ |---------|-------------|
40
+ | `yarn install` | Install all dependencies |
41
+ | `yarn lint` | Run ESLint (`airbnb-base` config) |
42
+ | `yarn test` | Run Jest test suite (`--runInBand --detectOpenHandles`) |
43
+ | `yarn increment` | Bump the patch version in `package.json` |
44
+ | `yarn build-txt` | Generate transformations documentation |
45
+ | `yarn compile` | Run the compiler CLI (`node src/cli.js`) |
46
+
47
+ ## Running the Compiler Locally
48
+
49
+ **With a configuration file** (full-featured mode):
50
+
51
+ ```bash
52
+ node src/cli.js -c examples/sdn/configuration.json -o filter.txt
53
+ ```
54
+
55
+ **Quick hosts conversion** (simple mode):
56
+
57
+ ```bash
58
+ node src/cli.js -i hosts.txt -o output.txt
59
+ ```
60
+
61
+ **With verbose logging**:
62
+
63
+ ```bash
64
+ node src/cli.js -c examples/sdn/configuration.json -o filter.txt -v
65
+ ```
66
+
67
+ The `examples/` directory contains several ready-made configurations (`sdn`,
68
+ `energized`, `china`, `whitelist`) that can be used for testing.
69
+
70
+ ## Development Workflow
71
+
72
+ ### Making Changes
73
+
74
+ 1. Create a feature branch from `master`.
75
+ 2. Make your changes in `src/`.
76
+ 3. Add or update tests in `test/` — mirror the source file structure.
77
+ 4. Run the full verification sequence:
78
+
79
+ ```bash
80
+ yarn lint
81
+ yarn test
82
+ ```
83
+
84
+ 5. Fix any issues until both commands pass.
85
+
86
+ ### Running Tests
87
+
88
+ Run the full test suite:
89
+
90
+ ```bash
91
+ yarn test
92
+ ```
93
+
94
+ Run a specific test file:
95
+
96
+ ```bash
97
+ npx jest test/rule.test.js --runInBand
98
+ ```
99
+
100
+ Run tests matching a pattern:
101
+
102
+ ```bash
103
+ npx jest --testNamePattern="compress" --runInBand
104
+ ```
105
+
106
+ Tests use `mock-fs` for filesystem mocking and `nock` for HTTP request
107
+ interception. Test fixtures live in `test/resources/`.
108
+
109
+ ### Linting
110
+
111
+ ```bash
112
+ yarn lint
113
+ ```
114
+
115
+ ESLint is configured in `.eslintrc.js` with the `airbnb-base` preset. The
116
+ `.eslintignore` file excludes non-source directories from linting.
117
+
118
+ ## Common Tasks
119
+
120
+ ### Adding a New Transformation
121
+
122
+ 1. Create a new module in `src/transformations/` exporting a single async
123
+ function.
124
+ 2. Register the transformation in `src/transformations/transform.js` — the
125
+ order in `transform.js` determines execution order (not configuration order).
126
+ 3. Add the transformation name to the JSON schema in
127
+ `src/schemas/configuration.schema.json`.
128
+ 4. Update `src/index.d.ts` with the new transformation name in the type union.
129
+ 5. Create a test file in `test/transformations/`.
130
+ 6. Update `CHANGELOG.md`.
131
+ 7. Run `yarn lint && yarn test`.
132
+
133
+ ### Updating the Configuration Schema
134
+
135
+ The configuration is validated by AJV against the JSON schema at
136
+ `src/schemas/configuration.schema.json`. When adding new fields:
137
+
138
+ 1. Update the schema with the new property definition.
139
+ 2. Update `src/index.d.ts` with the corresponding TypeScript type.
140
+ 3. Add test cases in `test/configuration.test.js`.
141
+
142
+ ## Troubleshooting
143
+
144
+ ### Tests Hang or Time Out
145
+
146
+ Tests run with `--runInBand --detectOpenHandles`. If a test hangs, it usually
147
+ means an HTTP mock (`nock`) was not set up correctly or a filesystem mock
148
+ (`mock-fs`) was not restored. Check that `nock.cleanAll()` and
149
+ `mock.restore()` are called in `afterEach` or `afterAll`.
150
+
151
+ ### ESLint Cache Issues
152
+
153
+ If linting gives unexpected results after changing `.eslintrc.js`, clear the
154
+ cache:
155
+
156
+ ```bash
157
+ rm -f .eslintcache
158
+ yarn lint
159
+ ```
160
+
161
+ ### Network Errors When Running the Compiler
162
+
163
+ The compiler downloads remote filter lists. If you're behind a proxy or
164
+ firewall, sources may fail to download. Use local file paths in your
165
+ configuration for offline development.
166
+
167
+ ## Additional Resources
168
+
169
+ - [README.md](README.md) — user documentation, configuration format, and
170
+ transformation reference
171
+ - [AGENTS.md](AGENTS.md) — code guidelines and contribution rules
172
+ - [CHANGELOG.md](CHANGELOG.md) — release history