@adguard/hostlist-compiler 2.1.1 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/prepare-release.yml +5 -0
- package/.markdownlint.json +29 -0
- package/.markdownlintignore +1 -0
- package/AGENTS.md +310 -98
- package/CHANGELOG.md +62 -8
- package/DEPLOYMENT.md +175 -0
- package/DEVELOPMENT.md +171 -72
- package/README.md +238 -143
- package/examples/china/README.md +2 -2
- package/examples/energized/README.md +4 -2
- package/examples/sdn/README.md +2 -2
- package/examples/whitelist/README.md +2 -2
- package/package.json +24 -18
- package/src/cli.js +53 -31
- package/src/compile-source.js +4 -3
- package/src/configuration.js +132 -2
- package/src/download.js +27 -0
- package/src/filter.js +2 -2
- package/src/index.d.ts +2 -0
- package/src/index.js +16 -50
- package/src/{transformations/ip-normalize.js → ip-normalize.js} +2 -2
- package/src/schemas/configuration.schema.js +137 -0
- package/src/source-types.js +17 -0
- package/src/transformations/enum.js +49 -0
- package/src/transformations/include.js +1 -1
- package/src/transformations/transform.js +40 -74
- package/src/transformations/validate-allow-ip-and-public-suffix.js +1 -1
- package/src/transformations/validate-allow-ip.js +1 -1
- package/src/utils.js +1284 -0
- package/src/validation-conflicts.js +106 -0
- package/test/cli-error-output.test.js +305 -0
- package/test/cli.test.js +47 -0
- package/test/compile-source.test.js +167 -7
- package/test/configuration.test.js +438 -0
- package/test/download.test.js +260 -0
- package/test/helpers/http-mock.js +94 -0
- package/test/index.test.js +497 -4
- package/test/{transformations/ip-normalize.test.js → ip-normalize.test.js} +1 -1
- package/test/resolve-include.test.js +8 -8
- package/test/transformations/covert-to-ascii.test.js +1 -1
- package/test/transformations/exclude.test.js +2 -2
- package/test/transformations/include.test.js +9 -3
- package/test/transformations/ip-alignment.test.js +1 -1
- package/test/transformations/transform.test.js +66 -1
- package/test/utils.test.js +1559 -0
- package/test/validation-conflicts.test.js +230 -0
- package/src/schemas/configuration.schema.json +0 -369
package/DEPLOYMENT.md
ADDED
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
# Deployment — @adguard/hostlist-compiler
|
|
2
|
+
|
|
3
|
+
- [Deployment Summary](#deployment-summary)
|
|
4
|
+
- [Release Pipeline](#release-pipeline)
|
|
5
|
+
- [CI/CD](#cicd)
|
|
6
|
+
- [Environment Variables](#environment-variables)
|
|
7
|
+
- [Infrastructure Dependencies](#infrastructure-dependencies)
|
|
8
|
+
- [Logging](#logging)
|
|
9
|
+
- [Integrations](#integrations)
|
|
10
|
+
- [External Service Dependencies](#external-service-dependencies)
|
|
11
|
+
- [External Filter Sources](#external-filter-sources)
|
|
12
|
+
- [Error Reporting](#error-reporting)
|
|
13
|
+
- [Docker Build](#docker-build)
|
|
14
|
+
- [Local Build Commands](#local-build-commands)
|
|
15
|
+
|
|
16
|
+
## Deployment Summary
|
|
17
|
+
|
|
18
|
+
| Parameter | Value |
|
|
19
|
+
| ----------------- | ------------------------------ |
|
|
20
|
+
| **npm package** | `@adguard/hostlist-compiler` |
|
|
21
|
+
| **Artifact** | `hostlist-compiler.tgz` |
|
|
22
|
+
| **Public mirror** | `AdguardTeam/HostlistCompiler` |
|
|
23
|
+
| **Slack channel** | `#adguard-filters-vcs` |
|
|
24
|
+
| **npm bin** | `hostlist-compiler` |
|
|
25
|
+
|
|
26
|
+
## Release Pipeline
|
|
27
|
+
|
|
28
|
+
Releases follow the shared [ext-shared-actions][ext-shared-actions] pipeline.
|
|
29
|
+
For the full step-by-step documentation, see
|
|
30
|
+
[publish-release.md](https://github.com/AdGuardSoftwareLimited/ext-shared-actions/blob/master/docs/publish-release.md).
|
|
31
|
+
|
|
32
|
+
In short:
|
|
33
|
+
|
|
34
|
+
1. A maintainer runs `prepare-release.yml` manually with a target tag
|
|
35
|
+
(e.g. `v2.1.0`) to open a release-bump PR that finalizes `CHANGELOG.md`.
|
|
36
|
+
2. Merging the release-bump PR triggers `publish-release.yml`, which tags
|
|
37
|
+
the release commit, builds and tests, publishes to npm via OIDC trusted
|
|
38
|
+
publishing, mirrors the tag to `AdguardTeam/HostlistCompiler`, drafts a
|
|
39
|
+
GitHub Release with the changelog entries, and notifies Slack
|
|
40
|
+
(`#adguard-filters-vcs`).
|
|
41
|
+
|
|
42
|
+
## CI/CD
|
|
43
|
+
|
|
44
|
+
| Workflow | Trigger | Purpose |
|
|
45
|
+
| --------------------- | -------------------------------------- | ------------------------------------------------------------------------ |
|
|
46
|
+
| `ci.yml` | PRs and pushes to `master` | Lint, test, build inside Docker; upload `hostlist-compiler.tgz` artifact |
|
|
47
|
+
| `prepare-release.yml` | Manual (`workflow_dispatch` with tag) | Open a release-bump PR that finalizes `CHANGELOG.md` |
|
|
48
|
+
| `publish-release.yml` | PR merged to `master` or manual re-run | Tag, build, publish to npm, mirror, draft GitHub Release, notify Slack |
|
|
49
|
+
| `mirror.yml` | Push to `master` | Mirror commits to `AdguardTeam/HostlistCompiler` |
|
|
50
|
+
|
|
51
|
+
A manual run of `prepare-release.yml` shows the requested tag in the Actions
|
|
52
|
+
run list (for example, `Prepare release v3.0.0-beta.0`), so runs for different
|
|
53
|
+
tags are easy to tell apart.
|
|
54
|
+
|
|
55
|
+
All workflows reuse the shared pipeline definitions from
|
|
56
|
+
[ext-shared-actions][ext-shared-actions] and [actions][actions]. The
|
|
57
|
+
`publish-release.yml` trigger fires for any closed PR against `master`; the
|
|
58
|
+
called workflow only proceeds when the PR was actually merged and its head ref
|
|
59
|
+
starts with `release-bump/`, so closing an unrelated PR is a safe no-op.
|
|
60
|
+
|
|
61
|
+
**Concurrency**: `ci.yml` uses a concurrency group
|
|
62
|
+
`ci-${{ github.workflow }}-${{ github.ref }}` with `cancel-in-progress: true`
|
|
63
|
+
to prevent redundant CI runs when a new push arrives for the same ref.
|
|
64
|
+
`publish-release.yml` uses a `publish-release` group with
|
|
65
|
+
`cancel-in-progress: false` to serialize release runs.
|
|
66
|
+
|
|
67
|
+
## Environment Variables
|
|
68
|
+
|
|
69
|
+
The compiler defines **no environment variables of its own**. To download
|
|
70
|
+
sources through a proxy, set `NODE_USE_ENV_PROXY=1` together with the
|
|
71
|
+
`HTTP_PROXY` (also used for `https://` when `HTTPS_PROXY` is unset),
|
|
72
|
+
`HTTPS_PROXY`, and `NO_PROXY` variables. The Node.js native `fetch` ignores
|
|
73
|
+
these variables unless `NODE_USE_ENV_PROXY=1` is set. All CI/CD
|
|
74
|
+
configuration (such as npm publish tokens, Octopass, and Slack webhooks) is
|
|
75
|
+
handled by the shared workflows and does not require per-project
|
|
76
|
+
configuration.
|
|
77
|
+
|
|
78
|
+
## Infrastructure Dependencies
|
|
79
|
+
|
|
80
|
+
The compiler is a **stateless CLI tool and library** with no database, cache,
|
|
81
|
+
or message queue dependencies. The only infrastructure requirement is:
|
|
82
|
+
|
|
83
|
+
| Dependency | Required | Purpose |
|
|
84
|
+
| --------------------------- | -------- | ---------------------------------------------------------------------------------------------- |
|
|
85
|
+
| **Outbound HTTP(S) access** | Yes | Downloading remote filter sources and `!#include` directives via `@adguard/filters-downloader` |
|
|
86
|
+
|
|
87
|
+
Remote sources are downloaded over HTTP or HTTPS with the Node.js native
|
|
88
|
+
`fetch` (inside `@adguard/filters-downloader`), and the downloaded content is
|
|
89
|
+
compiled entirely in memory — nothing is persisted between runs. The Docker
|
|
90
|
+
base image (`adguard/node-ssh:22.22--0`) includes everything else needed to
|
|
91
|
+
run the CLI.
|
|
92
|
+
|
|
93
|
+
## Logging
|
|
94
|
+
|
|
95
|
+
The compiler uses `consola` (v2) for all diagnostics.
|
|
96
|
+
|
|
97
|
+
| Aspect | Details |
|
|
98
|
+
| --------------- | ------------------------------------------------------------------------ |
|
|
99
|
+
| **Framework** | `consola` with its default reporters |
|
|
100
|
+
| **Output** | `stdout` for `info`/`warn`/`debug`/`trace`, `stderr` for `error`/`fatal` |
|
|
101
|
+
| **Format** | Plain text, consola default formatting |
|
|
102
|
+
| **Levels** | `fatal`, `error`, `warn`, `info`, `debug`, `trace` (consola levels 0-5) |
|
|
103
|
+
| **Default** | `info` level (3); `-v` / `--verbose` raises it to `trace` (5) |
|
|
104
|
+
| **Persistence** | No log files, no rotation, no remote log shipping |
|
|
105
|
+
|
|
106
|
+
There is no log configuration file — the only knob is the `-v` / `--verbose`
|
|
107
|
+
CLI flag. Failures are logged via `consola.error` and the CLI exits with
|
|
108
|
+
code 1.
|
|
109
|
+
|
|
110
|
+
## Integrations
|
|
111
|
+
|
|
112
|
+
### External Service Dependencies
|
|
113
|
+
|
|
114
|
+
| Integration | Purpose | Configuration |
|
|
115
|
+
| ------------------------------------------- | --------------------- | ----------------------------------------------------------------------------------------- |
|
|
116
|
+
| **npm registry** | Package distribution | OIDC trusted publishing via the shared `publish-release` workflow. No long-lived tokens. |
|
|
117
|
+
| **GitHub (`AdguardTeam/HostlistCompiler`)** | Public mirror | SSH push via Octopass OIDC. Workflows are disabled in the mirror repo. |
|
|
118
|
+
| **Slack** | Release notifications | `#adguard-filters-vcs` channel. Webhook managed by the shared `publish-release` workflow. |
|
|
119
|
+
|
|
120
|
+
### External Filter Sources
|
|
121
|
+
|
|
122
|
+
The compiler downloads filter lists at runtime from the URLs configured in
|
|
123
|
+
the compiler configuration (`sources`) and resolved `!#include` directives.
|
|
124
|
+
These URLs are user-defined — there are no hardcoded external services — but
|
|
125
|
+
the deployment environment must allow outbound HTTP(S) requests so that remote
|
|
126
|
+
sources can be fetched.
|
|
127
|
+
|
|
128
|
+
## Error Reporting
|
|
129
|
+
|
|
130
|
+
This project does **not** use an error reporting service (Sentry, Bugsnag,
|
|
131
|
+
or equivalent). Errors are logged to `stderr` via `consola.error` and the
|
|
132
|
+
CLI exits with code 1; when used as a library, errors are surfaced as thrown
|
|
133
|
+
`Error` / `TypeError` exceptions to the caller.
|
|
134
|
+
|
|
135
|
+
## Docker Build
|
|
136
|
+
|
|
137
|
+
The `Dockerfile` uses multi-stage builds based on `adguard/node-ssh:22.22--0`
|
|
138
|
+
(Node.js 22, pnpm 10 from the base image; the exact pnpm version is pinned in
|
|
139
|
+
the `engines` field of `package.json`):
|
|
140
|
+
|
|
141
|
+
| Stage | Purpose | Key Steps |
|
|
142
|
+
| ------------------------ | ----------------- | -------------------------------------------------------------------------------- |
|
|
143
|
+
| `base` | Shared foundation | Node.js 22, pnpm 10 (from base image), `npm_config_store_dir=/pnpm-store` |
|
|
144
|
+
| `deps` | Dependency cache | `pnpm install --frozen-lockfile --prefer-offline --ignore-scripts` |
|
|
145
|
+
| `source` | Full source | Copies project files over `deps` |
|
|
146
|
+
| `test` / `test-output` | CI validation | `pnpm lint && pnpm test`; `test-output` exports the passed-gate marker |
|
|
147
|
+
| `build` / `build-output` | Artifact creation | `pnpm pack --out hostlist-compiler.tgz`; outputs the `.tgz` |
|
|
148
|
+
|
|
149
|
+
The dependency stage is cached by `package.json` and `pnpm-lock.yaml`. The
|
|
150
|
+
pnpm store cache is mounted at `/pnpm-store` with id `hostlist-compiler-pnpm`.
|
|
151
|
+
|
|
152
|
+
The table above summarizes the Dockerfile; the `Dockerfile` and the workflow
|
|
153
|
+
files in `.github/workflows/` are the source of truth for the exact stages,
|
|
154
|
+
flags, triggers, and concurrency settings.
|
|
155
|
+
|
|
156
|
+
### Local Build Commands
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
# Run CI validation (lint + tests)
|
|
160
|
+
docker build --target test-output .
|
|
161
|
+
|
|
162
|
+
# Produce the release artifact. The repo's package.json has no `version`
|
|
163
|
+
# field and `pnpm pack` requires one, so set it first — CI stamps the
|
|
164
|
+
# release version into package.json before building the image.
|
|
165
|
+
npm pkg set version=0.0.0-dev
|
|
166
|
+
docker build --target build-output --output ./artifacts .
|
|
167
|
+
# → ./artifacts/hostlist-compiler.tgz
|
|
168
|
+
|
|
169
|
+
# Restore package.json — the version was only set for the local build, and
|
|
170
|
+
# CI stamps the release version into the tracked file.
|
|
171
|
+
git checkout -- package.json
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
[ext-shared-actions]: https://github.com/AdGuardSoftwareLimited/ext-shared-actions
|
|
175
|
+
[actions]: https://github.com/AdGuardSoftwareLimited/actions
|
package/DEVELOPMENT.md
CHANGED
|
@@ -1,47 +1,84 @@
|
|
|
1
1
|
# Development Guide
|
|
2
2
|
|
|
3
|
-
This document explains how to set up the development environment, run the
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
3
|
+
This document explains how to set up the development environment, run the project locally, and contribute code to
|
|
4
|
+
Hostlist Compiler. For code guidelines and architectural decisions, see [AGENTS.md](AGENTS.md).
|
|
5
|
+
|
|
6
|
+
## Table of Contents
|
|
7
|
+
|
|
8
|
+
- [Prerequisites](#prerequisites)
|
|
9
|
+
- [Getting Started](#getting-started)
|
|
10
|
+
- [Cloning the Repository](#cloning-the-repository)
|
|
11
|
+
- [Installing Dependencies](#installing-dependencies)
|
|
12
|
+
- [Verifying the Setup](#verifying-the-setup)
|
|
13
|
+
- [Project Structure](#project-structure)
|
|
14
|
+
- [Available Commands](#available-commands)
|
|
15
|
+
- [Running the Compiler Locally](#running-the-compiler-locally)
|
|
16
|
+
- [Development Workflow](#development-workflow)
|
|
17
|
+
- [Branching and Pull Requests](#branching-and-pull-requests)
|
|
18
|
+
- [Making Changes](#making-changes)
|
|
19
|
+
- [Running Tests](#running-tests)
|
|
20
|
+
- [Linting](#linting)
|
|
21
|
+
- [Commit Message Convention](#commit-message-convention)
|
|
22
|
+
- [Common Tasks](#common-tasks)
|
|
23
|
+
- [Adding a New Transformation](#adding-a-new-transformation)
|
|
24
|
+
- [Updating the Configuration Schema](#updating-the-configuration-schema)
|
|
25
|
+
- [Bumping the Version](#bumping-the-version)
|
|
26
|
+
- [Debugging and IDE Setup](#debugging-and-ide-setup)
|
|
27
|
+
- [Troubleshooting](#troubleshooting)
|
|
28
|
+
- [Tests Hang or Time Out](#tests-hang-or-time-out)
|
|
29
|
+
- [ESLint Cache Issues](#eslint-cache-issues)
|
|
30
|
+
- [Network Errors When Running the Compiler](#network-errors-when-running-the-compiler)
|
|
31
|
+
- [pnpm Version Mismatch](#pnpm-version-mismatch)
|
|
32
|
+
- [Additional Resources](#additional-resources)
|
|
7
33
|
|
|
8
34
|
## Prerequisites
|
|
9
35
|
|
|
10
|
-
- **Node.js** —
|
|
11
|
-
|
|
12
|
-
- **
|
|
36
|
+
- **Node.js 22.22.2 or newer** — required by the `engines` field in `package.json`; the CI pipeline and the
|
|
37
|
+
Docker image build with Node 22. Node.js 23.x is not supported by `@adguard/filters-downloader` 3.x.
|
|
38
|
+
- **pnpm 10.x** — required by the `engines` field in `package.json` (use the exact version pinned there if pnpm
|
|
39
|
+
reports a mismatch). Install with `npm install -g pnpm@10`.
|
|
40
|
+
- **Git** — for cloning the repository and managing branches.
|
|
13
41
|
|
|
14
42
|
## Getting Started
|
|
15
43
|
|
|
16
|
-
|
|
44
|
+
### Cloning the Repository
|
|
45
|
+
|
|
46
|
+
**Internal developers** (with access to the private repository where active development happens):
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
git clone git@github.com:AdGuardSoftwareLimited/filters-hostlist-compiler.git
|
|
50
|
+
cd filters-hostlist-compiler
|
|
51
|
+
```
|
|
17
52
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
53
|
+
**External contributors**: the public
|
|
54
|
+
[AdguardTeam/HostlistCompiler](https://github.com/AdguardTeam/HostlistCompiler)
|
|
55
|
+
mirror is read-only, so it cannot be used as a base for pull requests. Fork the
|
|
56
|
+
mirror, push your changes to the fork, and describe the proposed change in an
|
|
57
|
+
issue so maintainers can port it into the private repository.
|
|
58
|
+
|
|
59
|
+
### Installing Dependencies
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
pnpm install
|
|
63
|
+
```
|
|
22
64
|
|
|
23
|
-
|
|
65
|
+
### Verifying the Setup
|
|
24
66
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
67
|
+
Run the full verification sequence described in the [Build and Test Commands section of
|
|
68
|
+
AGENTS.md](AGENTS.md#build-and-test-commands) (`pnpm lint` and `pnpm test`) and make sure both finish without
|
|
69
|
+
errors.
|
|
28
70
|
|
|
29
|
-
|
|
71
|
+
## Project Structure
|
|
30
72
|
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
pnpm test
|
|
34
|
-
```
|
|
73
|
+
The full directory tree, including the layered architecture, is documented in the [Project Structure section of
|
|
74
|
+
AGENTS.md](AGENTS.md#project-structure).
|
|
35
75
|
|
|
36
76
|
## Available Commands
|
|
37
77
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
| `pnpm test` | Run Jest test suite (`--runInBand --detectOpenHandles`) |
|
|
43
|
-
| `pnpm increment` | Bump the patch version in `package.json` |
|
|
44
|
-
| `pnpm compile` | Run the compiler CLI (`node src/cli.js`) |
|
|
78
|
+
The [Build and Test Commands section of AGENTS.md](AGENTS.md#build-and-test-commands) is the single source of
|
|
79
|
+
truth for npm scripts. Running the compiler locally (`node src/cli.js` and its `pnpm compile` alias) and the
|
|
80
|
+
version bump (`pnpm increment`) are covered in [Running the Compiler Locally](#running-the-compiler-locally)
|
|
81
|
+
and [Bumping the Version](#bumping-the-version) below.
|
|
45
82
|
|
|
46
83
|
## Running the Compiler Locally
|
|
47
84
|
|
|
@@ -51,36 +88,41 @@ For code guidelines and architectural decisions, see [AGENTS.md](AGENTS.md).
|
|
|
51
88
|
node src/cli.js -c examples/sdn/configuration.json -o filter.txt
|
|
52
89
|
```
|
|
53
90
|
|
|
54
|
-
**Quick hosts conversion** (
|
|
91
|
+
**Quick hosts conversion** (no configuration file needed):
|
|
55
92
|
|
|
56
93
|
```bash
|
|
57
94
|
node src/cli.js -i hosts.txt -o output.txt
|
|
58
95
|
```
|
|
59
96
|
|
|
60
|
-
|
|
97
|
+
All CLI flags (`-i` repeatable, `-t` input type, required `-o`, etc.) are documented in the [Usage section of
|
|
98
|
+
README.md](README.md#usage).
|
|
99
|
+
|
|
100
|
+
**With verbose logging** (trace-level `consola` output):
|
|
61
101
|
|
|
62
102
|
```bash
|
|
63
103
|
node src/cli.js -c examples/sdn/configuration.json -o filter.txt -v
|
|
64
104
|
```
|
|
65
105
|
|
|
66
|
-
The `
|
|
67
|
-
|
|
106
|
+
The `pnpm compile` script is an alias for `node src/cli.js` and forwards its arguments:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
pnpm compile -i hosts.txt -o output.txt
|
|
110
|
+
```
|
|
68
111
|
|
|
69
112
|
## Development Workflow
|
|
70
113
|
|
|
71
|
-
###
|
|
114
|
+
### Branching and Pull Requests
|
|
72
115
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
116
|
+
- Create a feature branch from `master` and push it to the remote.
|
|
117
|
+
- Open a pull request against `master`.
|
|
118
|
+
- CI runs on every pull request and every push to `master`: it lints the code, runs the test suite, and builds the npm
|
|
119
|
+
package tarball. All checks must pass before merging.
|
|
77
120
|
|
|
78
|
-
|
|
79
|
-
pnpm lint
|
|
80
|
-
pnpm test
|
|
81
|
-
```
|
|
121
|
+
### Making Changes
|
|
82
122
|
|
|
83
|
-
|
|
123
|
+
Make your changes in `src/` and follow the contribution rules in [AGENTS.md](AGENTS.md#contribution-instructions):
|
|
124
|
+
add or update tests in `test/` mirroring the source file structure, keep `CHANGELOG.md` up to date for
|
|
125
|
+
user-facing changes, and run the full verification sequence before pushing.
|
|
84
126
|
|
|
85
127
|
### Running Tests
|
|
86
128
|
|
|
@@ -90,20 +132,20 @@ Run the full test suite:
|
|
|
90
132
|
pnpm test
|
|
91
133
|
```
|
|
92
134
|
|
|
93
|
-
Run a
|
|
135
|
+
Run a single test file:
|
|
94
136
|
|
|
95
137
|
```bash
|
|
96
138
|
npx jest test/rule.test.js --runInBand
|
|
97
139
|
```
|
|
98
140
|
|
|
99
|
-
Run tests matching a pattern:
|
|
141
|
+
Run only tests matching a name pattern:
|
|
100
142
|
|
|
101
143
|
```bash
|
|
102
144
|
npx jest --testNamePattern="compress" --runInBand
|
|
103
145
|
```
|
|
104
146
|
|
|
105
|
-
|
|
106
|
-
|
|
147
|
+
The test setup (Jest flags, `silent` mode, `mock-fs`/`undici` MockAgent mocking, fixture files) is
|
|
148
|
+
described in the [Testing section of AGENTS.md](AGENTS.md#testing).
|
|
107
149
|
|
|
108
150
|
### Linting
|
|
109
151
|
|
|
@@ -111,45 +153,94 @@ interception. Test fixtures live in `test/resources/`.
|
|
|
111
153
|
pnpm lint
|
|
112
154
|
```
|
|
113
155
|
|
|
114
|
-
ESLint
|
|
115
|
-
|
|
156
|
+
ESLint (`pnpm lint:code`) and markdownlint (`pnpm lint:md`) enforce the rules described in the [Code
|
|
157
|
+
Quality](AGENTS.md#code-quality) and [Markdown Formatting](AGENTS.md#markdown-formatting) sections of
|
|
158
|
+
AGENTS.md. Both linters support auto-fixing:
|
|
159
|
+
|
|
160
|
+
```bash
|
|
161
|
+
pnpm exec eslint . --fix
|
|
162
|
+
pnpm exec markdownlint . --fix
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
### Commit Message Convention
|
|
166
|
+
|
|
167
|
+
Commit messages MUST follow the ticket-prefix convention (`AG-XXX` + short present-tense description) described in
|
|
168
|
+
the [Contribution Instructions section of AGENTS.md](AGENTS.md#contribution-instructions).
|
|
116
169
|
|
|
117
170
|
## Common Tasks
|
|
118
171
|
|
|
119
172
|
### Adding a New Transformation
|
|
120
173
|
|
|
121
|
-
1. Create a new module in `src/transformations/` exporting a single async
|
|
122
|
-
|
|
123
|
-
2. Register the transformation in `src/transformations/
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
174
|
+
1. Create a new module in `src/transformations/` exporting a single async function with a `(rules, ...) => rules`
|
|
175
|
+
signature.
|
|
176
|
+
2. Register the transformation in the `TRANSFORMATIONS` enum in `src/transformations/enum.js` — the single source of
|
|
177
|
+
truth for transformation names; the configuration schema derives its accepted names from it. The declaration order
|
|
178
|
+
matters (it renders in the unknown-transformation error message's allowed-values list), so insert the entry in the
|
|
179
|
+
right place.
|
|
180
|
+
3. Register the transformation in `TRANSFORMATIONS_IN_ORDER` in `src/transformations/transform.js` — that list pairs
|
|
181
|
+
each enum name with its implementation and determines execution order, not the order in the configuration. A test
|
|
182
|
+
asserts the list covers every enum entry and pins its order, so a missing registration or an accidental reorder
|
|
183
|
+
fails the suite instead of being silently skipped.
|
|
184
|
+
4. If the new transformation is a *validation* transformation, also add it to `VALIDATION_TRANSFORMATIONS` in
|
|
185
|
+
`src/transformations/enum.js` — it sits right below the enum, so adding a validator is a single-file change; the
|
|
186
|
+
runtime conflict checks and the generated schema conflict rules are derived from that list.
|
|
187
|
+
5. If the transformation should be part of the quick `-i` conversion pipeline, add it to
|
|
188
|
+
`DEFAULT_TRANSFORMATIONS` in `src/transformations/transform.js`.
|
|
189
|
+
6. Apply the documentation-sync rules from [AGENTS.md](AGENTS.md#configuration--documentation)
|
|
190
|
+
(`src/schemas/configuration.schema.js`, `src/index.d.ts`, the order list in `README.md`, `CHANGELOG.md`).
|
|
191
|
+
7. Create a test file in `test/transformations/`.
|
|
192
|
+
8. Run `pnpm lint && pnpm test`.
|
|
131
193
|
|
|
132
194
|
### Updating the Configuration Schema
|
|
133
195
|
|
|
134
|
-
|
|
135
|
-
|
|
196
|
+
When adding or changing configuration fields, follow the documentation-sync rules in the
|
|
197
|
+
[Configuration & Documentation section of AGENTS.md](AGENTS.md#configuration--documentation), then:
|
|
136
198
|
|
|
137
|
-
1. Update the schema
|
|
138
|
-
|
|
199
|
+
1. Update the schema — it is a JS module, `src/schemas/configuration.schema.js`, not a JSON file. If the change
|
|
200
|
+
touches transformation names, update the `TRANSFORMATIONS` enum in `src/transformations/enum.js` instead (the
|
|
201
|
+
schema derives them from it); if it touches transformation conflicts, update `VALIDATION_TRANSFORMATIONS` in
|
|
202
|
+
`src/transformations/enum.js` (the schema conflict rules are generated from it); if it touches source types,
|
|
203
|
+
update `SOURCE_TYPES` in `src/source-types.js` (the schema derives the `type` enum from it).
|
|
204
|
+
2. Update the corresponding TypeScript types in `src/index.d.ts`.
|
|
139
205
|
3. Add test cases in `test/configuration.test.js`.
|
|
140
206
|
|
|
207
|
+
### Bumping the Version
|
|
208
|
+
|
|
209
|
+
```bash
|
|
210
|
+
pnpm increment
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
Bumps the patch version in `package.json` without creating a git tag. Preparing and publishing a release is automated
|
|
214
|
+
by the `prepare-release` and `publish-release` GitHub Actions workflows.
|
|
215
|
+
|
|
216
|
+
## Debugging and IDE Setup
|
|
217
|
+
|
|
218
|
+
- **Verbose logging**: pass `-v` to the CLI to enable trace-level `consola` output.
|
|
219
|
+
- **Node inspector**: run the CLI under the debugger:
|
|
220
|
+
|
|
221
|
+
```bash
|
|
222
|
+
node --inspect-brk src/cli.js -c examples/sdn/configuration.json -o filter.txt
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
- **VS Code**: install the ESLint extension for in-editor diagnostics, and add a `launch.json` configuration that runs
|
|
226
|
+
`src/cli.js` with the `-c`/`-o` arguments you want to debug.
|
|
227
|
+
- **Debugging a failing test**: run a single test file under the inspector:
|
|
228
|
+
|
|
229
|
+
```bash
|
|
230
|
+
node --inspect-brk node_modules/.bin/jest test/rule.test.js --runInBand
|
|
231
|
+
```
|
|
232
|
+
|
|
141
233
|
## Troubleshooting
|
|
142
234
|
|
|
143
235
|
### Tests Hang or Time Out
|
|
144
236
|
|
|
145
|
-
Tests run with `--runInBand --detectOpenHandles`. If a test hangs,
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
`mock.restore()` are called in `afterEach` or `afterAll`.
|
|
237
|
+
Tests run with `--runInBand --detectOpenHandles`. If a test hangs, an HTTP mock (`undici` MockAgent) was probably
|
|
238
|
+
not set up correctly or a filesystem mock (`mock-fs`) was not restored. Check that each mock scope's `done()`
|
|
239
|
+
assertion runs and that `mock.restore()` is called in `afterEach` or `afterAll`.
|
|
149
240
|
|
|
150
241
|
### ESLint Cache Issues
|
|
151
242
|
|
|
152
|
-
If linting gives unexpected results after changing `.eslintrc.js`, clear the
|
|
243
|
+
ESLint caches results in `.eslintcache`. If linting gives unexpected results after changing `.eslintrc.js`, clear the
|
|
153
244
|
cache:
|
|
154
245
|
|
|
155
246
|
```bash
|
|
@@ -159,13 +250,21 @@ pnpm lint
|
|
|
159
250
|
|
|
160
251
|
### Network Errors When Running the Compiler
|
|
161
252
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
configuration for offline development.
|
|
253
|
+
Remote filter lists are downloaded by `@adguard/filters-downloader`. Behind a proxy or firewall, sources may fail
|
|
254
|
+
to download. To use the standard proxy environment variables with the Node.js native `fetch`, set
|
|
255
|
+
`NODE_USE_ENV_PROXY=1`. Use local file paths in your configuration for offline development.
|
|
256
|
+
|
|
257
|
+
### pnpm Version Mismatch
|
|
258
|
+
|
|
259
|
+
The `engines` field in `package.json` requires a specific pnpm version. If pnpm reports a version mismatch, reinstall
|
|
260
|
+
the required version:
|
|
261
|
+
|
|
262
|
+
```bash
|
|
263
|
+
npm install -g pnpm@10
|
|
264
|
+
```
|
|
165
265
|
|
|
166
266
|
## Additional Resources
|
|
167
267
|
|
|
168
|
-
- [README.md](README.md) — user documentation, configuration format, and
|
|
169
|
-
transformation reference
|
|
268
|
+
- [README.md](README.md) — user documentation, configuration format, and transformation reference
|
|
170
269
|
- [AGENTS.md](AGENTS.md) — code guidelines and contribution rules
|
|
171
270
|
- [CHANGELOG.md](CHANGELOG.md) — release history
|