@adguard/hostlist-compiler 1.0.38 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +175 -0
- package/CHANGELOG.md +24 -0
- package/DEVELOPMENT.md +172 -0
- package/README.md +107 -94
- package/bamboo-specs/build.yaml +2 -10
- package/bamboo-specs/deploy.yaml +0 -7
- package/bamboo-specs/increment.yaml +1 -1
- package/bamboo-specs/test.yaml +0 -8
- package/package.json +2 -2
- package/src/compile-source.js +1 -1
- package/src/filter.js +2 -2
- package/src/index.d.ts +5 -2
- package/src/index.js +43 -1
- package/src/schemas/configuration.schema.json +102 -0
- package/src/transformations/transform.js +35 -1
- package/src/transformations/validate-allow-ip.js +3 -2
- package/src/transformations/validate-allow-public-suffix.js +20 -0
- package/src/transformations/validate.js +183 -19
- package/test/compile-source.test.js +44 -0
- package/test/configuration.test.js +64 -0
- package/test/index.test.js +64 -0
- package/test/resolve-include.test.js +98 -0
- package/test/transformations/transform.test.js +19 -0
- package/test/transformations/validate-allow-ip.test.js +44 -1
- package/test/transformations/validate-allow-public-suffix.test.js +264 -0
- package/test/transformations/validate.test.js +194 -0
package/AGENTS.md
ADDED
|
@@ -0,0 +1,175 @@
|
|
|
1
|
+
# AGENTS.md
|
|
2
|
+
|
|
3
|
+
Guidelines for AI agents working on this project.
|
|
4
|
+
|
|
5
|
+
## Project Overview
|
|
6
|
+
|
|
7
|
+
**Hostlist Compiler** (`@adguard/hostlist-compiler`) is a Node.js CLI tool and
|
|
8
|
+
library that compiles DNS hosts blocklists from multiple sources into a single
|
|
9
|
+
filter list compatible with AdGuard Home and other AdGuard products with DNS
|
|
10
|
+
filtering.
|
|
11
|
+
|
|
12
|
+
It supports multiple input formats (`/etc/hosts`, adblock-style), configurable
|
|
13
|
+
transformations (compress, deduplicate, validate, remove comments, etc.),
|
|
14
|
+
inclusion/exclusion rules, and `!#include` directive resolution via
|
|
15
|
+
`@adguard/filters-downloader`.
|
|
16
|
+
|
|
17
|
+
## Technical Context
|
|
18
|
+
|
|
19
|
+
- **Language**: JavaScript (Node.js, CommonJS modules)
|
|
20
|
+
- **Node Version**: Compatible with current LTS
|
|
21
|
+
- **Primary Dependencies**:
|
|
22
|
+
- `@adguard/filters-downloader` — resolves `!#include` directives and
|
|
23
|
+
downloads remote sources
|
|
24
|
+
- `ajv` / `ajv-errors` / `better-ajv-errors` — JSON schema validation for
|
|
25
|
+
configuration
|
|
26
|
+
- `axios` — HTTP requests for remote filter lists
|
|
27
|
+
- `lodash` — utility functions
|
|
28
|
+
- `tldts` — TLD parsing for domain validation
|
|
29
|
+
- `yargs` — CLI argument parsing
|
|
30
|
+
- `consola` — logging
|
|
31
|
+
- **Testing**: Jest (`jest --runInBand --detectOpenHandles`)
|
|
32
|
+
- **Linting**: ESLint with `airbnb-base` config, `@babel/eslint-parser`
|
|
33
|
+
- **Storage**: None (stateless CLI tool, reads remote/local files, writes output
|
|
34
|
+
files)
|
|
35
|
+
- **Target Platform**: Node.js (CLI and library; published to npm)
|
|
36
|
+
- **Project Type**: Single package (CLI + library)
|
|
37
|
+
- **License**: GPL-3.0
|
|
38
|
+
|
|
39
|
+
## Project Structure
|
|
40
|
+
|
|
41
|
+
```text
|
|
42
|
+
├── src/ # Main source code
|
|
43
|
+
│ ├── cli.js # CLI entry point (hashbang, yargs setup)
|
|
44
|
+
│ ├── index.js # Library entry point (compile function)
|
|
45
|
+
│ ├── index.d.ts # TypeScript type declarations
|
|
46
|
+
│ ├── compile-source.js # Compiles individual sources
|
|
47
|
+
│ ├── configuration.js # Configuration validation (AJV)
|
|
48
|
+
│ ├── filter.js # Filtering utilities
|
|
49
|
+
│ ├── rule.js # Rule parsing (adblock, /etc/hosts)
|
|
50
|
+
│ ├── utils.js # String and path utilities
|
|
51
|
+
│ ├── schemas/ # JSON schemas for configuration validation
|
|
52
|
+
│ └── transformations/ # Transformation modules (14 files)
|
|
53
|
+
├── test/ # Jest test suites mirroring src/
|
|
54
|
+
│ ├── resources/ # Test fixture files
|
|
55
|
+
│ └── transformations/ # Transformation-specific tests
|
|
56
|
+
├── tools/ # Build helper scripts
|
|
57
|
+
│ └── build-txt.js # Generates transformations docs
|
|
58
|
+
├── examples/ # Example configurations (sdn, energized, etc.)
|
|
59
|
+
├── bamboo-specs/ # CI/CD pipeline definitions (Bamboo)
|
|
60
|
+
├── package.json # Package manifest and scripts
|
|
61
|
+
├── jest.config.js # Jest configuration
|
|
62
|
+
├── .eslintrc.js # ESLint configuration
|
|
63
|
+
├── CHANGELOG.md # Release history
|
|
64
|
+
├── README.md # User documentation
|
|
65
|
+
└── LICENSE # GPL-3.0
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## Build and Test Commands
|
|
69
|
+
|
|
70
|
+
**Full verification sequence** (run all of these before considering a task done):
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
yarn lint
|
|
74
|
+
yarn test
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
| Action | Command |
|
|
78
|
+
|---------|---------|
|
|
79
|
+
| Install | `yarn install` |
|
|
80
|
+
| Lint | `yarn lint` |
|
|
81
|
+
| Test | `yarn test` |
|
|
82
|
+
| Run CLI (example) | `node src/cli.js -c examples/sdn/configuration.json -o filter.txt` |
|
|
83
|
+
| Version bump | `yarn increment` |
|
|
84
|
+
|
|
85
|
+
## Verification
|
|
86
|
+
|
|
87
|
+
You MUST follow these rules for EVERY task you perform:
|
|
88
|
+
|
|
89
|
+
1. **Run the linter**: `yarn lint`. Fix all warnings and errors before
|
|
90
|
+
proceeding. The project uses ESLint with `airbnb-base`.
|
|
91
|
+
2. **Run the tests**: `yarn test`. All tests must pass. If you changed
|
|
92
|
+
behavior, verify that existing tests still reflect correct expectations.
|
|
93
|
+
3. **Check the outcome**. If either command fails, fix the issues and re-run
|
|
94
|
+
until both pass. A task is not complete until all checks are green.
|
|
95
|
+
|
|
96
|
+
## Mandatory Task Actions
|
|
97
|
+
|
|
98
|
+
For every task you perform, you MUST:
|
|
99
|
+
|
|
100
|
+
- **Update or add unit tests** for any changed or new code. Mirror the source
|
|
101
|
+
file structure under `test/`.
|
|
102
|
+
- **Run the full verification sequence** (`yarn lint` and `yarn test`) and
|
|
103
|
+
confirm both pass.
|
|
104
|
+
- **Update `AGENTS.md`** if you change the project structure, add new build
|
|
105
|
+
commands, or discover a recurring agent mistake that should become a rule.
|
|
106
|
+
- **Update `CHANGELOG.md`** when making user-facing changes.
|
|
107
|
+
- **Update the JSON schema** (`src/schemas/configuration.schema.json`) when
|
|
108
|
+
adding or changing configuration options.
|
|
109
|
+
- **Update TypeScript declarations** (`src/index.d.ts`) when changing the
|
|
110
|
+
public API.
|
|
111
|
+
- **Verify your code follows the Code Guidelines** in this file after
|
|
112
|
+
completing the task.
|
|
113
|
+
- If a prompt asks you to refactor or improve existing code, check whether
|
|
114
|
+
the insight can be phrased as a code guideline. If so, add it to the
|
|
115
|
+
relevant Code Guidelines section below.
|
|
116
|
+
|
|
117
|
+
## Contribution Instructions
|
|
118
|
+
|
|
119
|
+
1. **Install** dependencies with `yarn install`.
|
|
120
|
+
2. **Lint** before committing: `yarn lint`. Fix all warnings and errors.
|
|
121
|
+
3. **Test** before committing: `yarn test`. All tests must pass. Add tests for
|
|
122
|
+
new functionality — mirror the source file structure under `test/`.
|
|
123
|
+
4. **Do not commit** generated files, `node_modules`, or editor configs.
|
|
124
|
+
5. **Do not create git commits or PRs** unless explicitly asked.
|
|
125
|
+
6. Keep the `CHANGELOG.md` up to date when making user-facing changes.
|
|
126
|
+
|
|
127
|
+
## Code Guidelines
|
|
128
|
+
|
|
129
|
+
### Architecture
|
|
130
|
+
|
|
131
|
+
- **CommonJS modules** — use `require`/`module.exports` throughout; do not
|
|
132
|
+
introduce ES module syntax.
|
|
133
|
+
- **Transformation pipeline** — each transformation is a standalone module in
|
|
134
|
+
`src/transformations/` exporting a single async function. Transformations are
|
|
135
|
+
orchestrated by `src/transformations/transform.js` and always run in a fixed
|
|
136
|
+
order regardless of the order specified in configuration.
|
|
137
|
+
- **Configuration-driven** — the compiler is controlled by a JSON configuration
|
|
138
|
+
object validated against `src/schemas/configuration.schema.json` via AJV.
|
|
139
|
+
|
|
140
|
+
### Code Quality
|
|
141
|
+
|
|
142
|
+
- **ESLint airbnb-base** rules apply — run `yarn lint` to check. See
|
|
143
|
+
`.eslintrc.js` for project-specific overrides.
|
|
144
|
+
- Use `lodash` helpers (e.g., `_.startsWith`, `_.isEmpty`, `_.trim`) where they
|
|
145
|
+
are already used — stay consistent with the existing style.
|
|
146
|
+
- Prefer `for...of` loops over `Array.forEach` for async iteration (the
|
|
147
|
+
codebase disables `no-restricted-syntax` where needed).
|
|
148
|
+
|
|
149
|
+
### Testing
|
|
150
|
+
|
|
151
|
+
- **Framework**: Jest, running in Node environment with `--runInBand`.
|
|
152
|
+
- **Mocking**: `mock-fs` for filesystem mocks, `nock` for HTTP request
|
|
153
|
+
interception.
|
|
154
|
+
- Test files follow the naming convention `<module>.test.js` under `test/`.
|
|
155
|
+
- Each transformation has its own test file under `test/transformations/`.
|
|
156
|
+
|
|
157
|
+
### Logging and Error Handling
|
|
158
|
+
|
|
159
|
+
- Use `consola` for all logging (`consola.info`, `consola.debug`,
|
|
160
|
+
`consola.error`). Do not use `console.log` / `console.error` directly.
|
|
161
|
+
- Throw descriptive `Error` (or `TypeError`) instances with messages that
|
|
162
|
+
include the invalid input value — this helps users diagnose configuration
|
|
163
|
+
and rule problems.
|
|
164
|
+
- Propagate errors up to the CLI entry point (`src/cli.js`), which catches
|
|
165
|
+
them, logs via `consola.error`, and exits with code 1.
|
|
166
|
+
|
|
167
|
+
### Other
|
|
168
|
+
|
|
169
|
+
- The `src/index.d.ts` file provides TypeScript type declarations for external
|
|
170
|
+
consumers. Keep it in sync when changing the public API.
|
|
171
|
+
- The JSON schema in `src/schemas/configuration.schema.json` must be updated
|
|
172
|
+
when adding new configuration options or transformations.
|
|
173
|
+
- Any recurring agent error is a defect in this document. If you had to be
|
|
174
|
+
corrected twice on the same issue, add a rule here to prevent it in the
|
|
175
|
+
future.
|
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [2.0.0] - 2026-04-02
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- `ValidateAllowPublicSuffix` transformation option that allows rules matching whole public suffixes (e.g. `||hl.cn^`, `||org^`). [#124]
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- `Validate` now rejects rules matching whole public suffixes consistently across syntactic variants such as `||org^`, `||*.org^`, `.org^`, and `*.org^`. Use `ValidateAllowPublicSuffix` if you need to keep such rules.
|
|
17
|
+
- The compiler now warns when incompatible validation transformations are combined at runtime, and configuration validation rejects combining `Validate`, `ValidateAllowIp`, and `ValidateAllowPublicSuffix` in the same transformations list.
|
|
18
|
+
|
|
19
|
+
[2.0.0]: https://github.com/AdguardTeam/HostlistCompiler/compare/v1.0.39...v2.0.0
|
|
20
|
+
[#124]: https://github.com/AdguardTeam/HostlistCompiler/issues/124
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
## [1.0.39] - 2025-03-13
|
|
24
|
+
|
|
25
|
+
### Changed
|
|
26
|
+
|
|
27
|
+
- Updated [@adguard/filters-downloader] to 2.3.1. This version handles empty sources correctly. [#85]
|
|
28
|
+
|
|
29
|
+
[1.0.39]: https://github.com/AdguardTeam/HostlistCompiler/compare/v1.0.38...v1.0.39
|
|
30
|
+
[#85]: https://github.com/AdguardTeam/HostlistCompiler/issues/85
|
|
31
|
+
|
|
8
32
|
## [1.0.38] - 2025-03-06
|
|
9
33
|
|
|
10
34
|
### Changed
|
package/DEVELOPMENT.md
ADDED
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
# Development Guide
|
|
2
|
+
|
|
3
|
+
This document explains how to set up the development environment, run the
|
|
4
|
+
project locally, and contribute code to Hostlist Compiler.
|
|
5
|
+
|
|
6
|
+
For code guidelines and architectural decisions, see [AGENTS.md](AGENTS.md).
|
|
7
|
+
|
|
8
|
+
## Prerequisites
|
|
9
|
+
|
|
10
|
+
- **Node.js** — current LTS version (20.x or later)
|
|
11
|
+
- **Yarn** — classic (1.x)
|
|
12
|
+
- **Git**
|
|
13
|
+
|
|
14
|
+
## Getting Started
|
|
15
|
+
|
|
16
|
+
1. Clone the repository:
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
git clone https://github.com/AdguardTeam/HostlistCompiler.git
|
|
20
|
+
cd HostlistCompiler
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
2. Install dependencies:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
yarn install
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
3. Verify the setup by running linter and tests:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
yarn lint
|
|
33
|
+
yarn test
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
## Available Commands
|
|
37
|
+
|
|
38
|
+
| Command | Description |
|
|
39
|
+
|---------|-------------|
|
|
40
|
+
| `yarn install` | Install all dependencies |
|
|
41
|
+
| `yarn lint` | Run ESLint (`airbnb-base` config) |
|
|
42
|
+
| `yarn test` | Run Jest test suite (`--runInBand --detectOpenHandles`) |
|
|
43
|
+
| `yarn increment` | Bump the patch version in `package.json` |
|
|
44
|
+
| `yarn build-txt` | Generate transformations documentation |
|
|
45
|
+
| `yarn compile` | Run the compiler CLI (`node src/cli.js`) |
|
|
46
|
+
|
|
47
|
+
## Running the Compiler Locally
|
|
48
|
+
|
|
49
|
+
**With a configuration file** (full-featured mode):
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
node src/cli.js -c examples/sdn/configuration.json -o filter.txt
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
**Quick hosts conversion** (simple mode):
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
node src/cli.js -i hosts.txt -o output.txt
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
**With verbose logging**:
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
node src/cli.js -c examples/sdn/configuration.json -o filter.txt -v
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
The `examples/` directory contains several ready-made configurations (`sdn`,
|
|
68
|
+
`energized`, `china`, `whitelist`) that can be used for testing.
|
|
69
|
+
|
|
70
|
+
## Development Workflow
|
|
71
|
+
|
|
72
|
+
### Making Changes
|
|
73
|
+
|
|
74
|
+
1. Create a feature branch from `master`.
|
|
75
|
+
2. Make your changes in `src/`.
|
|
76
|
+
3. Add or update tests in `test/` — mirror the source file structure.
|
|
77
|
+
4. Run the full verification sequence:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
yarn lint
|
|
81
|
+
yarn test
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
5. Fix any issues until both commands pass.
|
|
85
|
+
|
|
86
|
+
### Running Tests
|
|
87
|
+
|
|
88
|
+
Run the full test suite:
|
|
89
|
+
|
|
90
|
+
```bash
|
|
91
|
+
yarn test
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
Run a specific test file:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
npx jest test/rule.test.js --runInBand
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Run tests matching a pattern:
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
npx jest --testNamePattern="compress" --runInBand
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Tests use `mock-fs` for filesystem mocking and `nock` for HTTP request
|
|
107
|
+
interception. Test fixtures live in `test/resources/`.
|
|
108
|
+
|
|
109
|
+
### Linting
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
yarn lint
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
ESLint is configured in `.eslintrc.js` with the `airbnb-base` preset. The
|
|
116
|
+
`.eslintignore` file excludes non-source directories from linting.
|
|
117
|
+
|
|
118
|
+
## Common Tasks
|
|
119
|
+
|
|
120
|
+
### Adding a New Transformation
|
|
121
|
+
|
|
122
|
+
1. Create a new module in `src/transformations/` exporting a single async
|
|
123
|
+
function.
|
|
124
|
+
2. Register the transformation in `src/transformations/transform.js` — the
|
|
125
|
+
order in `transform.js` determines execution order (not configuration order).
|
|
126
|
+
3. Add the transformation name to the JSON schema in
|
|
127
|
+
`src/schemas/configuration.schema.json`.
|
|
128
|
+
4. Update `src/index.d.ts` with the new transformation name in the type union.
|
|
129
|
+
5. Create a test file in `test/transformations/`.
|
|
130
|
+
6. Update `CHANGELOG.md`.
|
|
131
|
+
7. Run `yarn lint && yarn test`.
|
|
132
|
+
|
|
133
|
+
### Updating the Configuration Schema
|
|
134
|
+
|
|
135
|
+
The configuration is validated by AJV against the JSON schema at
|
|
136
|
+
`src/schemas/configuration.schema.json`. When adding new fields:
|
|
137
|
+
|
|
138
|
+
1. Update the schema with the new property definition.
|
|
139
|
+
2. Update `src/index.d.ts` with the corresponding TypeScript type.
|
|
140
|
+
3. Add test cases in `test/configuration.test.js`.
|
|
141
|
+
|
|
142
|
+
## Troubleshooting
|
|
143
|
+
|
|
144
|
+
### Tests Hang or Time Out
|
|
145
|
+
|
|
146
|
+
Tests run with `--runInBand --detectOpenHandles`. If a test hangs, it usually
|
|
147
|
+
means an HTTP mock (`nock`) was not set up correctly or a filesystem mock
|
|
148
|
+
(`mock-fs`) was not restored. Check that `nock.cleanAll()` and
|
|
149
|
+
`mock.restore()` are called in `afterEach` or `afterAll`.
|
|
150
|
+
|
|
151
|
+
### ESLint Cache Issues
|
|
152
|
+
|
|
153
|
+
If linting gives unexpected results after changing `.eslintrc.js`, clear the
|
|
154
|
+
cache:
|
|
155
|
+
|
|
156
|
+
```bash
|
|
157
|
+
rm -f .eslintcache
|
|
158
|
+
yarn lint
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
### Network Errors When Running the Compiler
|
|
162
|
+
|
|
163
|
+
The compiler downloads remote filter lists. If you're behind a proxy or
|
|
164
|
+
firewall, sources may fail to download. Use local file paths in your
|
|
165
|
+
configuration for offline development.
|
|
166
|
+
|
|
167
|
+
## Additional Resources
|
|
168
|
+
|
|
169
|
+
- [README.md](README.md) — user documentation, configuration format, and
|
|
170
|
+
transformation reference
|
|
171
|
+
- [AGENTS.md](AGENTS.md) — code guidelines and contribution rules
|
|
172
|
+
- [CHANGELOG.md](CHANGELOG.md) — release history
|
package/README.md
CHANGED
|
@@ -9,18 +9,19 @@ This is a simple tool that makes it easier to compile a [hosts blocklist](https:
|
|
|
9
9
|
- [Command-line](#command-line)
|
|
10
10
|
- [API](#api)
|
|
11
11
|
- [Transformations](#transformations)
|
|
12
|
+
- [ConvertToAscii](#convert-to-ascii)
|
|
13
|
+
- [TrimLines](#trimlines)
|
|
12
14
|
- [RemoveComments](#remove-comments)
|
|
13
15
|
- [Compress](#compress)
|
|
14
16
|
- [RemoveModifiers](#remove-modifiers)
|
|
17
|
+
- [InvertAllow](#invertallow)
|
|
15
18
|
- [Validate](#validate)
|
|
16
19
|
- [ValidateAllowIp](#validate-allow-ip)
|
|
20
|
+
- [ValidateAllowPublicSuffix](#validate-allow-public-suffix)
|
|
17
21
|
- [Deduplicate](#deduplicate)
|
|
18
|
-
- [InvertAllow](#invertallow)
|
|
19
22
|
- [RemoveEmptyLines](#removeemptylines)
|
|
20
|
-
- [TrimLines](#trimlines)
|
|
21
23
|
- [InsertFinalNewLine](#insertfinalnewline)
|
|
22
|
-
|
|
23
|
-
- [How to build](#how-to-build)
|
|
24
|
+
- [Documentation](#documentation)
|
|
24
25
|
|
|
25
26
|
## <a name="usage"></a> Usage
|
|
26
27
|
|
|
@@ -162,8 +163,8 @@ Here is an example:
|
|
|
162
163
|
Rules in HOSTS syntax: `/hosts.txt`
|
|
163
164
|
|
|
164
165
|
```txt
|
|
165
|
-
0.0.0.0 ads.example.com
|
|
166
|
-
0.0.0.0 tracking.example1.com
|
|
166
|
+
0.0.0.0 ads.example.com
|
|
167
|
+
0.0.0.0 tracking.example1.com
|
|
167
168
|
0.0.0.0 example.com
|
|
168
169
|
```
|
|
169
170
|
|
|
@@ -195,7 +196,7 @@ Configuration of the final list:
|
|
|
195
196
|
Final filter output of `/hosts.txt` after applying the `Compress` transformation and exclusions:
|
|
196
197
|
|
|
197
198
|
```txt
|
|
198
|
-
||ads.example.com^
|
|
199
|
+
||ads.example.com^
|
|
199
200
|
||tracking.example1.com^
|
|
200
201
|
```
|
|
201
202
|
|
|
@@ -305,18 +306,66 @@ import { writeFileSync } from 'fs';
|
|
|
305
306
|
|
|
306
307
|
Here is the full list of transformations that are available:
|
|
307
308
|
|
|
309
|
+
1. `ConvertToAscii`
|
|
310
|
+
1. `TrimLines`
|
|
308
311
|
1. `RemoveComments`
|
|
309
312
|
1. `Compress`
|
|
310
313
|
1. `RemoveModifiers`
|
|
314
|
+
1. `InvertAllow`
|
|
311
315
|
1. `Validate`
|
|
312
316
|
1. `ValidateAllowIp`
|
|
317
|
+
1. `ValidateAllowPublicSuffix`
|
|
313
318
|
1. `Deduplicate`
|
|
314
|
-
1. `InvertAllow`
|
|
315
319
|
1. `RemoveEmptyLines`
|
|
316
|
-
1. `TrimLines`
|
|
317
320
|
1. `InsertFinalNewLine`
|
|
318
321
|
|
|
319
|
-
Please note that these transformations are
|
|
322
|
+
Please note that these transformations are always applied in the order specified here.
|
|
323
|
+
|
|
324
|
+
### <a name="convert-to-ascii"></a> ConvertToAscii
|
|
325
|
+
|
|
326
|
+
This transformation converts all non-ASCII characters to their ASCII equivalents. It is always performed first.
|
|
327
|
+
|
|
328
|
+
**Example:**
|
|
329
|
+
|
|
330
|
+
Original list:
|
|
331
|
+
|
|
332
|
+
```
|
|
333
|
+
||*.рус^
|
|
334
|
+
||*.कॉम^
|
|
335
|
+
||*.セール^
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
Here's what we will have after applying this transformation:
|
|
339
|
+
|
|
340
|
+
```
|
|
341
|
+
||*.xn--p1acf^
|
|
342
|
+
||*.xn--11b4c3d^
|
|
343
|
+
||*.xn--1qqw23a^
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
### <a name="trimlines"></a> TrimLines
|
|
347
|
+
|
|
348
|
+
This is a very simple transformation that removes leading and trailing spaces/tabs.
|
|
349
|
+
|
|
350
|
+
**Example:**
|
|
351
|
+
|
|
352
|
+
Original list:
|
|
353
|
+
|
|
354
|
+
```
|
|
355
|
+
rule1
|
|
356
|
+
rule2
|
|
357
|
+
rule3
|
|
358
|
+
rule4
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
Here's what we will have after applying this transformation:
|
|
362
|
+
|
|
363
|
+
```
|
|
364
|
+
rule1
|
|
365
|
+
rule2
|
|
366
|
+
rule3
|
|
367
|
+
rule4
|
|
368
|
+
```
|
|
320
369
|
|
|
321
370
|
### <a name="remove-comments"></a> RemoveComments
|
|
322
371
|
|
|
@@ -348,6 +397,39 @@ Here is the list of modifiers that will be removed:
|
|
|
348
397
|
> Blindly removing `$third-party` from traditional ad blocking rules leads to lots of false-positives.
|
|
349
398
|
>> This is exactly why there is an option to exclude rules - you may need to use it.
|
|
350
399
|
|
|
400
|
+
### <a name="invertallow"></a> InvertAllow
|
|
401
|
+
|
|
402
|
+
This transformation converts blocking rules to "allow" rules. Note, that it does nothing to /etc/hosts rules (unless they were previously converted to adblock-style syntax by a different transformation, for example [Compress](#compress)).
|
|
403
|
+
|
|
404
|
+
There are two important notes about this transformation:
|
|
405
|
+
|
|
406
|
+
1. It keeps the original rules order.
|
|
407
|
+
2. It ignores comments, empty lines, /etc/hosts rules and existing "allow" rules.
|
|
408
|
+
|
|
409
|
+
**Example:**
|
|
410
|
+
|
|
411
|
+
Original list:
|
|
412
|
+
|
|
413
|
+
```
|
|
414
|
+
! comment 1
|
|
415
|
+
rule1
|
|
416
|
+
|
|
417
|
+
# comment 2
|
|
418
|
+
192.168.11.11 test.local
|
|
419
|
+
@@rule2
|
|
420
|
+
```
|
|
421
|
+
|
|
422
|
+
Here's what we will have after applying this transformation:
|
|
423
|
+
|
|
424
|
+
```
|
|
425
|
+
! comment 1
|
|
426
|
+
@@rule1
|
|
427
|
+
|
|
428
|
+
# comment 2
|
|
429
|
+
192.168.11.11 test.local
|
|
430
|
+
@@rule2
|
|
431
|
+
```
|
|
432
|
+
|
|
351
433
|
### <a name="validate"></a> Validate
|
|
352
434
|
|
|
353
435
|
This transformation is really crucial if you're using a filter list for a traditional ad blocker as a source.
|
|
@@ -364,6 +446,7 @@ So here's what it does:
|
|
|
364
446
|
Examples:
|
|
365
447
|
- `||*.org^` - this rule will be removed
|
|
366
448
|
- `||*.org^$denyallow=example.com` - this rule will be kept because it has a limiting modifier
|
|
449
|
+
If such rules must be saved, use [ValidateAllowPublicSuffix](#validate-allow-public-suffix).
|
|
367
450
|
|
|
368
451
|
If there are comments preceding the invalid rule, they will be removed as well.
|
|
369
452
|
|
|
@@ -371,6 +454,16 @@ If there are comments preceding the invalid rule, they will be removed as well.
|
|
|
371
454
|
|
|
372
455
|
This transformation exactly repeats the behavior of [Validate](#validate), but leaves the IP addresses in the lists.
|
|
373
456
|
|
|
457
|
+
### <a name="validate-allow-public-suffix"></a> ValidateAllowPublicSuffix
|
|
458
|
+
|
|
459
|
+
This transformation exactly repeats the behavior of [Validate](#validate), but leaves rules that match whole public suffixes (e.g. `||hl.cn^`, `||org^`) in the list.
|
|
460
|
+
|
|
461
|
+
It still filters out invalid syntax rules and unsupported modifiers, but does not reject public-suffix rules unless the rule itself is malformed.
|
|
462
|
+
|
|
463
|
+
> **Note:** Combining any `Validate`, `ValidateAllowIp`, and `ValidateAllowPublicSuffix` in one transformation list is not allowed and will result in an error. Each runs its own validator on the already-filtered output of the previous one, so allow-modes become silently ineffective.
|
|
464
|
+
|
|
465
|
+
> **Important:** Validation transformations also cannot be used at both source-level and top-level simultaneously. For example, if a source uses `ValidateAllowPublicSuffix` and the top-level configuration uses `Validate`, the compiler will throw an error. This is because the top-level `Validate` would override the source-level validation, making `ValidateAllowPublicSuffix` ineffective. Use validation transformations at only one level.
|
|
466
|
+
|
|
374
467
|
### <a name="deduplicate"></a> Deduplicate
|
|
375
468
|
|
|
376
469
|
This transformation simply removes the duplicates from the specified source.
|
|
@@ -396,39 +489,6 @@ Here's what will be left after the transformation:
|
|
|
396
489
|
rule1
|
|
397
490
|
```
|
|
398
491
|
|
|
399
|
-
### <a name="invertallow"></a> InvertAllow
|
|
400
|
-
|
|
401
|
-
This transformation converts blocking rules to "allow" rules. Note, that it does nothing to /etc/hosts rules (unless they were previously converted to adblock-style syntax by a different transformation, for example [Compress](#compress)).
|
|
402
|
-
|
|
403
|
-
There are two important notes about this transformation:
|
|
404
|
-
|
|
405
|
-
1. It keeps the original rules order.
|
|
406
|
-
2. It ignores comments, empty lines, /etc/hosts rules and existing "allow" rules.
|
|
407
|
-
|
|
408
|
-
**Example:**
|
|
409
|
-
|
|
410
|
-
Original list:
|
|
411
|
-
|
|
412
|
-
```
|
|
413
|
-
! comment 1
|
|
414
|
-
rule1
|
|
415
|
-
|
|
416
|
-
# comment 2
|
|
417
|
-
192.168.11.11 test.local
|
|
418
|
-
@@rule2
|
|
419
|
-
```
|
|
420
|
-
|
|
421
|
-
Here's what we will have after applying this transformation:
|
|
422
|
-
|
|
423
|
-
```
|
|
424
|
-
! comment 1
|
|
425
|
-
@@rule1
|
|
426
|
-
|
|
427
|
-
# comment 2
|
|
428
|
-
192.168.11.11 test.local
|
|
429
|
-
@@rule2
|
|
430
|
-
```
|
|
431
|
-
|
|
432
492
|
### <a name="removeemptylines"></a> RemoveEmptyLines
|
|
433
493
|
|
|
434
494
|
This is a very simple transformation that removes empty lines.
|
|
@@ -454,30 +514,6 @@ rule2
|
|
|
454
514
|
rule3
|
|
455
515
|
```
|
|
456
516
|
|
|
457
|
-
### <a name="trimlines"></a> TrimLines
|
|
458
|
-
|
|
459
|
-
This is a very simple transformation that removes leading and trailing spaces/tabs.
|
|
460
|
-
|
|
461
|
-
**Example:**
|
|
462
|
-
|
|
463
|
-
Original list:
|
|
464
|
-
|
|
465
|
-
```
|
|
466
|
-
rule1
|
|
467
|
-
rule2
|
|
468
|
-
rule3
|
|
469
|
-
rule4
|
|
470
|
-
```
|
|
471
|
-
|
|
472
|
-
Here's what we will have after applying this transformation:
|
|
473
|
-
|
|
474
|
-
```
|
|
475
|
-
rule1
|
|
476
|
-
rule2
|
|
477
|
-
rule3
|
|
478
|
-
rule4
|
|
479
|
-
```
|
|
480
|
-
|
|
481
517
|
### <a name="insertfinalnewline"></a> InsertFinalNewLine
|
|
482
518
|
|
|
483
519
|
This is a very simple transformation that inserts a final newline.
|
|
@@ -503,31 +539,8 @@ rule3
|
|
|
503
539
|
|
|
504
540
|
`RemoveEmptyLines` doesn't delete this empty row due to the execution order.
|
|
505
541
|
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
This transformation converts all non-ASCII characters to their ASCII equivalents. It is always performed first.
|
|
509
|
-
|
|
510
|
-
**Example:**
|
|
511
|
-
|
|
512
|
-
Original list:
|
|
513
|
-
|
|
514
|
-
```
|
|
515
|
-
||*.рус^
|
|
516
|
-
||*.कॉम^
|
|
517
|
-
||*.セール^
|
|
518
|
-
```
|
|
519
|
-
|
|
520
|
-
Here's what we will have after applying this transformation:
|
|
521
|
-
|
|
522
|
-
```
|
|
523
|
-
||*.xn--p1acf^
|
|
524
|
-
||*.xn--11b4c3d^
|
|
525
|
-
||*.xn--1qqw23a^
|
|
526
|
-
```
|
|
527
|
-
|
|
528
|
-
## <a name="how-to-build"></a> How to build
|
|
542
|
+
## <a name="documentation"></a> Documentation
|
|
529
543
|
|
|
530
|
-
-
|
|
531
|
-
-
|
|
532
|
-
-
|
|
533
|
-
- `node src/cli.js -c examples/sdn/configuration.json -o filter.txt` - runs compiler with the example configuration
|
|
544
|
+
- [Development guide](DEVELOPMENT.md) — how to set up the environment, build, test, and contribute
|
|
545
|
+
- [LLM agent rules](AGENTS.md) — code guidelines and project conventions
|
|
546
|
+
- [Changelog](CHANGELOG.md) — release history
|
package/bamboo-specs/build.yaml
CHANGED
|
@@ -54,10 +54,10 @@ Build:
|
|
|
54
54
|
# Fix mixed logs
|
|
55
55
|
exec 2>&1
|
|
56
56
|
|
|
57
|
-
# Fail if version was not
|
|
57
|
+
# Fail if version was not bumped
|
|
58
58
|
if [ "$(git tag -l "v${bamboo.inject.version}")" ]; then
|
|
59
59
|
echo "Build failed!"
|
|
60
|
-
echo "Error: Tag v${bamboo.inject.version} already exists.
|
|
60
|
+
echo "Error: Tag v${bamboo.inject.version} already exists. Bump the version in package.json before building"
|
|
61
61
|
exit 1
|
|
62
62
|
fi
|
|
63
63
|
|
|
@@ -86,14 +86,6 @@ branches:
|
|
|
86
86
|
delete: never
|
|
87
87
|
link-to-jira: true
|
|
88
88
|
|
|
89
|
-
notifications:
|
|
90
|
-
- events:
|
|
91
|
-
- plan-status-changed
|
|
92
|
-
recipients:
|
|
93
|
-
- webhook:
|
|
94
|
-
name: Build webhook
|
|
95
|
-
url: http://prod.jirahub.service.eu.consul/v1/webhook/bamboo
|
|
96
|
-
|
|
97
89
|
labels: []
|
|
98
90
|
|
|
99
91
|
other:
|
package/bamboo-specs/deploy.yaml
CHANGED
|
@@ -36,10 +36,3 @@ npmjs:
|
|
|
36
36
|
npm publish hostlist-compiler.tgz --access public
|
|
37
37
|
requirements:
|
|
38
38
|
- adg-docker: true
|
|
39
|
-
notifications:
|
|
40
|
-
- events:
|
|
41
|
-
- deployment-started-and-finished
|
|
42
|
-
recipients:
|
|
43
|
-
- webhook:
|
|
44
|
-
name: Deploy webhook
|
|
45
|
-
url: http://prod.jirahub.service.eu.consul/v1/webhook/bamboo
|