@adguard/hostlist-compiler 1.0.34 → 1.0.37
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/README.md +72 -0
- package/package.json +3 -2
- package/src/compile-source.js +2 -3
- package/src/filter.js +2 -2
- package/src/rule.js +36 -0
- package/src/schemas/configuration.schema.json +4 -2
- package/src/transformations/covert-to-ascii.js +30 -0
- package/src/transformations/transform.js +5 -0
- package/src/transformations/validate.js +33 -26
- package/src/utils.js +0 -33
- package/test/compile-source.test.js +59 -10
- package/test/configuration.test.js +2 -0
- package/test/index.test.js +6 -2
- package/test/resolve-include.test.js +201 -0
- package/test/resources/includedfile.txt +2 -0
- package/test/resources/rules.txt +4 -0
- package/test/transformations/covert-to-ascii.test.js +30 -0
- package/test/transformations/exclude.test.js +12 -4
- package/test/transformations/include.test.js +12 -4
- package/test/transformations/transform.test.js +31 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,24 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [1.0.37] - 2025-02-28
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- Resolving the `!#include `directive by using `@adguard/filters-downloader`. [#78]
|
|
13
|
+
|
|
14
|
+
[1.0.37]: https://github.com/AdguardTeam/HostlistCompiler/compare/v1.0.35...v1.0.37
|
|
15
|
+
[#78]: https://github.com/AdguardTeam/HostlistCompiler/issues/78
|
|
16
|
+
|
|
17
|
+
## [1.0.35] - 2025-02-07
|
|
18
|
+
|
|
19
|
+
### Added
|
|
20
|
+
|
|
21
|
+
- `ConvertToAscii` transformation option [#62]
|
|
22
|
+
|
|
23
|
+
[1.0.35]: https://github.com/AdguardTeam/HostlistCompiler/compare/v1.0.34...v1.0.35
|
|
24
|
+
[#62]: https://github.com/AdguardTeam/HostlistCompiler/issues/62
|
|
25
|
+
|
|
8
26
|
## [1.0.34] - 2025-01-10
|
|
9
27
|
|
|
10
28
|
### Removed
|
package/README.md
CHANGED
|
@@ -19,6 +19,7 @@ This is a simple tool that makes it easier to compile a [hosts blocklist](https:
|
|
|
19
19
|
- [RemoveEmptyLines](#removeemptylines)
|
|
20
20
|
- [TrimLines](#trimlines)
|
|
21
21
|
- [InsertFinalNewLine](#insertfinalnewline)
|
|
22
|
+
- [ConvertToAscii](#convert-to-ascii)
|
|
22
23
|
- [How to build](#how-to-build)
|
|
23
24
|
|
|
24
25
|
## <a name="usage"></a> Usage
|
|
@@ -151,6 +152,55 @@ Please note, that exclusion or inclusion rules may be a plain string, wildcard,
|
|
|
151
152
|
- `/regex/` - every rule that matches this regular expression, will match the rule. By default, regular expressions are case-insensitive.
|
|
152
153
|
- `! comment` - comments will be ignored.
|
|
153
154
|
|
|
155
|
+
> [!IMPORTANT]
|
|
156
|
+
> Ensure that rules in the exclusion list match the format of the rules in the filter list.
|
|
157
|
+
> To maintain a consistent format, add the `Compress` transformation to convert `/etc/hosts` rules to adblock syntax.
|
|
158
|
+
> This is especially useful if you have multiple lists in different formats.
|
|
159
|
+
|
|
160
|
+
Here is an example:
|
|
161
|
+
|
|
162
|
+
Rules in HOSTS syntax: `/hosts.txt`
|
|
163
|
+
|
|
164
|
+
```txt
|
|
165
|
+
0.0.0.0 ads.example.com
|
|
166
|
+
0.0.0.0 tracking.example1.com
|
|
167
|
+
0.0.0.0 example.com
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
Exclusion rules in adblock syntax: `/exclusions.txt`
|
|
171
|
+
|
|
172
|
+
```txt
|
|
173
|
+
||example.com^
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
Configuration of the final list:
|
|
177
|
+
|
|
178
|
+
```json
|
|
179
|
+
{
|
|
180
|
+
"name": "List name",
|
|
181
|
+
"description": "List description",
|
|
182
|
+
"sources": [
|
|
183
|
+
{
|
|
184
|
+
"name": "HOSTS rules",
|
|
185
|
+
"source": "hosts.txt",
|
|
186
|
+
"type": "hosts",
|
|
187
|
+
"transformations": ["Compress"]
|
|
188
|
+
}
|
|
189
|
+
],
|
|
190
|
+
"transformations": ["Deduplicate", "Compress"],
|
|
191
|
+
"exclusions_sources": ["exclusions.txt"]
|
|
192
|
+
}
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
Final filter output of `/hosts.txt` after applying the `Compress` transformation and exclusions:
|
|
196
|
+
|
|
197
|
+
```txt
|
|
198
|
+
||ads.example.com^
|
|
199
|
+
||tracking.example1.com^
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
The last rule now `||example.com^` will correctly match the rule from the exclusion list and will be excluded.
|
|
203
|
+
|
|
154
204
|
### <a name="command-line"></a> Command-line
|
|
155
205
|
|
|
156
206
|
Command-line arguments.
|
|
@@ -453,6 +503,28 @@ rule3
|
|
|
453
503
|
|
|
454
504
|
`RemoveEmptyLines` doesn't delete this empty row due to the execution order.
|
|
455
505
|
|
|
506
|
+
### <a name="convert-to-ascii"></a> ConvertToAscii
|
|
507
|
+
|
|
508
|
+
This transformation converts all non-ASCII characters to their ASCII equivalents. It is always performed first.
|
|
509
|
+
|
|
510
|
+
**Example:**
|
|
511
|
+
|
|
512
|
+
Original list:
|
|
513
|
+
|
|
514
|
+
```
|
|
515
|
+
||*.рус^
|
|
516
|
+
||*.कॉम^
|
|
517
|
+
||*.セール^
|
|
518
|
+
```
|
|
519
|
+
|
|
520
|
+
Here's what we will have after applying this transformation:
|
|
521
|
+
|
|
522
|
+
```
|
|
523
|
+
||*.xn--p1acf^
|
|
524
|
+
||*.xn--11b4c3d^
|
|
525
|
+
||*.xn--1qqw23a^
|
|
526
|
+
```
|
|
527
|
+
|
|
456
528
|
## <a name="how-to-build"></a> How to build
|
|
457
529
|
|
|
458
530
|
- `yarn install` - installs dependencies
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@adguard/hostlist-compiler",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.37",
|
|
4
4
|
"description": "A simple tool that compiles hosts blocklists from multiple sources",
|
|
5
5
|
"main": "src/index.js",
|
|
6
6
|
"types": "src/index.d.ts",
|
|
@@ -23,11 +23,12 @@
|
|
|
23
23
|
"eslint": "^8.34.0",
|
|
24
24
|
"eslint-config-airbnb-base": "^15.0.0",
|
|
25
25
|
"eslint-plugin-import": "^2.27.5",
|
|
26
|
-
"jest": "^
|
|
26
|
+
"jest": "^29.7.0",
|
|
27
27
|
"mock-fs": "^4.11.0",
|
|
28
28
|
"nock": "^12.0.2"
|
|
29
29
|
},
|
|
30
30
|
"dependencies": {
|
|
31
|
+
"@adguard/filters-downloader": "^2.2.6",
|
|
31
32
|
"ajv": "^6.12.0",
|
|
32
33
|
"ajv-errors": "^1.0.1",
|
|
33
34
|
"axios": "^0.19.2",
|
package/src/compile-source.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
const consola = require('consola');
|
|
2
|
-
const
|
|
2
|
+
const { FiltersDownloader } = require('@adguard/filters-downloader');
|
|
3
3
|
const { transform } = require('./transformations/transform');
|
|
4
4
|
|
|
5
5
|
/**
|
|
@@ -21,8 +21,7 @@ const { transform } = require('./transformations/transform');
|
|
|
21
21
|
*/
|
|
22
22
|
async function compileSource(source) {
|
|
23
23
|
consola.info(`Start compiling ${source.source}`);
|
|
24
|
-
|
|
25
|
-
let rules = str.split(/\r?\n/);
|
|
24
|
+
let rules = await FiltersDownloader.download(source.source);
|
|
26
25
|
consola.info(`Original length is ${rules.length}`);
|
|
27
26
|
|
|
28
27
|
rules = await transform(rules, source, source.transformations);
|
package/src/filter.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
const _ = require('lodash');
|
|
2
|
+
const { FiltersDownloader } = require('@adguard/filters-downloader');
|
|
2
3
|
const utils = require('./utils');
|
|
3
4
|
const ruleUtils = require('./rule');
|
|
4
5
|
|
|
@@ -16,9 +17,8 @@ async function downloadAll(sources) {
|
|
|
16
17
|
}
|
|
17
18
|
|
|
18
19
|
await Promise.all(sources.map(async (s) => {
|
|
19
|
-
const rulesStr = await
|
|
20
|
+
const rulesStr = await FiltersDownloader.download(s);
|
|
20
21
|
const rules = rulesStr
|
|
21
|
-
.split(/\r?\n/)
|
|
22
22
|
.filter((el) => el.trim().length > 0 && !ruleUtils.isComment(el));
|
|
23
23
|
list = list.concat(rules);
|
|
24
24
|
}));
|
package/src/rule.js
CHANGED
|
@@ -1,9 +1,13 @@
|
|
|
1
1
|
const _ = require('lodash');
|
|
2
|
+
const { domainToASCII } = require('url');
|
|
2
3
|
const utils = require('./utils');
|
|
3
4
|
|
|
4
5
|
// eslint-disable-next-line max-len
|
|
5
6
|
const domainRegex = /^(?=.{1,255}$)[0-9A-Za-z](?:(?:[0-9A-Za-z]|-){0,61}[0-9A-Za-z])?(?:\.[0-9A-Za-z](?:(?:[0-9A-Za-z]|-){0,61}[0-9A-Za-z])?)*\.?$/;
|
|
6
7
|
const etcHostsRegex = /^([a-f0-9.:\][]+)(%[a-z0-9]+)?\s+([^#]+)(#.*)?$/;
|
|
8
|
+
const domainPatternRegex = /(\*\.|)([^\s^$|=]+(?:\.[^\s^$|=]+)+)/g;
|
|
9
|
+
// eslint-disable-next-line no-control-regex
|
|
10
|
+
const nonAsciiRegexp = /[^\x00-\x7F]/;
|
|
7
11
|
|
|
8
12
|
/**
|
|
9
13
|
* Helper utils for working with filtering rules
|
|
@@ -45,6 +49,36 @@ function isEtcHostsRule(ruleText) {
|
|
|
45
49
|
return etcHostsRegex.test(ruleText);
|
|
46
50
|
}
|
|
47
51
|
|
|
52
|
+
/**
|
|
53
|
+
* Check if the rule contains non-ASCII characters
|
|
54
|
+
* @param {string} ruleText - rule to check
|
|
55
|
+
* @returns {boolean} true if the rule contains non-latin characters
|
|
56
|
+
*/
|
|
57
|
+
function containsNonAsciiCharacters(ruleText) {
|
|
58
|
+
return nonAsciiRegexp.test(ruleText);
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
/**
|
|
62
|
+
* Function to convert all parts of the string that look like domain names with a possible wildcard (*.)
|
|
63
|
+
* and contain non-Latin characters to Punycode.
|
|
64
|
+
*
|
|
65
|
+
* @param {string} line The original string (adblock rule, regular domain, etc.)
|
|
66
|
+
* @returns {string} String with non-Latin domains converted to Punycode
|
|
67
|
+
*/
|
|
68
|
+
function convertNonAsciiToPunycode(line) {
|
|
69
|
+
return line.replace(domainPatternRegex, (match, wildcard, domain) => {
|
|
70
|
+
// Check if the domain contains non-ASCII characters
|
|
71
|
+
if (containsNonAsciiCharacters(domain)) {
|
|
72
|
+
// Convert only the domain (without '*.')
|
|
73
|
+
const punycodeDomain = domainToASCII(domain);
|
|
74
|
+
// Return with possible wildcard prefix '*.'
|
|
75
|
+
return wildcard + punycodeDomain;
|
|
76
|
+
}
|
|
77
|
+
// If there are no non-Latin characters, leave the original part
|
|
78
|
+
return match;
|
|
79
|
+
});
|
|
80
|
+
}
|
|
81
|
+
|
|
48
82
|
/**
|
|
49
83
|
* Represents a /etc/hosts rule
|
|
50
84
|
* @typedef {Object} AdblockRuleTokens
|
|
@@ -276,6 +310,8 @@ module.exports = {
|
|
|
276
310
|
isEtcHostsRule,
|
|
277
311
|
loadEtcHostsRuleProperties,
|
|
278
312
|
loadAdblockRuleProperties,
|
|
313
|
+
containsNonAsciiCharacters,
|
|
314
|
+
convertNonAsciiToPunycode,
|
|
279
315
|
findModifier,
|
|
280
316
|
removeModifier,
|
|
281
317
|
adblockRuleToString,
|
|
@@ -49,7 +49,8 @@
|
|
|
49
49
|
"InvertAllow",
|
|
50
50
|
"RemoveEmptyLines",
|
|
51
51
|
"TrimLines",
|
|
52
|
-
"InsertFinalNewLine"
|
|
52
|
+
"InsertFinalNewLine",
|
|
53
|
+
"ConvertToAscii"
|
|
53
54
|
]
|
|
54
55
|
}
|
|
55
56
|
},
|
|
@@ -125,7 +126,8 @@
|
|
|
125
126
|
"InvertAllow",
|
|
126
127
|
"RemoveEmptyLines",
|
|
127
128
|
"TrimLines",
|
|
128
|
-
"InsertFinalNewLine"
|
|
129
|
+
"InsertFinalNewLine",
|
|
130
|
+
"ConvertToAscii"
|
|
129
131
|
]
|
|
130
132
|
}
|
|
131
133
|
},
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
const consola = require('consola');
|
|
2
|
+
const ruleUtils = require('../rule');
|
|
3
|
+
|
|
4
|
+
/**
|
|
5
|
+
* Converts non-ASCII characters in the given rules to their ASCII punycode equivalents
|
|
6
|
+
*
|
|
7
|
+
* @param {Array<string>} rules - An array of rules to be converted.
|
|
8
|
+
* @returns {Array<string>} - The array of rules with non-ASCII characters converted to punycode.
|
|
9
|
+
*/
|
|
10
|
+
function convertToAscii(rules) {
|
|
11
|
+
return rules.map((rule) => {
|
|
12
|
+
// 1. Check if rule is comment or empty
|
|
13
|
+
if (ruleUtils.isComment(rule) || rule.length === 0) {
|
|
14
|
+
return rule;
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
// 2. Check if rule contains non ascii characters
|
|
18
|
+
if (!ruleUtils.containsNonAsciiCharacters(rule)) {
|
|
19
|
+
return rule;
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
// 3. Replace the rule with its punycode version
|
|
23
|
+
const punycodeRule = ruleUtils.convertNonAsciiToPunycode(rule);
|
|
24
|
+
consola.debug(`Converting non-ASCII line ${rule} to punycode ${punycodeRule}`);
|
|
25
|
+
|
|
26
|
+
return punycodeRule;
|
|
27
|
+
});
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
module.exports = convertToAscii;
|
|
@@ -10,6 +10,7 @@ const invertAllow = require('./invertallow');
|
|
|
10
10
|
const removeEmptyLines = require('./remove-empty-lines');
|
|
11
11
|
const trimLines = require('./trim-lines');
|
|
12
12
|
const insertFinalNewLine = require('./insert-final-newline');
|
|
13
|
+
const convertToAscii = require('./covert-to-ascii');
|
|
13
14
|
|
|
14
15
|
/**
|
|
15
16
|
* Enum with all available transformations
|
|
@@ -25,6 +26,7 @@ const TRANSFORMATIONS = Object.freeze({
|
|
|
25
26
|
RemoveEmptyLines: 'RemoveEmptyLines',
|
|
26
27
|
TrimLines: 'TrimLines',
|
|
27
28
|
InsertFinalNewLine: 'InsertFinalNewLine',
|
|
29
|
+
ConvertToAscii: 'ConvertToAscii',
|
|
28
30
|
});
|
|
29
31
|
|
|
30
32
|
/**
|
|
@@ -55,6 +57,9 @@ async function transform(rules, configuration, transformations) {
|
|
|
55
57
|
configuration.inclusions_sources,
|
|
56
58
|
);
|
|
57
59
|
|
|
60
|
+
if (transformations.indexOf(TRANSFORMATIONS.ConvertToAscii) !== -1) {
|
|
61
|
+
transformed = convertToAscii(transformed);
|
|
62
|
+
}
|
|
58
63
|
if (transformations.indexOf(TRANSFORMATIONS.TrimLines) !== -1) {
|
|
59
64
|
transformed = trimLines(transformed);
|
|
60
65
|
}
|
|
@@ -248,51 +248,58 @@ function valid(ruleText, allowedIP) {
|
|
|
248
248
|
* Class representing a validator for a list of rules.
|
|
249
249
|
*/
|
|
250
250
|
class Validator {
|
|
251
|
+
/**
|
|
252
|
+
* Creates a new rule validator.
|
|
253
|
+
* @param {boolean} allowedIP - Flag indicating whether IP addresses should be considered valid.
|
|
254
|
+
*/
|
|
251
255
|
constructor(allowedIP) {
|
|
252
256
|
/**
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
+
* Indicates that a rule was previously removed (the iteration processed an invalid rule).
|
|
258
|
+
* Used to remove preceding comments or empty lines.
|
|
259
|
+
* @type {boolean}
|
|
260
|
+
*/
|
|
261
|
+
this.previousRuleRemoved = false;
|
|
262
|
+
|
|
257
263
|
/**
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
264
|
+
* Flag to allow or disallow IP addresses during validation.
|
|
265
|
+
* @type {boolean}
|
|
266
|
+
*/
|
|
261
267
|
this.allowedIP = allowedIP;
|
|
262
268
|
}
|
|
263
269
|
|
|
264
270
|
/**
|
|
265
|
-
* Validates
|
|
271
|
+
* Validates a list of rules and removes invalid rules.
|
|
266
272
|
* If a rule is invalid, any preceding comments or empty lines are also removed.
|
|
267
273
|
*
|
|
268
|
-
* @
|
|
269
|
-
* @
|
|
274
|
+
* @param {string[]} rules - An array of strings (rules) to validate.
|
|
275
|
+
* @returns {string[]} A filtered list of valid rules.
|
|
270
276
|
*/
|
|
271
277
|
validate(rules) {
|
|
272
|
-
/**
|
|
273
|
-
* A filtered list of rules after validation.
|
|
274
|
-
* @type {Array<string>}
|
|
275
|
-
*/
|
|
276
278
|
const filtered = [...rules];
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
//
|
|
285
|
-
|
|
286
|
-
filtered.splice(
|
|
279
|
+
// Iterate from the end to the beginning so we can remove
|
|
280
|
+
// preceding comments/empty lines if needed
|
|
281
|
+
for (let i = filtered.length - 1; i >= 0; i -= 1) {
|
|
282
|
+
const isValidRule = valid(filtered[i], this.allowedIP);
|
|
283
|
+
const isCommentOrEmptyLine = ruleUtils.isComment(filtered[i]) || _.isEmpty(filtered[i]);
|
|
284
|
+
|
|
285
|
+
if (!isValidRule) {
|
|
286
|
+
// If the rule is invalid, remove it and set the flag to remove preceding lines
|
|
287
|
+
this.previousRuleRemoved = true;
|
|
288
|
+
filtered.splice(i, 1);
|
|
289
|
+
} else if (this.previousRuleRemoved && isCommentOrEmptyLine) {
|
|
290
|
+
// If a previous invalid rule was removed, remove this comment or empty line as well
|
|
291
|
+
consola.debug(`Removing a comment or empty line preceding an invalid rule: ${filtered[i]}`);
|
|
292
|
+
filtered.splice(i, 1);
|
|
287
293
|
} else {
|
|
288
|
-
//
|
|
289
|
-
this.
|
|
294
|
+
// If it's valid and doesn't match the "remove preceding" scenario, reset the removal flag
|
|
295
|
+
this.previousRuleRemoved = false;
|
|
290
296
|
}
|
|
291
297
|
}
|
|
292
298
|
|
|
293
299
|
return filtered;
|
|
294
300
|
}
|
|
295
301
|
}
|
|
302
|
+
|
|
296
303
|
/**
|
|
297
304
|
* Validates a list of rules.
|
|
298
305
|
*
|
package/src/utils.js
CHANGED
|
@@ -1,36 +1,4 @@
|
|
|
1
1
|
const _ = require('lodash');
|
|
2
|
-
const fs = require('fs').promises;
|
|
3
|
-
const axios = require('axios');
|
|
4
|
-
|
|
5
|
-
function isURL(str) {
|
|
6
|
-
try {
|
|
7
|
-
// eslint-disable-next-line no-unused-vars
|
|
8
|
-
const u = new URL(str);
|
|
9
|
-
return true;
|
|
10
|
-
} catch {
|
|
11
|
-
return false;
|
|
12
|
-
}
|
|
13
|
-
}
|
|
14
|
-
|
|
15
|
-
/**
|
|
16
|
-
* Downloads (or reads from the disk) the specified source
|
|
17
|
-
*
|
|
18
|
-
* @param {*} urlOrPath url or path to a file
|
|
19
|
-
* @returns {Promise<String>} contents of the files
|
|
20
|
-
*/
|
|
21
|
-
async function download(urlOrPath) {
|
|
22
|
-
let str = '';
|
|
23
|
-
|
|
24
|
-
if (isURL(urlOrPath)) {
|
|
25
|
-
const url = new URL(urlOrPath);
|
|
26
|
-
const response = await axios.get(url.toString(), { responseType: 'text' });
|
|
27
|
-
str = response.data;
|
|
28
|
-
} else {
|
|
29
|
-
str = (await fs.readFile(urlOrPath)).toString();
|
|
30
|
-
}
|
|
31
|
-
|
|
32
|
-
return str;
|
|
33
|
-
}
|
|
34
2
|
|
|
35
3
|
/**
|
|
36
4
|
* Extracts a substring between two tags.
|
|
@@ -169,7 +137,6 @@ class Wildcard {
|
|
|
169
137
|
}
|
|
170
138
|
|
|
171
139
|
module.exports = {
|
|
172
|
-
download,
|
|
173
140
|
Wildcard,
|
|
174
141
|
splitByDelimiterWithEscapeCharacter,
|
|
175
142
|
substringBetween,
|
|
@@ -1,8 +1,13 @@
|
|
|
1
1
|
const nock = require('nock');
|
|
2
2
|
const mock = require('mock-fs');
|
|
3
|
+
const path = require('path');
|
|
3
4
|
const compileSource = require('../src/compile-source');
|
|
4
5
|
const { TRANSFORMATIONS } = require('../src/transformations/transform');
|
|
5
6
|
|
|
7
|
+
const testDirPath = path.resolve(__dirname, 'test/dir');
|
|
8
|
+
const rulesFilePath = path.resolve(testDirPath, 'rules.txt');
|
|
9
|
+
const exclusionsFilePath = path.resolve(testDirPath, 'exclusions.txt');
|
|
10
|
+
|
|
6
11
|
describe('Source compiler', () => {
|
|
7
12
|
afterEach(() => {
|
|
8
13
|
// make sure FS is restored after running tests
|
|
@@ -12,7 +17,9 @@ describe('Source compiler', () => {
|
|
|
12
17
|
it('compile a simple URL source', async () => {
|
|
13
18
|
const scope = nock('https://example.org')
|
|
14
19
|
.get('/filter.txt')
|
|
15
|
-
.reply(200, 'testrule'
|
|
20
|
+
.reply(200, 'testrule', {
|
|
21
|
+
'Content-Type': 'text/plain',
|
|
22
|
+
});
|
|
16
23
|
|
|
17
24
|
const source = {
|
|
18
25
|
name: 'test source',
|
|
@@ -29,13 +36,13 @@ describe('Source compiler', () => {
|
|
|
29
36
|
|
|
30
37
|
it('compile a simple file source', async () => {
|
|
31
38
|
mock({
|
|
32
|
-
|
|
39
|
+
[testDirPath]: {
|
|
33
40
|
'rules.txt': 'testrule',
|
|
34
41
|
},
|
|
35
42
|
});
|
|
36
43
|
const source = {
|
|
37
44
|
name: 'test source',
|
|
38
|
-
source:
|
|
45
|
+
source: rulesFilePath,
|
|
39
46
|
};
|
|
40
47
|
|
|
41
48
|
const rules = await compileSource(source);
|
|
@@ -53,13 +60,15 @@ describe('Source compiler', () => {
|
|
|
53
60
|
||rule4`;
|
|
54
61
|
const scope = nock('https://example.org')
|
|
55
62
|
.get('/filter.txt')
|
|
56
|
-
.reply(200, rules
|
|
63
|
+
.reply(200, rules, {
|
|
64
|
+
'Content-Type': 'text/plain',
|
|
65
|
+
});
|
|
57
66
|
|
|
58
67
|
// STEP 2: MOCK EXCLUSIONS
|
|
59
68
|
const exclusions = `||rule1
|
|
60
69
|
||rule3`;
|
|
61
70
|
mock({
|
|
62
|
-
|
|
71
|
+
[testDirPath]: {
|
|
63
72
|
'exclusions.txt': exclusions,
|
|
64
73
|
},
|
|
65
74
|
});
|
|
@@ -68,7 +77,7 @@ describe('Source compiler', () => {
|
|
|
68
77
|
name: 'test source',
|
|
69
78
|
source: 'https://example.org/filter.txt',
|
|
70
79
|
exclusions: ['rule4'],
|
|
71
|
-
exclusions_sources: [
|
|
80
|
+
exclusions_sources: [exclusionsFilePath],
|
|
72
81
|
transformations: [
|
|
73
82
|
TRANSFORMATIONS.Deduplicate,
|
|
74
83
|
TRANSFORMATIONS.Validate,
|
|
@@ -88,18 +97,20 @@ describe('Source compiler', () => {
|
|
|
88
97
|
|
|
89
98
|
it('compile the source with transformations', async () => {
|
|
90
99
|
const testList = `
|
|
91
|
-
|
|
100
|
+
|
|
92
101
|
example.org
|
|
93
102
|
|
|
94
|
-
test1.com
|
|
103
|
+
test1.com
|
|
95
104
|
|
|
96
|
-
! comment
|
|
105
|
+
! comment
|
|
97
106
|
test1.com
|
|
98
107
|
|
|
99
108
|
test2.com`;
|
|
100
109
|
const scope = nock('https://example.org')
|
|
101
110
|
.get('/test-filter.txt')
|
|
102
|
-
.reply(200, testList
|
|
111
|
+
.reply(200, testList, {
|
|
112
|
+
'Content-Type': 'text/plain',
|
|
113
|
+
});
|
|
103
114
|
|
|
104
115
|
const source = {
|
|
105
116
|
name: 'test source',
|
|
@@ -121,4 +132,42 @@ describe('Source compiler', () => {
|
|
|
121
132
|
]);
|
|
122
133
|
scope.done();
|
|
123
134
|
});
|
|
135
|
+
|
|
136
|
+
it('compile the source with converToAscii transformation', async () => {
|
|
137
|
+
const testList = `
|
|
138
|
+
! comment
|
|
139
|
+
||*.ком^
|
|
140
|
+
||*.ком^
|
|
141
|
+
||*.укр^
|
|
142
|
+
||*.мон^
|
|
143
|
+
||*.ευ^
|
|
144
|
+
||*.ελ^
|
|
145
|
+
||*.հայ^`;
|
|
146
|
+
const scope = nock('https://example.org')
|
|
147
|
+
.get('/test-filter.txt')
|
|
148
|
+
.reply(200, testList, {
|
|
149
|
+
'Content-Type': 'text/plain',
|
|
150
|
+
});
|
|
151
|
+
|
|
152
|
+
const source = {
|
|
153
|
+
name: 'test source',
|
|
154
|
+
source: 'https://example.org/test-filter.txt',
|
|
155
|
+
transformations: [
|
|
156
|
+
TRANSFORMATIONS.ConvertToAscii,
|
|
157
|
+
TRANSFORMATIONS.Deduplicate,
|
|
158
|
+
TRANSFORMATIONS.RemoveComments,
|
|
159
|
+
TRANSFORMATIONS.TrimLines,
|
|
160
|
+
],
|
|
161
|
+
};
|
|
162
|
+
const compiled = await compileSource(source);
|
|
163
|
+
expect(compiled).toEqual([
|
|
164
|
+
'||*.xn--j1aef^',
|
|
165
|
+
'||*.xn--j1amh^',
|
|
166
|
+
'||*.xn--l1acc^',
|
|
167
|
+
'||*.xn--qxa6a^',
|
|
168
|
+
'||*.xn--qxam^',
|
|
169
|
+
'||*.xn--y9a3aq^',
|
|
170
|
+
]);
|
|
171
|
+
scope.done();
|
|
172
|
+
});
|
|
124
173
|
});
|
|
@@ -35,6 +35,7 @@ describe('Configuration', () => {
|
|
|
35
35
|
'Validate',
|
|
36
36
|
'Deduplicate',
|
|
37
37
|
'InvertAllow',
|
|
38
|
+
'ConvertToAscii',
|
|
38
39
|
],
|
|
39
40
|
},
|
|
40
41
|
],
|
|
@@ -45,6 +46,7 @@ describe('Configuration', () => {
|
|
|
45
46
|
'Validate',
|
|
46
47
|
'Deduplicate',
|
|
47
48
|
'InvertAllow',
|
|
49
|
+
'ConvertToAscii',
|
|
48
50
|
],
|
|
49
51
|
});
|
|
50
52
|
expect(ret.valid).toBe(true);
|
package/test/index.test.js
CHANGED
|
@@ -7,9 +7,13 @@ describe('Hostlist compiler', () => {
|
|
|
7
7
|
// Prepare source 1
|
|
8
8
|
const scope = nock('https://example.org')
|
|
9
9
|
.get('/source1.txt')
|
|
10
|
-
.reply(200, '||example.org'
|
|
10
|
+
.reply(200, '||example.org', {
|
|
11
|
+
'Content-Type': 'text/plain',
|
|
12
|
+
})
|
|
11
13
|
.get('/source2.txt')
|
|
12
|
-
.reply(200, '||example.com'
|
|
14
|
+
.reply(200, '||example.com', {
|
|
15
|
+
'Content-Type': 'text/plain',
|
|
16
|
+
});
|
|
13
17
|
|
|
14
18
|
// compiler configuration
|
|
15
19
|
const configuration = {
|
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
const nock = require('nock');
|
|
2
|
+
const consola = require('consola');
|
|
3
|
+
const path = require('path');
|
|
4
|
+
const compile = require('../src/index');
|
|
5
|
+
|
|
6
|
+
const testDirPath = path.resolve(__dirname, 'resources');
|
|
7
|
+
const filterFilePath = path.resolve(testDirPath, 'rules.txt');
|
|
8
|
+
|
|
9
|
+
describe('Hostlist compiler', () => {
|
|
10
|
+
it('compile from one source with nested includes', async () => {
|
|
11
|
+
// Prepare filters content
|
|
12
|
+
const filterContent1 = `! this is a source
|
|
13
|
+
||example.org
|
|
14
|
+
||example.com
|
|
15
|
+
!#include https://example.org/source2.txt`;
|
|
16
|
+
|
|
17
|
+
const filterContent2 = `
|
|
18
|
+
||example.net
|
|
19
|
+
||example.io
|
|
20
|
+
!#include https://example.org/source3.txt`;
|
|
21
|
+
|
|
22
|
+
const filterContent3 = `
|
|
23
|
+
last.include.com
|
|
24
|
+
non/valid_rule`;
|
|
25
|
+
|
|
26
|
+
// Prepare source
|
|
27
|
+
const scope = nock('https://example.org')
|
|
28
|
+
.get('/source1.txt')
|
|
29
|
+
.reply(200, filterContent1, {
|
|
30
|
+
'Content-Type': 'text/plain',
|
|
31
|
+
})
|
|
32
|
+
.get('/source2.txt')
|
|
33
|
+
.reply(200, filterContent2, {
|
|
34
|
+
'Content-Type': 'text/plain',
|
|
35
|
+
})
|
|
36
|
+
.get('/source3.txt')
|
|
37
|
+
.reply(200, filterContent3, {
|
|
38
|
+
'Content-Type': 'text/plain',
|
|
39
|
+
});
|
|
40
|
+
|
|
41
|
+
// compiler configuration
|
|
42
|
+
const configuration = {
|
|
43
|
+
name: 'Test filter',
|
|
44
|
+
description: 'Our test filter',
|
|
45
|
+
version: '1.0.0.9',
|
|
46
|
+
sources: [
|
|
47
|
+
{
|
|
48
|
+
name: 'source 1',
|
|
49
|
+
source: 'https://example.org/source1.txt',
|
|
50
|
+
},
|
|
51
|
+
],
|
|
52
|
+
transformations: [
|
|
53
|
+
'RemoveComments',
|
|
54
|
+
'Compress',
|
|
55
|
+
'InsertFinalNewLine',
|
|
56
|
+
'Validate',
|
|
57
|
+
],
|
|
58
|
+
};
|
|
59
|
+
|
|
60
|
+
// compile the final list
|
|
61
|
+
const list = await compile(configuration);
|
|
62
|
+
|
|
63
|
+
// assert
|
|
64
|
+
const expectedRules = [
|
|
65
|
+
'||example.org',
|
|
66
|
+
'||example.com',
|
|
67
|
+
'||example.net',
|
|
68
|
+
'||example.io',
|
|
69
|
+
'||last.include.com^',
|
|
70
|
+
];
|
|
71
|
+
|
|
72
|
+
expectedRules.forEach((rule) => {
|
|
73
|
+
expect(list).toContain(rule);
|
|
74
|
+
});
|
|
75
|
+
|
|
76
|
+
expect(list).not.toContain('non/valid_rule');
|
|
77
|
+
|
|
78
|
+
expect(list).toContain('! Version: 1.0.0.9');
|
|
79
|
+
|
|
80
|
+
const str = list.join('\n');
|
|
81
|
+
consola.info(str);
|
|
82
|
+
|
|
83
|
+
scope.done();
|
|
84
|
+
});
|
|
85
|
+
|
|
86
|
+
it('compile from one source with include and exclude file', async () => {
|
|
87
|
+
// Prepare filters content
|
|
88
|
+
const filterContent1 = `! this is a source
|
|
89
|
+
ads.example.com
|
|
90
|
+
trackers.exclude.com
|
|
91
|
+
exclude.com
|
|
92
|
+
!#include https://example.org/source2.txt`;
|
|
93
|
+
|
|
94
|
+
const filterContent2 = `
|
|
95
|
+
popups.com
|
|
96
|
+
event.com`;
|
|
97
|
+
|
|
98
|
+
const exclusionsContent = `
|
|
99
|
+
exclude.com
|
|
100
|
+
non/valid_rule`;
|
|
101
|
+
|
|
102
|
+
// Prepare source
|
|
103
|
+
const scope = nock('https://example.org')
|
|
104
|
+
.get('/source1.txt')
|
|
105
|
+
.reply(200, filterContent1, {
|
|
106
|
+
'Content-Type': 'text/plain',
|
|
107
|
+
})
|
|
108
|
+
.get('/source2.txt')
|
|
109
|
+
.reply(200, filterContent2, {
|
|
110
|
+
'Content-Type': 'text/plain',
|
|
111
|
+
})
|
|
112
|
+
.get('/exclusions.txt')
|
|
113
|
+
.reply(200, exclusionsContent, {
|
|
114
|
+
'Content-Type': 'text/plain',
|
|
115
|
+
});
|
|
116
|
+
|
|
117
|
+
// compiler configuration
|
|
118
|
+
const configuration = {
|
|
119
|
+
name: 'Test filter',
|
|
120
|
+
description: 'Our test filter',
|
|
121
|
+
version: '1.0.0.9',
|
|
122
|
+
sources: [
|
|
123
|
+
{
|
|
124
|
+
name: 'source 1',
|
|
125
|
+
source: 'https://example.org/source1.txt',
|
|
126
|
+
},
|
|
127
|
+
],
|
|
128
|
+
transformations: [
|
|
129
|
+
'RemoveComments',
|
|
130
|
+
'Compress',
|
|
131
|
+
'InsertFinalNewLine',
|
|
132
|
+
'Validate',
|
|
133
|
+
],
|
|
134
|
+
exclusions_sources: [
|
|
135
|
+
'https://example.org/exclusions.txt',
|
|
136
|
+
],
|
|
137
|
+
};
|
|
138
|
+
|
|
139
|
+
// compile the final list
|
|
140
|
+
const list = await compile(configuration);
|
|
141
|
+
|
|
142
|
+
// assert
|
|
143
|
+
const expectedRules = [
|
|
144
|
+
'||ads.example.com^',
|
|
145
|
+
'||popups.com^',
|
|
146
|
+
'||event.com^',
|
|
147
|
+
];
|
|
148
|
+
|
|
149
|
+
expectedRules.forEach((rule) => {
|
|
150
|
+
expect(list).toContain(rule);
|
|
151
|
+
});
|
|
152
|
+
|
|
153
|
+
expect(list).not.toContain('||exclude.com^', 'non/valid_rule');
|
|
154
|
+
|
|
155
|
+
expect(list).toContain('! Version: 1.0.0.9');
|
|
156
|
+
|
|
157
|
+
const str = list.join('\n');
|
|
158
|
+
consola.info(str);
|
|
159
|
+
|
|
160
|
+
scope.done();
|
|
161
|
+
});
|
|
162
|
+
|
|
163
|
+
it('compile from local source with local include', async () => {
|
|
164
|
+
// compiler configuration
|
|
165
|
+
const configuration = {
|
|
166
|
+
name: 'Test filter',
|
|
167
|
+
description: 'Our test filter',
|
|
168
|
+
version: '1.0.0.9',
|
|
169
|
+
sources: [
|
|
170
|
+
{
|
|
171
|
+
name: 'filter',
|
|
172
|
+
source: filterFilePath,
|
|
173
|
+
},
|
|
174
|
+
],
|
|
175
|
+
transformations: [
|
|
176
|
+
'RemoveComments',
|
|
177
|
+
'Compress',
|
|
178
|
+
'InsertFinalNewLine',
|
|
179
|
+
'Validate',
|
|
180
|
+
],
|
|
181
|
+
};
|
|
182
|
+
|
|
183
|
+
// compile the final list
|
|
184
|
+
const list = await compile(configuration);
|
|
185
|
+
|
|
186
|
+
const str = list.join('\n');
|
|
187
|
+
consola.info(str);
|
|
188
|
+
|
|
189
|
+
const expectedRules = [
|
|
190
|
+
'||sat.terithrow2.net^',
|
|
191
|
+
'||sat.fevilsor5.net^',
|
|
192
|
+
'||leegreemula.net^',
|
|
193
|
+
'||should.be.included.com^',
|
|
194
|
+
'||last.rule^',
|
|
195
|
+
];
|
|
196
|
+
|
|
197
|
+
expectedRules.forEach((rule) => {
|
|
198
|
+
expect(list).toContain(rule);
|
|
199
|
+
});
|
|
200
|
+
});
|
|
201
|
+
});
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
const convertToAscii = require('../../src/transformations/covert-to-ascii');
|
|
2
|
+
|
|
3
|
+
describe('Exclusions', () => {
|
|
4
|
+
it('convert non latin symbols in full domain to punycode', () => {
|
|
5
|
+
const rules = `||*.рус^$denyallow=example.com
|
|
6
|
+
||*.कॉम^$denyallow=example.com
|
|
7
|
+
||*.セール^$denyallow=example.com`.split(/\r?\n/);
|
|
8
|
+
const filtered = convertToAscii(rules);
|
|
9
|
+
|
|
10
|
+
expect(filtered).toEqual([
|
|
11
|
+
'||*.xn--p1acf^$denyallow=example.com',
|
|
12
|
+
'||*.xn--11b4c3d^$denyallow=example.com',
|
|
13
|
+
'||*.xn--1ck2e1b^$denyallow=example.com',
|
|
14
|
+
]);
|
|
15
|
+
});
|
|
16
|
+
|
|
17
|
+
it('convert non latin symbols in modifiers to punycode', () => {
|
|
18
|
+
const rules = `||example.com^$denyallow=*.рф
|
|
19
|
+
||example.com^$denyallow=пример.рф
|
|
20
|
+
||example.com^$client=пример.рф`.split(/\r?\n/);
|
|
21
|
+
|
|
22
|
+
const filtered = convertToAscii(rules);
|
|
23
|
+
|
|
24
|
+
expect(filtered).toEqual([
|
|
25
|
+
'||example.com^$denyallow=*.xn--p1ai',
|
|
26
|
+
'||example.com^$denyallow=xn--e1afmkfd.xn--p1ai',
|
|
27
|
+
'||example.com^$client=xn--e1afmkfd.xn--p1ai',
|
|
28
|
+
]);
|
|
29
|
+
});
|
|
30
|
+
});
|
|
@@ -1,7 +1,11 @@
|
|
|
1
1
|
const nock = require('nock');
|
|
2
2
|
const mock = require('mock-fs');
|
|
3
|
+
const path = require('path');
|
|
3
4
|
const exclude = require('../../src/transformations/exclude');
|
|
4
5
|
|
|
6
|
+
const testDirPath = path.resolve(__dirname, 'test/dir');
|
|
7
|
+
const exclusionsFilePath = path.resolve(testDirPath, 'exclusions.txt');
|
|
8
|
+
|
|
5
9
|
describe('Exclusions', () => {
|
|
6
10
|
afterEach(() => {
|
|
7
11
|
// make sure FS is restored after running tests
|
|
@@ -20,11 +24,15 @@ describe('Exclusions', () => {
|
|
|
20
24
|
// Mock exclusions
|
|
21
25
|
const scope = nock('https://example.org')
|
|
22
26
|
.get('/exclusions.txt')
|
|
23
|
-
.reply(200, 'rule1'
|
|
27
|
+
.reply(200, 'rule1', {
|
|
28
|
+
'Content-Type': 'text/plain',
|
|
29
|
+
})
|
|
24
30
|
.get('/exclusions2.txt')
|
|
25
|
-
.reply(200, 'rule2'
|
|
31
|
+
.reply(200, 'rule2', {
|
|
32
|
+
'Content-Type': 'text/plain',
|
|
33
|
+
});
|
|
26
34
|
mock({
|
|
27
|
-
|
|
35
|
+
[testDirPath]: {
|
|
28
36
|
'exclusions.txt': 'rule3',
|
|
29
37
|
},
|
|
30
38
|
});
|
|
@@ -36,7 +44,7 @@ describe('Exclusions', () => {
|
|
|
36
44
|
const filtered = await exclude(
|
|
37
45
|
rules,
|
|
38
46
|
['rule4'],
|
|
39
|
-
['https://example.org/exclusions.txt', 'https://example.org/exclusions2.txt',
|
|
47
|
+
['https://example.org/exclusions.txt', 'https://example.org/exclusions2.txt', exclusionsFilePath],
|
|
40
48
|
);
|
|
41
49
|
|
|
42
50
|
// Assert
|
|
@@ -1,7 +1,11 @@
|
|
|
1
1
|
const nock = require('nock');
|
|
2
2
|
const mock = require('mock-fs');
|
|
3
|
+
const path = require('path');
|
|
3
4
|
const include = require('../../src/transformations/include');
|
|
4
5
|
|
|
6
|
+
const testDirPath = path.resolve(__dirname, 'test/dir');
|
|
7
|
+
const inclusionsFilePath = path.resolve(testDirPath, 'inclusions.txt');
|
|
8
|
+
|
|
5
9
|
describe('Exclusions', () => {
|
|
6
10
|
afterEach(() => {
|
|
7
11
|
// make sure FS is restored after running tests
|
|
@@ -39,11 +43,15 @@ describe('Exclusions', () => {
|
|
|
39
43
|
// Mock inclusions sources
|
|
40
44
|
const scope = nock('https://example.org')
|
|
41
45
|
.get('/inclusions.txt')
|
|
42
|
-
.reply(200, 'rule1'
|
|
46
|
+
.reply(200, 'rule1', {
|
|
47
|
+
'Content-Type': 'text/plain',
|
|
48
|
+
})
|
|
43
49
|
.get('/inclusions2.txt')
|
|
44
|
-
.reply(200, 'rule2'
|
|
50
|
+
.reply(200, 'rule2', {
|
|
51
|
+
'Content-Type': 'text/plain',
|
|
52
|
+
});
|
|
45
53
|
mock({
|
|
46
|
-
|
|
54
|
+
[testDirPath]: {
|
|
47
55
|
'inclusions.txt': 'rule3',
|
|
48
56
|
},
|
|
49
57
|
});
|
|
@@ -56,7 +64,7 @@ describe('Exclusions', () => {
|
|
|
56
64
|
const inclusionsSources = [
|
|
57
65
|
'https://example.org/inclusions.txt',
|
|
58
66
|
'https://example.org/inclusions2.txt',
|
|
59
|
-
|
|
67
|
+
inclusionsFilePath,
|
|
60
68
|
];
|
|
61
69
|
|
|
62
70
|
// Include!
|
|
@@ -73,4 +73,35 @@ rule2
|
|
|
73
73
|
expect(filtered).toHaveLength(4);
|
|
74
74
|
expect(filtered).toEqual(['! test comment', 'rule1', '||185.149.120.173^', '||example.com^']);
|
|
75
75
|
});
|
|
76
|
+
|
|
77
|
+
it('simple transformations with ConvertToAscii', async () => {
|
|
78
|
+
const rules = `! test comment
|
|
79
|
+
||*.рус^
|
|
80
|
+
||*.कॉम^
|
|
81
|
+
||*.セール^
|
|
82
|
+
||*.佛山^
|
|
83
|
+
||*.ಭಾರತ^
|
|
84
|
+
||*.慈善^
|
|
85
|
+
||*.集团^
|
|
86
|
+
||*.在线^`.split(/\r?\n/);
|
|
87
|
+
const exclusions = [
|
|
88
|
+
'', // empty exclusions are ignored
|
|
89
|
+
'||*.セール^',
|
|
90
|
+
];
|
|
91
|
+
const configuration = {
|
|
92
|
+
exclusions,
|
|
93
|
+
};
|
|
94
|
+
const filtered = await transform(rules, configuration, ['ConvertToAscii']);
|
|
95
|
+
expect(filtered).toHaveLength(8);
|
|
96
|
+
expect(filtered).toEqual([
|
|
97
|
+
'! test comment',
|
|
98
|
+
'||*.xn--p1acf^',
|
|
99
|
+
'||*.xn--11b4c3d^',
|
|
100
|
+
'||*.xn--1qqw23a^',
|
|
101
|
+
'||*.xn--2scrj9c^',
|
|
102
|
+
'||*.xn--30rr7y^',
|
|
103
|
+
'||*.xn--3bst00m^',
|
|
104
|
+
'||*.xn--3ds443g^',
|
|
105
|
+
]);
|
|
106
|
+
});
|
|
76
107
|
});
|