@adguard/hostlist-compiler 1.0.16 → 1.0.19

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -20,7 +20,7 @@ This is a simple tool that makes it easier to compile a [hosts blocklist](https:
20
20
  - [InsertFinalNewLine](#insertfinalnewline)
21
21
  - [How to build](#how-to-build)
22
22
 
23
- ## <a id="usage"></a> Usage
23
+ ## <a name="usage"></a> Usage
24
24
 
25
25
  First of all, install the `hostlist-compiler`:
26
26
 
@@ -40,13 +40,36 @@ hostlist-compiler -i hosts.txt -i hosts2.txt -o output.txt
40
40
 
41
41
  **Build a configurable blocklist from multiple sources**
42
42
 
43
- Prepare the list configuration (read more about that below) and run the compiler:
43
+ Prepare the list configuration (read more about that [below](#configuration)) and run the compiler:
44
44
 
45
45
  ```bash
46
46
  hostlist-compiler -c configuration.json -o output.txt
47
47
  ```
48
48
 
49
- ### <a id="configuration"></a> Configuration
49
+ **All command line options**
50
+
51
+ ```
52
+ Usage: hostlist-compiler [options]
53
+
54
+ Options:
55
+ --config, -c Path to the compiler configuration file [string]
56
+ --input, -i URL (or path to a file) to convert to an AdGuard-syntax
57
+ blocklist. Can be specified multiple times. [array]
58
+ --input-type, -t Type of the input file (/etc/hosts, adguard) [string]
59
+ --output, -o Path to the output file [string] [required]
60
+ --verbose, -v Run with verbose logging [boolean]
61
+ --version Show version number [boolean]
62
+ -h, --help Show help [boolean]
63
+
64
+ Examples:
65
+ hostlist-compiler -c config.json -o compile a blocklist and write the
66
+ output.txt output to output.txt
67
+ hostlist-compiler -i compile a blocklist from the URL and
68
+ https://example.org/hosts.txt -o write the output to output.txt
69
+ output.txt
70
+ ```
71
+
72
+ ### <a name="configuration"></a> Configuration
50
73
 
51
74
  Configuration defines your filter list sources, and the transformations that are applied to the sources.
52
75
 
@@ -125,7 +148,7 @@ Please note, that exclusion or inclusion rules may be a plain string, wildcard,
125
148
  - `/regex/` - every rule that matches this regular expression, will match the rule. By default, regular expressions are case-insensitive.
126
149
  - `! comment` - comments will be ignored.
127
150
 
128
- ### <a id="command-line"></a> Command-line
151
+ ### <a name="command-line"></a> Command-line
129
152
 
130
153
  Command-line arguments.
131
154
 
@@ -144,7 +167,7 @@ Examples:
144
167
  output.txt output to output.txt
145
168
  ```
146
169
 
147
- ### <a id="api"></a> API
170
+ ### <a name="api"></a> API
148
171
 
149
172
  Install: `npm i @adguard/hostlist-compiler` or `yarn add @adguard/hostlist-compiler`
150
173
 
@@ -166,7 +189,7 @@ const compile = require("@adguard/hostlist-compiler");
166
189
  ],
167
190
  transformations: ['Deduplicate'],
168
191
  });
169
-
192
+
170
193
  // Write to file
171
194
  writeFileSync('your-hostlist.txt', result.join('\n'));
172
195
  })();
@@ -225,7 +248,7 @@ import { writeFileSync } from 'fs';
225
248
  })();
226
249
  ```
227
250
 
228
- ## <a id="transformations"></a> Transformations
251
+ ## <a name="transformations"></a> Transformations
229
252
 
230
253
  Here is the full list of transformations that are available:
231
254
 
@@ -241,11 +264,11 @@ Here is the full list of transformations that are available:
241
264
 
242
265
  Please note that these transformations are are always applied in the order specified here.
243
266
 
244
- ### <a id="remove-comments"></a> RemoveComments
267
+ ### <a name="remove-comments"></a> RemoveComments
245
268
 
246
269
  This is a very simple transformation that simply removes comments (e.g. all rules starting with `!` or `#`).
247
270
 
248
- ### <a id="compress"></a> Compress
271
+ ### <a name="compress"></a> Compress
249
272
 
250
273
  > **IMPORTANT:** this transformation converts `hosts` lists into `adblock` lists.
251
274
 
@@ -254,7 +277,7 @@ Here's what it does:
254
277
  1. It converts all rules to adblock-style rules. For instance, `0.0.0.0 example.org` will be converted to `||example.org^`.
255
278
  2. It discards the rules that are now redundant because of other existing rules. For instance, `||example.org` blocks `example.org` and all it's subdomains, therefore additional rules for the subdomains are now redundant.
256
279
 
257
- ### <a id="remove-modifiers"></a> RemoveModifiers
280
+ ### <a name="remove-modifiers"></a> RemoveModifiers
258
281
 
259
282
  By default, [AdGuard Home](https://github.com/AdguardTeam/AdGuardHome) will ignore rules with unsupported modifiers, and all of the modifiers listed here are unsupported. However, the rules with these modifiers are likely to be okay for DNS-level blocking, that's why you might want to remove them when importing rules from a traditional filter list.
260
283
 
@@ -267,7 +290,7 @@ Here is the list of modifiers that will be removed:
267
290
 
268
291
  **IMPORTANT:** please, be cautious with this. Blindly removing `$third-party` from traditional ad blocking rules leads to lots of false-positives. This is exactly why there is an option to exclude rules - you may need to use it.
269
292
 
270
- ### <a id="validate"></a> Validate
293
+ ### <a name="validate"></a> Validate
271
294
 
272
295
  This transformation is really crucial if you're using a filter list for a traditional ad blocker as a source.
273
296
 
@@ -281,7 +304,7 @@ So here's what it does:
281
304
 
282
305
  If there are comments preceding the invalid rule, they will be removed as well.
283
306
 
284
- ### <a id="deduplicate"></a> Deduplicate
307
+ ### <a name="deduplicate"></a> Deduplicate
285
308
 
286
309
  This transformation simply removes the duplicates from the specified source.
287
310
 
@@ -306,7 +329,7 @@ Here's what will be left after the transformation:
306
329
  rule1
307
330
  ```
308
331
 
309
- ### <a id="invertallow"></a> InvertAllow
332
+ ### <a name="invertallow"></a> InvertAllow
310
333
 
311
334
  This transformation converts blocking rules to "allow" rules. Note, that it does nothing to /etc/hosts rules (unless they were previously converted to adblock-style syntax by a different transformation, for example [Compress](#compress)).
312
335
 
@@ -339,7 +362,7 @@ Here's what we will have after applying this transformation:
339
362
  @@rule2
340
363
  ```
341
364
 
342
- ### <a id="removeemptylines"></a> RemoveEmptyLines
365
+ ### <a name="removeemptylines"></a> RemoveEmptyLines
343
366
 
344
367
  This is a very simple transformation that removes empty lines.
345
368
 
@@ -364,7 +387,7 @@ rule2
364
387
  rule3
365
388
  ```
366
389
 
367
- ### <a id="trimlines"></a> TrimLines
390
+ ### <a name="trimlines"></a> TrimLines
368
391
 
369
392
  This is a very simple transformation that removes leading and trailing spaces/tabs.
370
393
 
@@ -388,7 +411,7 @@ rule3
388
411
  rule4
389
412
  ```
390
413
 
391
- ### <a id="insertfinalnewline"></a> InsertFinalNewLine
414
+ ### <a name="insertfinalnewline"></a> InsertFinalNewLine
392
415
 
393
416
  This is a very simple transformation that inserts a final newline.
394
417
 
@@ -413,7 +436,7 @@ rule3
413
436
 
414
437
  `RemoveEmptyLines` doesn't delete this empty row due to the execution order.
415
438
 
416
- ## <a id="how-to-build"></a> How to build
439
+ ## <a name="how-to-build"></a> How to build
417
440
 
418
441
  - `yarn install` - installs dependencies
419
442
  - `yarn lint` - runs eslint
@@ -1,58 +1,14 @@
1
1
  ---
2
- version: 2
3
- plan:
4
- project-key: AJL
5
- key: HBRTSPECS
6
- name: hostlist-compiler - test
7
- variables:
8
- dockerJslBuilder: adguard/jsl-builder:15.8.0--1.22.5--1
2
+ !include 'build.yaml'
9
3
 
10
- stages:
11
- - Test:
12
- manual: 'false'
13
- final: 'false'
14
- jobs:
15
- - Test
16
-
17
- Test:
18
- key: TEST
19
- docker:
20
- image: "${bamboo.dockerJslBuilder}"
21
- volumes:
22
- ${system.YARN_DIR}: "${bamboo.cacheYarn}"
23
- tasks:
24
- - checkout:
25
- force-clean-build: 'true'
26
- - script:
27
- interpreter: SHELL
28
- scripts:
29
- - |-
30
- set -e
31
- set -x
32
- yarn install
33
- yarn lint
34
- yarn test
35
- rm -rf node_modules
36
- final-tasks:
37
- - clean
38
- requirements:
39
- - adg-docker: 'true'
4
+ ---
5
+ !include 'deploy.yaml'
40
6
 
41
- branches:
42
- create: for-pull-request
43
- delete:
44
- after-deleted-days: '1'
45
- after-inactive-days: '5'
46
- link-to-jira: 'true'
7
+ ---
8
+ !include 'increment.yaml'
47
9
 
48
- notifications:
49
- - events:
50
- - plan-status-changed
51
- recipients:
52
- - webhook:
53
- name: Build webhook
54
- url: http://prod.jirahub.service.eu.consul/v1/webhook/bamboo?channel=adguard-qa
10
+ ---
11
+ !include 'permissions.yaml'
55
12
 
56
- labels: []
57
- other:
58
- concurrent-build-plugin: system-default
13
+ ---
14
+ !include 'test.yaml'
@@ -0,0 +1,101 @@
1
+ ---
2
+ version: 2
3
+ plan:
4
+ project-key: AJL
5
+ key: HOSTLISTCMPLRBUILD
6
+ name: hostlist-compiler - build
7
+ variables:
8
+ dockerContainer: adguard/node-ssh:18.13--0
9
+
10
+ stages:
11
+ - Build:
12
+ manual: false
13
+ final: false
14
+ jobs:
15
+ - Build
16
+
17
+ Build:
18
+ key: BUILD
19
+ other:
20
+ clean-working-dir: true
21
+ docker:
22
+ image: "${bamboo.dockerContainer}"
23
+ volumes:
24
+ ${system.YARN_DIR}: "${bamboo.cacheYarn}"
25
+ tasks:
26
+ - checkout:
27
+ force-clean-build: true
28
+ - script:
29
+ interpreter: SHELL
30
+ scripts:
31
+ - |-
32
+ set -e
33
+ set -x
34
+
35
+ # Fix mixed logs
36
+ exec 2>&1
37
+
38
+ # install dependencies at the beginning
39
+ yarn install ${bamboo.varsYarn}
40
+
41
+ yarn build-txt
42
+ - inject-variables:
43
+ file: dist/build.txt
44
+ scope: RESULT
45
+ namespace: inject
46
+ - script:
47
+ interpreter: SHELL
48
+ scripts:
49
+ - |-
50
+ #!/bin/bash
51
+ set -e
52
+ set -x
53
+
54
+ # Fix mixed logs
55
+ exec 2>&1
56
+
57
+ # Fail if version was not incremented
58
+ if [ "$(git tag -l "v${bamboo.inject.version}")" ]; then
59
+ echo "Build failed!"
60
+ echo "Error: Tag v${bamboo.inject.version} already exists. Increment version before build"
61
+ exit 1
62
+ fi
63
+
64
+ yarn pack --filename hostlist-compiler.tgz
65
+
66
+ # clean up at the end
67
+ rm -rf node_modules
68
+ - any-task:
69
+ plugin-key: com.atlassian.bamboo.plugins.vcs:task.vcs.tagging
70
+ configuration:
71
+ selectedRepository: defaultRepository
72
+ tagName: v${bamboo.inject.version}
73
+ artifacts:
74
+ - name: hostlist-compiler.tgz
75
+ location: ./
76
+ pattern: hostlist-compiler.tgz
77
+ shared: true
78
+ required: true
79
+ requirements:
80
+ - adg-docker: true
81
+
82
+ triggers: []
83
+
84
+ branches:
85
+ create: manually
86
+ delete: never
87
+ link-to-jira: true
88
+
89
+ notifications:
90
+ - events:
91
+ - plan-status-changed
92
+ recipients:
93
+ - webhook:
94
+ name: Build webhook
95
+ # TODO: maybe channel should be 'adguard-filters'
96
+ url: http://prod.jirahub.service.eu.consul/v1/webhook/bamboo?channel=adguard-qa
97
+
98
+ labels: []
99
+
100
+ other:
101
+ concurrent-build-plugin: system-default
@@ -0,0 +1,46 @@
1
+ ---
2
+ version: 2
3
+ deployment:
4
+ name: hostlist-compiler - deploy
5
+ source-plan: AJL-HOSTLISTCMPLRBUILD
6
+ release-naming: ${bamboo.inject.version}
7
+ environments:
8
+ - npmjs
9
+
10
+ npmjs:
11
+ docker:
12
+ image: adguard/node-ssh:18.13--0
13
+ volumes:
14
+ ${system.YARN_DIR}: "${bamboo.cacheYarn}"
15
+ triggers: []
16
+ tasks:
17
+ - checkout:
18
+ force-clean-build: true
19
+ - artifact-download:
20
+ artifacts:
21
+ - name: hostlist-compiler.tgz
22
+ - script:
23
+ interpreter: SHELL
24
+ scripts:
25
+ - |-
26
+ set -e
27
+ set -x
28
+
29
+ # Fix mixed logs
30
+ exec 2>&1
31
+
32
+ ls -alt
33
+
34
+ export NPM_TOKEN=${bamboo.npmSecretToken}
35
+ echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc
36
+ npm publish hostlist-compiler.tgz --access public
37
+ requirements:
38
+ - adg-docker: true
39
+ notifications:
40
+ - events:
41
+ - deployment-started-and-finished
42
+ recipients:
43
+ - webhook:
44
+ name: Deploy webhook
45
+ # TODO: maybe channel should be 'adguard-filters'
46
+ url: http://prod.jirahub.service.eu.consul/v1/webhook/bamboo?channel=adguard-qa
@@ -0,0 +1,54 @@
1
+ ---
2
+ version: 2
3
+ plan:
4
+ project-key: AJL
5
+ key: HOSTLISTCMPLRINCR
6
+ name: hostlist-compiler - increment
7
+ variables:
8
+ dockerContainer: adguard/node-ssh:18.13--0
9
+
10
+ stages:
11
+ - Increment:
12
+ manual: false
13
+ final: false
14
+ jobs:
15
+ - Increment
16
+
17
+ Increment:
18
+ key: INCR
19
+ docker:
20
+ image: "${bamboo.dockerContainer}"
21
+ volumes:
22
+ ${system.YARN_DIR}: "${bamboo.cacheYarn}"
23
+ other:
24
+ clean-working-dir: true
25
+ tasks:
26
+ - checkout:
27
+ force-clean-build: true
28
+ - script:
29
+ interpreter: SHELL
30
+ scripts:
31
+ - |-
32
+ set -e
33
+ set -x
34
+
35
+ # Fix mixed logs
36
+ exec 2>&1
37
+
38
+ yarn increment
39
+ - any-task:
40
+ plugin-key: com.atlassian.bamboo.plugins.vcs:task.vcs.commit
41
+ configuration:
42
+ commitMessage: 'skipci: Automatic increment build number'
43
+ selectedRepository: defaultRepository
44
+ requirements:
45
+ - adg-docker: true
46
+
47
+ branches:
48
+ create: manually
49
+ delete: never
50
+ link-to-jira: true
51
+
52
+ labels: []
53
+ other:
54
+ concurrent-build-plugin: system-default
@@ -0,0 +1,16 @@
1
+ ---
2
+ version: 2
3
+ deployment:
4
+ name: hostlist-compiler - deploy
5
+ deployment-permissions:
6
+ - groups:
7
+ - extensions-developers
8
+ permissions:
9
+ - view
10
+ environment-permissions:
11
+ - npmjs:
12
+ - groups:
13
+ - extensions-developers
14
+ permissions:
15
+ - view
16
+ - deploy
@@ -0,0 +1,65 @@
1
+ ---
2
+ version: 2
3
+ plan:
4
+ project-key: AJL
5
+ key: HOSTLISTCMPLRTEST
6
+ name: hostlist-compiler - tests
7
+ variables:
8
+ dockerContainer: adguard/node-ssh:18.13--0
9
+
10
+ stages:
11
+ - Test:
12
+ manual: false
13
+ final: false
14
+ jobs:
15
+ - Test
16
+
17
+ Test:
18
+ key: TEST
19
+ docker:
20
+ image: "${bamboo.dockerContainer}"
21
+ volumes:
22
+ ${system.YARN_DIR}: "${bamboo.cacheYarn}"
23
+ tasks:
24
+ - checkout:
25
+ force-clean-build: true
26
+ - script:
27
+ interpreter: SHELL
28
+ scripts:
29
+ - |-
30
+ set -e
31
+ set -x
32
+
33
+ # Fix mixed logs
34
+ exec 2>&1
35
+
36
+ yarn install ${bamboo.varsYarn}
37
+
38
+ yarn lint
39
+ yarn test
40
+
41
+ rm -rf node_modules
42
+ final-tasks:
43
+ - clean
44
+ requirements:
45
+ - adg-docker: true
46
+
47
+ branches:
48
+ create: for-pull-request
49
+ delete:
50
+ after-deleted-days: '1'
51
+ after-inactive-days: '5'
52
+ link-to-jira: true
53
+
54
+ notifications:
55
+ - events:
56
+ - plan-status-changed
57
+ recipients:
58
+ - webhook:
59
+ name: Build webhook
60
+ # TODO: maybe channel should be 'adguard-filters'
61
+ url: http://prod.jirahub.service.eu.consul/v1/webhook/bamboo?channel=adguard-qa
62
+
63
+ labels: []
64
+ other:
65
+ concurrent-build-plugin: system-default
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@adguard/hostlist-compiler",
3
- "version": "1.0.16",
3
+ "version": "1.0.19",
4
4
  "description": "A simple tool that compiles hosts blocklists from multiple sources",
5
5
  "main": "src/index.js",
6
6
  "types": "src/index.d.ts",
@@ -11,8 +11,10 @@
11
11
  "hostlist-compiler": "src/cli.js"
12
12
  },
13
13
  "scripts": {
14
- "lint": "eslint .",
14
+ "lint": "eslint . --cache",
15
15
  "test": "jest --runInBand .",
16
+ "increment": "yarn version --patch --no-git-tag-version",
17
+ "build-txt": "node tools/build-txt.js",
16
18
  "compile": "node src/cli.js"
17
19
  },
18
20
  "devDependencies": {
@@ -0,0 +1,19 @@
1
+ const fs = require('fs');
2
+ const path = require('path');
3
+ const { version } = require('../package.json');
4
+
5
+ const PATH = '../dist';
6
+ const FILENAME = 'build.txt';
7
+
8
+ const buildTxt = () => {
9
+ const dir = path.resolve(__dirname, PATH);
10
+
11
+ if (!fs.existsSync(dir)) {
12
+ fs.mkdirSync(dir);
13
+ }
14
+
15
+ const content = `version=${version}`;
16
+ fs.writeFileSync(path.resolve(__dirname, PATH, FILENAME), content);
17
+ };
18
+
19
+ buildTxt();