@adguard/filters-compiler 3.0.1

package/LICENSE

@@ -0,0 +1,166 @@
+GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
+

package/README.md

@@ -0,0 +1,242 @@
+# AdGuard Filters Compiler
+
+Filters compiler package is a tool for compiling ad blocking filters into a supported format.
+It is used in [FiltersRegistry].
+
+- [Usage](#usage)
+- [Tests](#tests)
+- [Development](#development)
+    - [Schemas maintenance](#schemas-maintenance)
+- [Filters metadata](#filters-metadata)
+- [`@include` directive and its options](#include-directive)
+
+## Usage
+
+This package is suggested to be used with filters repository with directory structure presented in tests here.
+
+The package could be run with the following command:
+
+```javascript
+const whitelist = [1, 3];
+const blacklist = [2];
+
+const path = require('path');
+const compiler = require("adguard-filters-compiler");
+
+const filtersDir = path.join(__dirname, './filters');
+const logPath = path.join(__dirname, './log.txt');
+const reportPath = path.join(__dirname, './report.txt');
+
+const platformsPath = path.join(__dirname, './platforms');
+
+const customPlatformsConfig = {
+    // Here you can redefine some of the platforms from platforms.json
+    // or add new platforms if you need it.
+    "MAC_V3": {
+        "platform": "mac",
+        "path": "mac_v3",
+        "configuration": {
+            "ignoreRuleHints": false,
+            "removeRulePatterns": [
+                "^\\/.*" // remove regex rules for some reason.
+            ],
+            "replacements": [
+                {
+                    "from": "regex",
+                    "to": "repl"
+                }
+            ]
+        },
+        "defines": {
+            "adguard": true,
+            "adguard_app_mac": true
+        }
+    },
+};
+
+compiler.compile(filtersDir, logPath, reportPath, platformsPath, whitelist, blacklist, customPlatformsConfig);
+```
+
+The built filters for the platforms can be validated by schemas.
+And there is `validateJSONSchema()` method for that:
+
+```javascript
+const compiler = require("adguard-filters-compiler");
+
+const validationResult = compiler.validateJSONSchema(<platformsPath>, <FILTERS_REQUIRED_AMOUNT>);
+```
+
+where `<platformsPath>` is the path to the platforms directory
+and `<FILTERS_REQUIRED_AMOUNT>` is an expected minimum number of filters.
+
+## Tests
+
+```bash
+pnpm test
+```
+
+## Development
+
+> No new fields should be added to the metadata files for old `mac` and current `mac_v2` platforms,
+> check [generator.js](./src/main/platforms/generator.js) for more details.
+
+In order to add support for new scriptlets and redirects,
+you should update `@adguard/tsurlfilter` with updated scriptlets.
+
+For fixing scriptlets converting or validation you should update `@adguard/scriptlets`.
+
+### Schemas maintenance
+
+Schemas which are used for `validateJSONSchema()` method are located in `schemas/` directory:
+
+- `filters.schema.json` — schema for *filters* metadata;
+- `filters_i18n.schema.json` — schema for *filters_i18n* metadata.
+
+> Schemas in `schemas/mac/` directory are needed for legacy macOS v1 platform, so they should not be changed.
+> The same is true for `schemas/mac_v2/` directory.
+
+If any changes should be made in the schemas, e.g. adding a new locale or filter or tag,
+**never edit them directly in `schemas/` manually**.
+
+Instead of that, you should edit scripts in `tasks/build-schemas/` directory
+and use the following command to generate the schemas:
+
+```bash
+pnpm build-schemas
+```
+
+## Filters metadata
+
+Description of the filters metadata is available in the [FiltersRegistry][filters-metadata] repository.
+
+## <a name="include-directive"></a> `@include` directive and its options
+
+The `@include` directive provides the ability to include content from the specified address.
+
+### Syntax
+
+```text
+@include <filepath> [<options>]
+```
+
+where:
+
+- `<filepath>` — required, URL or same origin relative file path to be included;
+- `<options>` — optional, a list of options separated by spaces.
+  Available options:
+
+    - `/stripComments` removes AdBlock-style syntax comments from the included file — lines which start with `!`;
+    - `/notOptimized` adds a `!+ NOT_OPTIMIZED` hint to the rules;
+    - `/exclude="<filepath>"` excludes from the included file rules
+      listed in the exceptions file available by `filepath`;
+    - `/addModifiers="<modifiers>"` adds the specified `modifiers` (string as is) to the rules in the included file.
+      The addModifiers option can also work with the host-rule format files.
+      In this case, host-file comments are to be replaced `#` by AdBlock-style syntax comments `!`;
+    - `/ignoreTrustLevel` disables the check of the trust level of the included file.
+      Allowed only for the same origin files.
+    - `/optimizeDomainBlockingRules` remove redundant rules for domain blocking of the included file.
+      Base rules with modifiers and rules of other format will be ignored.
+      Comments are not checked separately and may not be relevant after optimization.
+
+> [!IMPORTANT]
+> The content of the included file is formatted by the options due to the order of their mention in the directive,
+> except `/ignoreTrustLevel`.
+
+### Examples
+
+- Include a file with domains, add modifiers to the rules, exclude some rules,
+   add a hint to the rules, and remove comments from the prepared rules:
+
+    ```adblock
+    @include ../input.txt /addModifiers="script" /exclude="../exclusions.txt" /notOptimized /stripComments /optimizeDomainBlockingRules
+    ```
+
+    The order of execution of the options is as follows:
+
+    1. `@include ../input.txt`: Includes the content of the file named `input.txt` from the parent directory.
+
+        ```adblock
+        # comment
+        example.com
+        example.org
+        ```
+
+    1. `/addModifiers="script"`: Adds the `$script` modifier to all rules in the included file.
+
+        Result of adding modifiers:
+
+        ```adblock
+        ! comment
+        example.com$script
+        example.org$script
+        ```
+
+        > Used to restrict rules with modifiers when blocking the entire domain would result in a breakage.
+        > [issue example](https://github.com/AdguardTeam/FiltersCompiler/issues/190)
+
+    1. `/exclude="../exclusions.txt"`: Excludes rules listed in the exception list from the file named `exclusions.txt`, if they match.
+
+        Due to the content of `exclusions.txt`:
+
+        ```adblock
+        example.com$script
+        example2.com$script
+        ```
+
+        Result of excluding:
+
+        ```adblock
+        ! comment
+        example.org$script
+        ```
+
+        > Used to exclude problematic rules in the filter
+
+    1. `/notOptimized`: Adds the `!+ NOT_OPTIMIZED` hint to the rules.
+
+        Result of adding the hint:
+
+        ```adblock
+        ! comment
+        !+ NOT_OPTIMIZED
+        example.org$script
+        ```
+
+        > Used in cases where the filter is designed for mobile site layout and some rules may be removed,
+        > due to the lack of ability to collect statistics on mobile platforms.
+
+    1. `/stripComments`: Removes comments in AdBlock style from the included file.
+
+        ```adblock
+        !+ NOT_OPTIMIZED
+        example.org$script
+        ```
+
+    1. `/optimizeDomainBlockingRules`: Remove only domain blocking redundant rules from the included file.
+
+        Due to the optimization:
+
+        ```adblock
+        ||example.com^
+        ||sub.example.com^
+        ||domain.com^
+        ||test.com^$script
+        ```
+
+        Result of optimization:
+
+        ```adblock
+        ||example.com^
+        ||domain.com^
+        ||test.com^$script
+        ```
+
+- Ignore the trust level of the filter list (specified in the metadata) during the file including —
+  include the file rules as is:
+
+    ```adblock
+    @include ./input.txt /ignoreTrustLevel
+    ```
+
+[FiltersRegistry]: https://github.com/AdguardTeam/FiltersRegistry/
+[filters-metadata]: https://github.com/AdguardTeam/FiltersRegistry/blob/master/README.md#filters-metadata

package/dist/build.txt

@@ -0,0 +1 @@
+version=3.0.1