@opentermsarchive/engine 0.15.0

package/README.md

@@ -0,0 +1,438 @@
+# Open Terms Archive
+
+**Services** have **terms** that can change over time. _Open Terms Archive_ enables users rights advocates, regulatory bodies and any interested citizen to follow the **changes** to these **terms** by being **notified** whenever a new **version** is published, and exploring their entire **history**.
+
+> Les services ont des conditions générales qui évoluent dans le temps. _Open Terms Archive_ permet aux défenseurs des droits des utilisateurs, aux régulateurs et à toute personne intéressée de suivre les évolutions de ces conditions générales en étant notifiée à chaque publication d'une nouvelle version, et en explorant leur historique.
+
+[🇫🇷 Manuel en français](README.fr.md).
+
+## Table of Contents
+
+- [How it works](#how-it-works)
+- [Exploring the versions history](#exploring-the-versions-history)
+- [Be notified](#be-notified)
+  - [By email](#by-email)
+  - [By RSS](#by-rss)
+- [Importing as a module](#importing-as-a-module)
+  - [CLI](#cli)
+  - [Features exposed](#features-exposed)
+    - [fetch](#fetch)
+    - [filter](#filter)
+- [Using locally](#using-locally)
+  - [Installing](#installing)
+    - [Declarations repository](#declarations-repository)
+    - [Core](#core)
+  - [Configuring](#configuring)
+    - [Configuration file](#configuration-file)
+      - [Storage repositories](#storage-repositories)
+    - [Environment variables](#environment-variables)
+  - [Running](#running)
+- [Deploying](#deploying)
+- [Publishing](#publishing)
+- [Contributing](#contributing)
+  - [Adding or updating a service](#adding-a-new-service-or-updating-an-existing-service)
+  - [Core engine](#core-engine)
+  - [Funding and partnerships](#funding-and-partnerships)
+- [License](#license)
+
+## How it works
+
+_Note: Words in bold are [business domain names](https://en.wikipedia.org/wiki/Domain-driven_design)._
+
+**Services** are **declared** within _Open Terms Archive_ with a **declaration file** listing all the **documents** that, together, constitute the **terms** under which this **service** can be used. These **documents** all have a **type**, such as “terms and conditions”, “privacy policy”, “developer agreement”…
+
+In order to **track** their **changes**, **documents** are periodically obtained by **fetching** a web **location** and **selecting content** within the **web page** to remove the **noise** (ads, navigation menu, login fields…). Beyond selecting a subset of a page, some **documents** have additional **noise** (hashes in links, CSRF tokens…) that would be false positives for **changes**. _Open Terms Archive_ thus supports specific **filters** for each **document**.
+
+However, the shape of that **noise** can change over time. In order to recover in case of information loss during the **noise filtering** step, a **snapshot** is **recorded** every time there is a **change**. After the **noise** is **filtered out** from the **snapshot**, if there are **changes** in the resulting **document**, a new **version** of the **document** is **recorded**.
+
+Anyone can run their own **private** instance and track changes on their own. However, we also **publish** each **version** on a [**public** instance](https://github.com/OpenTermsArchive/contrib-versions) that makes it easy to explore the entire **history** and enables **notifying** over email whenever a new **version** is **recorded**.
+Users can [**subscribe** to **notifications**](#be-notified).
+
+_Note: For now, when multiple versions coexist, **terms** are only **tracked** in their English version and for the European jurisdiction._
+
+## Exploring the versions history
+
+We offer a public database of versions recorded each time there is a change in the terms of service and other contractual documents of tracked services: [contrib-versions](https://github.com/OpenTermsArchive/contrib-versions).
+
+From the **repository homepage** [contrib-versions](https://github.com/OpenTermsArchive/contrib-versions), open the folder of the **service of your choice** (e.g. [WhatsApp](https://github.com/OpenTermsArchive/contrib-versions/tree/main/WhatsApp)).
+
+You will see the **set of documents tracked** for that service, now click **on the document of your choice** (e.g. [WhatsApp's Privacy Policy](https://github.com/OpenTermsArchive/contrib-versions/blob/main/WhatsApp/Privacy%20Policy.md)). The **latest version** (updated hourly) will be displayed.
+
+To view the **history of changes** made to this document, click on **History** at the top right of the document (for our previous [example](https://github.com/OpenTermsArchive/contrib-versions/commits/main/WhatsApp/Privacy%20Policy.md)). The **changes** are ordered **by date**, with the latest first.
+
+Click on a change to see what it consists of (for example [this one](https://github.com/OpenTermsArchive/contrib-versions/commit/58a1d2ae4187a3260ac58f3f3c7dcd3aeacaebcd)). There are **two types of display** you can choose from the icons in the gray bar above the document.
+
+- The first one, named _source diff_ (button with chevrons) allows you to **display the old version and the new one side by side** (for our [example](https://github.com/OpenTermsArchive/contrib-versions/commit/58a1d2ae4187a3260ac58f3f3c7dcd3aeacaebcd#diff-e8bdae8692561f60aeac9d27a55e84fc)). This display has the merit of **explicitly showing** all additions and deletions.
+- The second one, named _rich diff_ (button with a document icon) allows you to **unify all the changes in a single document** (for our [example](https://github.com/OpenTermsArchive/contrib-versions/commit/58a1d2ae4187a3260ac58f3f3c7dcd3aeacaebcd?short_path=e8bdae8#diff-e8bdae8692561f60aeac9d27a55e84fc)). The **red** color shows **deleted** elements, the **yellow** color shows **modified** paragraphs, and the **green** color shows **added** elements. Be careful, this display **does not show some changes** such as hyperlinks and text style's changes.
+
+### Notes
+
+- For long documents, unchanged **paragraphs will not be displayed by default**. You can manually make them appear by clicking on the small arrows just above or just below the displayed paragraphs.
+- You can use the **History button anywhere** in the repository contrib-versions, which will then display the **history of changes made to all documents in the folder** where you are (including sub-folders).
+
+## Be notified
+
+### By email
+
+#### Document per document
+
+You can go on the official front website [opentermsarchive.org](https://opentermsarchive.org). From there, you can select a service and then the corresponding document type.
+After you enter your email and click on subscribe, we will add your email to the correspondning mailing list in [SendInBlue](https://www.sendinblue.com/) and will not store your email anywhere else.
+Then, everytime a modification is found on the correspondning document, we will send you an email.
+
+You can unsubscribe at any moment by clicking on the `unsubscribe` link at the bottom of the received email.
+
+#### For all documents at once
+
+You can [subscribe](https://59692a77.sibforms.com/serve/MUIEAKuTv3y67e27PkjAiw7UkHCn0qVrcD188cQb-ofHVBGpvdUWQ6EraZ5AIb6vJqz3L8LDvYhEzPb2SE6eGWP35zXrpwEFVJCpGuER9DKPBUrifKScpF_ENMqwE_OiOZ3FdCV2ra-TXQNxB2sTEL13Zj8HU7U0vbbeF7TnbFiW8gGbcOa5liqmMvw_rghnEB2htMQRCk6A3eyj) to receive an email whenever a document is updated in the database.
+
+**Beware, you are likely to receive a large amount of notifications!** You can unsubscribe by replying to any email you will receive.
+
+### By RSS
+
+You can receive notification for a specific service or document by subscribing to RSS feeds.
+
+> An RSS feed is a type of web page that contains information about the latest content published by a website, such as the date of publication and the address where you can view it. When this resource is updated, a feed reader app automatically notifies you and you can see the update.
+
+To find out the address of the RSS feed you want to subscribe to:
+
+1. [Navigate](#exploring-the-versions-history) to the page with the history of changes you are interested in. _In the WhatsApp example above, this would be [this page](https://github.com/OpenTermsArchive/contrib-versions/commits/main/WhatsApp/Privacy%20Policy.md)._
+2. Copy the address of that page from your browser’s address bar. _In the WhatsApp example, this would be `https://github.com/OpenTermsArchive/contrib-versions/commits/main/WhatsApp/Privacy%20Policy.md`._
+3. Append `.atom` at the end of this address. _In the WhatsApp example, this would become `https://github.com/OpenTermsArchive/contrib-versions/commits/main/WhatsApp/Privacy%20Policy.md.atom`._
+4. Subscribe your RSS feed reader to the resulting address.
+
+#### Recap of available RSS feeds
+
+| Updated for                         | URL                                                                                                                                                                                            |
+| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| all services and documents          | `https://github.com/OpenTermsArchive/contrib-versions/commits.atom`                                                                                                                            |
+| all the documents of a service      | Replace `$serviceId` with the service ID:<br>`https://github.com/OpenTermsArchive/contrib-versions/commits/main/$serviceId.atom.`                                                            |
+| a specific document of a service | Replace `$serviceId` with the service ID and `$documentType` with the document type:<br>`https://github.com/OpenTermsArchive/contrib-versions/commits/main/$serviceId/$documentType.md.atom` |
+
+For example:
+
+- To receive all updates of `Facebook` documents, the URL is `https://github.com/OpenTermsArchive/contrib-versions/commits/main/Facebook.atom`.
+- To receive all updates of the `Privacy Policy` from `Google`, the URL is `https://github.com/OpenTermsArchive/contrib-versions/commits/main/Google/Privacy%20Policy.md.atom`.
+
+## Importing as a module
+
+Open Terms Archive exposes a JavaScript API to make some of its capabilities available in NodeJS. You can install it as an NPM module:
+
+```
+npm install "ambanum/OpenTermsArchive#main"
+```
+
+### CLI
+
+The following commands are available where the package is installed:
+
+- `./node_modules/.bin/ota-lint-declarations`: check and normalise the format of declarations.
+- `./node_modules/.bin/ota-validate-declarations`: validate declarations.
+- `./node_modules/.bin/ota-track`: track services. Recorded snapshots and versions will be stored in the `data` folder at the root of the module where the package is installed.
+
+In order to have them available globally in your command line, install it with the `--global` option.
+
+### Features exposed
+
+#### fetch
+
+The `fetch` module gets the MIME type and content of a document from its URL.
+
+You can use it in your code by using `import fetch from 'open-terms-archive/fetch';`.
+
+Documentation on how to use `fetch` is provided as JSDoc within [./src/archivist/fetcher/index.js](./src/archivist/fetcher/index.js).
+
+If you plan to use `executeClientScripts` as a parameter of `fetch`, the fetching will be done using a headless browser.
+In order to not instantiate this browser at each fetch, the starting and stopping of the browser is your responsibility.
+
+Here is an example on how to use it:
+
+```js
+import fetch, { launchHeadlessBrowser, stopHeadlessBrowser } from 'open-terms-archive/fetch';
+
+await launchHeadlessBrowser();
+await fetch({ executeClientScripts: true, ... });
+await fetch({ executeClientScripts: true, ... });
+await fetch({ executeClientScripts: true, ... });
+await stopHeadlessBrowser();
+```
+
+The `fetch` module can also be configured as a [`node-config` submodule](https://github.com/node-config/node-config/wiki/Sub-Module-Configuration).
+If [`node-config`](https://github.com/node-config/node-config) is used in the project, the default `fetcher` configuration can be overridden by adding a `fetcher` object to the local config. See [Configuration file](#configuration-file) for full reference.
+
+#### filter
+
+The `filter` module transforms HTML or PDF content into a Markdown string.
+It will filter content based on the [document declaration](https://github.com/OpenTermsArchive/contrib-declarations/blob/main/CONTRIBUTING.md#declaring-a-new-service).
+
+You can use the filter in your code by using `import filter from 'open-terms-archive/filter';`.
+
+The `filter` function documentation is available as JSDoc within [./src/archivist/filter/index.js](./src/archivist/filter/index.js).
+
+#### page-declaration
+
+PageDeclaration object is used to describe a page to be tracked by Open Terms Archive.
+
+You can use the page-declaration in your code by using `import pageDeclaration from 'open-terms-archive/page-declaration';`.
+
+## Using locally
+
+### Installing
+
+This module is built with [Node](https://nodejs.org/en/) and is tested on macOS, UNIX and Windows. You will need to [install Node >= v16.x](https://nodejs.org/en/download/) to run it.
+
+#### Declarations repository
+
+1. Locally clone your declarations repository, e.g., `git@github.com:OpenTermsArchive/contrib-declarations.git`.
+2. Go into your folder and initialize it, e.g., `cd contrib-declarations; npm install`.
+3. You can now modify your declarations in the `./declarations/` folder, following [these instructions](https://github.com/OpenTermsArchive/contrib-declarations/blob/main/CONTRIBUTING.md).
+4. When you want to test:
+    - If you want to test every declaration, run `npm test`.
+    - If you want to test a specific declaration, run `npm test $serviceId`, e.g., `npm test HER`.
+    - If you want to have faster feedback on the structure of a specific declaration, run `npm run test:schema $serviceId`, e.g., `npm run test:schema HER`.
+5. Once you have done that, if you have any error, it will be prompted and detailed at the end of the test.
+    - E.g., `InaccessibleContentError`: Your selector is wrong and should be fixed.
+    - E.g., `TypeError`: The file declaration is invalid.
+    - E.g., if you have a weird error, you may want to contact OTA, if may be a bug.
+
+##### Note: Testing
+
+Testing works with multiple tests (e.g., checking the validity of the file, that the URL is correct and reachable, that the content is correctly gathered, etc.); as it may take a bit of time, that's why you may want to use `npm run test:schema`.
+
+#### Core
+
+When refering to the base folder, it means the folder where you will be `git pull`ing everything.
+
+1. If not done already, follow the previous part with the repo of your choice.
+2. In the base folder of the previous step (i.e., not _in_ the previous folder, but _where the previous folder is_), clone the core engine: `git clone git@github.com:ambanum/OpenTermsArchive.git`.
+3. Go into the cloned folder and install dependencies: `cd contrib-declarations; npm install`.
+4. If you are using the main repo, you are done, go to step 6.
+5. If you are using a special repo instance (e.g., `dating-declarations`), create a new [config file](#configuring), `config/development.json`, and add:
+    ```json
+    {
+
+      "services": {
+        "declarationsPath": "../<name of the repo>/declarations"
+      }
+    }
+    ```
+    e.g.,
+    ```json
+    {
+      "services": {
+        "declarationsPath": "../dating-declarations/declarations"
+      }
+    }
+    ```
+6. In the folder of the repo (i.e., `OpenTermsArchive`), use `npm start`.
+    - It will first do a refiltering to check whenever everything works properly.
+    - You will then start to see everything being downloaded under `data/`.
+    - More details in [Running](#running).
+
+##### Notes: Tips
+
+- You may want to regularly `git pull` to have the latest updates, both in the core engine and in the declarations repos.
+- You have to `npm install` in the declarations repo at least once, and a least once each time `package.json` changes.
+- Be careful, it doesn't download the history! If you want that, you need to git clone `snapshots` and `versions` in `data/`.
+
+You can clone as many declarations repositories as you want. The one that will be loaded at execution will be defined through configuration.
+
+### Configuring
+
+#### Configuration file
+
+The default configuration can be found in `config/default.json`. The full reference is given below. You are unlikely to want to edit all of these elements.
+
+```js
+{
+  "services": {
+    "declarationsPath": "Directory containing services declarations and associated filters"
+  },
+  "recorder": {
+    "versions": {
+      "storage": {
+        "<storage-repository>": "Storage repository configuration object; see below"
+      }
+    },
+    "snapshots": {
+      "storage": {
+        "<storage-repository>": "Storage repository configuration object; see below"
+      }
+    }
+  },
+  "fetcher": {
+    "waitForElementsTimeout": "Maximum time (in milliseconds) to wait for elements to be present in the page when fetching document in a headless browser"
+    "navigationTimeout": "Maximum time (in milliseconds) to wait for page to load",
+    "language": "Language (in ISO 639-1 format) to pass in request headers"
+  },
+  "notifier": { // Notify specified mailing lists when new versions are recorded
+    "sendInBlue": { // SendInBlue API Key is defined in environment variables, see the “Environment variables” section below
+      "updatesListId": "SendInBlue contacts list ID of persons to notify on document updates",
+      "updateTemplateId": "SendInBlue email template ID used for updates notifications"
+    }
+  },
+  "logger": { // Logging mechanism to be notified upon error
+    "smtp": {
+      "host": "SMTP server hostname",
+      "username": "User for server authentication" // Password for server authentication is defined in environment variables, see the “Environment variables” section below
+    },
+    "sendMailOnError": { // Can be set to `false` if you do not want to send email on error
+      "to": "The address to send the email to in case of an error",
+      "from": "The address from which to send the email",
+      "sendWarnings": "Boolean. Set to true to also send email in case of warning",
+    }
+  },
+  "tracker": { // Tracking mechanism to create GitHub issues when document content is inaccessible
+    "githubIssues": {
+      "repository": "GitHub repository where to create isssues",
+      "label": {
+        "name": "Label to attach to bot-created issues. This specific label will be created automatically in the target repository",
+        "color": "The hexadecimal color code for the label, without the leading #",
+        "description": "A short description of the label"
+      }
+    }
+  },
+  "dataset": { // Release mechanism to create dataset periodically
+    "title": "Title of the dataset; recommended to be the name of the instance that generated it",
+    "versionsRepositoryURL": "GitHub repository where the dataset will be published as a release; recommended to be the versions repository for discoverability and tagging purposes"
+  }
+}
+```
+
+The default configuration is merged with (and overridden by) environment-specific configuration that can be specified at startup with the `NODE_ENV` environment variable. For example, you would run `NODE_ENV=development npm start` to load the `development.json` configuration file.
+
+If you want to change your local configuration, we suggest you create a `config/development.json` file with overridden values. Example production configuration files can be found in the `config` folder.
+
+##### Storage repositories
+
+Two storage repositories are currently supported: Git and MongoDB. Each one can be used independently for versions and snapshots.
+
+###### Git
+
+```json
+{
+  …
+  "storage": {
+    "git": {
+      "path": "Versions database directory path, relative to the root of this project",
+      "publish": "Boolean. Set to true to push changes to the origin of the cloned repository at the end of every run. Recommended for production only.",
+      "snapshotIdentiferTemplate": "Text. Template used to explicit where to find the referenced snapshot id. Must contain a %SNAPSHOT_ID that will be replaced by the snapshot ID. Only useful for versions",
+      "author": {
+        "name": "Name to which changes in tracked documents will be credited",
+        "email": "Email to which changes in tracked documents will be credited"
+      }
+    }
+  }
+  …
+}
+```
+###### MongoDB
+
+```json
+{
+    …
+  "storage": {
+    "mongo": {
+      "connectionURI": "URI for defining connection to the MongoDB instance. See https://docs.mongodb.com/manual/reference/connection-string/",
+      "database": "Database name",
+      "collection": "Collection name"
+    }
+  }
+  …
+}
+```
+
+#### Environment variables
+
+Environment variables can be passed in the command-line or provided in a `.env` file at the root of the repository. See `.env.example` for an example of such a file.
+
+- `SMTP_PASSWORD`: a password for email server authentication, in order to send email notifications.
+- `SENDINBLUE_API_KEY`: a SendInBlue API key, in order to send email notifications with that service.
+- `GITHUB_TOKEN`: a token with repository privileges to access the [GitHub API](https://github.com/settings/tokens).
+
+If your infrastructure requires using an outgoing HTTP/HTTPS proxy to access the Internet, you can provide it through the `HTTP_PROXY` and `HTTPS_PROXY` environment variable.
+
+### Running
+
+To get the latest versions of all documents:
+
+```
+npm start
+```
+
+The latest version of a document will be available in the versions path defined in your configuration, under `$versions_folder/$service_provider_name/$document_type.md`.
+
+To update documents automatically:
+
+```
+npm run start:scheduler
+```
+
+To get the latest version of a specific service's terms:
+
+```
+npm start -- --services <service_id>
+```
+
+> The service ID is the case sensitive name of the service declaration file without the extension. For example, for `Twitter.json`, the service ID is `Twitter`.
+
+
+To get the latest version of a specific service's terms and document type:
+
+```
+npm start -- --services <service_id> --documentTypes <document_type>
+```
+
+To display help:
+
+```
+npm start -- --help
+```
+
+## Deploying
+
+See [Ops Readme](ops/README.md).
+
+## Publishing
+
+To generate a dataset:
+
+```
+npm run dataset:generate
+```
+
+To release a dataset:
+
+```
+npm run dataset:release
+```
+
+To weekly release a dataset:
+
+```
+npm run dataset:scheduler
+```
+
+## Contributing
+
+Thanks for wanting to contribute! There are different ways to contribute to Open Terms Archive. We describe the most common below. If you want to explore other venues for contributing, please contact us over email (contact@[our domain name]) or [Twitter](https://twitter.com/OpenTerms).
+
+### Adding a new service or updating an existing service
+
+See the [CONTRIBUTING](https://github.com/OpenTermsArchive/contrib-declarations/blob/main/CONTRIBUTING.md) of repository [`OpenTermsArchive/contrib-declarations`](https://github.com/OpenTermsArchive/contrib-declarations). You will need knowledge of JSON and web DOM.
+
+### Core engine
+
+To contribute to the core engine of Open Terms Archive, see the [CONTRIBUTING](CONTRIBUTING.md) file of this repository. You will need knowledge of JavaScript and NodeJS.
+
+### Funding and partnerships
+
+Beyond individual contributions, we need funds and committed partners to pay for a core team to maintain and grow Open Terms Archive. If you know of opportunities, please let us know! You can find [on our website](https://opentermsarchive.org/en/about) an up-to-date list of the partners and funders that make Open Terms Archive possible.
+
+
+---
+
+## License
+
+The code for this software is distributed under the European Union Public Licence (EUPL) v1.2.
+Contact the author if you have any specific need or question regarding licensing.

package/Vagrantfile

@@ -0,0 +1,38 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+  config.vm.hostname = "vagrant"
+
+  config.vm.box = "debian/bullseye64" # Unable to locate package mongodb-org
+
+  # in order to have the same config for both Docker and VirtualBox providers, we load the key manually
+  # if necessary, create the key with `ssh-keygen -f ~/.ssh/ota-vagrant -q -N ""`
+  # CAUTION: use of `~` in path causes problems with ssh
+  config.vm.provision "file", source: File.join(ENV['HOME'], ".ssh", "ota-vagrant.pub"), destination: "/home/vagrant/.ssh/authorized_keys"
+
+  # based on https://github.com/rofrano/vagrant-docker-provider#example-vagrantfile
+  config.vm.provider :docker do |docker, override|
+    override.vm.box = nil
+    docker.image = "rofrano/vagrant-provider:debian"
+    docker.remains_running = true
+    docker.has_ssh = true
+    docker.privileged = true
+    docker.volumes = ["/sys/fs/cgroup:/sys/fs/cgroup:rw"]
+    docker.create_args = ["--cgroupns=host"]
+
+    # python is not installed by default in the vagrant-provider image
+    # and deploying results in  /bin/sh: 1: /usr/bin/python: not found
+    # use a provision to fix that
+    # only with debian, no need with ubuntu
+    # Also need to name the provisioner, so that it runs only once https://github.com/hashicorp/vagrant/issues/7685#issuecomment-308281283
+    config.vm.provision "install_python3", type: "shell", inline: $installPython3
+  end
+end
+
+$installPython3 = <<-SCRIPT
+echo Updating apt...
+sudo apt-get update --fix-missing # Needed to fix "No package matching 'chromium' is available"
+echo Installing python...
+sudo apt-get --assume-yes install python3 python3-pip
+SCRIPT

package/ansible.cfg

@@ -0,0 +1,13 @@
+[defaults]
+
+inventory = ops/inventories/dev.yml
+roles_path = ops/roles
+
+
+# The two following lines allow to have human readable output
+# Use the YAML callback plugin.
+stdout_callback = yaml
+# Use the stdout_callback when running ad-hoc commands.
+bin_ansible_callbacks = true
+
+vault_password_file = vault.key

package/bin/.env.js

@@ -0,0 +1 @@
+process.env.SUPPRESS_NO_CONFIG_WARNING = 'y';

package/bin/lint-declarations.js

@@ -0,0 +1,31 @@
+#! /usr/bin/env node
+// makes it easy to lint all files relative to one service ID, which would have been
+// more difficult to achieve using an eslint based command directly defined in the package.json.
+// It also ensures that the same version of eslint is used in the OpenTermsArchive core and declarations repositories.
+import './.env.js'; // Workaround to ensure `SUPPRESS_NO_CONFIG_WARNING` is set before config is imported
+
+import fs from 'fs';
+import path from 'path';
+import { fileURLToPath, pathToFileURL } from 'url';
+
+import { program } from 'commander';
+import config from 'config';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+// Initialise configs to allow clients of this module to use it without requiring node-config in their own application.
+// see https://github.com/lorenwest/node-config/wiki/Sub-Module-Configuration
+
+config.util.setModuleDefaults('services', { declarationsPath: path.resolve(process.cwd(), './declarations') });
+const { version } = JSON.parse(fs.readFileSync(new URL('../package.json', import.meta.url)).toString());
+
+program
+  .name('ota-lint-declarations')
+  .description('Check format and stylistic errors in declarations and auto fix them')
+  .version(version)
+  .option('-s, --services [serviceId...]', 'service IDs of services to handle')
+  .option('-m, --modified', 'to only lint modified services already commited to git');
+
+const lintDeclarations = (await import(pathToFileURL(path.resolve(__dirname, '../scripts/declarations/lint/index.js')))).default;
+
+lintDeclarations(program.parse().opts());

package/bin/track.js

@@ -0,0 +1,26 @@
+#! /usr/bin/env node
+import './.env.js'; // Workaround to ensure `SUPPRESS_NO_CONFIG_WARNING` is set before config is imported
+
+import fs from 'fs';
+import path from 'path';
+import { fileURLToPath, pathToFileURL } from 'url';
+
+import config from 'config';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+const defaultConfigs = JSON.parse(fs.readFileSync(path.resolve(__dirname, '../config/default.json')));
+
+// Initialise configs to allow clients of this module to use it without requiring node-config in their own application.
+// see https://github.com/lorenwest/node-config/wiki/Sub-Module-Configuration
+config.util.setModuleDefaults('services', { declarationsPath: path.resolve(process.cwd(), './declarations') });
+config.util.setModuleDefaults('fetcher', defaultConfigs.fetcher);
+config.util.setModuleDefaults('recorder', config.util.extendDeep({}, defaultConfigs.recorder, {
+  versions: { storage: { git: { path: path.resolve(process.cwd(), './data/versions') } } },
+  snapshots: { storage: { git: { path: path.resolve(process.cwd(), './data/snapshots') } } },
+}));
+config.util.setModuleDefaults('logger', defaultConfigs.logger);
+// we do not want any tracker when launching through this command line
+config.util.setModuleDefaults('tracker', {});
+
+import(pathToFileURL(path.resolve(__dirname, '../src/main.js')));

package/bin/validate-declarations.js

@@ -0,0 +1,68 @@
+#! /usr/bin/env node
+import './.env.js'; // Workaround to ensure `SUPPRESS_NO_CONFIG_WARNING` is set before config is imported
+
+import fs from 'fs';
+import path from 'path';
+import { fileURLToPath } from 'url';
+
+import { program } from 'commander';
+import config from 'config';
+import Mocha from 'mocha';
+
+const { version } = JSON.parse(fs.readFileSync(new URL('../package.json', import.meta.url)).toString());
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+const defaultConfigs = JSON.parse(fs.readFileSync(path.resolve(__dirname, '../config/default.json')));
+
+// Initialise configs to allow clients of this module to use it without requiring node-config in their own application.
+// see https://github.com/lorenwest/node-config/wiki/Sub-Module-Configuration
+config.util.setModuleDefaults('services', { declarationsPath: path.resolve(process.cwd(), './declarations') });
+config.util.setModuleDefaults('fetcher', defaultConfigs.fetcher);
+
+const VALIDATE_PATH = path.resolve(__dirname, '../scripts/declarations/validate/index.mocha.js');
+
+// Mocha catches unhandled rejection from the user code and re-emits them to the process (see https://github.com/mochajs/mocha/blob/master/lib/runner.js#L198)
+process.on('unhandledRejection', reason => {
+  // Re-throw them so that the validation command fails in these cases (for example, if there is a syntax error when parsing JSON declaration files)
+  throw reason;
+});
+
+program
+  .name('ota-validate-declarations')
+  .description('Run a series of tests to check the validity of document declarations')
+  .version(version)
+  .option('-s, --services [serviceId...]', 'service IDs of services to handle')
+  .option('-d, --documentTypes [documentType...]', 'document types to handle')
+  .option('-m, --modified', 'to only lint modified services already commited to git')
+  .option('-so, --schema-only', 'only refilter exisiting snapshots with last declarations and engine\'s updates');
+
+const mocha = new Mocha({
+  delay: true, // as the validation script performs an asynchronous load before running the tests, the execution of the tests are delayed until run() is called
+  failZero: true, // consider that being called with no service to validate is a failure
+});
+
+(async () => {
+  mocha.addFile(VALIDATE_PATH); // As `delay` has been called, this statement will not load the file directly, `loadFilesAsync` is required.
+  await mocha.loadFilesAsync() // Load files previously added to the Mocha cache with `addFile`.
+    .catch(error => {
+      console.error(error);
+      process.exit(2);
+    });
+
+  let hasFailedTests = false;
+
+  const generateValidationTestSuite = (await import('../scripts/declarations/validate/index.mocha.js')).default;
+
+  generateValidationTestSuite(program.parse().opts());
+
+  mocha.run()
+    .on('fail', () => { hasFailedTests = true; })
+    .on('end', () => {
+      if (hasFailedTests) {
+        process.exit(1);
+      }
+
+      process.exit(0);
+    });
+})();

package/config/ci.json

@@ -0,0 +1,5 @@
+{
+  "services": {
+    "declarationsPath": "./contrib-declarations/declarations"
+  }
+}

package/config/contrib.json

@@ -0,0 +1,35 @@
+{
+  "services": {
+    "repository": "https://github.com/OpenTermsArchive/contrib-declarations.git"
+  },
+  "recorder": {
+    "versions": {
+      "storage": {
+        "git": {
+          "snapshotIdentiferTemplate": "mongo://contrib/open-terms-archive/snapshots/%SNAPSHOT_ID",
+          "repository": "git@github.com:OpenTermsArchive/contrib-versions.git"
+        }
+      }
+    },
+    "snapshots": {
+      "storage": {
+        "type": "mongo"
+      }
+    }
+  },
+  "notifier": {
+    "sendInBlue": {
+      "updatesListId": 596,
+      "updateTemplateId": 39
+    }
+  },
+  "tracker": {
+    "githubIssues": {
+      "repository": "OpenTermsArchive/contrib-declarations"
+    }
+  },
+  "dataset": {
+    "title": "contrib",
+    "versionsRepositoryURL": "https://github.com/OpenTermsArchive/contrib-versions"
+  }
+}