reffy 3.1.0 → 4.0.3

package/README.md

@@ -1,49 +1,37 @@
 # Reffy
 
-Reffy is a **Web spec crawler and analyzer** tool. It is notably used to update [Webref](https://github.com/w3c/webref#webref) every 6 hours.
+Reffy is a **Web spec crawler** tool. It is notably used to update [Webref](https://github.com/w3c/webref#webref) every 6 hours.
 
-The code features a generic crawler that can fetch Web specifications and generate machine-readable extracts out of them, and a set of individual tools to study these extracts and create human-readable reports (such as the [crawl report in Webref](https://w3c.github.io/webref/ed/)). Created extracts include lists of CSS properties, definitions, IDL, links and references contained in the specification.
+The code features a generic crawler that can fetch Web specifications and generate machine-readable extracts out of them. Created extracts include lists of CSS properties, definitions, IDL, links and references contained in the specification.
+
+The code also currently includes a set of individual tools to study extracts and create human-readable reports (such as the [crawl report in Webref](https://w3c.github.io/webref/ed/)). Please note the on-going plan to move this part out of Reffy into a dedicated companion analysis tool (see [issue #747](https://github.com/w3c/reffy/issues/747)).
 
 
 ## How to use
 
 ### Pre-requisites
 
-- To install Reffy, you need [Node.js](https://nodejs.org/en/).
-- If you want to generate HTML reports, you need to install [Pandoc](http://pandoc.org/).
+To install Reffy, you need [Node.js](https://nodejs.org/en/).
 
 ### Installation
 
-Reffy is available as an NPM package. To install, run:
-
-`npm install reffy`
-
-This should install Reffy's command-line interface tools to Node.js path.
-
-### Launch Reffy
-
-To crawl all specs, generate a crawl report and an anomaly report, follow these steps:
-
-1. To produce a report using Editor's Drafts, run `reffy run ed`.
-2. To produce a report using latest published versions in `/TR/`, run `reffy run tr`.
+Reffy is available as an NPM package. To install the package globally, run:
 
-Under the hoods, these commands run the following steps (and related commands) in turn:
-1. **Crawling**: Crawls a list of spec and outputs relevant information extracts in the specified folder. See [Specs crawler](#specs-crawler) below for details.
-2. **Analysis**: Analyses the result of the crawling step, and produces a study report. `study-crawl reports/ed/crawl.json [url]`. When the `url` parameter is given, the resulting analysis will only contain the results for the spec at that URL (multiple URLs may be given as a comma-separated value list without spaces). You will probably want to redirect the output to a file, e.g. using `study-crawl reports/ed/crawl.json > reports/ed/study.json`.
-3. **Markdown report generation**: Produces a human-readable report in Markdown format out of the report returned by the analysis step, or directly out of results of the crawling step. `generate-report reports/ed/study.json [perspec|dep]`. By default, the tool generates a report per anomaly, pass `perspec` to create a report per specification and `dep` to generate a dependencies report. You will probably want to redirect the output to a file, e.g. using `generate-report reports/ed/study.json > reports/ed/index.md`.
-4. **Conversion to HTML**: Takes the Markdown analysis per specification and prepares an HTML report with expandable sections. `pandoc reports/ed/index.md -f markdown -t html5 --section-divs -s --template report-template.html -o reports/ed/index.html` (where `report.md` is the Markdown report)
-5. **Diff with latest published version of the crawl report**: Compares a crawl analysis with the latest published crawl analysis and produce a human-readable diff in Markdown format. `generate-report reports/ed/study.json diff https://w3c.github.io/webref/ed/study.json`
+```bash
+npm install -g reffy`
+```
 
-Some notes:
+This will install Reffy as a command-line interface tool.
 
-* The crawler may take some time
-* The crawler uses a local cache for HTTP exchanges. It will create and fill a `.cache` subfolder in particular.
+The list of specs crawled by default evolves regularly. To make sure that you run the latest version, use:
 
-## Reffy's tools
+```bash
+npm update -g reffy
+```
 
-### Specs crawler
+### Launch Reffy
 
-The **Specs crawler** crawls requested specifications and runs a set of processing modules on the content fetched to create relevant extracts from each spec. Which specs get crawled, and which processing modules get run depend on how the crawler gets called. By default, the crawler crawls all specs defined in [browser-specs](https://github.com/w3c/browser-specs/) and runs all core processing modules defined in the [`browserlib`](https://github.com/w3c/reffy/tree/main/src/browserlib) folder.
+Reffy crawls requested specifications and runs a set of processing modules on the content fetched to create relevant extracts from each spec. Which specs get crawled, and which processing modules get run depend on how the crawler gets called. By default, the crawler crawls all specs defined in [browser-specs](https://github.com/w3c/browser-specs/) and runs all core processing modules defined in the [`browserlib`](https://github.com/w3c/reffy/tree/main/src/browserlib) folder.
 
 Crawl results will either be returned to the console or saved in individual files in a report folder when the `--output` parameter is set.
 
@@ -60,31 +48,68 @@ The crawler can be fully parameterized to crawl a specific list of specs and run
 
 - To extract the raw IDL defined in Fetch, run:
   ```bash
-  crawl-specs --spec fetch --module idl
+  reffy --spec fetch --module idl
   ```
 - To retrieve the list of specs that the HTML spec references, run (noting that crawling the HTML spec takes some time due to it being a multipage spec):
   ```bash
-  crawl-specs --spec html --module refs`
+  reffy --spec html --module refs`
   ```
 - To extract the list of CSS properties defined in CSS Flexible Box Layout Module Level 1, run:
   ```bash
-  crawl-specs --spec css-flexbox-1 --module css
+  reffy --spec css-flexbox-1 --module css
   ```
 - To extract the list of terms defined in WAI ARIA 1.2, run:
   ```bash
-  crawl-specs --spec wai-aria-1.2 --module dfns
+  reffy --spec wai-aria-1.2 --module dfns
   ```
 - To run an hypothetical `extract-editors.mjs` processing module and create individual spec extracts with the result of the processing under an `editors` folder for all specs in browser-specs, run:
   ```bash
-  crawl-specs --output reports/test --module editors:extract-editors.mjs
+  reffy --output reports/test --module editors:extract-editors.mjs
   ```
 
 You may add `--terse` (or `-t`) to the above commands to access the extracts directly.
 
-Run `crawl-specs -h` for a complete list of options and usage details.
+Run `reffy -h` for a complete list of options and usage details.
+
+
+Some notes:
+
+* The crawler may take a few minutes, depending on the number of specs it needs to crawl.
+* The crawler uses a local cache for HTTP exchanges. It will create and fill a `.cache` subfolder in particular.
+* If you cloned the repo instead of installing Reffy globally, replace `reffy` width `node reffy.js` in the above example to run Reffy.
+
+
+## Additional tools
+
+Additional CLI tools in the `src/cli` folder complete the main specs crawler.
+
+
+### WebIDL parser
+
+The **WebIDL parser** takes the relative path to an IDL extract and generates a JSON structure that describes WebIDL term definitions and references that the spec contains. The parser uses [WebIDL2](https://github.com/darobin/webidl2.js/) to parse the WebIDL content found in the spec. To run the WebIDL parser: `node src/cli/parse-webidl.js [idlfile]`
+
+To create the WebIDL extract in the first place, you will need to run the `idl` module in Reffy, as in:
+
+```bash
+reffy --spec fetch --module idl > fetch.idl
+```
+
 
+### WebIDL names generator
 
-### Study tool
+The **WebIDL names generator** takes the results of a crawl as input and creates a report per referenceable IDL name, that details the complete parsed IDL structure that defines the name across all specs. To run the generator: `node src/cli/generate-idlnames.js [crawl folder] [save folder]`
+
+
+### Crawl results merger
+
+The **crawl results merger** merges a new JSON crawl report into a reference one. This tool is typically useful to replace the crawl results of a given specification with the results of a new run of the crawler on that specification. To run the crawl results merger: `node src/cli/merge-crawl-results.js [new crawl report] [reference crawl report] [crawl report to create]`
+
+
+### Analysis tools
+
+**Note:** Plan is to move analysis tools out of Reffy's codebase into a dedicated companion analysis tool (see [issue #747](https://github.com/w3c/reffy/issues/747)).
+
+#### Study tool
 
 **Reffy's report study tool** takes the machine-readable report generated by the crawler, and creates a study report of *potential* anomalies found in the report. The study report can then easily be converted to a human-readable Markdown report. Reported potential anomalies are:
 
@@ -98,32 +123,63 @@ Run `crawl-specs -h` for a complete list of options and usage details.
 8. specs that link to another spec but do not include a reference to that other spec;
 9. specs that link to another spec inconsistently in the body of the document and in the list of references (e.g. because the body of the document references the Editor's draft while the reference is to the latest published version).
 
-### WebIDL terms explorer
+For instance:
 
-See the related **[WebIDLPedia](https://dontcallmedom.github.io/webidlpedia)** project and its [repo](https://github.com/dontcallmedom/webidlpedia).
+```bash
+node src/cli/study-crawl.js reports/ed/crawl.json > reports/ed/study.json.
+```
 
-### Other tools
+#### Markdown report generator
+
+The **markdown report generator** produces a human-readable report in Markdown format out of the report returned by the study step, or directly out of the results of the crawling step. To run the generator:
+
+```bash
+node src/cli/generate-report.js reports/ed/study.json [perspec|dep]`
+```
 
-Some of the tools that compose Reffy may also be used directly.
+By default, the tool generates a report per anomaly, pass `perspec` to create a report per specification and `dep` to generate a dependencies report. You will probably want to redirect the output to a file, e.g. using `node src/cli/generate-report.js reports/ed/study.json > reports/ed/index.md`.
 
-The **WebIDL parser** takes the relative path to an IDL extract and generates a JSON structure that describes WebIDL term definitions and references that the spec contains. The parser uses [WebIDL2](https://github.com/darobin/webidl2.js/) to parse the WebIDL content found in the spec. To run the WebIDL parser: `parse-webidl [idlfile]`
+The markdown report generator may also produce diff reports, e.g.:
 
-The **WebIDL names generator** takes the results of a crawl as input and creates a report per referenceable IDL name, that details the complete parsed IDL structure that defines the name across all specs. To run the generator: `generate-idlnames [crawl folder] [save folder]`
+```bash
+node src/cli/generate-report.js reports/ed/study.json diff https://w3c.github.io/webref/ed/study.json
+```
 
-The **crawl results merger** merges a new JSON crawl report into a reference one. This tool is typically useful to replace the crawl results of a given specification with the results of a new run of the crawler on that specification. To run the crawl results merger: `merge-crawl-results [new crawl report] [reference crawl report] [crawl report to create]`
+#### Spec checker
 
-The **spec checker** takes the URL of a spec, a reference crawl report and the name of the study report to create as inputs. It crawls and studies the given spec against the reference crawl report. Essentially, it applies the **crawler**, the **merger** and the **study** tool in order, to produces the anomalies report for the given spec. Note the URL can check multiple specs at once, provided the URLs are passed as a comma-separated value list without spaces. To run the spec checker: `check-specs [url] [reference crawl report] [study report to create]`
+The **spec checker** takes the URL of a spec, a reference crawl report and the name of the study report to create as inputs. It crawls and studies the given spec against the reference crawl report. Essentially, it applies the **crawler**, the **merger** and the **study** tool in order, to produces the anomalies report for the given spec. Note the URL can check multiple specs at once, provided the URLs are passed as a comma-separated value list without spaces. To run the spec checker: `node src/cli/check-specs.js [url] [reference crawl report] [study report to create]`
 
 For instance:
 
 ```bash
-parse-webidl ed/idl/fetch.idl
-check-specs https://www.w3.org/TR/webstorage/ reports/ed/crawl.json reports/study-webstorage.json
+node src/cli/check-specs.js https://www.w3.org/TR/webstorage/ reports/ed/crawl.json reports/study-webstorage.json
 ```
 
+#### Crawl and study all at once
+
+**Note:** You will need to install [Pandoc](http://pandoc.org/) for HTML report generation to succeed.
+
+To crawl all specs, generate a crawl report and an anomaly report, follow these steps:
+
+1. To produce a report using Editor's Drafts, run `npm run ed`.
+2. To produce a report using latest published versions in `/TR/`, run `npm run tr`.
+
+These commands run the `src/cli/crawl-and-study.js` script. Under the hoods, this script runs the following tools in turn:
+1. **Crawler**: crawls all specs with [Reffy](#launch-reffy)
+2. **Analysis**: Runs the [study tool](#study-tool)
+3. **Markdown report generation**: Runs the [markdown report generator](#markdown-report-generator)
+4. **Conversion to HTML**: Runs `pandoc` to prepare an HTML report with expandable sections out of the Takes the markdown report per specification. Typically runs `pandoc reports/ed/index.md -f markdown -t html5 --section-divs -s --template report-template.html -o reports/ed/index.html` (where `report.md` is the Markdown report)
+5. **Diff with latest published version of the crawl report**: Compares a crawl analysis with the latest published crawl analysis and produce a human-readable diff in Markdown format with the [markdown report generator](#markdown-report-generator)
+
+
+### WebIDL terms explorer
+
+See the related **[WebIDLPedia](https://dontcallmedom.github.io/webidlpedia)** project and its [repo](https://github.com/dontcallmedom/webidlpedia).
+
+
 ## Technical notes
 
-Reffy should be able to parse most of the W3C/WHATWG specifications that define CSS and/or WebIDL terms (both published versions and Editor's Drafts). The tool may work with other types of specs, but has not been tested with any of them.
+Reffy should be able to parse most of the W3C/WHATWG specifications that define CSS and/or WebIDL terms (both published versions and Editor's Drafts), and more generally speaking specs authored with one of [Bikeshed](https://tabatkins.github.io/bikeshed/) or [ReSpec](https://respec.org/docs/). Reffy can also parse certain IETF specs to some extent, and may work with other types of specs as well.
 
 ### List of specs to crawl
 

package/index.js

@@ -1,5 +1,4 @@
 module.exports = {
-  extractIdl: require("./src/cli/extract-webidl.js").extract,
   parseIdl: require("./src/cli/parse-webidl").parse,
-  crawlSpecs: require("./src/cli/crawl-specs").crawlList
+  crawlSpecs: require("./src/lib/specs-crawler").crawlList
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "reffy",
-  "version": "3.1.0",
+  "version": "4.0.3",
   "description": "W3C/WHATWG spec dependencies exploration companion. Features a short set of tools to study spec references as well as WebIDL term definitions and references found in W3C specifications.",
   "repository": {
     "type": "git",
@@ -29,51 +29,42 @@
     "node": ">=14"
   },
   "main": "index.js",
-  "bin": {
-    "reffy": "./reffy.js",
-    "check-specs": "./src/cli/check-specs.js",
-    "crawl-specs": "./src/cli/crawl-specs.js",
-    "generate-idlnames": "./src/cli/generate-idlnames.js",
-    "generate-report": "./src/cli/generate-report.js",
-    "merge-crawl-results": "./src/cli/merge-crawl-results.js",
-    "parse-webidl": "./src/cli/parse-webidl.js",
-    "study-crawl": "./src/cli/study-crawl.js"
-  },
+  "bin": "./reffy.js",
   "dependencies": {
     "abortcontroller-polyfill": "1.7.3",
-    "browser-specs": "2.10.0",
-    "commander": "8.1.0",
+    "browser-specs": "2.14.1",
+    "commander": "8.2.0",
     "fetch-filecache-for-crawling": "4.0.2",
     "node-pandoc": "0.3.0",
-    "puppeteer": "10.2.0",
+    "puppeteer": "10.4.0",
     "webidl2": "24.1.2"
   },
   "devDependencies": {
     "chai": "4.3.4",
-    "mocha": "9.1.1",
+    "mocha": "9.1.2",
     "nock": "13.1.3",
-    "respec": "26.13.3",
+    "respec": "26.13.4",
     "respec-hljs": "2.1.1",
-    "rollup": "2.56.3"
+    "rollup": "2.58.0"
   },
   "scripts": {
-    "all": "node reffy.js run ed all && node reffy.js run tr all",
-    "diff": "node reffy.js run ed diff && node reffy.js run tr diff",
-    "diffnew": "node reffy.js run ed diffnew && node reffy.js run tr diffnew",
-    "tr": "node reffy.js run tr all",
-    "tr-crawl": "node reffy.js run tr crawl",
-    "tr-study": "node reffy.js run tr study",
-    "tr-markdown": "node reffy.js run tr markdown",
-    "tr-html": "node reffy.js run tr html",
-    "tr-diff": "node reffy.js run tr diff",
-    "tr-diffnew": "node reffy.js run tr diffnew",
-    "ed": "node reffy.js run ed all",
-    "ed-crawl": "node --max-old-space-size=8192 reffy.js run ed crawl",
-    "ed-study": "node reffy.js run ed study",
-    "ed-markdown": "node reffy.js run ed markdown",
-    "ed-html": "node reffy.js run ed html",
-    "ed-diff": "node reffy.js run ed diff",
-    "ed-diffnew": "node reffy.js run ed diffnew",
+    "all": "node src/cli/crawl-and-study.js run ed all && node src/cli/crawl-and-study.js run tr all",
+    "diff": "node src/cli/crawl-and-study.js run ed diff && node src/cli/crawl-and-study.js run tr diff",
+    "diffnew": "node src/cli/crawl-and-study.js run ed diffnew && node src/cli/crawl-and-study.js run tr diffnew",
+    "tr": "node src/cli/crawl-and-study.js run tr all",
+    "tr-crawl": "node src/cli/crawl-and-study.js run tr crawl",
+    "tr-study": "node src/cli/crawl-and-study.js run tr study",
+    "tr-markdown": "node src/cli/crawl-and-study.js run tr markdown",
+    "tr-html": "node src/cli/crawl-and-study.js run tr html",
+    "tr-diff": "node src/cli/crawl-and-study.js run tr diff",
+    "tr-diffnew": "node src/cli/crawl-and-study.js run tr diffnew",
+    "ed": "node src/cli/crawl-and-study.js run ed all",
+    "ed-crawl": "node --max-old-space-size=8192 src/cli/crawl-and-study.js run ed crawl",
+    "ed-study": "node src/cli/crawl-and-study.js run ed study",
+    "ed-markdown": "node src/cli/crawl-and-study.js run ed markdown",
+    "ed-html": "node src/cli/crawl-and-study.js run ed html",
+    "ed-diff": "node src/cli/crawl-and-study.js run ed diff",
+    "ed-diffnew": "node src/cli/crawl-and-study.js run ed diffnew",
     "test": "mocha --recursive tests/"
   }
 }