reffy 4.0.5 → 5.2.1

package/README.md

@@ -1,12 +1,11 @@
 # Reffy
 
+<img align="right" width="256" height="256" src="images/reffy-512.png" alt="Reffy, represented as a brave little worm with a construction helmet, ready to crawl specs">
+
 Reffy is a **Web spec crawler** tool. It is notably used to update [Webref](https://github.com/w3c/webref#webref) every 6 hours.
 
 The code features a generic crawler that can fetch Web specifications and generate machine-readable extracts out of them. Created extracts include lists of CSS properties, definitions, IDL, links and references contained in the specification.
 
-The code also currently includes a set of individual tools to study extracts and create human-readable reports (such as the [crawl report in Webref](https://w3c.github.io/webref/ed/)). Please note the on-going plan to move this part out of Reffy into a dedicated companion analysis tool (see [issue #747](https://github.com/w3c/reffy/issues/747)).
-
-
 ## How to use
 
 ### Pre-requisites
@@ -107,69 +106,7 @@ The **crawl results merger** merges a new JSON crawl report into a reference one
 
 ### Analysis tools
 
-**Note:** Plan is to move analysis tools out of Reffy's codebase into a dedicated companion analysis tool (see [issue #747](https://github.com/w3c/reffy/issues/747)).
-
-#### Study tool
-
-**Reffy's report study tool** takes the machine-readable report generated by the crawler, and creates a study report of *potential* anomalies found in the report. The study report can then easily be converted to a human-readable Markdown report. Reported potential anomalies are:
-
-1. specs that do not seem to reference any other spec normatively;
-2. specs that define WebIDL terms but do not normatively reference the WebIDL spec;
-3. specs that contain invalid WebIDL terms definitions;
-4. specs that use obsolete WebIDL constructs (e.g. `[]` instead of `FrozenArray`);
-5. specs that define WebIDL terms that are *also* defined in another spec;
-6. specs that use WebIDL terms defined in another spec without referencing that spec normatively;
-7. specs that use WebIDL terms for which the crawler could not find any definition in any of the specs it studied;
-8. specs that link to another spec but do not include a reference to that other spec;
-9. specs that link to another spec inconsistently in the body of the document and in the list of references (e.g. because the body of the document references the Editor's draft while the reference is to the latest published version).
-
-For instance:
-
-```bash
-node src/cli/study-crawl.js reports/ed/crawl.json > reports/ed/study.json.
-```
-
-#### Markdown report generator
-
-The **markdown report generator** produces a human-readable report in Markdown format out of the report returned by the study step, or directly out of the results of the crawling step. To run the generator:
-
-```bash
-node src/cli/generate-report.js reports/ed/study.json [perspec|dep]`
-```
-
-By default, the tool generates a report per anomaly, pass `perspec` to create a report per specification and `dep` to generate a dependencies report. You will probably want to redirect the output to a file, e.g. using `node src/cli/generate-report.js reports/ed/study.json > reports/ed/index.md`.
-
-The markdown report generator may also produce diff reports, e.g.:
-
-```bash
-node src/cli/generate-report.js reports/ed/study.json diff https://w3c.github.io/webref/ed/study.json
-```
-
-#### Spec checker
-
-The **spec checker** takes the URL of a spec, a reference crawl report and the name of the study report to create as inputs. It crawls and studies the given spec against the reference crawl report. Essentially, it applies the **crawler**, the **merger** and the **study** tool in order, to produces the anomalies report for the given spec. Note the URL can check multiple specs at once, provided the URLs are passed as a comma-separated value list without spaces. To run the spec checker: `node src/cli/check-specs.js [url] [reference crawl report] [study report to create]`
-
-For instance:
-
-```bash
-node src/cli/check-specs.js https://www.w3.org/TR/webstorage/ reports/ed/crawl.json reports/study-webstorage.json
-```
-
-#### Crawl and study all at once
-
-**Note:** You will need to install [Pandoc](http://pandoc.org/) for HTML report generation to succeed.
-
-To crawl all specs, generate a crawl report and an anomaly report, follow these steps:
-
-1. To produce a report using Editor's Drafts, run `npm run ed`.
-2. To produce a report using latest published versions in `/TR/`, run `npm run tr`.
-
-These commands run the `src/cli/crawl-and-study.js` script. Under the hoods, this script runs the following tools in turn:
-1. **Crawler**: crawls all specs with [Reffy](#launch-reffy)
-2. **Analysis**: Runs the [study tool](#study-tool)
-3. **Markdown report generation**: Runs the [markdown report generator](#markdown-report-generator)
-4. **Conversion to HTML**: Runs `pandoc` to prepare an HTML report with expandable sections out of the Takes the markdown report per specification. Typically runs `pandoc reports/ed/index.md -f markdown -t html5 --section-divs -s --template report-template.html -o reports/ed/index.html` (where `report.md` is the Markdown report)
-5. **Diff with latest published version of the crawl report**: Compares a crawl analysis with the latest published crawl analysis and produce a human-readable diff in Markdown format with the [markdown report generator](#markdown-report-generator)
+Starting with Reffy v5, analysis tools that used to be part of Reffy's suite of tools to study extracts and create human-readable reports of potential spec anomalies migrated to a companion tool named [Strudy](https://github.com/w3c/strudy). The actual reports get published in a separate [w3c/webref-analysis](https://github.com/w3c/webref-analysis) repository as well.
 
 
 ### WebIDL terms explorer

package/index.js

@@ -1,4 +1,7 @@
 module.exports = {
   parseIdl: require("./src/cli/parse-webidl").parse,
-  crawlSpecs: require("./src/lib/specs-crawler").crawlList
+  crawlSpecs: require("./src/lib/specs-crawler").crawlList,
+  expandCrawlResult: require("./src/lib/util").expandCrawlResult,
+  mergeCrawlResults: require("./src/lib/util").mergeCrawlResults,
+  isLatestLevelThatPasses: require("./src/lib/util").isLatestLevelThatPasses
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "reffy",
-  "version": "4.0.5",
+  "version": "5.2.1",
   "description": "W3C/WHATWG spec dependencies exploration companion. Features a short set of tools to study spec references as well as WebIDL term definitions and references found in W3C specifications.",
   "repository": {
     "type": "git",
@@ -32,40 +32,22 @@
   "bin": "./reffy.js",
   "dependencies": {
     "abortcontroller-polyfill": "1.7.3",
-    "browser-specs": "2.15.1",
-    "commander": "8.2.0",
+    "browser-specs": "2.18.0",
+    "commander": "8.3.0",
     "fetch-filecache-for-crawling": "4.0.2",
-    "node-pandoc": "0.3.0",
-    "puppeteer": "10.4.0",
+    "puppeteer": "11.0.0",
     "semver": "^7.3.5",
-    "webidl2": "24.1.2"
+    "webidl2": "24.2.0"
   },
   "devDependencies": {
     "chai": "4.3.4",
-    "mocha": "9.1.2",
-    "nock": "13.1.3",
-    "respec": "26.14.0",
+    "mocha": "9.1.3",
+    "nock": "13.2.1",
+    "respec": "28.0.6",
     "respec-hljs": "2.1.1",
-    "rollup": "2.58.0"
+    "rollup": "2.60.2"
   },
   "scripts": {
-    "all": "node src/cli/crawl-and-study.js run ed all && node src/cli/crawl-and-study.js run tr all",
-    "diff": "node src/cli/crawl-and-study.js run ed diff && node src/cli/crawl-and-study.js run tr diff",
-    "diffnew": "node src/cli/crawl-and-study.js run ed diffnew && node src/cli/crawl-and-study.js run tr diffnew",
-    "tr": "node src/cli/crawl-and-study.js run tr all",
-    "tr-crawl": "node src/cli/crawl-and-study.js run tr crawl",
-    "tr-study": "node src/cli/crawl-and-study.js run tr study",
-    "tr-markdown": "node src/cli/crawl-and-study.js run tr markdown",
-    "tr-html": "node src/cli/crawl-and-study.js run tr html",
-    "tr-diff": "node src/cli/crawl-and-study.js run tr diff",
-    "tr-diffnew": "node src/cli/crawl-and-study.js run tr diffnew",
-    "ed": "node src/cli/crawl-and-study.js run ed all",
-    "ed-crawl": "node --max-old-space-size=8192 src/cli/crawl-and-study.js run ed crawl",
-    "ed-study": "node src/cli/crawl-and-study.js run ed study",
-    "ed-markdown": "node src/cli/crawl-and-study.js run ed markdown",
-    "ed-html": "node src/cli/crawl-and-study.js run ed html",
-    "ed-diff": "node src/cli/crawl-and-study.js run ed diff",
-    "ed-diffnew": "node src/cli/crawl-and-study.js run ed diffnew",
     "test": "mocha --recursive tests/"
   }
 }

package/src/cli/parse-webidl.js

@@ -350,9 +350,7 @@ function parseType(idltype, idlReport, contextName) {
         return;
     }
     var wellKnownTypes = ["undefined", "any", "boolean", "byte", "octet", "short", "unsigned short", "long", "unsigned long", "long long", "unsigned long long", "float", "unrestricted float", "double", "unrestricted double", "DOMString", "ByteString", "USVString", "object",
-                          "RegExp", "Error", "DOMException", "ArrayBuffer", "DataView", "Int8Array", "Int16Array", "Int32Array", "Uint8Array", "Uint16Array", "Uint32Array", "Uint8ClampedArray", "Float32Array", "Float64Array",
-                          "BigUint64Array", "BigInt64Array",
-                          "ArrayBufferView", "BufferSource", "DOMTimeStamp", "Function", "VoidFunction"];
+                          "RegExp", "Error", "DOMException"];
     if (wellKnownTypes.indexOf(idltype.idlType) === -1) {
         addDependency(idltype.idlType, idlReport.idlNames, idlReport.externalDependencies);
         if (contextName) {

package/src/lib/nock-server.js

@@ -74,9 +74,9 @@ nock("https://api.specref.org")
 
 nock("https://www.w3.org")
   .persist()
-  .get("/scripts/TR/2016/fixup.js").reply(200, '')
-  .get("/StyleSheets/TR/2016/logos/W3C").reply(200, '')
-  .get("/StyleSheets/TR/2016/base.css").reply(200, '')
+  .get("/scripts/TR/2021/fixup.js").reply(200, '')
+  .get("/StyleSheets/TR/2021/logos/W3C").reply(200, '')
+  .get("/StyleSheets/TR/2021/base.css").reply(200, '')
   .get("/Tools/respec/respec-highlight").replyWithFile(200, path.join(modulesFolder, "respec-hljs", "dist", "respec-highlight.js"), {"Content-Type": "application/js"})
   .get("/Tools/respec/respec-w3c").replyWithFile(200, path.join(modulesFolder, "respec", "builds", "respec-w3c.js"), {"Content-Type": "application/js"});
 

package/src/lib/util.js

@@ -525,8 +525,8 @@ async function processSpecification(spec, processFunction, args, options) {
                 if (counter > 60) {
                     throw new Error('Respec generation took too long');
                 }
-                if (window.document.respecIsReady) {
-                    await window.document.respecIsReady;
+                if (window.document.respec?.ready) {
+                    await window.document.respec.ready;
                 }
                 else if (usesRespec) {
                     await sleep(1000);

package/src/cli/check-specs.js

@@ -1,148 +0,0 @@
-#!/usr/bin/env node
-/**
- * The spec checker crawls a spec (or a list of specs) and creates an anomalies
- * report for it (or for them). The analysis is made against a knowledge base
- * that must also be provided as input under the form of a reference crawl
- * report.
- *
- * Essentially, the spec checker runs the [spec crawler]{@link module:crawler}
- * on the given spec(s), applies the [crawl results merger]{@link module:merger}
- * to update the reference knowledge with the newly crawled results and run the
- * [crawl study]{@link module:study} tool to produce the anomalies report.
- *
- * The spec checker can be called directly through:
- *
- * `node check-specs.js [url] [ref crawl report] [study report] [option]`
- *
- * where `url` is the URL of the spec to check, or a comma-separated value list
- * (without spaces) of URLs, `ref crawl report` is the local name of the
- * reference crawl report file to use as knowledge base, `study report` is the
- * name the of the anomalies report file to create (JSON file), and `option`
- * gives the crawl options (see the spec crawler for details).
- *
- * @module checker
- */
-
-const fs = require('fs');
-const path = require('path');
-const browserSpecs = require('browser-specs');
-const requireFromWorkingDirectory = require('../lib/util').requireFromWorkingDirectory;
-const expandCrawlResult = require('../lib/util').expandCrawlResult;
-const crawlList = require('../lib/specs-crawler').crawlList;
-const mergeCrawlResults = require('./merge-crawl-results').mergeCrawlResults;
-const studyCrawl = require('./study-crawl').studyCrawl;
-
-
-/**
- * Shortcut that returns a property extractor iterator
- */
-const prop = p => x => x[p];
-
-
-/**
- * Crawl one or more specs and study them against a reference crawl report.
- *
- * The reference crawl report acts as the knowledge database. Knowledge about
- * the specs given as parameter is automatically replaced by the knowledge
- * obtained by crawling these specs.
- *
- * @function
- * @param {Array(Object)} speclist The list of specs to check. Each spec should
- *   have a "url" and/or an "html" property.
- * @param {Object} refCrawl The reference crawl report against which the specs
- *   should be checked
- * @param {Object} options Crawl options
- * @return {Promise} The promise to get the study report for the requested list
- *   of specs
- */
-async function checkSpecs(speclist, refCrawl, options) {
-    const specs = speclist.map(spec => (typeof spec === 'string') ?
-            browserSpecs.find(s => s.url === spec || s.shortname === spec) :
-            spec)
-        .filter(spec => !!spec);
-
-    const crawl = await crawlList(specs, options);
-    const report = {
-        type: 'crawl',
-        title: 'Anomalies in spec: ' + specs.map(prop('url')).join(', '),
-        description: 'Study of anomalies in the given spec against a reference crawl report',
-        date: (new Date()).toJSON(),
-        options: options,
-        stats: {
-            crawled: crawl.length,
-            errors: crawl.filter(spec => !!spec.error).length
-        },
-        results: crawl
-    };
-    const mergedReport = await mergeCrawlResults(report, refCrawl);
-    const study = await studyCrawl(mergedReport, { include: specs });
-    return study;
-}
-
-
-/**
- * Crawl the given spec and study it against a reference crawl report.
- *
- * Shortcut for the checkSpecs method when there is only one spec to check.
- *
- * @function
- * @param {Object} spec The spec to check. It should have a "url" and/or an
- *   "html" property.
- * @param {Object} refCrawl The reference crawl report against which the spec
- *   should be checked
- * @param {Object} options Crawl options
- * @return {Promise} The promise to get the study report for the requested spec
- */
-function checkSpec(spec, refCrawl, options) {
-    return checkSpecs([spec], refCrawl, options);
-}
-
-
-/**************************************************
-Export methods for use as module
-**************************************************/
-module.exports.checkSpecs = checkSpecs;
-module.exports.checkSpec = checkSpec;
-
-
-/**************************************************
-Code run if the code is run as a stand-alone module
-**************************************************/
-if (require.main === module) {
-    const specUrls = (process.argv[2] ? process.argv[2].split(',') : []);
-    const refCrawlPath = process.argv[3];
-    const resPath = process.argv[4];
-    const crawlOptions = { publishedVersion: (process.argv[5] === 'tr') };
-
-    if (specUrls.length === 0) {
-        console.error('URL(s) of the specification(s) to check must be passed as first parameter');
-        process.exit(2);
-    }
-    if (!refCrawlPath) {
-        console.error('A reference crawl results must be passed as second parameter');
-        process.exit(2);
-    }
-    if (!resPath) {
-        console.error('Result file to create must be passed as third parameter');
-        process.exit(3);
-    }
-
-    let refCrawl;
-    try {
-        refCrawl = requireFromWorkingDirectory(refCrawlPath);
-        refCrawl = expandCrawlResult(refCrawl, path.dirname(refCrawlPath));
-    } catch(e) {
-        console.error("Impossible to read " + crawlResultsPath + ": " + e);
-        process.exit(3);
-    }
-
-    checkSpecs(specUrls, refCrawl, crawlOptions)
-        .then(study => new Promise((resolve, reject) =>
-            fs.writeFile(resPath, JSON.stringify(study, null, 2),
-                         err => { if (err) return reject(err); resolve();})))
-        .then(_ => console.log('Finished'))
-        .catch(err => {
-            console.error(err);
-            process.exit(64);
-        });
-}

package/src/cli/crawl-and-study.js

@@ -1,212 +0,0 @@
-#!/usr/bin/env node
-/**
- * Reffy's command line interface that you can use to crawl and study spec
- * references. The tool runs the crawler, then the study tools to create the
- * full reports that typically show up under w3c/webref.
- *
- * Tool can be called directly through:
- *
- * `node crawl-and-study.js [command]`
- *
- * Run `node crawl-and-study.js -h` for help
- *
- * @module crawler
- */
-
-const program = require('commander');
-const version = require('../../package.json').version;
-const fs = require('fs');
-const path = require('path');
-const crawlSpecs = require('../lib/specs-crawler').crawlSpecs;
-const studyCrawl = require('./study-crawl').studyCrawl;
-const generateReport = require('./generate-report').generateReport;
-const pandoc = require('node-pandoc');
-
-
-// List of possible perspectives and associated parameters
-// Note the "ed" perspective produces reports under "whatwg" for backward
-// compatibility reason.
-const perspectives = {
-  'ed': {
-    description: 'Crawls the latest Editor\'s Drafts',
-    refStudy: 'https://w3c.github.io/webref/ed/study.json'
-  },
-  'tr': {
-    description: 'Crawls the latest published versions of specifications in /TR/ space instead of the latest Editor\'s Drafts',
-    publishedVersion: true,
-    refStudy: 'https://w3c.github.io/webref/tr/study.json'
-  }
-};
-
-// List of possible actions for each perspective
-const possibleActions = {
-  'all': 'crawl specs, study report and generate markdown, HTML and diff reports. Default action',
-  'crawl': 'crawl specs and generate a machine-readable report with facts about each spec',
-  'study': 'parse the machine-readable report generated by the crawler, and create a study report of potential anomalies found in the report',
-  'markdown': 'produce a human-readable report in Markdown format out of the anomalies report returned by the study action',
-  'html': 'produce an HTML report out of the Markdown report generated by the markdown action',
-  'diff': 'compare the anomalies report with the latest published anomalies report and generate diff report',
-  'diffnew': 'compare the anomalies report with the latest published anomalies report and generate diff report that only contains new anomalies'
-};
-
-let command = null; 
-program
-  .version(version)
-  .option('-d, --debug', 'run crawl in debug mode (single process, one spec at a time)');
-
-program
-  .command('run <perspective> [action]')
-  .description('run a new crawl and study from the given perspective')
-  .option('-d, --debug', 'run crawl in debug mode (single process, one spec at a time)')
-  .action(async (perspective, action, cmdObj) => {
-    command = 'run';
-    if (!(perspective in perspectives)) {
-      return program.help();
-    }
-    if (action && !(action in possibleActions)) {
-      return program.help();
-    }
-
-    let debug = cmdObj.debug || program.debug;
-    let publishedVersion = perspectives[perspective].publishedVersion;
-    let refStudy = perspectives[perspective].refStudy;
-    let reportFolder = perspectives[perspective].reportFolder ||
-      'reports/' + perspective;
-    let crawlReport = path.join(reportFolder, 'index.json');
-    let studyReport = path.join(reportFolder, 'study.json');
-
-    let promise = Promise.resolve();
-    let actions = (!action || (action === 'all')) ?
-      ['crawl', 'study', 'markdown', 'html', 'diff', 'diffnew'] :
-      [action];
-
-    actions.forEach(action => {
-      switch (action) {
-      case 'crawl':
-        promise = promise
-          .then(_ => crawlSpecs(
-            { publishedVersion, debug, output: reportFolder }));
-        break;
-
-      case 'study':
-        const options = {};
-        if (perspective === 'ed') {
-          const trFolder = perspectives.tr.reportFolder || 'reports/tr';
-          const trReport = path.join(trFolder, 'index.json');
-          if (fs.existsSync(trReport)) {
-            options.trResults = trReport;
-          }
-        }
-        promise = promise
-          .then(_ => studyCrawl(crawlReport, options))
-          .then(results => {
-            fs.writeFileSync(path.join(reportFolder, 'study.json'),
-              JSON.stringify(results, null, 2));
-          });
-        break;
-
-      case 'markdown':
-        promise = promise
-          .then(_ => generateReport(studyReport, { perSpec: true }))
-          .then(report => fs.writeFileSync(path.join(reportFolder, 'index.md'), report))
-          .then(_ => generateReport(studyReport, { perSpec: false }))
-          .then(report => fs.writeFileSync(path.join(reportFolder, 'perissue.md'), report));
-        break;
-
-      case 'html':
-        promise = promise
-          .then(_ => new Promise((resolve, reject) => {
-            let args = [
-              '-f', 'markdown', '-t', 'html5', '--section-divs', '-s',
-              '--template', path.join(__dirname, '..', 'templates', 'report-template.html'),
-              '-o', path.join(reportFolder, 'index.html')
-            ];
-            pandoc(path.join(reportFolder, 'index.md'), args,
-              (err => {
-                if (err) {
-                  return reject(err);
-                }
-                args = [
-                  '-f', 'markdown', '-t', 'html5', '--section-divs', '-s',
-                  '--template', path.join(__dirname, '..', 'templates', 'report-perissue-template.html'),
-                  '-o', path.join(reportFolder, 'perissue.html')];
-                pandoc(path.join(reportFolder, 'perissue.md'), args,
-                  (err => {
-                    if (err) {
-                      return reject(err);
-                    }
-                    return resolve();
-                  }));
-              }));
-            }));
-        break;
-
-      case 'diff':
-        promise = promise
-          .then(_ => generateReport(studyReport, {
-            diffReport: true,
-            refStudyFile: refStudy
-          }))
-          .then(report => fs.writeFileSync(path.join(reportFolder, 'diff.md'), report));
-        break;
-
-      case 'diffnew':
-        promise = promise
-          .then(_ => generateReport(studyReport, {
-            diffReport: true,
-            refStudyFile: refStudy,
-            onlyNew: true
-          }))
-          .then(report => fs.writeFileSync(path.join(reportFolder, 'diffnew.md'), report));
-        break;
-      }
-    });
-
-    return promise;
-  });
-
-program.on('--help', function() {
-  console.log('');
-  console.log('  Possible perspectives:');
-  console.log('');
-  Object.keys(perspectives).forEach(perspective => {
-    console.log('    ' + perspective + ': ' + perspectives[perspective].description);
-  });
-  console.log('');
-
-  console.log('  Possible actions:');
-  console.log('');
-  Object.keys(possibleActions).forEach(action => {
-    console.log('    ' + action + ': ' + possibleActions[action]);
-  });
-  console.log('');
-
-  console.log('  Possible options:');
-  console.log('');
-  console.log('    -d, --debug: run crawl in debug mode (single process, one spec at a time)');
-  console.log('');
-});
-
-program.on('command:*', function () {
-  console.error('Invalid command: %s.\n', program.args.join(' '));
-  program.outputHelp();
-  process.exit(1);
-});
-
-if (!process.argv.slice(2).length) {
-  console.error('Cannot run program without arguments.\n');
-  program.outputHelp();
-  process.exit(1);
-}
-
-program
-  .parseAsync(process.argv)
-  .then(_ => {
-    console.log('-- THE END -- ');
-    process.exit(0);
-  })
-  .catch(err => {
-    console.error('-- ERROR CAUGHT --');
-    console.error(err);
-    process.exit(1);
-  });