reffy 20.0.16 → 21.0.0

package/README.md

@@ -1,151 +1,151 @@
-# Reffy
-
-<img align="right" width="256" height="256" src="images/reffy-512.png" alt="Reffy, represented as a brave little worm with a construction helmet, ready to crawl specs">
-
-Reffy is a **Web spec crawler** tool. It is notably used to update [Webref](https://github.com/w3c/webref#webref) every 6 hours.
-
-The code features a generic crawler that can fetch Web specifications and generate machine-readable extracts out of them. Created extracts include lists of CSS properties, definitions, IDL, links and references contained in the specification.
-
-## How to use
-
-### Pre-requisites
-
-To install Reffy, you need [Node.js](https://nodejs.org/en/) 20.12.1 or greater (the crawler itself may still run with earlier versions of Node.js but without any guarantee).
-
-### Installation
-
-Reffy is available as an NPM package. To install the package globally, run:
-
-```bash
-npm install -g reffy
-```
-
-This will install Reffy as a command-line interface tool.
-
-The list of specs crawled by default evolves regularly. To make sure that you run the latest version, use:
-
-```bash
-npm update -g reffy
-```
-
-### Launch Reffy
-
-Reffy crawls requested specifications and runs a set of processing modules on the content fetched to create relevant extracts from each spec. Which specs get crawled, and which processing modules get run depend on how the crawler gets called. By default, the crawler crawls all specs defined in [browser-specs](https://github.com/w3c/browser-specs/) and runs all core processing modules defined in the [`browserlib`](https://github.com/w3c/reffy/tree/main/src/browserlib) folder.
-
-Reffy can also run post-processing modules on the results of the crawl to create additional views of the data extracted from the spec during the crawl.
-
-Crawl results will either be returned to the console or saved in individual files in a report folder when the `--output` parameter is set.
-
-Examples of information that can be extracted from the specs:
-
-1. Generic information such as the title of the spec or the URL of the Editor's Draft. This information is typically copied over from [browser-specs](https://github.com/w3c/browser-specs/).
-2. The list of terms that the spec defines, in a format suitable for ingestion in cross-referencing tools such as [ReSpec](https://respec.org/xref/).
-3. The list of IDs, the list of headings and the list of links in the spec.
-4. The list of normative/informative references found in the spec.
-5. Extended information about WebIDL term definitions and references that the spec contains
-6. For CSS specs, the list of CSS properties, descriptors and value spaces that the spec defines.
-
-The crawler can be fully parameterized to crawl a specific list of specs and run a custom set of processing modules on them. For example:
-
-- To extract the raw IDL defined in Fetch, run:
-  ```bash
-  reffy --spec fetch --module idl
-  ```
-- To retrieve the list of specs that the HTML spec references, run (noting that crawling the HTML spec takes some time due to it being a multipage spec):
-  ```bash
-  reffy --spec html --module refs
-  ```
-- To extract the list of CSS properties defined in CSS Flexible Box Layout Module Level 1, run:
-  ```bash
-  reffy --spec css-flexbox-1 --module css
-  ```
-- To extract the list of terms defined in WAI ARIA 1.2, run:
-  ```bash
-  reffy --spec wai-aria-1.2 --module dfns
-  ```
-- To run an hypothetical `extract-editors.mjs` processing module and create individual spec extracts with the result of the processing under an `editors` folder for all specs in browser-specs, run:
-  ```bash
-  reffy --output reports/test --module editors:extract-editors.mjs
-  ```
-
-You may add `--terse` (or `-t`) to the above commands to access the extracts directly.
-
-Run `reffy -h` for a complete list of options and usage details.
-
-
-Some notes:
-
-* The crawler may take a few minutes, depending on the number of specs it needs to crawl.
-* The crawler uses a local cache for HTTP exchanges. It will create and fill a `.cache` subfolder in particular.
-* If you cloned the repo instead of installing Reffy globally, replace `reffy` width `node reffy.js` in the above example to run Reffy.
-
-
-## Additional tools
-
-Additional CLI tools in the `src/cli` folder complete the main specs crawler.
-
-
-### WebIDL parser
-
-The **WebIDL parser** takes the relative path to an IDL extract and generates a JSON structure that describes WebIDL term definitions and references that the spec contains. The parser uses [WebIDL2](https://github.com/darobin/webidl2.js/) to parse the WebIDL content found in the spec. To run the WebIDL parser: `node src/cli/parse-webidl.js [idlfile]`
-
-To create the WebIDL extract in the first place, you will need to run the `idl` module in Reffy, as in:
-
-```bash
-reffy --spec fetch --module idl > fetch.idl
-```
-
-
-### Crawl results merger
-
-The **crawl results merger** merges a new JSON crawl report into a reference one. This tool is typically useful to replace the crawl results of a given specification with the results of a new run of the crawler on that specification. To run the crawl results merger: `node src/cli/merge-crawl-results.js [new crawl report] [reference crawl report] [crawl report to create]`
-
-
-### Analysis tools
-
-Starting with Reffy v5, analysis tools that used to be part of Reffy's suite of tools to study extracts and create human-readable reports of potential spec anomalies migrated to a companion tool named [Strudy](https://github.com/w3c/strudy). The actual reports get published in a separate [w3c/webref-analysis](https://github.com/w3c/webref-analysis) repository as well.
-
-
-### WebIDL terms explorer
-
-See the related **[WebIDLPedia](https://dontcallmedom.github.io/webidlpedia)** project and its [repo](https://github.com/dontcallmedom/webidlpedia).
-
-
-## Technical notes
-
-Reffy should be able to parse most of the W3C/WHATWG specifications that define CSS and/or WebIDL terms (both published versions and Editor's Drafts), and more generally speaking specs authored with one of [Bikeshed](https://tabatkins.github.io/bikeshed/) or [ReSpec](https://respec.org/docs/). Reffy can also parse certain IETF specs to some extent, and may work with other types of specs as well.
-
-### List of specs to crawl
-
-Reffy crawls specs defined in [w3c/browser-specs](https://github.com/w3c/browser-specs/). If you believe a spec is missing, please check the [Spec selection criteria](https://github.com/w3c/browser-specs/#spec-selection-criteria) and create an issue (or prepare a pull request) against the [w3c/browser-specs](https://github.com/w3c/browser-specs/) repository.
-
-### Crawling a spec
-
-Given some spec info, the crawler basically goes through the following steps:
-
-1. Load the URL through Puppeteer.
-2. If the document contains a "head" section that includes a link whose label looks like "single page", go back to step 2 and load the target of that link instead. This makes the crawler load the single page version of multi-page specifications such as HTML5.
-3. If the document is a multi-page spec without a "single page" version, load the individual subpage and add their content to the bottom of the first page to create a single page version.
-4. If the document uses ReSpec, let ReSpec finish its generation work.
-5. Run internal tools on the generated document to build the relevant information.
-
-The crawler processes 4 specifications at a time. Network and parsing errors should be reported in the crawl results.
-
-### Config parameters
-
-The crawler reads parameters from the `config.json` file. Optional parameters:
-
-* `cacheRefresh`: set this flag to `never` to tell the crawler to use the cache entry for a URL directly, instead of sending a conditional HTTP request to check whether the entry is still valid. This parameter is typically useful when developing Reffy's code to work offline.
-* `resetCache`: set this flag to `true` to tell the crawler to reset the contents of the local cache when it starts.
-
-
-## Contributing
-
-Authors so far are [François Daoust](https://github.com/tidoust/) and [Dominique Hazaël-Massieux](https://github.com/dontcallmedom/).
-
-Additional ideas, bugs and/or code contributions are most welcome. Create [issues on GitHub](https://github.com/w3c/reffy/issues) as needed!
-
-
-## Licensing
-
-The code is available under an [MIT license](LICENSE).
+# Reffy
+
+<img align="right" width="256" height="256" src="images/reffy-512.png" alt="Reffy, represented as a brave little worm with a construction helmet, ready to crawl specs">
+
+Reffy is a **Web spec crawler** tool. It is notably used to update [Webref](https://github.com/w3c/webref#webref) every 6 hours.
+
+The code features a generic crawler that can fetch Web specifications and generate machine-readable extracts out of them. Created extracts include lists of CSS properties, definitions, IDL, links and references contained in the specification.
+
+## How to use
+
+### Pre-requisites
+
+To install Reffy, you need [Node.js](https://nodejs.org/en/) 22.19.0 or greater (the crawler itself may still run with earlier versions of Node.js but without any guarantee).
+
+### Installation
+
+Reffy is available as an NPM package. To install the package globally, run:
+
+```bash
+npm install -g reffy
+```
+
+This will install Reffy as a command-line interface tool.
+
+The list of specs crawled by default evolves regularly. To make sure that you run the latest version, use:
+
+```bash
+npm update -g reffy
+```
+
+### Launch Reffy
+
+Reffy crawls requested specifications and runs a set of processing modules on the content fetched to create relevant extracts from each spec. Which specs get crawled, and which processing modules get run depend on how the crawler gets called. By default, the crawler crawls all specs defined in [browser-specs](https://github.com/w3c/browser-specs/) and runs all core processing modules defined in the [`browserlib`](https://github.com/w3c/reffy/tree/main/src/browserlib) folder.
+
+Reffy can also run post-processing modules on the results of the crawl to create additional views of the data extracted from the spec during the crawl.
+
+Crawl results will either be returned to the console or saved in individual files in a report folder when the `--output` parameter is set.
+
+Examples of information that can be extracted from the specs:
+
+1. Generic information such as the title of the spec or the URL of the Editor's Draft. This information is typically copied over from [browser-specs](https://github.com/w3c/browser-specs/).
+2. The list of terms that the spec defines, in a format suitable for ingestion in cross-referencing tools such as [ReSpec](https://respec.org/xref/).
+3. The list of IDs, the list of headings and the list of links in the spec.
+4. The list of normative/informative references found in the spec.
+5. Extended information about WebIDL term definitions and references that the spec contains
+6. For CSS specs, the list of CSS properties, descriptors and value spaces that the spec defines.
+
+The crawler can be fully parameterized to crawl a specific list of specs and run a custom set of processing modules on them. For example:
+
+- To extract the raw IDL defined in Fetch, run:
+  ```bash
+  reffy --spec fetch --module idl
+  ```
+- To retrieve the list of specs that the HTML spec references, run (noting that crawling the HTML spec takes some time due to it being a multipage spec):
+  ```bash
+  reffy --spec html --module refs
+  ```
+- To extract the list of CSS properties defined in CSS Flexible Box Layout Module Level 1, run:
+  ```bash
+  reffy --spec css-flexbox-1 --module css
+  ```
+- To extract the list of terms defined in WAI ARIA 1.2, run:
+  ```bash
+  reffy --spec wai-aria-1.2 --module dfns
+  ```
+- To run an hypothetical `extract-editors.mjs` processing module and create individual spec extracts with the result of the processing under an `editors` folder for all specs in browser-specs, run:
+  ```bash
+  reffy --output reports/test --module editors:extract-editors.mjs
+  ```
+
+You may add `--terse` (or `-t`) to the above commands to access the extracts directly.
+
+Run `reffy -h` for a complete list of options and usage details.
+
+
+Some notes:
+
+* The crawler may take a few minutes, depending on the number of specs it needs to crawl.
+* The crawler uses a local cache for HTTP exchanges. It will create and fill a `.cache` subfolder in particular.
+* If you cloned the repo instead of installing Reffy globally, replace `reffy` width `node reffy.js` in the above example to run Reffy.
+
+
+## Additional tools
+
+Additional CLI tools in the `src/cli` folder complete the main specs crawler.
+
+
+### WebIDL parser
+
+The **WebIDL parser** takes the relative path to an IDL extract and generates a JSON structure that describes WebIDL term definitions and references that the spec contains. The parser uses [WebIDL2](https://github.com/darobin/webidl2.js/) to parse the WebIDL content found in the spec. To run the WebIDL parser: `node src/cli/parse-webidl.js [idlfile]`
+
+To create the WebIDL extract in the first place, you will need to run the `idl` module in Reffy, as in:
+
+```bash
+reffy --spec fetch --module idl > fetch.idl
+```
+
+
+### Crawl results merger
+
+The **crawl results merger** merges a new JSON crawl report into a reference one. This tool is typically useful to replace the crawl results of a given specification with the results of a new run of the crawler on that specification. To run the crawl results merger: `node src/cli/merge-crawl-results.js [new crawl report] [reference crawl report] [crawl report to create]`
+
+
+### Analysis tools
+
+Starting with Reffy v5, analysis tools that used to be part of Reffy's suite of tools to study extracts and create human-readable reports of potential spec anomalies migrated to a companion tool named [Strudy](https://github.com/w3c/strudy). The actual reports get published in a separate [w3c/webref-analysis](https://github.com/w3c/webref-analysis) repository as well.
+
+
+### WebIDL terms explorer
+
+See the related **[WebIDLPedia](https://dontcallmedom.github.io/webidlpedia)** project and its [repo](https://github.com/dontcallmedom/webidlpedia).
+
+
+## Technical notes
+
+Reffy should be able to parse most of the W3C/WHATWG specifications that define CSS and/or WebIDL terms (both published versions and Editor's Drafts), and more generally speaking specs authored with one of [Bikeshed](https://tabatkins.github.io/bikeshed/) or [ReSpec](https://respec.org/docs/). Reffy can also parse certain IETF specs to some extent, and may work with other types of specs as well.
+
+### List of specs to crawl
+
+Reffy crawls specs defined in [w3c/browser-specs](https://github.com/w3c/browser-specs/). If you believe a spec is missing, please check the [Spec selection criteria](https://github.com/w3c/browser-specs/#spec-selection-criteria) and create an issue (or prepare a pull request) against the [w3c/browser-specs](https://github.com/w3c/browser-specs/) repository.
+
+### Crawling a spec
+
+Given some spec info, the crawler basically goes through the following steps:
+
+1. Load the URL through Puppeteer.
+2. If the document contains a "head" section that includes a link whose label looks like "single page", go back to step 2 and load the target of that link instead. This makes the crawler load the single page version of multi-page specifications such as HTML5.
+3. If the document is a multi-page spec without a "single page" version, load the individual subpage and add their content to the bottom of the first page to create a single page version.
+4. If the document uses ReSpec, let ReSpec finish its generation work.
+5. Run internal tools on the generated document to build the relevant information.
+
+The crawler processes 4 specifications at a time. Network and parsing errors should be reported in the crawl results.
+
+### Config parameters
+
+The crawler reads parameters from the `config.json` file. Optional parameters:
+
+* `cacheRefresh`: set this flag to `never` to tell the crawler to use the cache entry for a URL directly, instead of sending a conditional HTTP request to check whether the entry is still valid. This parameter is typically useful when developing Reffy's code to work offline.
+* `resetCache`: set this flag to `true` to tell the crawler to reset the contents of the local cache when it starts.
+
+
+## Contributing
+
+Authors so far are [François Daoust](https://github.com/tidoust/) and [Dominique Hazaël-Massieux](https://github.com/dontcallmedom/).
+
+Additional ideas, bugs and/or code contributions are most welcome. Create [issues on GitHub](https://github.com/w3c/reffy/issues) as needed!
+
+
+## Licensing
+
+The code is available under an [MIT license](LICENSE).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "reffy",
-  "version": "20.0.16",
+  "version": "21.0.0",
   "description": "W3C/WHATWG spec dependencies exploration companion. Features a short set of tools to study spec references as well as WebIDL term definitions and references found in W3C specifications.",
   "repository": {
     "type": "git",
@@ -27,7 +27,7 @@
   ],
   "license": "MIT",
   "engines": {
-    "node": ">=20.18.1"
+    "node": ">=22.19.0"
   },
   "type": "module",
   "main": "index.js",
@@ -37,16 +37,16 @@
     "ajv-formats": "3.0.1",
     "commander": "14.0.3",
     "fetch-filecache-for-crawling": "5.1.1",
-    "puppeteer": "24.42.0",
+    "puppeteer": "24.43.0",
     "semver": "^7.3.5",
-    "web-specs": "3.84.0",
+    "web-specs": "3.85.0",
     "webidl2": "24.5.0"
   },
   "devDependencies": {
     "respec": "37.0.0",
     "respec-hljs": "2.1.1",
-    "rollup": "4.60.2",
-    "undici": "^7.0.0"
+    "rollup": "4.60.3",
+    "undici": "^8.2.0"
   },
   "overrides": {
     "puppeteer": "$puppeteer"

package/src/lib/mock-server.js

@@ -1,218 +1,202 @@
-/**
- * Setup a proxy server that intercepts some network requests. To be used in
- * tests not to hit the network.
- *
- * @module mock-server
- */
-
-import { MockAgent, setGlobalDispatcher } from 'undici';
-import path from 'node:path';
-import { existsSync, readFileSync } from 'node:fs';
-import { fileURLToPath } from 'node:url';
-
-const scriptPath = path.dirname(fileURLToPath(import.meta.url));
-
-/**
- * Determine the path to the "node_modules" folder. The path depends on whether
- * Reffy is run directly, or installed as a library.
- *
- * @function
- * @return {String} Path to the node_modules folder.
- */
-function getModulesFolder() {
-    const rootFolder = path.resolve(scriptPath, '../..');
-    let folder = path.resolve(rootFolder, 'node_modules');
-    if (existsSync(folder)) {
-        return folder;
-    }
-    folder = path.resolve(rootFolder, '..');
-    return folder;
-}
-const modulesFolder = getModulesFolder();
-
-const mockSpecs = {
-  "/woff/woff2/": {
-    html: `
-      <title>WOFF2</title>
-      <body>
-        <dfn id='foo' data-dfn-type="dfn">Foo</dfn>
-        <a href="https://www.w3.org/TR/bar/#baz">bar</a>
-        <ul class='toc'><li><a href='page.html'>page</a></ul>`,
-    pages: {
-      "page.html": `<h2 id='bar'>Heading in subpage</h2>`
-    }
-  },
-  "/mediacapture-output/": `
-    <script>respecConfig = { shortName: 'test' };</script>
-    <script src='https://www.w3.org/Tools/respec/respec-w3c'></script>
-    <div id=abstract></div>
-    <pre class='idl'>[Exposed=Window] interface Foo { attribute DOMString bar; };</pre>`,
-  "/accelerometer/": `<html><meta name='document-revision' content='c0917d216986f88bdd43c72c0b13352c71f283aa'>
-    <h2>Normative references</h2>
-    <dl>
-      <dt>FOO</dt>
-      <dd><a href='https://www.w3.org/TR/Foo'>Foo</a></dd>
-    </dl>`,
-  "/pointerlock/": `<html>
-    <h1>Pointer Lock 2.0`,
-  "/TR/remote-playback/": {
-    html: `<title>Published version</title>
-      <body><h1>Published version</h1></body>`,
-    domain: 'https://www.w3.org'
-  }
-};
-
-const respecHiglight = readFileSync(
-  path.join(modulesFolder, "respec-hljs", "dist", "respec-highlight.js"),
-  'utf8'
-);
-const respecW3C = readFileSync(
-  path.join(modulesFolder, "respec", "builds", "respec-w3c.js"),
-  'utf8'
-);
-
-const mockAgent = new MockAgent();
-setGlobalDispatcher(mockAgent);
-mockAgent.disableNetConnect();
-// for chrome devtool protocol
-mockAgent.enableNetConnect('127.0.0.1');
-
-for (const [path, desc] of Object.entries(mockSpecs)) {
-  mockAgent.get(desc.domain || "https://w3c.github.io")
-    .intercept({ method: "GET", path })
-    .reply(200, desc.html || desc, {
-      headers: { "Content-Type": "text/html" }
-    })
-    .persist();
-
-  for (const [page, pageContent] of Object.entries(desc.pages || {})) {
-    mockAgent.get(desc.domain || "https://w3c.github.io")
-      .intercept({ method: "GET", path: path + page })
-      .reply(200, pageContent, {
-        headers: { "Content-Type": "text/html" }
-      })
-      .persist();
-  }
-}
-
-
-// Handling requests generated by ReSpec documents
-mockAgent
-  .get("https://api.specref.org")
-  .intercept({ method: "GET", path: "/bibrefs?refs=webidl" })
-  .reply(200, { webidl: { href: "https://webidl.spec.whatwg.org/" } }, {
-    headers: {
-      "Content-Type": "application/json",
-      "Access-Control-Allow-Origin": "*"
-    }
-  })
-  .persist();
-
-mockAgent
-  .get("https://www.w3.org")
-  .intercept({ method: "GET", path: "/scripts/TR/2021/fixup.js" })
-  .reply(200, '')
-  .persist();
-
-mockAgent
-  .get("https://www.w3.org")
-  .intercept({ method: "GET", path: "/StyleSheets/TR/2021/logos/W3C" })
-  .reply(200, '')
-  .persist();
-
-mockAgent
-  .get("https://www.w3.org")
-  .intercept({ method: "GET", path: "/Tools/respec/respec-highlight" })
-  .reply(200, respecHiglight, {
-    headers: { "Content-Type": "application/js" }
-  })
-  .persist();
-
-mockAgent
-  .get("https://www.w3.org")
-  .intercept({ method: "GET", path: "/Tools/respec/respec-w3c" })
-  .reply(200, respecW3C, {
-    headers: { "Content-Type": "application/js" }
-  })
-  .persist();
-
-mockAgent
-  .get("https://www.w3.org")
-  .intercept({ method: "GET", path: "/TR/idontexist/" })
-  .reply(404, '');
-
-mockAgent
-  .get("https://www.w3.org")
-  .intercept({ method: "GET", path: "/TR/ididnotchange/" })
-  .reply(({ headers }) => {
-    // NB: Before Node.js v18.17.0, the headers parameters is not an instance
-    // of Headers as suggested in examples, but rather an array that alternates
-    // header names and header values. Bug detailed at:
-    // https://github.com/nodejs/undici/issues/2078
-    // Bug fix was integrated in Node.js v18.17.0.
-    // Code below can be simplified when support for Node.js v18 gets dropped.
-    let value;
-    if (Array.isArray(headers)) {
-      const pos = headers.findIndex(h => h === 'If-Modified-Since');
-      if (pos === -1) {
-        return { statusCode: 200, data: 'Unexpected If-Modified-Since header' };
-      }
-      value = headers[pos+1];
-    }
-    else {
-      value = headers['If-Modified-Since'];
-    }
-    if (value === "Fri, 11 Feb 2022 00:00:42 GMT") {
-      return { statusCode: 304 };
-    } else {
-      return { statusCode: 200, data: 'Unexpected If-Modified-Since header' };
-    }
-  });
-
-mockAgent
-  .get("https://www.w3.org")
-  .intercept({ method: "GET", path: "/TR/iredirect/" })
-  .reply(200,
-    `<!DOCTYPE html><script>window.location = '/TR/recentlyupdated/';</script>`,
-    {
-      headers: {
-        "Content-Type": "text/html",
-        "Last-Modified": "Fri, 11 Feb 2022 00:00:42 GMT"
-      }
-    }
-  );
-
-mockAgent
-  .get("https://www.w3.org")
-  .intercept({ method: "GET", path: "/TR/recentlyupdated/" })
-  .reply(200,
-    `<html><title>Recently updated</title>
-    <h1>Recently updated</h1>`,
-    {
-      headers: {
-        "Content-Type": "text/html",
-        "Last-Modified": (new Date()).toString()
-      }
-    }
-  );
-
-mockAgent
-  .get("https://drafts.csswg.org")
-  .intercept({ method: "GET", path: "/server-hiccup/" })
-  .reply(200,
-    `<html><title>Server hiccup</title>
-    <h1> Index of Server Hiccup Module Level 42 </h1>`,
-    { headers: { "Content-Type": "text/html" } })
-  .persist();
-
-/*nock.emitter.on('error', function (err) {
-  console.error(err);
-});
-nock.emitter.on('no match', function(req, options, requestBody) {
-  // 127.0.0.1 is used by the devtool protocol, we ignore it
-  if (req && req.hostname !== '127.0.0.1') {
-    console.error("No match for nock request on " + (options ? options.href : req.href));
-  }
-});*/
-
-export default mockAgent;
+/**
+ * Setup a proxy server that intercepts some network requests. To be used in
+ * tests not to hit the network.
+ *
+ * @module mock-server
+ */
+
+import { MockAgent, setGlobalDispatcher, install } from 'undici';
+import path from 'node:path';
+import { existsSync, readFileSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+
+const scriptPath = path.dirname(fileURLToPath(import.meta.url));
+
+/**
+ * Determine the path to the "node_modules" folder. The path depends on whether
+ * Reffy is run directly, or installed as a library.
+ *
+ * @function
+ * @return {String} Path to the node_modules folder.
+ */
+function getModulesFolder() {
+    const rootFolder = path.resolve(scriptPath, '../..');
+    let folder = path.resolve(rootFolder, 'node_modules');
+    if (existsSync(folder)) {
+        return folder;
+    }
+    folder = path.resolve(rootFolder, '..');
+    return folder;
+}
+const modulesFolder = getModulesFolder();
+
+const mockSpecs = {
+  "/woff/woff2/": {
+    html: `
+      <title>WOFF2</title>
+      <body>
+        <dfn id='foo' data-dfn-type="dfn">Foo</dfn>
+        <a href="https://www.w3.org/TR/bar/#baz">bar</a>
+        <ul class='toc'><li><a href='page.html'>page</a></ul>`,
+    pages: {
+      "page.html": `<h2 id='bar'>Heading in subpage</h2>`
+    }
+  },
+  "/mediacapture-output/": `
+    <script>respecConfig = { shortName: 'test' };</script>
+    <script src='https://www.w3.org/Tools/respec/respec-w3c'></script>
+    <div id=abstract></div>
+    <pre class='idl'>[Exposed=Window] interface Foo { attribute DOMString bar; };</pre>`,
+  "/accelerometer/": `<html><meta name='document-revision' content='c0917d216986f88bdd43c72c0b13352c71f283aa'>
+    <h2>Normative references</h2>
+    <dl>
+      <dt>FOO</dt>
+      <dd><a href='https://www.w3.org/TR/Foo'>Foo</a></dd>
+    </dl>`,
+  "/pointerlock/": `<html>
+    <h1>Pointer Lock 2.0`,
+  "/TR/remote-playback/": {
+    html: `<title>Published version</title>
+      <body><h1>Published version</h1></body>`,
+    domain: 'https://www.w3.org'
+  }
+};
+
+const respecHiglight = readFileSync(
+  path.join(modulesFolder, "respec-hljs", "dist", "respec-highlight.js"),
+  'utf8'
+);
+const respecW3C = readFileSync(
+  path.join(modulesFolder, "respec", "builds", "respec-w3c.js"),
+  'utf8'
+);
+
+install();
+const mockAgent = new MockAgent();
+setGlobalDispatcher(mockAgent);
+mockAgent.disableNetConnect();
+// for chrome devtool protocol
+mockAgent.enableNetConnect('127.0.0.1');
+
+for (const [path, desc] of Object.entries(mockSpecs)) {
+  mockAgent.get(desc.domain || "https://w3c.github.io")
+    .intercept({ method: "GET", path })
+    .reply(200, desc.html || desc, {
+      headers: { "Content-Type": "text/html" }
+    })
+    .persist();
+
+  for (const [page, pageContent] of Object.entries(desc.pages || {})) {
+    mockAgent.get(desc.domain || "https://w3c.github.io")
+      .intercept({ method: "GET", path: path + page })
+      .reply(200, pageContent, {
+        headers: { "Content-Type": "text/html" }
+      })
+      .persist();
+  }
+}
+
+
+// Handling requests generated by ReSpec documents
+mockAgent
+  .get("https://api.specref.org")
+  .intercept({ method: "GET", path: "/bibrefs?refs=webidl" })
+  .reply(200, { webidl: { href: "https://webidl.spec.whatwg.org/" } }, {
+    headers: {
+      "Content-Type": "application/json",
+      "Access-Control-Allow-Origin": "*"
+    }
+  })
+  .persist();
+
+mockAgent
+  .get("https://www.w3.org")
+  .intercept({ method: "GET", path: "/scripts/TR/2021/fixup.js" })
+  .reply(200, '')
+  .persist();
+
+mockAgent
+  .get("https://www.w3.org")
+  .intercept({ method: "GET", path: "/StyleSheets/TR/2021/logos/W3C" })
+  .reply(200, '')
+  .persist();
+
+mockAgent
+  .get("https://www.w3.org")
+  .intercept({ method: "GET", path: "/Tools/respec/respec-highlight.js" })
+  .reply(200, respecHiglight, {
+    headers: { "Content-Type": "application/js" }
+  })
+  .persist();
+
+mockAgent
+  .get("https://www.w3.org")
+  .intercept({ method: "GET", path: "/Tools/respec/respec-w3c" })
+  .reply(200, respecW3C, {
+    headers: { "Content-Type": "application/js" }
+  })
+  .persist();
+
+mockAgent
+  .get("https://www.w3.org")
+  .intercept({ method: "GET", path: "/TR/idontexist/" })
+  .reply(404, '');
+
+mockAgent
+  .get("https://www.w3.org")
+  .intercept({ method: "GET", path: "/TR/ididnotchange/" })
+  .reply(({ headers }) => {
+    if (headers['If-Modified-Since'] === "Fri, 11 Feb 2022 00:00:42 GMT") {
+      return { statusCode: 304 };
+    } else {
+      return { statusCode: 200, data: 'Unexpected If-Modified-Since header' };
+    }
+  });
+
+mockAgent
+  .get("https://www.w3.org")
+  .intercept({ method: "GET", path: "/TR/iredirect/" })
+  .reply(200,
+    `<!DOCTYPE html><script>window.location = '/TR/recentlyupdated/';</script>`,
+    {
+      headers: {
+        "Content-Type": "text/html",
+        "Last-Modified": "Fri, 11 Feb 2022 00:00:42 GMT"
+      }
+    }
+  );
+
+mockAgent
+  .get("https://www.w3.org")
+  .intercept({ method: "GET", path: "/TR/recentlyupdated/" })
+  .reply(200,
+    `<html><title>Recently updated</title>
+    <h1>Recently updated</h1>`,
+    {
+      headers: {
+        "Content-Type": "text/html",
+        "Last-Modified": (new Date()).toString()
+      }
+    }
+  );
+
+mockAgent
+  .get("https://drafts.csswg.org")
+  .intercept({ method: "GET", path: "/server-hiccup/" })
+  .reply(200,
+    `<html><title>Server hiccup</title>
+    <h1> Index of Server Hiccup Module Level 42 </h1>`,
+    { headers: { "Content-Type": "text/html" } })
+  .persist();
+
+/*nock.emitter.on('error', function (err) {
+  console.error(err);
+});
+nock.emitter.on('no match', function(req, options, requestBody) {
+  // 127.0.0.1 is used by the devtool protocol, we ignore it
+  if (req && req.hostname !== '127.0.0.1') {
+    console.error("No match for nock request on " + (options ? options.href : req.href));
+  }
+});*/
+
+export default mockAgent;