@markuplint/spec-generator 4.8.0 → 4.8.2

package/docs/scraping.md

@@ -0,0 +1,320 @@
+# Scraping Details
+
+This document describes the web scraping targets, CSS selectors, caching strategy, and error handling used by `@markuplint/spec-generator`. The build is network-dependent, issuing 200+ HTTP requests to MDN and W3C specifications.
+
+## MDN Element Scraping
+
+**Module:** `scraping.ts`
+
+### URL Patterns
+
+HTML elements:
+
+```
+https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/<name>
+```
+
+SVG elements:
+
+```
+https://developer.mozilla.org/en-US/docs/Web/SVG/Reference/Element/<name>
+```
+
+**Special case:** Heading elements (`h1`-`h6`) are mapped to a single page:
+
+```
+https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/Heading_Elements
+```
+
+### Extracted Data
+
+For each element, `fetchHTMLElement()` extracts:
+
+| Data          | Selector / Method                                                            | Notes                                                              |
+| ------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------ |
+| Description   | `main#content .reference-layout__header .content-section`                    | Text content, whitespace-normalized                                |
+| Compatibility | `.bc-table tbody tr:first-child th` (icons)                                  | Falls back to notecard-based indicators if BC table is unavailable |
+| Categories    | `#technical_summary ~ figure.table-container > table` ("Content categories") | Matched against known category keywords                            |
+| Attributes    | `.content-section[aria-labelledby="<id>"] > dl > dt`                         | Parsed from definition lists in multiple sections                  |
+
+### Compatibility Flag Detection
+
+Two strategies are used, depending on whether the browser compatibility table is available:
+
+**Strategy 1: Browser Compatibility Table** (when `<code>` in the first row matches the element name)
+
+| Flag           | Selector within `tbody tr:first-child th` |
+| -------------- | ----------------------------------------- |
+| `experimental` | `.ic-experimental`                        |
+| `obsolete`     | `.ic-obsolete`                            |
+| `deprecated`   | `.ic-deprecated`                          |
+| `nonStandard`  | `.ic-non-standard`                        |
+
+**Strategy 2: Fallback indicators** (when BC table is missing or doesn't match)
+
+| Flag           | Selector                                                                                         |
+| -------------- | ------------------------------------------------------------------------------------------------ |
+| `experimental` | `.blockIndicator.experimental` or `> div .notecard.experimental`                                 |
+| `obsolete`     | `.obsoleteHeader` or `h1` text contains "obsolete" or `> div:first-child .notecard.obsolete`     |
+| `deprecated`   | `.deprecatedHeader` or `> div:first-child .notecard.deprecated` or `h1 + * .notecard.deprecated` |
+| `nonStandard`  | `.nonStandardHeader` or `h4#Non-standard`                                                        |
+
+### Content Category Parsing
+
+The "Content categories" property is extracted from the technical summary table. The text is matched against these keywords (case-insensitive):
+
+| Keyword               | Category             |
+| --------------------- | -------------------- |
+| `metadata content`    | `#metadata`          |
+| `flow content`        | `#flow`              |
+| `sectioning content`  | `#sectioning`        |
+| `heading content`     | `#heading`           |
+| `phrasing content`    | `#phrasing`          |
+| `embedded content`    | `#embedded`          |
+| `interactive content` | `#interactive`       |
+| `palpable content`    | `#palpable`          |
+| `script-supporting`   | `#script-supporting` |
+
+### Attribute Extraction
+
+Attributes are extracted from up to 5 sections identified by `aria-labelledby` IDs:
+
+| Section ID                | Status Flags Applied            |
+| ------------------------- | ------------------------------- |
+| `attributes`              | Per-attribute flags from icons  |
+| `deprecated_attributes`   | `deprecated: true` from heading |
+| `individual_attributes`   | Per-attribute flags from icons  |
+| `non-standard_attributes` | Per-attribute flags from icons  |
+| `obsolete_attributes`     | `obsolete: true` from heading   |
+
+For each `<dt>` entry:
+
+1. Extract attribute name from `<code>` text
+2. Extract description from the next `<dd>` sibling(s)
+3. Detect status flags from icon classes:
+   - `.icon-beaker`, `.icon.experimental`, `.icon.icon-experimental` -- experimental
+   - `.icon-trash`, `.icon.obsolete`, `.icon.icon-obsolete`, `.obsolete` -- obsolete
+   - `.icon-thumbs-down-alt`, `.icon.deprecated`, `.icon.icon-deprecated` -- deprecated
+   - `.icon-warning-sign`, `.icon.non-standard`, `.icon.icon-nonstandard` -- non-standard
+4. Check heading context (`getItsHeading()`) for section-level flags
+
+All extracted attributes are merged and sorted by key.
+
+---
+
+## MDN SVG Index Scraping
+
+**Module:** `svg.ts`
+
+### Target
+
+```
+https://developer.mozilla.org/en-US/docs/Web/SVG/Element
+```
+
+### Extraction Process
+
+1. Unwrap all `<section>` elements (replace with children) to flatten the document structure
+2. Find the heading with `id="obsolete_and_deprecated_elements"`
+3. Use `getThisOutline()` to collect all siblings until the next `<h2>`
+4. Extract element names from `div > a` elements, stripping angle brackets
+5. Prefix each name with `svg_` (e.g., `altGlyph` becomes `svg_altGlyph`)
+
+---
+
+## WAI-ARIA Scraping
+
+**Module:** `aria.ts`
+
+### Specification URLs
+
+| Version | URL                                   | Status         |
+| ------- | ------------------------------------- | -------------- |
+| 1.1     | `https://www.w3.org/TR/wai-aria-1.1/` | Recommendation |
+| 1.2     | `https://www.w3.org/TR/wai-aria-1.2/` | Recommendation |
+| 1.3     | `https://w3c.github.io/aria/`         | Working Draft  |
+
+### Role Extraction
+
+**Selector:** `#role_definitions section.role`
+
+For each role section:
+
+| Data                         | Selector                                            |
+| ---------------------------- | --------------------------------------------------- |
+| Name                         | `.role-name[title]`                                 |
+| Description                  | `.role-description p` (joined with `\n\n`)          |
+| Is Abstract                  | `.role-abstract` text equals "true"                 |
+| Generalization               | `.role-parent a`                                    |
+| Required Properties          | `.role-required-properties li` (fallback to parent) |
+| Inherited Properties         | `.role-inherited li`                                |
+| Owned Properties             | `.role-properties li` or `.role-properties > a`     |
+| Required Context Role        | `.role-scope li` or `.role-scope a`                 |
+| Required Owned Elements      | `.role-mustcontain li` or `.role-mustcontain a`     |
+| Accessible Name Required     | `.role-namerequired` contains "true"                |
+| Accessible Name From Author  | `.role-namefrom` contains "author"                  |
+| Accessible Name From Content | `.role-namefrom` contains "content"                 |
+| Accessible Name Prohibited   | `.role-namefrom` contains "prohibited"              |
+| Children Presentational      | `.role-childpresentational` "true"/"false"          |
+| Prohibited Properties        | `.role-disallowed li code`                          |
+
+**Role synonym handling:**
+
+- ARIA 1.1/1.2: `none` inherits properties from `presentation`
+- ARIA 1.3: `presentation` inherits from `none`; `img` inherits from `image`
+
+### Property/State Extraction
+
+Properties are discovered from the `ownedProperties` of all scraped roles. For each property:
+
+**Selector base:** `#<property-name>` (e.g., `#aria-label`)
+
+| Data               | Selector                                                                               |
+| ------------------ | -------------------------------------------------------------------------------------- |
+| Type               | Section class: `/property/i` matches → `"property"`, else `"state"`                    |
+| Deprecated         | Section class contains "deprecated"                                                    |
+| Value type         | `table .${type}-value` or `table .property-value` or `.state-features .property-value` |
+| Value descriptions | `table:is(.value-descriptions, .def:has(.value-description)) tbody tr`                 |
+| Enum values        | From `.value-name` elements (only for `token` or `token list` value types)             |
+| Default value      | `.value-name .default` text                                                            |
+| Is Global          | Listed in `#global_states li a`                                                        |
+
+**Conditional value overrides:**
+
+- `aria-checked`: Value set to `"true/false"` with conditional `"tristate"` for `checkbox` and `menuitemcheckbox` roles
+- `aria-hidden`: The `hidden` HTML attribute equivalent is marked as `isNotStrictEquivalent`
+
+### Global States/Properties
+
+Global ARIA attributes are identified by collecting all `<a>` links under `#global_states li`. The hash fragment of each link is used as the property name.
+
+---
+
+## Graphics ARIA Scraping
+
+**Module:** `aria.ts`
+
+Graphics ARIA roles are fetched using the same `getRoles()` function with `graphicsAria = true`.
+
+| Version | URL                                        |
+| ------- | ------------------------------------------ |
+| 1.1     | `https://www.w3.org/TR/graphics-aria-1.0/` |
+| 1.2     | `https://w3c.github.io/graphics-aria/`     |
+| 1.3     | `https://w3c.github.io/graphics-aria/`     |
+
+The same CSS selectors used for standard ARIA roles apply to Graphics ARIA roles.
+
+---
+
+## HTML-ARIA Mapping
+
+**Module:** `aria.ts` (`getAriaInHtml()`)
+
+### Target
+
+```
+https://www.w3.org/TR/html-aria/
+```
+
+### Selector
+
+```
+#requirements-for-use-of-aria-attributes-in-place-of-equivalent-html-attributes table tbody tr
+```
+
+For each row:
+
+- HTML attribute name: `th:nth-of-type(1) a` (first link text)
+- Implicit ARIA property: `td:nth-of-type(1) code` (first code element text)
+- The property string is split on `=` to get the ARIA property name and value
+
+**Skipped:** The `contenteditable` attribute is excluded because it requires ancestor evaluation.
+
+---
+
+## Caching
+
+### In-Process Cache
+
+Two `Map` caches exist in `fetch.ts`:
+
+| Cache      | Key | Value           | Scope                               |
+| ---------- | --- | --------------- | ----------------------------------- |
+| `cache`    | URL | Raw HTML string | Single build run (process lifetime) |
+| `domCache` | URL | `CheerioAPI`    | Single build run (process lifetime) |
+
+- The same URL is never fetched twice within a single build
+- Failed fetches are cached as empty strings, preventing retry
+- There is **no persistence between builds** -- every `yarn up:gen` fetches all URLs fresh
+
+### Cache Behavior on Failure
+
+When `globalThis.fetch()` throws:
+
+1. An empty string is cached for that URL
+2. The build continues (does not abort)
+3. Elements whose pages failed to fetch will have empty/missing metadata
+
+---
+
+## Error Handling
+
+| Scenario              | Behavior                                                            |
+| --------------------- | ------------------------------------------------------------------- |
+| HTTP fetch failure    | Empty string cached, build continues, metadata is empty             |
+| Missing DOM element   | Cheerio returns empty selection, fields default to empty            |
+| MDN page restructured | CSS selectors fail silently, data is lost for affected elements     |
+| W3C spec URL changed  | Fetch returns error page HTML, scraping extracts garbage or nothing |
+
+The generator does not validate scraped data against expected shapes. Incorrect or missing data will propagate silently into `index.json`.
+
+---
+
+## Known Fragile Points
+
+These CSS selectors are sensitive to upstream page structure changes:
+
+### MDN Pages
+
+| Selector                                                        | Used For            | Risk Level |
+| --------------------------------------------------------------- | ------------------- | ---------- |
+| `main#content`                                                  | Main article        | Low        |
+| `.reference-layout__header .content-section`                    | Description         | Medium     |
+| `.bc-table tbody tr:first-child th`                             | Compatibility flags | Medium     |
+| `#technical_summary ~ figure.table-container > table`           | Technical summary   | High       |
+| `.content-section[aria-labelledby="attributes"]`                | Attribute section   | Medium     |
+| `.icon-beaker`, `.icon.experimental`, `.icon.icon-experimental` | Experimental flag   | High       |
+| `.icon-trash`, `.icon.obsolete`, `.icon.icon-obsolete`          | Obsolete flag       | High       |
+
+### W3C ARIA Spec Pages
+
+| Selector                         | Used For            | Risk Level |
+| -------------------------------- | ------------------- | ---------- |
+| `#role_definitions section.role` | Role sections       | Low        |
+| `.role-name[title]`              | Role name           | Low        |
+| `.role-required-properties li`   | Required properties | Low        |
+| `.role-properties li`            | Owned properties    | Low        |
+| `#global_states li a`            | Global properties   | Low        |
+
+W3C specs use more structured class names, making them less likely to change than MDN selectors.
+
+---
+
+## Diagnosing Scraping Failures
+
+After running `yarn up:gen`, check the diff of `index.json`:
+
+```bash
+git diff packages/@markuplint/html-spec/index.json
+```
+
+**Symptoms of scraping failure:**
+
+- **Massive data loss** -- Large chunks of specification data disappear from `index.json`. This almost certainly indicates a scraping failure, not an actual spec change.
+- **Empty descriptions** -- Multiple elements suddenly have empty `description` fields
+- **Missing attributes** -- Attributes that were previously present are gone
+- **Empty ARIA data** -- Role or property definitions are empty or significantly reduced
+
+**Root cause:** The referenced site (MDN, W3C) has changed its HTML structure, information layout, or element IDs/classes.
+
+**Resolution:** Identify which module's CSS selectors are broken by inspecting the actual page structure, then update the selectors in `scraping.ts` or `aria.ts` to match the new structure. Re-run `yarn up:gen` to verify.

package/lib/aria.d.ts

@@ -1,4 +1,10 @@
 import type { ARIAProperty, ARIARoleInSchema } from '@markuplint/ml-spec';
+/**
+ * Fetches and assembles the complete ARIA specification data for all supported versions (1.1, 1.2, 1.3).
+ * For each version, gathers roles, properties/states, and graphics ARIA roles by scraping the W3C specs.
+ *
+ * @returns An object keyed by ARIA version, each containing `roles`, `props`, and `graphicsRoles`
+ */
 export declare function getAria(): Promise<{
     '1.3': {
         roles: ARIARoleInSchema[];

package/lib/aria.js

@@ -1,5 +1,11 @@
 import { fetch } from './fetch.js';
 import { arrayUnique, nameCompare } from './utils.js';
+/**
+ * Fetches and assembles the complete ARIA specification data for all supported versions (1.1, 1.2, 1.3).
+ * For each version, gathers roles, properties/states, and graphics ARIA roles by scraping the W3C specs.
+ *
+ * @returns An object keyed by ARIA version, each containing `roles`, `props`, and `graphicsRoles`
+ */
 export async function getAria() {
     const roles13 = await getRoles('1.3');
     const roles12 = await getRoles('1.2');
@@ -22,6 +28,13 @@ export async function getAria() {
         },
     };
 }
+/**
+ * Returns the URL of the WAI-ARIA specification for a given version.
+ *
+ * @param version - The ARIA specification version
+ * @param graphicsAria - Whether to return the Graphics ARIA module URL instead
+ * @returns The specification URL string
+ */
 function getARIASpecURLByVersion(version, graphicsAria = false) {
     switch (version) {
         case '1.3': {
@@ -44,6 +57,15 @@ function getARIASpecURLByVersion(version, graphicsAria = false) {
         }
     }
 }
+/**
+ * Scrapes ARIA role definitions from the W3C specification page for a given version.
+ * Extracts role metadata including generalization, owned properties, required context,
+ * accessible name requirements, and more. Handles role synonyms (e.g., `none`/`presentation`).
+ *
+ * @param version - The ARIA specification version to scrape
+ * @param graphicsAria - Whether to scrape the Graphics ARIA module instead
+ * @returns A sorted array of ARIA role schema objects
+ */
 async function getRoles(version, graphicsAria = false) {
     const $ = await fetch(getARIASpecURLByVersion(version, graphicsAria));
     const $roleList = $('#role_definitions section.role');
@@ -178,6 +200,15 @@ async function getRoles(version, graphicsAria = false) {
     roles.sort(nameCompare);
     return roles;
 }
+/**
+ * Scrapes ARIA properties and states from the specification for a given version.
+ * Builds a list of all ARIA properties referenced by the provided roles, enriches each
+ * with value types, enum values, default values, global status, and equivalent HTML attributes.
+ *
+ * @param version - The ARIA specification version to scrape
+ * @param roles - The array of role definitions used to discover which properties exist
+ * @returns A sorted array of ARIA property definitions
+ */
 async function getProps(version, roles) {
     const $ = await fetch(getARIASpecURLByVersion(version));
     const ariaNameList = new Set();
@@ -278,6 +309,12 @@ async function getProps(version, roles) {
     arias.sort(nameCompare);
     return arias;
 }
+/**
+ * Scrapes the W3C HTML-ARIA specification to extract the mapping between
+ * HTML attributes and their equivalent implicit ARIA properties.
+ *
+ * @returns An object containing `implicitProps`, an array of mappings from HTML attribute names to ARIA property names and values
+ */
 async function getAriaInHtml() {
     const $ = await fetch('https://www.w3.org/TR/html-aria/');
     const implicitProps = [];
@@ -306,6 +343,14 @@ async function getAriaInHtml() {
         implicitProps,
     };
 }
+/**
+ * Tries multiple CSS selectors on a Cheerio element and returns the first non-empty match.
+ * Falls back to the last selector's result if none match.
+ *
+ * @param $el - The Cheerio element to search within
+ * @param selectors - An ordered list of CSS selectors to try
+ * @returns The Cheerio selection from the first matching selector, or the last selector's (empty) result
+ */
 function $$(
 // eslint-disable-next-line @typescript-eslint/prefer-readonly-parameter-types
 $el, selectors) {

package/lib/fetch.d.ts

@@ -1,4 +1,25 @@
 import * as cheerio from 'cheerio';
+/**
+ * Fetches a URL and returns a parsed Cheerio DOM instance.
+ * Results are cached so subsequent calls with the same URL avoid re-fetching and re-parsing.
+ *
+ * @param url - The URL to fetch and parse as HTML
+ * @returns A Cheerio API instance for querying the fetched document
+ */
 export declare function fetch(url: string): Promise<cheerio.CheerioAPI>;
+/**
+ * Fetches the raw text content of a URL.
+ * Results are cached so repeated requests for the same URL return the cached response.
+ * Updates the CLI progress bar on each call.
+ *
+ * @param url - The URL to fetch
+ * @returns The raw text content of the HTTP response, or an empty string on failure
+ */
 export declare function fetchText(url: string): Promise<string>;
+/**
+ * Finalizes the fetch progress bar and returns a sorted list of all URLs that were fetched.
+ * Should be called after all fetch operations are complete.
+ *
+ * @returns A sorted array of all fetched URL strings (used as reference citations)
+ */
 export declare function getReferences(): string[];

package/lib/fetch.js

@@ -1,6 +1,12 @@
 import * as cheerio from 'cheerio';
 import { Bar, Presets } from 'cli-progress';
+/**
+ * In-memory cache mapping URLs to their raw HTML text responses.
+ */
 const cache = new Map();
+/**
+ * In-memory cache mapping URLs to their parsed Cheerio DOM instances.
+ */
 const domCache = new Map();
 let total = 1;
 let current = 0;
@@ -8,6 +14,13 @@ const bar = new Bar({
     format: '🔎 Fetch references... {bar} {percentage}% | ETA: {eta}s | {value}/{total} {process}',
 }, Presets.shades_grey);
 bar.start(total, current, { process: '🚀 Started.' });
+/**
+ * Fetches a URL and returns a parsed Cheerio DOM instance.
+ * Results are cached so subsequent calls with the same URL avoid re-fetching and re-parsing.
+ *
+ * @param url - The URL to fetch and parse as HTML
+ * @returns A Cheerio API instance for querying the fetched document
+ */
 export async function fetch(url) {
     if (domCache.has(url)) {
         return domCache.get(url);
@@ -17,10 +30,18 @@ export async function fetch(url) {
     domCache.set(url, $);
     return $;
 }
+/**
+ * Fetches the raw text content of a URL.
+ * Results are cached so repeated requests for the same URL return the cached response.
+ * Updates the CLI progress bar on each call.
+ *
+ * @param url - The URL to fetch
+ * @returns The raw text content of the HTTP response, or an empty string on failure
+ */
 export async function fetchText(url) {
     total += 1;
     bar.setTotal(total);
-    let text = '';
+    let text;
     if (cache.has(url)) {
         text = cache.get(url);
     }
@@ -39,6 +60,12 @@ export async function fetchText(url) {
     bar.update(current, { process: `🔗 ${url.length > 30 ? `${url.slice(0, 15)}...${url.slice(-15)}` : url}` });
     return text;
 }
+/**
+ * Finalizes the fetch progress bar and returns a sorted list of all URLs that were fetched.
+ * Should be called after all fetch operations are complete.
+ *
+ * @returns A sorted array of all fetched URL strings (used as reference citations)
+ */
 export function getReferences() {
     current += 1;
     bar.update(current, { process: '🎉 Finished.' });

package/lib/global-attrs.d.ts

@@ -1,3 +1,9 @@
+/**
+ * Reads the global HTML attributes definition from a JSON file.
+ *
+ * @param filePath - The absolute path to the common attributes JSON file
+ * @returns The parsed global attributes specification object
+ */
 export declare function getGlobalAttrs(filePath: string): {
     readonly [category: string]: Readonly<Record<string, Partial<import("@markuplint/ml-spec").Attribute>>>;
 };

package/lib/global-attrs.js

@@ -1,4 +1,10 @@
 import { readJson } from './read-json.js';
+/**
+ * Reads the global HTML attributes definition from a JSON file.
+ *
+ * @param filePath - The absolute path to the common attributes JSON file
+ * @returns The parsed global attributes specification object
+ */
 export function getGlobalAttrs(filePath) {
     const gAttrs = readJson(filePath);
     return gAttrs;

package/lib/html-elements.d.ts

@@ -1,2 +1,10 @@
 import type { ExtendedElementSpec } from '@markuplint/ml-spec';
+/**
+ * Builds the complete list of HTML and SVG element specifications by reading local JSON spec files,
+ * enriching them with data scraped from MDN, and appending obsolete/deprecated elements.
+ * Elements are sorted alphabetically with SVG elements placed after HTML elements.
+ *
+ * @param filePattern - An absolute glob pattern matching the per-element JSON spec files
+ * @returns A sorted array of extended element specification objects
+ */
 export declare function getElements(filePattern: string): Promise<ExtendedElementSpec[]>;

package/lib/html-elements.js

@@ -3,6 +3,8 @@ import { fetchHTMLElement, fetchObsoleteElements } from './scraping.js';
 import { getSVGElementList } from './svg.js';
 import { getName, nameCompare, sortObjectByKey } from './utils.js';
 /**
+ * List of non-conforming (obsolete) HTML elements.
+ *
  * @see https://html.spec.whatwg.org/multipage/obsolete.html#non-conforming-features
  */
 const obsoleteList = [
@@ -36,6 +38,14 @@ const obsoleteList = [
     'spacer',
     'tt',
 ];
+/**
+ * Builds the complete list of HTML and SVG element specifications by reading local JSON spec files,
+ * enriching them with data scraped from MDN, and appending obsolete/deprecated elements.
+ * Elements are sorted alphabetically with SVG elements placed after HTML elements.
+ *
+ * @param filePattern - An absolute glob pattern matching the per-element JSON spec files
+ * @returns A sorted array of extended element specification objects
+ */
 export async function getElements(filePattern) {
     let specs = await readJsons(filePattern, (file, body) => {
         const name = file.replace(/^.+spec\.([\w-]+)\.json$/i, '$1');
@@ -88,15 +98,27 @@ export async function getElements(filePattern) {
                         };
                         continue;
                     }
-                    if (typeof current === 'object' && 'name' in current) {
-                        attrs[mdnAttr.name] = {
-                            // @ts-ignore for key order that "name" is first
-                            name: mdnAttr.name,
-                            // @ts-ignore for key order that "description" is second
-                            ...mdnData.attributes,
-                            // @ts-ignore
-                            ...current,
-                        };
+                    if (typeof current === 'object') {
+                        if ('name' in current) {
+                            attrs[mdnAttr.name] = {
+                                // @ts-ignore for key order that "name" is first
+                                name: mdnAttr.name,
+                                // @ts-ignore for key order that "description" is second
+                                ...mdnData.attributes,
+                                // @ts-ignore
+                                ...current,
+                            };
+                        }
+                        else {
+                            attrs[mdnAttr.name] = {
+                                description: mdnAttr.description,
+                                experimental: mdnAttr.experimental,
+                                obsolete: mdnAttr.obsolete,
+                                deprecated: mdnAttr.deprecated,
+                                nonStandard: mdnAttr.nonStandard,
+                                ...current,
+                            };
+                        }
                     }
                 }
                 return attrs;

package/lib/index.d.ts

@@ -1,7 +1,29 @@
+/**
+ * @module @markuplint/spec-generator
+ *
+ * Generates the markuplint extended specification JSON file by scraping W3C and MDN web standards
+ * documentation. Aggregates HTML/SVG element specs, global attributes, ARIA roles and properties,
+ * and content model definitions into a single output file consumed by the markuplint linter.
+ */
+/**
+ * Configuration options for the spec generator.
+ */
 export type Options = {
+    /** The absolute file path where the generated JSON spec will be written. */
     readonly outputFilePath: string;
+    /** An absolute glob pattern matching per-element HTML spec JSON files. */
     readonly htmlFilePattern: string;
+    /** The absolute file path to the common (global) attributes JSON definition. */
     readonly commonAttrsFilePath: string;
+    /** The absolute file path to the common content models JSON definition. */
     readonly commonContentsFilePath: string;
 };
+/**
+ * Main entry point for the spec generator. Fetches and aggregates all specification data
+ * (HTML/SVG elements, global attributes, ARIA definitions, content models, and reference URLs)
+ * then writes the combined result as a JSON file to the specified output path.
+ *
+ * @param options - The configuration options controlling input sources and output destination
+ * @returns A promise that resolves when the output file has been written
+ */
 export declare function main({ outputFilePath, htmlFilePattern, commonAttrsFilePath, commonContentsFilePath }: Options): Promise<void>;

package/lib/index.js

@@ -1,9 +1,24 @@
+/**
+ * @module @markuplint/spec-generator
+ *
+ * Generates the markuplint extended specification JSON file by scraping W3C and MDN web standards
+ * documentation. Aggregates HTML/SVG element specs, global attributes, ARIA roles and properties,
+ * and content model definitions into a single output file consumed by the markuplint linter.
+ */
 import { writeFile } from 'node:fs/promises';
 import { getAria } from './aria.js';
 import { getReferences } from './fetch.js';
 import { getGlobalAttrs } from './global-attrs.js';
 import { getElements } from './html-elements.js';
 import { readJson } from './read-json.js';
+/**
+ * Main entry point for the spec generator. Fetches and aggregates all specification data
+ * (HTML/SVG elements, global attributes, ARIA definitions, content models, and reference URLs)
+ * then writes the combined result as a JSON file to the specified output path.
+ *
+ * @param options - The configuration options controlling input sources and output destination
+ * @returns A promise that resolves when the output file has been written
+ */
 export async function main({ outputFilePath, htmlFilePattern, commonAttrsFilePath, commonContentsFilePath }) {
     const [specs, globalAttrs, aria] = await Promise.all([
         getElements(htmlFilePattern),

package/lib/read-json.d.ts

@@ -1,2 +1,20 @@
+/**
+ * Reads and parses a single JSON file (with support for JSON comments) from an absolute file path.
+ *
+ * @template T - The expected shape of the parsed JSON data
+ * @param filePath - The absolute file path to the JSON file
+ * @returns The parsed JSON content
+ * @throws If the provided path is not absolute
+ */
 export declare function readJson<T = Record<string, any>>(filePath: string): T;
+/**
+ * Reads multiple JSON files matching a glob pattern and optionally transforms each result.
+ * All matched files are read and parsed in parallel.
+ *
+ * @template T - The expected shape of each parsed JSON file
+ * @param pattern - An absolute glob pattern to match JSON files
+ * @param hook - An optional transformation function called with each file path and its parsed body
+ * @returns An array of parsed (and optionally transformed) JSON objects
+ * @throws If the provided pattern is not an absolute path
+ */
 export declare function readJsons<T = Record<string, any>>(pattern: string, hook?: (fileName: string, body: T) => T | Promise<T>): Promise<T[]>;