@gmb/bitmark-parser 6.11.1 → 7.0.0

package/README.md

@@ -11,7 +11,7 @@ npm install @gmb/bitmark-parser
 ```js
 import {
   convert,
-  lex,
+  semanticTokens,
   breakscapeText,
   unbreakscapeText,
   info,
@@ -32,11 +32,18 @@ console.log(parsedJson[0].bitmark);
 // Generate bitmark from JSON
 const bitmark = convert(json, { inputFormat: "json", outputFormat: "bitmark" });
 
-// Lex bitmark to token dump
-const tokens = lex("[.article]\nHello **bold**");
+// Parser-derived highlighting in the LSP semantic-tokens shape
+const highlights = semanticTokens("[.article]\nHello **bold**", {
+  tokensLayout: "absolute",
+});
 
-// Lex with a specific stage
-const tokenJson = lex("[.article]\nHello **bold**", { stage: "lex-json" });
+// Dump the lexer's token stream (debug tooling; bitmark input only)
+const tokens = JSON.parse(
+  convert("[.article]\nHello **bold**", {
+    inputFormat: "bitmark",
+    outputFormat: "lex",
+  }),
+);
 
 // Breakscape / unbreakscape text
 const escaped = breakscapeText("[!example]", {
@@ -60,23 +67,37 @@ console.log(version());
 The package ships **three wasm builds of the same engine** and selects one at
 runtime — the API surface never changes:
 
-| Feature | Contents | Size (approx.) |
-| --- | --- | --- |
-| `full` (Node default) | everything: rich `info` metadata + built-in translations + the semantic `diff` | ~1082 KB (~423 KB gzip) |
-| `browser-full` (browser default) | the same conversions and `diff`, without either | ~935 KB (~374 KB gzip) |
-| `bitmark-json` | bitmark ↔ JSON only: `convert`/`canonicalize`/`transform` (formats `auto`/`bitmark`/`json`, plus the `text` output), `info`, breakscape, text fragments — no `diff` | ~705 KB (~277 KB gzip) |
-
-Not in `bitmark-json`: the markup formats (`html`, `xml`, `xml-niso-iec`, …)
-as input or output (they return an `error: … not supported in this build`
-string), the `mappingReport` option (same error), and `lex` (throws
-`UnsupportedFeatureError`).
+| Feature                          | Contents                                                                                                                                                                                               | Size (approx.)         |
+| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- |
+| `full` (Node default)            | everything: rich `info` metadata + built-in translations + the semantic `diff`                                                                                                                         | ~990 KB (~378 KB gzip) |
+| `browser-full` (browser default) | the same conversions and `diff`, without either                                                                                                                                                        | ~783 KB (~317 KB gzip) |
+| `bitmark-json`                   | bitmark ↔ JSON only: `convert`/`canonicalize`/`transform` (formats `auto`/`bitmark`/`json`, plus the `text` and `semantic-tokens` outputs), `info` (JSON only), breakscape, text fragments — no `diff` | ~563 KB (~224 KB gzip) |
+
+**The rule.** _Every build includes every feature, except the two
+browser-targeted variants, which each carry an explicit exclusion list._ That
+is the whole story — a new capability is in every build unless it is named
+below:
+
+| build                                              | features                                                                            |
+| -------------------------------------------------- | ----------------------------------------------------------------------------------- |
+| native CLI, daemon, docgen, schemagen, wasm `full` | **all**                                                                             |
+| wasm `browser-full`                                | all **except** `info-meta` (bit + tag prose) and `translations` (the baked table)   |
+| wasm `bitmark-json`                                | all **except** the above and `markup`, `mapping-report`, `diff`, `lex`, `info-text` |
+
+So, not in `bitmark-json`: the markup formats (`html`, `xml`,
+`xml-niso-iec`, …) as input or output (they throw an `unsupported` error), the
+`lex` output format and the `mappingReport` option (same error), `diff`
+(throws `UnsupportedFeatureError`), and the human-readable rendering of `info` —
+`info({ format: "text" })` throws `UnsupportedFeatureError` there, so pass
+`format: "json"` (the JSON views are the contractual ones and are in every
+variant).
 
 `full` and `browser-full` convert **identically** — they differ only in what
-`info` can report: the META fields (tag descriptions, group provenance and raw
-mapping patterns) and the built-in translations table. Bit titles, bit
-descriptions and the bit-group / resource-group catalogs are in every variant,
-in English; `register` supplies translations to the lean ones (see *Display
-names and languages*).
+`info` can report: the META fields (bit and tag descriptions, group provenance
+and raw mapping patterns) and the built-in translations table. Bit titles and
+the bit-group / resource-group catalogs are in every variant, in English;
+`register` supplies translations to the lean ones (see _Display names and
+languages_).
 
 Why the defaults differ: only the browser pays a download-latency cost for
 those strings, and a Node backend loading wasm from disk has the same
@@ -89,15 +110,15 @@ browser, select `browser-full` explicitly.**
 > **Migrating from ≤ 6.10:** `full` used to mean what `browser-full` means now.
 > Browser code that explicitly selected `"full"` should move to
 > `"browser-full"` to keep its download size close to what it was; leave it on
-> `"full"` to gain the metadata and built-in translations for ~49 KB gzip.
+> `"full"` to gain the metadata and built-in translations for ~62 KB gzip.
 > Nothing else changes — the capabilities that were in `full` are in both.
 
 ```js
 import { init, convert, variant } from "@gmb/bitmark-parser";
 
 await init({ feature: "bitmark-json" }); // load the minimal wasm
-convert("[.article]\nHello");            // works
-variant();                               // "bitmark-json"
+convert("[.article]\nHello"); // works
+variant(); // "bitmark-json"
 
 // Upgrade in place: the interface stays valid while the full wasm loads —
 // calls keep being served by the active module, then swap atomically.
@@ -130,6 +151,39 @@ Notes:
 
 ### API
 
+#### Errors
+
+**Errors are thrown, never returned.** Every fallible WASM export raises a
+JavaScript `Error` whose `message` has one shape:
+
+```text
+<kind> at <path>: <message>
+```
+
+A returned string is therefore always a result — there is no prefix or
+sentinel to test for, and every call site is a `try`/`catch`.
+
+`<kind>` is a **stable kebab-case identifier** — the part to match on.
+`<path>` locates the failure (a JSON pointer-ish path, `offset N` for a JSON
+syntax error, or empty) and `<message>` is human-readable prose that may
+change in any release. The kinds:
+
+| kind                       | meaning                                                                                                                       |
+| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| `invalid-json`             | the input string is not well-formed JSON                                                                                      |
+| `compose`                  | the reverse composer rejected the JSON (shape / field / type)                                                                 |
+| `unsupported`              | the requested conversion is not available (an output-only format used as input, or a format compiled out of this variant)     |
+| `patch-parse-error`        | a patch or patch document is malformed                                                                                        |
+| `patch-index-out-of-range` | a patch path addressed an array element that does not exist                                                                   |
+| `patch-type-mismatch`      | a patch value's JSON type is incompatible with its target                                                                     |
+| `patch-conflict`           | a patch document's entries contradict each other (a bit removed or moved twice, or an anchor that is itself removed or moved) |
+
+Every TypeScript wrapper propagates that `Error` unchanged — `convert`,
+`transform`, `canonicalize`, `bitmarkToObjects`, `countBits`, `diff`, … — as
+does the parallel worker pool, which rejects with the first failing bit's
+error. `register` re-wraps it as a `RegisterError`; a capability the loaded
+variant compiled out raises `UnsupportedFeatureError` before any wasm call.
+
 #### convert(input: string, options?: ConvertOptions): string
 
 Convert between bitmark, JSON, and the mapped markup formats (html/xml/…).
@@ -153,7 +207,7 @@ Options:
   the bit is `_`-prefixed. A bare tag the bit does not declare (`[!…]`,
   `[?…]`, `[#…]`, … on a bit whose config has no such tag) is an unknown
   too, under the key a declaring bit would use (`[!x]` → `"instruction":
-  ["x"]`); a rejected resource attachment is not — it is still emitted, with
+["x"]`); a rejected resource attachment is not — it is still emitted, with
   a warning. Unknown properties are never converted back to bitmark, and
   each occurrence warns whether or not it is included.
   Also available on `bitmarkToObjects` and as the CLI's
@@ -175,6 +229,13 @@ Options:
 > format, set `inputFormat` explicitly. The same applies to `canonicalize`
 > and `transform`, whose `inputFormat` also defaults to `"auto"`.
 
+> **Input limits:** JSON objects/arrays and markup elements may nest at most
+> **256** levels deep. Deeper JSON is rejected like any other malformed JSON
+> (`"error: … nesting depth exceeds 256"`); a deeper markup open tag is kept
+> as literal text. The bound keeps a crafted document from exhausting the
+> WASM stack — an instance that rejects a too-deep input stays usable for the
+> next call.
+
 `convert` is the single string-based conversion surface — parsing is
 `{ inputFormat: "bitmark", outputFormat: "json" }`, generating is
 `{ inputFormat: "json", outputFormat: "bitmark" }`. For typed objects use
@@ -184,7 +245,15 @@ For bitmark input, output is a JSON array of entries with:
 
 - `bit` — serialized bit payload
 - `parser` — parser metadata, `errors` / `warnings`, and `infos` (informational notices for recoveries that are not necessarily wrong, e.g. text kept as text because it only looks like an unclosed tag). Each issue carries a stable machine-readable `code` (`"unknown-property"`, `"missing-required-tag"`, …) beside its human-readable `message`, plus `text` and `location`. **Branch on `code`** — a shipped code never changes meaning, whereas the message text is not contractual and may be reworded in any release
-- `bitmark` — source bitmark text
+- `bitmark` — source bitmark text (a convenience echo; every bit is fully described by `bit` alone)
+
+A region the parser could not read — an unknown bit type, or non-blank text
+that never opened a bit — comes back as a bit of type `_error` beside its
+`parser.errors` entry. It carries `originalType` (the header exactly as
+written between the level dots and the `]`, including a leading `|` for a
+commented-out bit and any `:format` / `&resource` suffix; absent when there
+was no header) and `body` (the raw text after it, as a plain string), so
+generating bitmark from it reproduces the original region.
 
 #### canonicalize(input: string, options?: CanonicalizeOptions): string
 
@@ -212,9 +281,15 @@ A patch document addresses bits by their position (or id) in the input
 {
   "version": 1,
   "entries": [
-    { "select": { "index": 3, "id": "mc-014" }, "patch": [{ "path": "title", "op": "set", "value": "New" }] },
+    {
+      "select": { "index": 3, "id": "mc-014" },
+      "patch": [{ "path": "title", "op": "set", "value": "New" }]
+    },
     { "select": { "index": 9 }, "remove": true },
-    { "insert": { "after": 4 }, "bit": { "type": "article", "body": "Inserted" } },
+    {
+      "insert": { "after": 4 },
+      "bit": { "type": "article", "body": "Inserted" }
+    },
     { "select": { "index": 7 }, "move": { "after": 12 } },
     { "select": "all", "patch": [{ "lang": "de" }] }
   ]
@@ -222,8 +297,10 @@ A patch document addresses bits by their position (or id) in the input
 ```
 
 `after` names an anchor — an input bit that is neither removed nor moved,
-or `-1` for the document start. `diff` produces such a document from two
-versions of a file (below).
+or `-1` for the document start. A bit may be the target of at most one
+`remove` or `move` entry: moving a bit twice, removing it twice, or removing
+and moving it are all rejected, naming the second of the conflicting entries.
+`diff` produces such a document from two versions of a file (below).
 
 #### diff(a: string, b: string, options?: DiffOptions): string
 
@@ -281,14 +358,74 @@ issue codes gained or lost, and the rendered `hunks`.
 
 Count or split the bits of a bitmark/JSON input without full parsing.
 
-#### lex(input: string, options?: LexOptions): string
-
-Lex bitmark text.
-
-Stages (`options.stage`):
-
-- `"lex"` — Unified lexer token stream, one token per line (default)
-- `"lex-json"` — Unified lexer token stream as JSON
+#### semanticTokens(input: string, options?: SemanticTokensOptions): SemanticTokens
+
+Parser-derived highlighting of bitmark source in the LSP semantic-tokens
+shape — the typed form of
+`convert(input, { inputFormat: "bitmark", outputFormat: "semantic-tokens" })`.
+The parser runs (never the validator; diagnostics stay on their own channel)
+and the AST is walked loss-lessly, so every character of the source is
+covered exactly once and the highlighting agrees with the JSON the same
+input produces: an unpaired `**` is text, an abandoned `[!tag` is text and
+the `====` / `--` it would have swallowed are dividers, the body of a bit
+whose body format is not bitmark (`[.code]`, `==== text ====`) is plain
+text. Bitmark input only.
+
+Options (`SemanticTokensOptions`, also `ConvertOptions` keys):
+
+- `positionEncoding` — `"utf-16"` (default; the LSP default, what browser
+  editors index by) or `"utf-8"` (bytes). Echoed in the response.
+- `tokensLayout` — `"lsp"` (default): `data`, the LSP relative encoding,
+  five integers per token (`deltaLine, deltaStart, length, tokenType,
+tokenModifiers`; the type is a legend index, the modifiers a bitset); or
+  `"absolute"`: `tokens`, one `{ line, start, length, type, modifiers }`
+  per token, 0-based, by name. Both carry `legend` and `positionEncoding`.
+
+Lines break at `\n`, `\r\n` and `\r`; no token contains a line terminator.
+The legend orders are stable (a change is semver-major; new entries may be
+appended). Token types name sigil classes and syntax positions, never a bit
+or tag name; each has a recommended standard super type so an editor's
+default theme colours it — paste the table into a VS Code extension's
+`semanticTokenTypes`:
+
+| type                                                                               | covers                                                   | super type                         |
+| ---------------------------------------------------------------------------------- | -------------------------------------------------------- | ---------------------------------- |
+| `frontmatter`                                                                      | the region before the first bit                          | `comment`                          |
+| `bitSigil`                                                                         | `[.`, level dots, `\|`, `:`, `&`, `]` of a bit header    | `keyword`                          |
+| `bitType` / `bitFormat` / `bitResourceType`                                        | the header's three fields                                | `type` / `modifier` / `type`       |
+| `tagSigil`                                                                         | every tag's own syntax (`[@`, `[!`, `[##`, `:`, `]`, …)  | `keyword`                          |
+| `propertyKey` / `resourceType`                                                     | a property tag's key / a resource tag's type             | `property` / `type`                |
+| `tagText`                                                                          | plain text inside any tag value                          | `string`                           |
+| `cardDivider` / `sideDivider` / `variantDivider` / `footerDivider` / `textDivider` | `====`, `--`, `++`, `==== footer ====`, `==== text ====` | `operator`                         |
+| `paragraphBreak`                                                                   | a `\|` paragraph break                                   | `operator`                         |
+| `headingSigil` / `heading`                                                         | `# ` / the heading's plain text                          | `keyword` / `string`               |
+| `listMarker`                                                                       | `• `, `•1 `, `•+ `, the indent                           | `keyword`                          |
+| `codeSigil` / `codeLanguage` / `codeBody`                                          | `\|code:` / the language / the body                      | `keyword` / `type` / `string`      |
+| `imageSigil` / `imageSrc`                                                          | `\|image:` / the source                                  | `keyword` / `string`               |
+| `markSigil`                                                                        | a mark's open and close markers                          | `operator`                         |
+| `bold` / `italic` / `highlight` / `light` / `inline`                               | the marked text                                          | `string`                           |
+| `attrSigil` / `attrKey` / `attrValue`                                              | an attr chain's `\|` and `:` / key / value               | `operator` / `property` / `string` |
+| `url`                                                                              | a bare URL the parser auto-linked                        | `string`                           |
+| `text`                                                                             | plain body / card / footer text, and every top-level gap | (none)                             |
+| `plainText`                                                                        | a plain region (after `==== text ====`, or a raw body)   | `string`                           |
+
+Modifiers, in bit order: the tag classes `property`, `resource`, `title`,
+`reference`, `anchor`, `item`, `instruction`, `hint`, `true`, `false`,
+`gap`, `mark`, `solution` (every token of a tag of that class, sigils
+included — exactly one per tag); `comment` (every token of a commented-out
+bit); `unclosed` (tokens inside a parser unclosed-mark / unclosed-tag
+notice). The full contract is `.zen/specs/API-SEM-semantic-tokens.tsp`.
+
+#### The `lex` output format
+
+`convert(input, { inputFormat: "bitmark", outputFormat: "lex" })` returns the
+lexer's token stream as a JSON array: one `{ kind, span, text }` per token
+(the `LexToken` type), `span` in bytes. Debug tooling — the token kinds are the
+engine's own names, not a stable vocabulary, and the parser is never run
+(pairing, abandoned-tag repair and raw-body handling all happen after lexing),
+so a highlighter should use `semantic-tokens` instead. Bitmark input only; a
+JSON or markup document has no source text to lex. The former `lex()` export
+and `bitmark lex` command were removed in 7.0 (see the migration guide).
 
 #### breakscapeText(input: string, options?: BreakscapeOptions): string
 
@@ -325,13 +462,13 @@ version and (separately) any migration target.
 - `full` — complete detail for `"bit"`/`"all"` (default: `false` = compact view)
 - `language` — BCP-47 tag(s) for display names: one (`"de"`), several
   (`["de", "fr"]` or `"de,fr"`), or `"all"` — exported as `ALL_LANGUAGES`
-  (default: English). More than one adds a `titles` map. See *Display names
-  and languages* below
+  (default: English). More than one adds a `titles` map. See _Display names
+  and languages_ below
 
 `"bit-groups"` / `"resource-groups"` return the search/filter catalogs —
 each group's key, translated title, description, optional aliases and
 `subgroupOf`, and its members. They are available in EVERY build: the group
-catalog and per-bit titles/descriptions are "descriptive" data, which the
+catalog and per-bit titles are "descriptive" data, which the
 lean wasm variants carry too. Deprecated members are excluded by default and
 MARKED when included with `includeDeprecated: true`; a search index matching
 already-published content needs them, because a migrating bit is re-emitted
@@ -341,13 +478,13 @@ To derive quiz categories, intersect a bit's `bitGroups` with the groups
 carrying `subgroupOf: "quizzes"` — `subgroupOf` is metadata and never implies
 membership, so every member of a subgroup also declares the parent.
 
-Only the META fields — TAG descriptions, group-inheritance provenance and raw
-mapping patterns (the `info-meta` cargo feature) — depend on the build. They
-are reported by the native CLI and the wasm `full` variant; `browser-full` and
-`bitmark-json` omit them to stay small. Those are absent structurally: the key
-is missing, never `null` or empty. Everything else `info` returns — including
-bit titles, bit descriptions and the group catalogs — is identical in every
-variant.
+Only the META fields — BIT and TAG descriptions, group-inheritance provenance
+and raw mapping patterns (the `info-meta` cargo feature) — depend on the build.
+They are reported by the native CLI and the wasm `full` variant; `browser-full`
+and `bitmark-json` omit them to stay small (the bit descriptions alone are
+14 KB gzip of the download). Those are absent structurally: the key is
+missing, never `null` or empty. Everything else `info` returns — including bit
+titles and the group catalogs — is identical in every variant.
 
 #### Display names and languages
 
@@ -390,7 +527,12 @@ which categories it is in":
 
 ```js
 const groups = JSON.parse(
-  info({ infoType: "bit-groups", format: "json", includeDeprecated: true, language: "all" }),
+  info({
+    infoType: "bit-groups",
+    format: "json",
+    includeDeprecated: true,
+    language: "all",
+  }),
 );
 groups[0].bitTypes[0]; // { name: "assignment", title: "Assignment", titles: {…} }
 ```
@@ -508,7 +650,7 @@ const patch = patchEntry("id", "append", "1234");
 
 See `examples/` for runnable scripts covering the whole API (executed by
 `npm test`, so they stay correct). Generated API documentation (TypeDoc) is
-published at <https://getmorebrain.github.io/bitmark-parser-rust/docs/api/>
+published at <https://getmorebrain.github.io/bitmark-parser/docs/api/>
 with each release (the site root hosts the bitmark language docs), or build
 it locally with `npm run docs` (→ `docs/api/`) and preview it with
 `npm run docs:serve` (http://localhost:8080, `--port` to change).
@@ -536,6 +678,28 @@ Note: deprecated bit types with a configured migration target (the
 `isCollapsible: true` — consumers keyed on the old `type` strings see the
 base type instead. `collapsible` itself is unaffected.
 
+### Property number values
+
+A tag whose configured format is `number` accepts what JavaScript's
+`Number(value)` accepts, minus the values that are not finite:
+
+- surrounding whitespace, a leading `+`, leading zeros (`01`), a trailing
+  point (`1.`) and a leading point (`.5`);
+- an exponent in either case (`1e2`, `1E2`);
+- hexadecimal, binary and octal literals (`0x1F`, `0b101`, `0o701`) — prefix
+  case-insensitive, with no sign, fraction or exponent.
+
+`NaN`, `Infinity`, an exponent beyond ±4000, and a radix literal past 64 bits
+are **rejected**: the tag is dropped from the output and a
+`property-format-mismatch` warning names it, exactly like any other value that
+does not fit its format.
+
+Accepted values print in JavaScript's `JSON.stringify` form — `0.5`, `1.5`,
+`100`, `1e+21`, `1e-7` — so `[@width: 1.50]` emits `1.5` and `[@width:0x10]`
+emits `16`. Values with more than 17 significant digits, including integers at
+or above 2^63, keep the digits you wrote rather than being rounded through a
+floating-point double.
+
 ## Legacy API (bpg-compatible)
 
 A compatibility facade for consumers migrating from
@@ -643,8 +807,12 @@ bitmark diff old.bitmark new.bitmark --output-format patch > d.patch
 bitmark transform old.bitmark --patch d.patch --output-format bitmark   # reproduces new
 # --context N, --locate, --similarity F, --bbox-tolerance N, --color auto|always|never
 
-# Dump the lexer token stream
-bitmark lex input.bitmark
+# Parser-derived highlighting: LSP semantic tokens (bitmark input only)
+bitmark convert input.bitmark --input-format bitmark --output-format semantic-tokens
+bitmark convert input.bitmark --output-format semantic-tokens --tokens-layout absolute --position-encoding utf-8 --pretty
+
+# Dump the lexer token stream as JSON (debug tooling; bitmark input only)
+bitmark convert input.bitmark --input-format bitmark --output-format lex --pretty
 
 # Breakscape / unbreakscape text
 bitmark breakscape input.txt
@@ -687,13 +855,13 @@ The package includes pre-built browser bundles with the WASM module.
 The CDN bundle carries all three variants' JS glue and fetches only the
 selected variant's `.wasm` (from `dist/browser/wasm/`). A bare `init()` loads
 `browser-full`; `init({ feature: "bitmark-json" })` fetches the smallest
-build, and a later `init({ feature: "full" })` upgrades in place (see *WASM
-variants*).
+build, and a later `init({ feature: "full" })` upgrades in place (see _WASM
+variants_).
 
 ### Bundler (webpack / vite)
 
 ```js
-import init, { convert, lex } from "@gmb/bitmark-parser/browser";
+import init, { convert } from "@gmb/bitmark-parser/browser";
 
 await init();
 const json = convert("[.article]\nHello", { inputFormat: "bitmark" });