@gmb/bitmark-parser 6.11.1 → 7.1.0

package/README.md

@@ -11,7 +11,7 @@ npm install @gmb/bitmark-parser
 ```js
 import {
   convert,
-  lex,
+  semanticTokens,
   breakscapeText,
   unbreakscapeText,
   info,
@@ -32,11 +32,18 @@ console.log(parsedJson[0].bitmark);
 // Generate bitmark from JSON
 const bitmark = convert(json, { inputFormat: "json", outputFormat: "bitmark" });
 
-// Lex bitmark to token dump
-const tokens = lex("[.article]\nHello **bold**");
+// Parser-derived highlighting in the LSP semantic-tokens shape
+const highlights = semanticTokens("[.article]\nHello **bold**", {
+  tokensLayout: "absolute",
+});
 
-// Lex with a specific stage
-const tokenJson = lex("[.article]\nHello **bold**", { stage: "lex-json" });
+// Dump the lexer's token stream (debug tooling; bitmark input only)
+const tokens = JSON.parse(
+  convert("[.article]\nHello **bold**", {
+    inputFormat: "bitmark",
+    outputFormat: "lex",
+  }),
+);
 
 // Breakscape / unbreakscape text
 const escaped = breakscapeText("[!example]", {
@@ -60,23 +67,37 @@ console.log(version());
 The package ships **three wasm builds of the same engine** and selects one at
 runtime — the API surface never changes:
 
-| Feature | Contents | Size (approx.) |
-| --- | --- | --- |
-| `full` (Node default) | everything: rich `info` metadata + built-in translations + the semantic `diff` | ~1082 KB (~423 KB gzip) |
-| `browser-full` (browser default) | the same conversions and `diff`, without either | ~935 KB (~374 KB gzip) |
-| `bitmark-json` | bitmark ↔ JSON only: `convert`/`canonicalize`/`transform` (formats `auto`/`bitmark`/`json`, plus the `text` output), `info`, breakscape, text fragments — no `diff` | ~705 KB (~277 KB gzip) |
-
-Not in `bitmark-json`: the markup formats (`html`, `xml`, `xml-niso-iec`, …)
-as input or output (they return an `error: … not supported in this build`
-string), the `mappingReport` option (same error), and `lex` (throws
-`UnsupportedFeatureError`).
+| Feature                          | Contents                                                                                                                                                                                               | Size (approx.)         |
+| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- |
+| `full` (Node default)            | everything: rich `info` metadata + built-in translations + the semantic `diff`                                                                                                                         | ~1025 KB (~395 KB gzip) |
+| `browser-full` (browser default) | the same conversions and `diff`, without either                                                                                                                                                        | ~818 KB (~333 KB gzip) |
+| `bitmark-json`                   | bitmark ↔ JSON only: `convert`/`canonicalize`/`transform` (formats `auto`/`bitmark`/`json`, plus the `text`, `semantic-tokens` and `diagnostics` outputs), the editor services, `info` (JSON only), breakscape, text fragments — no `diff` | ~599 KB (~239 KB gzip) |
+
+**The rule.** _Every build includes every feature, except the two
+browser-targeted variants, which each carry an explicit exclusion list._ That
+is the whole story — a new capability is in every build unless it is named
+below:
+
+| build                                              | features                                                                            |
+| -------------------------------------------------- | ----------------------------------------------------------------------------------- |
+| native CLI, daemon, docgen, schemagen, wasm `full` | **all**                                                                             |
+| wasm `browser-full`                                | all **except** `info-meta` (bit + tag prose) and `translations` (the baked table)   |
+| wasm `bitmark-json`                                | all **except** the above and `markup`, `mapping-report`, `diff`, `lex`, `info-text` |
+
+So, not in `bitmark-json`: the markup formats (`html`, `xml`,
+`xml-niso-iec`, …) as input or output (they throw an `unsupported` error), the
+`lex` output format and the `mappingReport` option (same error), `diff`
+(throws `UnsupportedFeatureError`), and the human-readable rendering of `info` —
+`info({ format: "text" })` throws `UnsupportedFeatureError` there, so pass
+`format: "json"` (the JSON views are the contractual ones and are in every
+variant).
 
 `full` and `browser-full` convert **identically** — they differ only in what
-`info` can report: the META fields (tag descriptions, group provenance and raw
-mapping patterns) and the built-in translations table. Bit titles, bit
-descriptions and the bit-group / resource-group catalogs are in every variant,
-in English; `register` supplies translations to the lean ones (see *Display
-names and languages*).
+`info` can report: the META fields (bit and tag descriptions, group provenance
+and raw mapping patterns) and the built-in translations table. Bit titles and
+the bit-group / resource-group catalogs are in every variant, in English;
+`register` supplies translations to the lean ones (see _Display names and
+languages_).
 
 Why the defaults differ: only the browser pays a download-latency cost for
 those strings, and a Node backend loading wasm from disk has the same
@@ -89,15 +110,15 @@ browser, select `browser-full` explicitly.**
 > **Migrating from ≤ 6.10:** `full` used to mean what `browser-full` means now.
 > Browser code that explicitly selected `"full"` should move to
 > `"browser-full"` to keep its download size close to what it was; leave it on
-> `"full"` to gain the metadata and built-in translations for ~49 KB gzip.
+> `"full"` to gain the metadata and built-in translations for ~62 KB gzip.
 > Nothing else changes — the capabilities that were in `full` are in both.
 
 ```js
 import { init, convert, variant } from "@gmb/bitmark-parser";
 
 await init({ feature: "bitmark-json" }); // load the minimal wasm
-convert("[.article]\nHello");            // works
-variant();                               // "bitmark-json"
+convert("[.article]\nHello"); // works
+variant(); // "bitmark-json"
 
 // Upgrade in place: the interface stays valid while the full wasm loads —
 // calls keep being served by the active module, then swap atomically.
@@ -114,8 +135,10 @@ Notes:
   the selected variant's `.wasm` is fetched (all three small JS glue modules
   are in the bundle). Overlapping `init` calls: same-feature calls coalesce; a
   different-feature call supersedes an unfinished one (last call wins); every
-  promise resolves once the finally-active module is live. In the browser,
-  `initSync(bytes, { feature })` instantiates from bytes you supply.
+  promise resolves once the finally-active module is live. `init({})` is the
+  same request as `init()`. In the browser, `initSync(bytes, { feature })`
+  instantiates from bytes you supply, and takes part in the same last-call-wins
+  order: an `init` still loading when `initSync` runs never swaps in afterwards.
 - **Supplying your own `.wasm`**: glue and module are a pair, so name the
   feature the bytes belong to — `init({ feature, module_or_path })`. The
   back-compat positional form `init(module_or_path)` stays on `full`, since
@@ -130,14 +153,56 @@ Notes:
 
 ### API
 
+#### Errors
+
+**Errors are thrown, never returned.** Every fallible WASM export raises a
+JavaScript `Error` whose `message` has one shape:
+
+```text
+<kind> at <path>: <message>
+```
+
+A returned string is therefore always a result — there is no prefix or
+sentinel to test for, and every call site is a `try`/`catch`.
+
+`<kind>` is a **stable kebab-case identifier** — the part to match on.
+`<path>` locates the failure (a JSON pointer-ish path, `offset N` for a JSON
+syntax error, or empty) and `<message>` is human-readable prose that may
+change in any release. The kinds:
+
+| kind                       | meaning                                                                                                                       |
+| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| `invalid-json`             | the input string is not well-formed JSON                                                                                      |
+| `compose`                  | the reverse composer rejected the JSON (shape / field / type)                                                                 |
+| `unsupported`              | the requested conversion is not available (an output-only format used as input, or a format compiled out of this variant)     |
+| `patch-parse-error`        | a patch or patch document is malformed                                                                                        |
+| `patch-index-out-of-range` | a patch path addressed an array element that does not exist                                                                   |
+| `patch-type-mismatch`      | a patch value's JSON type is incompatible with its target                                                                     |
+| `patch-conflict`           | a patch document's entries contradict each other (a bit removed or moved twice, or an anchor that is itself removed or moved) |
+
+Every TypeScript wrapper propagates that `Error` unchanged — `convert`,
+`transform`, `canonicalize`, `bitmarkToObjects`, `countBits`, `diff`, … — as
+does the parallel worker pool, which rejects with the first failing bit's
+error. `register` re-wraps it as a `RegisterError`; a capability the loaded
+variant compiled out raises `UnsupportedFeatureError` before any wasm call.
+
 #### convert(input: string, options?: ConvertOptions): string
 
 Convert between bitmark, JSON, and the mapped markup formats (html/xml/…).
 
+A JSON input may be an array or a single value, each entry a `{ bit: {…} }`
+envelope (the forward output) or a bare bit object. An entry that is not a
+bit produces no bitmark: it is skipped and reported as an `unknown-bit-type`
+issue naming its JSON path (`$[2] is not a bit: no "type" — skipped`) on the
+warnings channel of the CLI (`--warnings`) and the daemon; the bits beside it
+still convert, and a document in which nothing is a bit converts to empty
+output. This `convert` returns a bare string and so cannot show the skip.
+Only malformed JSON *text* throws (`invalid-json`).
+
 Options:
 
 - `inputFormat` — `"auto"` (default), `"bitmark"`, `"json"`, or a mapping id (`"html"`, `"xml"`, …)
-- `outputFormat` — `"auto"` (default: the opposite direction), `"text"` (lossy plain-text extraction; output-only), or as above
+- `outputFormat` — `"auto"` (default: the opposite direction), `"text"` (lossy plain-text extraction; output-only), `"semantic-tokens"` / `"diagnostics"` / `"lex"` (output-only, bitmark input only — see their sections), or as above
 - `mode` — `"optimized"` (default) or `"full"`
 - `warnings` — include validation warnings (default: `false`)
 - `plainText` — output text as plain text (default: `false`)
@@ -153,7 +218,7 @@ Options:
   the bit is `_`-prefixed. A bare tag the bit does not declare (`[!…]`,
   `[?…]`, `[#…]`, … on a bit whose config has no such tag) is an unknown
   too, under the key a declaring bit would use (`[!x]` → `"instruction":
-  ["x"]`); a rejected resource attachment is not — it is still emitted, with
+["x"]`); a rejected resource attachment is not — it is still emitted, with
   a warning. Unknown properties are never converted back to bitmark, and
   each occurrence warns whether or not it is included.
   Also available on `bitmarkToObjects` and as the CLI's
@@ -175,6 +240,13 @@ Options:
 > format, set `inputFormat` explicitly. The same applies to `canonicalize`
 > and `transform`, whose `inputFormat` also defaults to `"auto"`.
 
+> **Input limits:** JSON objects/arrays and markup elements may nest at most
+> **256** levels deep. Deeper JSON is rejected like any other malformed JSON
+> (`"error: … nesting depth exceeds 256"`); a deeper markup open tag is kept
+> as literal text. The bound keeps a crafted document from exhausting the
+> WASM stack — an instance that rejects a too-deep input stays usable for the
+> next call.
+
 `convert` is the single string-based conversion surface — parsing is
 `{ inputFormat: "bitmark", outputFormat: "json" }`, generating is
 `{ inputFormat: "json", outputFormat: "bitmark" }`. For typed objects use
@@ -183,8 +255,16 @@ Options:
 For bitmark input, output is a JSON array of entries with:
 
 - `bit` — serialized bit payload
-- `parser` — parser metadata, `errors` / `warnings`, and `infos` (informational notices for recoveries that are not necessarily wrong, e.g. text kept as text because it only looks like an unclosed tag). Each issue carries a stable machine-readable `code` (`"unknown-property"`, `"missing-required-tag"`, …) beside its human-readable `message`, plus `text` and `location`. **Branch on `code`** — a shipped code never changes meaning, whereas the message text is not contractual and may be reworded in any release
-- `bitmark` — source bitmark text
+- `parser` — parser metadata, `errors` / `warnings`, and `infos` (informational notices for recoveries that are not necessarily wrong: text kept as text because it only looks like an unclosed tag, `unclosed-tag`, or a formatting mark with no partner, `unclosed-formatting` — in bitmark malformed markup is text, not an error). Each issue carries a stable machine-readable `code` (`"unknown-property"`, `"missing-required-tag"`, …) beside its human-readable `message`, plus `text` and `location`. **Branch on `code`** — a shipped code never changes meaning, whereas the message text is not contractual and may be reworded in any release
+- `bitmark` — source bitmark text (a convenience echo; every bit is fully described by `bit` alone)
+
+A region the parser could not read — an unknown bit type, or non-blank text
+that never opened a bit — comes back as a bit of type `_error` beside its
+`parser.errors` entry. It carries `originalType` (the header exactly as
+written between the level dots and the `]`, including a leading `|` for a
+commented-out bit and any `:format` / `&resource` suffix; absent when there
+was no header) and `body` (the raw text after it, as a plain string), so
+generating bitmark from it reproduces the original region.
 
 #### canonicalize(input: string, options?: CanonicalizeOptions): string
 
@@ -203,7 +283,13 @@ Apply a patch document to a document, bit by bit, and re-emit it.
   before the document reshapes anything; a hook's patches apply after the
   document's
 - `inputFormat`, `outputFormat`, `mode`, `pretty`, `indent`,
-  `spacesAroundValues` — as for `convert`
+  `spacesAroundValues` — as for `convert`. `outputFormat` (default `"json"`)
+  takes every per-bit format: `"json"`, `"bitmark"`, `"text"` or a mapping id
+  such as `"html"` (per-bit text outputs are joined with newlines). The
+  whole-document outputs `"lex"`, `"semantic-tokens"` and `"diagnostics"` are
+  refused with an `unsupported` error in every driver — use `convert`. An
+  explicit `inputFormat` is honoured as given, patch or hooks or not; it is
+  never re-detected from the content
 
 A patch document addresses bits by their position (or id) in the input
 **before** any entry is applied, so it reads like a diff:
@@ -212,9 +298,15 @@ A patch document addresses bits by their position (or id) in the input
 {
   "version": 1,
   "entries": [
-    { "select": { "index": 3, "id": "mc-014" }, "patch": [{ "path": "title", "op": "set", "value": "New" }] },
+    {
+      "select": { "index": 3, "id": "mc-014" },
+      "patch": [{ "path": "title", "op": "set", "value": "New" }]
+    },
     { "select": { "index": 9 }, "remove": true },
-    { "insert": { "after": 4 }, "bit": { "type": "article", "body": "Inserted" } },
+    {
+      "insert": { "after": 4 },
+      "bit": { "type": "article", "body": "Inserted" }
+    },
     { "select": { "index": 7 }, "move": { "after": 12 } },
     { "select": "all", "patch": [{ "lang": "de" }] }
   ]
@@ -222,8 +314,10 @@ A patch document addresses bits by their position (or id) in the input
 ```
 
 `after` names an anchor — an input bit that is neither removed nor moved,
-or `-1` for the document start. `diff` produces such a document from two
-versions of a file (below).
+or `-1` for the document start. A bit may be the target of at most one
+`remove` or `move` entry: moving a bit twice, removing it twice, or removing
+and moving it are all rejected, naming the second of the conflicting entries.
+`diff` produces such a document from two versions of a file (below).
 
 #### diff(a: string, b: string, options?: DiffOptions): string
 
@@ -279,16 +373,195 @@ issue codes gained or lost, and the rendered `hunks`.
 
 #### countBits(input: string): number / splitBits(input: string): BitSlice[]
 
-Count or split the bits of a bitmark/JSON input without full parsing.
+Count or split the bits of a bitmark/JSON input without full parsing. A JSON
+input may be an array or a single value, each entry a `{ bit: {…} }` envelope
+or a bare bit object (the four shapes `convert` accepts); `splitJsonBits`
+always returns envelopes. An entry that is not a bit is not counted and not
+returned — these calls have no warnings channel; `convert` reports the skip.
+
+#### semanticTokens(input: string, options?: SemanticTokensOptions): SemanticTokens
+
+Parser-derived highlighting of bitmark source in the LSP semantic-tokens
+shape — the typed form of
+`convert(input, { inputFormat: "bitmark", outputFormat: "semantic-tokens" })`.
+The parser runs (never the validator; diagnostics stay on their own channel)
+and the AST is walked loss-lessly, so every character of the source is
+covered exactly once and the highlighting agrees with the JSON the same
+input produces: an unpaired `**` is text, an abandoned `[!tag` is text and
+the `====` / `--` it would have swallowed are dividers, the body of a bit
+whose body format is not bitmark (`[.code]`, `==== text ====`) is plain
+text. Bitmark input only.
+
+Options (`SemanticTokensOptions`, also `ConvertOptions` keys):
+
+- `positionEncoding` — `"utf-16"` (default; the LSP default, what browser
+  editors index by) or `"utf-8"` (bytes). Echoed in the response.
+- `tokensLayout` — `"lsp"` (default): `data`, the LSP relative encoding,
+  five integers per token (`deltaLine, deltaStart, length, tokenType,
+tokenModifiers`; the type is a legend index, the modifiers a bitset); or
+  `"absolute"`: `tokens`, one `{ line, start, length, type, modifiers }`
+  per token, 0-based, by name. Both carry `legend` and `positionEncoding`.
+
+Lines break at `\n`, `\r\n` and `\r`; no token contains a line terminator.
+The legend orders are stable (a change is semver-major; new entries may be
+appended). Token types name sigil classes and syntax positions, never a bit
+or tag name; each has a recommended standard super type so an editor's
+default theme colours it — paste the table into a VS Code extension's
+`semanticTokenTypes`:
+
+| type                                                                               | covers                                                   | super type                         |
+| ---------------------------------------------------------------------------------- | -------------------------------------------------------- | ---------------------------------- |
+| `frontmatter`                                                                      | the region before the first bit                          | `comment`                          |
+| `bitSigil`                                                                         | `[.`, level dots, `\|`, `:`, `&`, `]` of a bit header    | `keyword`                          |
+| `bitType` / `bitFormat` / `bitResourceType`                                        | the header's three fields                                | `type` / `modifier` / `type`       |
+| `tagSigil`                                                                         | every tag's own syntax (`[@`, `[!`, `[##`, `:`, `]`, …)  | `keyword`                          |
+| `propertyKey` / `resourceType`                                                     | a property tag's key / a resource tag's type             | `property` / `type`                |
+| `tagText`                                                                          | plain text inside any tag value                          | `string`                           |
+| `cardDivider` / `sideDivider` / `variantDivider` / `footerDivider` / `textDivider` | `====`, `--`, `++`, `==== footer ====`, `==== text ====` | `operator`                         |
+| `paragraphBreak`                                                                   | a `\|` paragraph break                                   | `operator`                         |
+| `headingSigil` / `heading`                                                         | `# ` / the heading's plain text                          | `keyword` / `string`               |
+| `listMarker`                                                                       | `• `, `•1 `, `•+ `, the indent                           | `keyword`                          |
+| `codeSigil` / `codeLanguage` / `codeBody`                                          | `\|code:` / the language / the body                      | `keyword` / `type` / `string`      |
+| `imageSigil` / `imageSrc`                                                          | `\|image:` / the source                                  | `keyword` / `string`               |
+| `markSigil`                                                                        | a mark's open and close markers                          | `operator`                         |
+| `bold` / `italic` / `highlight` / `light` / `inline`                               | the marked text (in body text and `bitmark++` values; a string value is one literal `tagText`) | `string`                           |
+| `attrSigil` / `attrKey` / `attrValue`                                              | an attr chain's `\|` and `:` / key / value               | `operator` / `property` / `string` |
+| `url`                                                                              | a bare URL the parser auto-linked                        | `string`                           |
+| `text`                                                                             | plain body / card / footer text, and every top-level gap | (none)                             |
+| `plainText`                                                                        | a plain region (after `==== text ====`, or a raw body)   | `string`                           |
+
+Modifiers, in bit order: the tag classes `property`, `resource`, `title`,
+`reference`, `anchor`, `item`, `instruction`, `hint`, `true`, `false`,
+`gap`, `mark`, `solution` (every token of a tag of that class, sigils
+included — exactly one per tag); `comment` (every token of a commented-out
+bit); `unclosed` (tokens inside a parser unclosed-mark / unclosed-tag
+notice); `bold`, `italic`, `highlight`, `light` (the text of an inline mark
+`==…==\|…\|` whose chain carries that bare key — style it as you style the
+`bold` / `italic` / `highlight` / `light` token types). The full contract is
+`.zen/specs/API-SEM-semantic-tokens.tsp`.
+
+#### diagnostics(input: string, options?: DiagnosticsOptions): Diagnostics
+
+Every issue the JSON `parser` envelope would carry for bitmark source, as
+LSP 3.17 `Diagnostic`s with editor positions — the typed form of
+`convert(input, { inputFormat: "bitmark", outputFormat: "diagnostics" })`.
+Parse and validate, no serialization: cheap enough for every keystroke. The
+validator owns every issue, so this is exactly the envelope's set — the same
+`code`s, the same spans, the same order (bit by bit) — with the envelope's
+buckets as LSP severities: `errors` → 1 (`DiagnosticSeverity.Error`),
+`warnings` → 2, `infos` → 3. Bitmark input only.
 
-#### lex(input: string, options?: LexOptions): string
+```js
+import { diagnostics, DiagnosticSeverity } from "@gmb/bitmark-parser";
+
+const { positionEncoding, diagnostics: list } = diagnostics(source);
+// list[i] = {
+//   range: { start: { line, character }, end: { line, character } },  // 0-based, UTF-16 units
+//   severity: 2,                        // DiagnosticSeverity.Warning
+//   code: "unknown-property",           // the contract — branch on this
+//   source: "bitmark",
+//   message: "[@nope] is an unknown property …",   // prose, may change
+//   data: { bit: 0 },                   // the bit's index in the JSON array
+// }
+```
+
+Option: `positionEncoding` — `"utf-16"` (default) or `"utf-8"`, as for
+`semanticTokens`; echoed in the response. The types (`Diagnostic`, `Range`,
+`Position`, `DiagnosticSeverity`) are written field for field as the
+protocol defines them, so a `Diagnostic` assigns to Monaco's marker input
+and to `vscode-languageserver-types` without a cast, and the package keeps
+its zero runtime dependencies. In every wasm variant. The contract is
+`.zen/specs/API-EDT-editor-services.tsp`.
+
+#### complete(input, position, options?) / resolve(input, position, item, options?) / hover(input, position, options?)
+
+The editor position queries, in LSP 3.17 shapes (PLAN-196, PLAN-202). Positions are
+0-based `{ line, character }` in `positionEncoding` units — `"utf-16"` by
+default, as for `semanticTokens`.
+
+`complete` returns an LSP `CompletionList` of everything valid at the cursor:
+
+| where the cursor is | what you get |
+| --- | --- |
+| a bit header's type (`[.art`) | every bit type (`kind: Class`), with its title |
+| a header's `:format` / `&resource` | the body formats / the resource types this bit allows |
+| a tag, or body / card / footer text | the tags valid in THAT scope — the bit body, a card variant, a chain — minus those already at their maximum, the ones the scope still requires sorted first (the first `preselect`ed) |
+| a tag's value | an `enum` tag's vocabulary, a `boolean` tag's `true` / `false` |
+| after a tag's `]` | that tag's chain children |
+| an inline attribute chain (`==x==\|bo`) | the attribute keys, and the closed value set of one that has it |
+| a line holding a prefix of one (empty, `=`, `==== f`) | the structural lines the bit's card set allows there (`====`, `--`, `++`), `==== footer ====`, `==== text ====` — each documented |
+| bitmark text (body text, a `bitmark++` tag's value), after none, `=` or `==` | the inline mark `==…==\|…\|` as a snippet (`insertTextFormat: 2`); never in a string value or a plain region |
+
+Items carry `kind`, `detail`, `tags: [1]` when deprecated, `sortText`,
+`insertText`, and — in LSP's own `data` slot — the `info` record behind
+them, so you can build UI from fields rather than prose. `isIncomplete` is
+always `false`: every candidate for the context comes back and filtering by
+the typed prefix is the editor's job. A half-typed `[@wi`, `[.art` or
+`==x==\|bo` resolves as if it were closed. A header resource attachment
+(`[.flashcard&image]`) is offered at the bit's level until it is there; a
+tag whose label is only its sigil (`%`, `!`) leads its `detail` with its
+name (`item`, `instruction`). The result adds one field beyond LSP,
+`context`, saying where the query resolved.
+
+Pass `triggerCharacter` in the options when a character opened the query
+(LSP `CompletionContext.triggerCharacter`; Monaco and VS Code hand it to the
+provider). A trigger that opens nothing where the cursor is — a `.` or a
+`-` typed in prose — answers an empty list; `[` opens a tag anywhere, and
+`=` / `-` / `+` narrow the list to the structural lines they begin. An
+explicit invocation (Ctrl+Space) is never narrowed. Editors that open the
+list on every letter (Monaco's `quickSuggestions`) should turn that off for
+bitmark: it is prose with markup in it.
+
+Items carry NO `documentation` (PLAN-202): a header's list is every bit type,
+and its prose would be most of the bytes of every list. `resolve` is LSP's
+`completionItem/resolve` — pass the same `input` and `position` you gave
+`complete` plus one item as it was listed, and get that item back with its
+Markdown `documentation` (a bit's title and description; a tag's format,
+count, default, JSON key, chain and description in this scope; an inline
+attribute's shape). Wire it to Monaco's `resolveCompletionItem` or an LSP
+server's resolve handler, so only the item about to be shown is rendered.
+When the list at that position no longer holds the item, the item comes
+back as given.
+
+`hover` returns an LSP `Hover` — Markdown `contents` plus the record in
+`data` — for the bit type, a tag AS RESOLVED IN ITS SCOPE (the same record
+`info({ infoType: "bit", full: true })` shows for it there), a tag value, a
+header resource type, or an inline attribute key or value. It is `null` on
+body text, dividers and whitespace.
+
+```js
+import { complete, resolve, hover, CompletionItemKind } from "@gmb/bitmark-parser";
+
+const list = complete("[.article]\n[@", { line: 1, character: 2 });
+list.context; // { bit: "article", scope: "bit" }
+list.items.filter((i) => i.kind === CompletionItemKind.Property).map((i) => i.label);
 
-Lex bitmark text.
+const id = list.items.find((i) => i.label === "@id");
+resolve("[.article]\n[@", { line: 1, character: 2 }, id).documentation.value; // Markdown for @id here
 
-Stages (`options.stage`):
+const h = hover("[.article]\n[@id:1]", { line: 1, character: 2 });
+h.contents.value; // Markdown: format, count, default, JSON key, description
+h.data.tag; // the info record for @id in this bit
+```
 
-- `"lex"` — Unified lexer token stream, one token per line (default)
-- `"lex-json"` — Unified lexer token stream as JSON
+Both need a variant built with `editor` (all three are) — otherwise they
+throw `UnsupportedFeatureError`. All three read bitmark SOURCE, so a JSON
+document is refused; an empty buffer, or one that is still only frontmatter,
+is source being typed and is answered normally (an empty one's `complete`
+offers the bit types). The CLI has the same three services under one
+command: `bitmark editor diagnostics|complete|resolve|hover`. The contract is
+`.zen/specs/API-EDT-editor-services.tsp`.
+
+#### The `lex` output format
+
+`convert(input, { inputFormat: "bitmark", outputFormat: "lex" })` returns the
+lexer's token stream as a JSON array: one `{ kind, span, text }` per token
+(the `LexToken` type), `span` in bytes. Debug tooling — the token kinds are the
+engine's own names, not a stable vocabulary, and the parser is never run
+(pairing, abandoned-tag repair and raw-body handling all happen after lexing),
+so a highlighter should use `semantic-tokens` instead. Bitmark input only; a
+JSON or markup document has no source text to lex. The former `lex()` export
+and `bitmark lex` command were removed in 7.0 (see the migration guide).
 
 #### breakscapeText(input: string, options?: BreakscapeOptions): string
 
@@ -325,13 +598,13 @@ version and (separately) any migration target.
 - `full` — complete detail for `"bit"`/`"all"` (default: `false` = compact view)
 - `language` — BCP-47 tag(s) for display names: one (`"de"`), several
   (`["de", "fr"]` or `"de,fr"`), or `"all"` — exported as `ALL_LANGUAGES`
-  (default: English). More than one adds a `titles` map. See *Display names
-  and languages* below
+  (default: English). More than one adds a `titles` map. See _Display names
+  and languages_ below
 
 `"bit-groups"` / `"resource-groups"` return the search/filter catalogs —
 each group's key, translated title, description, optional aliases and
 `subgroupOf`, and its members. They are available in EVERY build: the group
-catalog and per-bit titles/descriptions are "descriptive" data, which the
+catalog and per-bit titles are "descriptive" data, which the
 lean wasm variants carry too. Deprecated members are excluded by default and
 MARKED when included with `includeDeprecated: true`; a search index matching
 already-published content needs them, because a migrating bit is re-emitted
@@ -341,13 +614,13 @@ To derive quiz categories, intersect a bit's `bitGroups` with the groups
 carrying `subgroupOf: "quizzes"` — `subgroupOf` is metadata and never implies
 membership, so every member of a subgroup also declares the parent.
 
-Only the META fields — TAG descriptions, group-inheritance provenance and raw
-mapping patterns (the `info-meta` cargo feature) — depend on the build. They
-are reported by the native CLI and the wasm `full` variant; `browser-full` and
-`bitmark-json` omit them to stay small. Those are absent structurally: the key
-is missing, never `null` or empty. Everything else `info` returns — including
-bit titles, bit descriptions and the group catalogs — is identical in every
-variant.
+Only the META fields — BIT and TAG descriptions, group-inheritance provenance
+and raw mapping patterns (the `info-meta` cargo feature) — depend on the build.
+They are reported by the native CLI and the wasm `full` variant; `browser-full`
+and `bitmark-json` omit them to stay small (the bit descriptions alone are
+14 KB gzip of the download). Those are absent structurally: the key is
+missing, never `null` or empty. Everything else `info` returns — including bit
+titles and the group catalogs — is identical in every variant.
 
 #### Display names and languages
 
@@ -390,7 +663,12 @@ which categories it is in":
 
 ```js
 const groups = JSON.parse(
-  info({ infoType: "bit-groups", format: "json", includeDeprecated: true, language: "all" }),
+  info({
+    infoType: "bit-groups",
+    format: "json",
+    includeDeprecated: true,
+    language: "all",
+  }),
 );
 groups[0].bitTypes[0]; // { name: "assignment", title: "Assignment", titles: {…} }
 ```
@@ -508,7 +786,7 @@ const patch = patchEntry("id", "append", "1234");
 
 See `examples/` for runnable scripts covering the whole API (executed by
 `npm test`, so they stay correct). Generated API documentation (TypeDoc) is
-published at <https://getmorebrain.github.io/bitmark-parser-rust/docs/api/>
+published at <https://getmorebrain.github.io/bitmark-parser/docs/api/>
 with each release (the site root hosts the bitmark language docs), or build
 it locally with `npm run docs` (→ `docs/api/`) and preview it with
 `npm run docs:serve` (http://localhost:8080, `--port` to change).
@@ -536,6 +814,28 @@ Note: deprecated bit types with a configured migration target (the
 `isCollapsible: true` — consumers keyed on the old `type` strings see the
 base type instead. `collapsible` itself is unaffected.
 
+### Property number values
+
+A tag whose configured format is `number` accepts what JavaScript's
+`Number(value)` accepts, minus the values that are not finite:
+
+- surrounding whitespace, a leading `+`, leading zeros (`01`), a trailing
+  point (`1.`) and a leading point (`.5`);
+- an exponent in either case (`1e2`, `1E2`);
+- hexadecimal, binary and octal literals (`0x1F`, `0b101`, `0o701`) — prefix
+  case-insensitive, with no sign, fraction or exponent.
+
+`NaN`, `Infinity`, an exponent beyond ±4000, and a radix literal past 64 bits
+are **rejected**: the tag is dropped from the output and a
+`property-format-mismatch` warning names it, exactly like any other value that
+does not fit its format.
+
+Accepted values print in JavaScript's `JSON.stringify` form — `0.5`, `1.5`,
+`100`, `1e+21`, `1e-7` — so `[@width: 1.50]` emits `1.5` and `[@width:0x10]`
+emits `16`. Values with more than 17 significant digits, including integers at
+or above 2^63, keep the digits you wrote rather than being rounded through a
+floating-point double.
+
 ## Legacy API (bpg-compatible)
 
 A compatibility facade for consumers migrating from
@@ -643,8 +943,12 @@ bitmark diff old.bitmark new.bitmark --output-format patch > d.patch
 bitmark transform old.bitmark --patch d.patch --output-format bitmark   # reproduces new
 # --context N, --locate, --similarity F, --bbox-tolerance N, --color auto|always|never
 
-# Dump the lexer token stream
-bitmark lex input.bitmark
+# Parser-derived highlighting: LSP semantic tokens (bitmark input only)
+bitmark convert input.bitmark --input-format bitmark --output-format semantic-tokens
+bitmark convert input.bitmark --output-format semantic-tokens --tokens-layout absolute --position-encoding utf-8 --pretty
+
+# Dump the lexer token stream as JSON (debug tooling; bitmark input only)
+bitmark convert input.bitmark --input-format bitmark --output-format lex --pretty
 
 # Breakscape / unbreakscape text
 bitmark breakscape input.txt
@@ -687,13 +991,13 @@ The package includes pre-built browser bundles with the WASM module.
 The CDN bundle carries all three variants' JS glue and fetches only the
 selected variant's `.wasm` (from `dist/browser/wasm/`). A bare `init()` loads
 `browser-full`; `init({ feature: "bitmark-json" })` fetches the smallest
-build, and a later `init({ feature: "full" })` upgrades in place (see *WASM
-variants*).
+build, and a later `init({ feature: "full" })` upgrades in place (see _WASM
+variants_).
 
 ### Bundler (webpack / vite)
 
 ```js
-import init, { convert, lex } from "@gmb/bitmark-parser/browser";
+import init, { convert } from "@gmb/bitmark-parser/browser";
 
 await init();
 const json = convert("[.article]\nHello", { inputFormat: "bitmark" });