@gmb/bitmark-parser 7.2.0 → 7.4.0

package/README.md

@@ -69,9 +69,9 @@ runtime — the API surface never changes:
 
 | Feature                          | Contents                                                                                                                                                                                               | Size (approx.)         |
 | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------- |
-| `full` (Node default)            | everything: rich `info` metadata + built-in translations + the semantic `diff`                                                                                                                         | ~1025 KB (~395 KB gzip) |
-| `browser-full` (browser default) | the same conversions and `diff`, without either                                                                                                                                                        | ~818 KB (~333 KB gzip) |
-| `bitmark-json`                   | bitmark ↔ JSON only: `convert`/`canonicalize`/`transform` (formats `auto`/`bitmark`/`json`, plus the `text`, `semantic-tokens` and `diagnostics` outputs), the editor services, `info` (JSON only), breakscape, text fragments — no `diff` | ~599 KB (~239 KB gzip) |
+| `full` (Node default)            | everything: rich `info` metadata + built-in translations + the semantic `diff`                                                                                                                         | ~1044 KB (~410 KB gzip) |
+| `browser-full` (browser default) | the same conversions and `diff`, without either                                                                                                                                                        | ~855 KB (~353 KB gzip) |
+| `bitmark-json`                   | bitmark ↔ JSON only: `convert`/`canonicalize`/`transform` (formats `auto`/`bitmark`/`json`, plus the `text`, `semantic-tokens` and `diagnostics` outputs), the editor services, `info` (JSON only), breakscape, text fragments — no `diff` | ~629 KB (~256 KB gzip) |
 
 **The rule.** _Every build includes every feature, except the two
 browser-targeted variants, which each carry an explicit exclusion list._ That
@@ -82,7 +82,7 @@ below:
 | -------------------------------------------------- | ----------------------------------------------------------------------------------- |
 | native CLI, daemon, docgen, schemagen, wasm `full` | **all**                                                                             |
 | wasm `browser-full`                                | all **except** `info-meta` (bit + tag prose) and `translations` (the baked table)   |
-| wasm `bitmark-json`                                | all **except** the above and `markup`, `mapping-report`, `diff`, `lex`, `info-text` |
+| wasm `bitmark-json`                                | all **except** the above and `markup`, `mapping-report`, `diff`, `lex` (with `ast`), `info-text` |
 
 So, not in `bitmark-json`: the markup formats (`html`, `xml`,
 `xml-niso-iec`, …) as input or output (they throw an `unsupported` error), the
@@ -192,9 +192,10 @@ Convert between bitmark, JSON, and the mapped markup formats (html/xml/…).
 
 A JSON input may be an array or a single value, each entry a `{ bit: {…} }`
 envelope (the forward output) or a bare bit object. An entry that is not a
-bit produces no bitmark: it is skipped and reported as an `unknown-bit-type`
-issue naming its JSON path (`$[2] is not a bit: no "type" — skipped`) on the
-warnings channel of the CLI (`--warnings`) and the daemon; the bits beside it
+bit produces no bitmark: it is skipped and reported as a `bit-invalid` issue
+(or `bit-type-invalid` when it has a `type` the registry does not know) naming
+its JSON path (`$[2] is not a bit: expected a bit object with "type" —
+ignored`) on the warnings channel of the CLI (`--warnings`) and the daemon; the bits beside it
 still convert, and a document in which nothing is a bit converts to empty
 output. This `convert` returns a bare string and so cannot show the skip.
 Only malformed JSON *text* throws (`invalid-json`).
@@ -202,7 +203,7 @@ Only malformed JSON *text* throws (`invalid-json`).
 Options:
 
 - `inputFormat` — `"auto"` (default), `"bitmark"`, `"json"`, or a mapping id (`"html"`, `"xml"`, …)
-- `outputFormat` — `"auto"` (default: the opposite direction), `"text"` (lossy plain-text extraction; output-only), `"semantic-tokens"` / `"diagnostics"` / `"lex"` (output-only, bitmark input only — see their sections), or as above
+- `outputFormat` — `"auto"` (default: the opposite direction), `"text"` (lossy plain-text extraction; output-only), `"semantic-tokens"` / `"diagnostics"` / `"lex"` / `"ast"` (output-only, bitmark input only — see their sections), or as above
 - `mode` — `"optimized"` (default) or `"full"`
 - `warnings` — include validation warnings (default: `false`)
 - `plainText` — output text as plain text (default: `false`)
@@ -255,7 +256,7 @@ Options:
 For bitmark input, output is a JSON array of entries with:
 
 - `bit` — serialized bit payload
-- `parser` — parser metadata, `errors` / `warnings`, and `infos` (informational notices for recoveries that are not necessarily wrong: text kept as text because it only looks like an unclosed tag, `unclosed-tag`, or a formatting mark with no partner, `unclosed-formatting` — in bitmark malformed markup is text, not an error). Each issue carries a stable machine-readable `code` (`"unknown-property"`, `"missing-required-tag"`, …) beside its human-readable `message`, plus `text` and `location`. **Branch on `code`** — a shipped code never changes meaning, whereas the message text is not contractual and may be reworded in any release
+- `parser` — parser metadata, `errors` / `warnings`, and `infos` (informational notices for recoveries that are not necessarily wrong: text kept as text because it only looks like an unclosed tag, `tag-unclosed`, or a formatting mark with no partner, `formatting-unclosed` — in bitmark malformed markup is text, not an error). Each issue carries a stable machine-readable `code` (`"tag-invalid"`, `"tag-missing"`, …) and a `scope` (`"bit"`, `"card"`, `"bit-chain"` or `"card-chain"` — where the issue sits) beside its human-readable `message`, plus `text` and `location`. **Branch on `code` and `scope`** — a shipped code never changes meaning, whereas the message text is not contractual and may be reworded in any release. `bitmark info error-codes` lists every code with its severity and message template (see `info` below)
 - `bitmark` — source bitmark text (a convenience echo; every bit is fully described by `bit` alone)
 
 A region the parser could not read — an unknown bit type, or non-blank text
@@ -266,6 +267,25 @@ commented-out bit and any `:format` / `&resource` suffix; absent when there
 was no header) and `body` (the raw text after it, as a plain string), so
 generating bitmark from it reproduces the original region.
 
+The other direction reports too. Going **to** bitmark (from JSON or from
+markup), an inline mark that has no bitmark form is dropped — the forward
+parser would not read it back as that mark, so writing it would corrupt the
+surrounding text. The text it decorated is always kept, and the loss is
+reported: `mark-dropped` when the whole mark goes, `mark-reduced` when it
+survives in a lesser form (its value cleared, or one of its segments dropped).
+Both arrive on the conversion's warnings, like every other issue. In practice
+they are rare — a `href` holding a line terminator, a mark kind the grammar has
+no tag for — so treat either as a signal that the JSON holds something bitmark
+cannot express, not as routine noise. Likewise `list-number-invalid`: a list
+`start` or item `value` that is not a valid number for its list (`4000` on a
+Roman list), or a `value` on a bullet or task item, is ignored and reported.
+
+Reading bitmark, an inline property list (`==text==|link:…|?hint|`) reports two
+things (PLAN-217): `text-tag-cardinality-exceeded` when a child is given more
+often than it may be (`|?a|?b|` — the later value wins), and
+`text-tag-value-invalid` when a value does not fit its format but is kept
+(`|symbol:…|width:wide|`).
+
 #### canonicalize(input: string, options?: CanonicalizeOptions): string
 
 Re-emit input in its own format in canonical form — the single same-format surface. `mode`: `"optimized"` (default, omit natural defaults) or `"full"` (all keys).
@@ -277,7 +297,17 @@ Apply a patch document to a document, bit by bit, and re-emit it.
 
 - `patch` — a `PatchDocument`, or the shorthand: a bare array of entries
   applied to every bit (build entries with `patchEntry`). Either form may be
-  a JSON string
+  a JSON string. An entry's `path` is in one of two languages, told apart by
+  its first character (one language per call; a mixed call is refused):
+  - a **JSON path** addresses the bit JSON (`title`,
+    `quizzes[0].choices[1].choice`, `table.data[1][1]`, `id[-1]`) and the
+    value is a JSON value;
+  - a **bitmark path** addresses the bit as an author writes it and the value
+    is bitmark source (see *Bitmark paths* below). Bitmark paths need
+    bitmark input
+- a skipped entry — a bad path, an index out of range, a value the engine
+  refuses — **throws `PatchError`** (`diagnostics: { path, kind, message }[]`)
+  rather than returning a document with the edit silently missing
 - `preHook` / `postHook` — per-bit hooks; the pre-hook may return patches, an
   output-format override or `drop`. Hooks see the original bits, in order,
   before the document reshapes anything; a hook's patches apply after the
@@ -286,13 +316,14 @@ Apply a patch document to a document, bit by bit, and re-emit it.
   `spacesAroundValues` — as for `convert`. `outputFormat` (default `"json"`)
   takes every per-bit format: `"json"`, `"bitmark"`, `"text"` or a mapping id
   such as `"html"` (per-bit text outputs are joined with newlines). The
-  whole-document outputs `"lex"`, `"semantic-tokens"` and `"diagnostics"` are
-  refused with an `unsupported` error in every driver — use `convert`. An
-  explicit `inputFormat` is honoured as given, patch or hooks or not; it is
-  never re-detected from the content
+  whole-document outputs `"lex"`, `"ast"`, `"semantic-tokens"` and
+  `"diagnostics"` are refused with an `unsupported` error in every driver —
+  use `convert`. An explicit `inputFormat` is honoured as given, patch or
+  hooks or not; it is never re-detected from the content
 
 A patch document addresses bits by their position (or id) in the input
-**before** any entry is applied, so it reads like a diff:
+**before** any entry is applied, so it reads like a diff. An inserted `bit`
+is a JSON bit, or a string holding one bit of bitmark source:
 
 ```json
 {
@@ -319,6 +350,53 @@ or `-1` for the document start. A bit may be the target of at most one
 and moving it are all rejected, naming the second of the conflicting entries.
 `diff` produces such a document from two versions of a file (below).
 
+#### Bitmark paths
+
+A bitmark path spells the thing it names the way an author writes it, so the
+same path works for every bit type and no JSON key name is needed:
+
+| Path | Addresses |
+| --- | --- |
+| `[@id]`, `[!]`, `[#]`, `[&image]` | the bit's first such tag; `[@id][-1]` the last, `[-][1]` the second `[-…]` |
+| `[&image][@width]` | `[@width]` in the run after `[&image]` — a chain is spelled like the chain |
+| `$body` | the bit's own text (tags, cards and footer untouched) |
+| `$raw` | everything after the bit header |
+| `$card[1]` | the whole second card; `$card[1][0]` its first side, `$card[1][0][1]` a variant |
+| `$card[1][!]`, `$card[1][-][1]` | tags inside a card, by occurrence |
+| `$card[1]$body`, `$card[1]$raw` | a card's own text / whole content |
+| `$footer` | the `==== footer ====` section |
+
+Values are bitmark source, breakscaped as an author would write it (a bare
+`]` or a trailing `^` in a tag value is refused — use `breakscapeText` with
+`location: "tag"` for user data, and add `subLocation: "attrValue"` when the
+value is an inline-attr chain value such as a `link:` href); `null` is a
+valueless tag (`[@x]`). The
+four ops are the JSON engine's: `set` replaces a tag's value, a section's
+content or a scope's text; `create` sets only when absent; `delete` removes
+a tag (**a run head takes its whole run** — `[&image:u][@width:1]` minus
+`[&image]` leaves nothing behind), a section, a body; `append` adds another
+occurrence after the last one on its own line (`$card[1][-]` twice gives
+white, green, blue, red), a sibling section after `$card[c]`, or a new last
+card for a bare `$card`.
+
+Every edit is a splice of the bit's own source: the parser's tree is the
+address map, the rest of the bit keeps its formatting, and an edit that
+would change the structure outside the addressed scope — a `==== text ====`
+or `==== footer ====` that would swallow the cards after it, a bit header in
+a value — is refused with the bit untouched. `convert(…, { outputFormat:
+"ast" })` shows the tree a path resolves against.
+
+```js
+transform(doc, {
+  outputFormat: "bitmark",
+  patch: [
+    { path: "$card[1][-]", op: "append", value: "green" },
+    { path: "$card[0][!]", op: "set", value: "Milk (cold)" },
+    { path: "$body", op: "set", value: "Think **twice**." },
+  ],
+});
+```
+
 #### diff(a: string, b: string, options?: DiffOptions): string
 
 Semantic diff of two documents. Both sides are re-canonicalized and
@@ -458,10 +536,10 @@ const { positionEncoding, diagnostics: list } = diagnostics(source);
 // list[i] = {
 //   range: { start: { line, character }, end: { line, character } },  // 0-based, UTF-16 units
 //   severity: 2,                        // DiagnosticSeverity.Warning
-//   code: "unknown-property",           // the contract — branch on this
+//   code: "tag-invalid",                // the contract — branch on this
 //   source: "bitmark",
-//   message: "[@nope] is an unknown property …",   // prose, may change
-//   data: { bit: 0 },                   // the bit's index in the JSON array
+//   message: "[@nope] is not valid for [.article]",   // prose, may change
+//   data: { bit: 0, scope: "bit" },     // the bit's index in the JSON array; the issue's scope
 // }
 ```
 
@@ -552,7 +630,7 @@ offers the bit types). The CLI has the same three services under one
 command: `bitmark editor diagnostics|complete|resolve|hover`. The contract is
 `.zen/specs/API-EDT-editor-services.tsp`.
 
-#### The `lex` output format
+#### The `lex` and `ast` output formats
 
 `convert(input, { inputFormat: "bitmark", outputFormat: "lex" })` returns the
 lexer's token stream as a JSON array: one `{ kind, span, text }` per token
@@ -563,19 +641,59 @@ so a highlighter should use `semantic-tokens` instead. Bitmark input only; a
 JSON or markup document has no source text to lex. The former `lex()` export
 and `bitmark lex` command were removed in 7.0 (see the migration guide).
 
+`convert(input, { inputFormat: "bitmark", outputFormat: "ast" })` returns the
+parser's tree as JSON — bits, their blocks (tag runs, body text, dividers)
+and inlines, every node with its byte `span`. Debug tooling like `lex`: the
+shape follows the parser and is not a contract. It shows what a bitmark
+patch path addresses (a run is one `TagChain`; a card is what sits between
+divider blocks). Not to be confused with the legacy entry point's "ast",
+which is bit JSON. Bitmark input only; in the same builds as `lex`.
+
 #### breakscapeText(input: string, options?: BreakscapeOptions): string
 
 Breakscape text (escape bitmark special characters).
 
 - `format` — `"bitmark++"` (default) or `"plainText"`
-- `location` — `"body"` (default) or `"tag"`
+- `location` — `"body"` (default), `"cardBody"` or `"tag"`
+- `subLocation` — `"text"` (default) or `"attrValue"`
+
+`subLocation` REFINES `location`, it does not replace it. `"attrValue"` means
+the string is the value half of an inline-attr chain segment
+(`==t==|link:VALUE|`): it adds the chain delimiter `|` to the escaped set and
+drops the running-text rules (inline doubles, block markers), while `location`
+keeps deciding the tag rules — a `[`+trigger is escaped in `body` and inert in
+`tag`, a `]` is inert in `body` and escaped in `tag`.
+
+`"cardBody"` is a card side or variant body. It has the `body` rules plus the
+card dividers `--` and `++`, which are markup only inside a card set and only
+at the start of a line, where they break to `-^-` and `+^+`.
+
+Reach for `subLocation: "attrValue"` when building a chain value for the patch
+API by hand; without it, an href containing `[@` cannot be written correctly.
+
+```ts
+breakscapeText("http://x/a[@b]", { location: "body", subLocation: "attrValue" });
+// → "http://x/a[^@b]"
+```
+
+Carets themselves follow one rule: **a literal `^` is written `^^` except with
+`format: "plainText"` and `location: "body"`, where it is written `^`.** Three
+cases land on the `^^` side although they read as though they would not — a tag
+value in a plain-format bit (the `:text` suffix switches the body only), an
+inline `code` mark (ordinary bitmark text carrying a mark), and an attr value
+or bare URL. A `|code` *block* body is raw, but it has no public `format`: the
+parser and generator apply that rule themselves at the block boundary.
 
 #### unbreakscapeText(input: string, options?: BreakscapeOptions): string
 
 Unbreakscape text (unescape bitmark special characters).
 
 - `format` — `"bitmark++"` (default) or `"plainText"`
-- `location` — `"body"` (default) or `"tag"`
+- `location` — `"body"` (default), `"cardBody"` or `"tag"`
+- `subLocation` — accepted and **ignored**
+
+For `bitmark++` the rule is "remove one caret from each run" whatever the
+context, so a single inverse undoes every breakscape mode.
 
 #### info(options?: InfoOptions): string
 
@@ -588,7 +706,10 @@ defaults), JSON keys, per-context overrides, and the card set structure. `"depre
 version and (separately) any migration target.
 
 - `infoType` — `"list"` (default), `"bit"`, `"all"`, `"deprecated"`,
-  `"bit-groups"`, `"resource-groups"`, or `"languages"`
+  `"bit-groups"`, `"resource-groups"`, `"languages"`, or `"error-codes"`
+  (every issue code the parser can emit, with its severity and its message
+  templates per scope — `ErrorCodesInfo`; read from the validator's own
+  message table, so it is what the envelope emits, in every variant)
 - `format` — `"text"` (default) or `"json"`
 - `bit` — filter to a specific bit type (when `infoType` is `"bit"`)
 - `pretty` — prettify JSON output (default: `false`)
@@ -791,6 +912,21 @@ with each release (the site root hosts the bitmark language docs), or build
 it locally with `npm run docs` (→ `docs/api/`) and preview it with
 `npm run docs:serve` (http://localhost:8080, `--port` to change).
 
+### Resolved config document
+
+The package ships the resolved bitmark configuration its parser was compiled
+against — every bit type, tag group and card set with descriptions, formats,
+defaults and mapping keys — as `bitmark.json`, with its TypeSpec contract
+beside it:
+
+```ts
+import config from "@gmb/bitmark-parser/bitmark.json" with { type: "json" };
+// contract: node_modules/@gmb/bitmark-parser/config/bitmark-json.tsp
+```
+
+It is generated by the build (`bitmark-confgen`) from the repo's jsonc config
+tree, never hand-edited, and its `label` is a content hash of that tree.
+
 ### JSON Schema
 
 The package also ships a JSON Schema (Draft 2020-12) describing the parser's
@@ -827,7 +963,7 @@ A tag whose configured format is `number` accepts what JavaScript's
 
 `NaN`, `Infinity`, an exponent beyond ±4000, and a radix literal past 64 bits
 are **rejected**: the tag is dropped from the output and a
-`property-format-mismatch` warning names it, exactly like any other value that
+`tag-value-invalid` warning names it, exactly like any other value that
 does not fit its format.
 
 Accepted values print in JavaScript's `JSON.stringify` form — `0.5`, `1.5`,
@@ -953,6 +1089,10 @@ bitmark convert input.bitmark --input-format bitmark --output-format lex --prett
 # Breakscape / unbreakscape text
 bitmark breakscape input.txt
 bitmark breakscape input.txt --format plainText --location tag
+# a card side / variant body, where the card dividers are live
+bitmark breakscape input.txt --location cardBody
+# a value for an inline-attr chain segment (==t==|link:VALUE|)
+bitmark breakscape input.txt --location body --sub-location attrValue
 bitmark unbreakscape input.txt
 
 # Query bit type information