umath 0.1.0

data/docs/model-contract.adoc

@@ -0,0 +1,184 @@
+= UnicodeMath model contract
+:toc:
+
+== Scope
+
+`Umath.parse(input, decimal: ".", engine: :auto)` returns a
+`Umath::Model::Document`, the typed UnicodeMath language model. This
+document describes that model: how it is built from the grammar, what its
+fields hold, and how it is checked. The rules shared by all stage gems
+(API shape, errors, engines, artifacts, corpus pins) are in the stage
+profile, amath's
+https://github.com/plurimath/amath/blob/main/docs/stage-contract.adoc[`docs/stage-contract.adoc`];
+this document only adds what is specific to UnicodeMath. The grammar,
+preprocessing, the capture-shape normalization and the raw tree are
+described in link:parser-contract.adoc[the parser contract].
+
+The model is the UnicodeMath model, not a shared one. Mapping it to
+Plurimath's `Formula` classes is Plurimath's job.
+
+== Construction
+
+The grammar is normalized so that every capture always holds one kind of
+value: text, a node (hash) or a list (see the parser contract). The grammar
+binds its one root capture twice. The `raw` binding keeps the tree with its
+slices and offsets, for parity replay and error positions. The `any`
+binding (path `model`) gives the same tree as plain hashes, arrays and
+strings. `Model::Document.new` receives that tree, and lutaml-model builds
+every nested node from the declared attribute types. No gem code walks the
+parse tree to build the model, and no class has a cast hook.
+
+After construction, one separate pass resolves symbols (see below).
+
+=== Classes and fields
+
+`scripts/generate_model.rb` writes the classes in
+`lib/umath/model/nodes.rb` from the normalized rules
+(`grammars/umath.normalized-rules.json`). The mapping is mechanical,
+because every capture holds one kind:
+
+* A capture that holds a node becomes a class named after it, for example
+  `frac` becomes `Model::Frac`. Three names are changed so they do not
+  shadow Ruby or model classes: `array` becomes `ArrayNode`, `symbol`
+  becomes `SymbolNode` and `root` becomes `RootNode`.
+* A capture that holds a list becomes a collection attribute of the class
+  `<Capture>Item`, for example `prime_accent_symbols` holds
+  `PrimeAccentSymbolsItem` elements.
+* A capture that holds text becomes a String attribute.
+* Each class declares one attribute for every capture that can appear
+  directly inside it. An attribute has the same name and the same type in
+  every class that declares it, because its type is the type of the
+  capture it names.
+* Captures that hold a whole expression (25 or more possible inner
+  captures, such as `expr`, `exp`, `factor`, `numerator`, `base`, `sub`,
+  `sup`, `first_value` and the root) share one class, `Model::Expression`,
+  which declares the union of their inner captures. The mini-script
+  captures (`mini_expr`, `sub_recursions` and the like) share
+  `Model::MiniExpression`, and `open_paren` and `close_paren` share
+  `Model::Paren`. lutaml-model compiles a serialization transformation for
+  every path through the class graph, and with one class per expression
+  capture that number grows past any practical time. The generator fails if
+  any other cycle of classes is left outside `Expression`.
+* The captures added by the normalization appear as ordinary fields:
+  `literal` (text that legacy folded into its parent), `empty` (an absent
+  optional part, legacy `""`), `list` (a list in a node slot) and `present`
+  (a lookahead that matched).
+
+These rules give 74 classes plus `Document`. The exported declarations,
+with every field and its type, are in `model/umath.lml.json`.
+
+Text fields keep the text the grammar matched. The grammar runs on the
+preprocessed, entity-encoded input, so non-ASCII text appears as character
+references: a `text` capture for `"αβ"` holds `&#x3b1;&#x3b2;`. This is the
+same text the legacy transform received. The `value` of a symbol gives its
+Unicode text.
+
+=== Symbols
+
+The symbol captures are the captures whose legacy transform rules build a
+symbol through Plurimath's symbol catalog (`alphanumeric`, `operator`,
+`ordinary_symbols`, `relational_symbols`, `nary_class`, `unicode_symbols`,
+`accent_symbols`, the paren prefixes and others; the list is in
+`scripts/generate_grammar.rb`). The grammar captures their text as
+`spelling`, so each holds a `Model::MathSymbol` with `spelling`, `value`
+and `kind`. The symbol captures that hold a node (`symbol`, `open_paren`,
+`close_paren`, `prime_accent_symbols`) capture their own text the same way,
+and their classes (`SymbolNode`, `Paren`, `PrimeAccentSymbolsItem`) also
+declare `spelling`, `value` and `kind`. For example, the base of `x^2` is
+a `SymbolNode` with spelling `x`.
+
+=== Document
+
+`expression`:: The formula's tree, an `Expression`. Nil for input without
+a formula: an empty string, or a string made only of `#` label
+separators. The legacy facade fails on such input with `NoMethodError`, so
+it has no defined result to keep; like amath, the gem returns an empty
+result. An empty formula before a label, such as `#b`, reaches the grammar
+and is refused there.
+`label`:: The equation label: the text after the last `#`, or nil when the
+input has none. This follows the legacy facade. A `#` inside `✎(...)` does
+not start a label.
+`decimal`:: The decimal marker the input was parsed with.
+
+=== Numbers
+
+Numbers are kept as written. `number` is the digit string, and
+`DecimalNumber` has `whole`, `decimal` (the marker as matched, for example
+`.` or `&#x66b;`) and `fractional`. Nothing is converted to a float.
+
+== Symbol resolution
+
+`Umath::Symbols.resolve` visits every node that has a `spelling` and
+looks it up in `data/umath-symbols.json`. That file is the legacy
+symbol catalog exported from the pinned Plurimath commit. Like the legacy
+transform, the lookup uses the spelling with surrounding whitespace
+removed.
+
+`spelling`:: The token as the grammar matched it, for example `&#x3b1;` for
+`α` or `alpha` for `\alpha`.
+`value`:: The Unicode text the catalog entry writes as UnicodeMath, for
+example `α`.
+`kind`:: `symbol` or `paren` for catalog entries. When the catalog has no
+entry the kind is `unlisted`, and the value is the spelling with its
+character references decoded. The legacy transform keeps such a spelling
+as a plain symbol.
+
+== Errors and positions
+
+Errors follow the stage profile. A `ParseError` reports `input` (the
+caller's text as UTF-8), `char_offset` and `byte_offset`, both measured in
+the caller's text. The grammar reports positions in the preprocessed text.
+`Umath::SourceMap` repeats each preprocessing step on the caller's
+text and records where every character came from, so a grammar position can
+be mapped back. Characters produced by one replacement, such as the
+`&#x3b1;` that stands for `α`, all map to the position of the original
+character. The end of the grammar input maps to the end of the formula part.
+
+`NestingTooDeep` is raised when the engine cannot handle the input's depth.
+That includes the Ruby stack running out while the model is built. In
+measurements on the Ruby engine the grammar reached its limit first in
+every case: 400 chained terms and 100 nested parentheses failed in the
+grammar, and every input the grammar accepted (200 chained terms, 50 nested
+parentheses, 100 nested fractions) was built and serialized without error.
+
+== Checks
+
+* `spec/model/model_shape_spec.rb` computes, from the normalized rules and
+  Parslet's flattening rules, which values each capture can hold, with code
+  written independently of the normalizer. It requires that every capture
+  holds exactly one kind and never nil, and that every capture is declared
+  with the type of its kind and every class declares exactly the captures
+  it can hold.
+* `scripts/generate_grammar.rb --check` and `scripts/generate_model.rb
+  --check` require the committed grammar, wrapper table, normalized rules
+  and model classes to equal a fresh generation.
+* `scripts/model_tree.rb` turns a model back into plain data and compares it
+  with the tree the bindings produced. lutaml-model ignores keys that a
+  class does not declare, so this is how a missing declaration is caught.
+  It also fails when an attribute holds a value of the wrong type.
+* `scripts/replay_corpus.rb --staging corpora/model` replays the
+  representative rows in `corpora/model/`. For each row it checks that the
+  model holds the bound tree and compares the document with the stored
+  expectation. The expectations are the gem's own output, so they guard
+  against regressions; they are not legacy parity.
+* In the replay workflow, every fixture and authored oracle row runs, on
+  the Ruby and the native engine, through the raw-tree parity spec
+  (`canonical(unwrap(raw tree)) == legacy tree`) and through
+  `scripts/replay_corpus.rb --inventory ... --rows ...`, which checks raw
+  parity again and that the model holds the bound tree. A row the legacy
+  grammar refused must raise `ParseError`, and a row without a formula
+  must give an empty document.
+  On the native engine, the rows in
+  `spec/parser/fixtures/oracle/native-divergences.yml` give another tree
+  than legacy, because Parsanol's native engine takes another alternative
+  of the same grammar. The grammar before the shape normalization gives the
+  same native trees, so the normalization does not cause them. Both checks
+  require those rows to still differ, so a fixed engine shows up as a
+  failure that asks to shorten the list.
+* `scripts/verify_lml.rb` exports every model class with
+  `Lutaml::Model::Lml.export`, requires the committed
+  `model/umath.lml.json` and `model/umath.lml.checksums.txt` to
+  be byte-identical to a fresh export, and runs `Lutaml::Model::Lml.verify`
+  on each document. `Lml.verify` re-exports the loaded Ruby class, so it
+  does not prove that another implementation can rebuild the model. The
+  model uses no union types, which LML export rejects.

data/docs/parser-contract.adoc

@@ -0,0 +1,341 @@
+= UnicodeMath parser contract
+:toc:
+
+== Status and scope
+
+This document covers syntax only. It records what the legacy UnicodeMath
+parser inside Plurimath does today, which parts of it this gem takes over,
+and how that legacy behaviour is captured so the gem can be compared against
+it.
+
+The gem has one public entry point, `Umath.parse(input, ...)`, which
+returns a typed language model declared with LutaML Model (LML). That model
+is described in link:model-contract.adoc[the model contract].
+
+The raw parse tree that the grammar produces is not public API. It is an
+internal boundary that the gem and its tests use, and it is the boundary
+the parity captures compare first.
+
+The repository holds the grammar, its compiled artifacts, the internal
+raw-tree parser (`Umath::RawParser`), the typed model and the public
+parser, the exported symbol catalog and the capture harness in
+`scripts/oracle/`.
+
+== Legacy source
+
+All legacy facts below come from one pinned commit:
+
+* Repository: https://github.com/plurimath/plurimath
+* Commit: `e7ab4a5186c9be96f62d670574bc2c72f9365ad4` (Plurimath 0.11.17)
+
+The capture harness refuses to run against any other commit or against a
+checkout with local changes.
+
+The UnicodeMath code at that commit (line counts from `wc -l`):
+
+[cols="2,1,4"]
+|===
+| File | Lines | Role
+
+| `lib/plurimath/unicode_math.rb` | 22 | Facade class: `new(text).to_formula`
+| `lib/plurimath/unicode_math/parser.rb` | 56 | Preprocessing, then grammar, then transform
+| `lib/plurimath/unicode_math/parse.rb` | 302 | Grammar class (`Parsanol::Parser`), root rule `expression`
+| `lib/plurimath/unicode_math/parsing_rules/*.rb` | 800 | Grammar rule modules: masked, sub/sup, common, constants, absence, helper
+| `lib/plurimath/unicode_math/constants.rb` | 1029 | Symbol tables used by the grammar and the transform
+| `lib/plurimath/unicode_math/transform.rb` | 4058 | Tree to `Plurimath::Math::Formula` transform
+| `lib/plurimath/unicode_math/utility.rb` | 294 | Transform helpers; subclass of `Plurimath::Utility`
+|===
+
+== Boundaries
+
+A UnicodeMath parse passes four boundaries. The capture harness records each
+one separately, so a difference can be traced to the first boundary where it
+appears.
+
+=== 1. Caller input and options
+
+The caller passes a string and, optionally, `locale:`. Through
+`Plurimath::Math.parse(text, :unicode, locale: ...)` the locale sets
+`Plurimath.configuration.locale` for the duration of the call, and the
+grammar reads its decimal marker from that configuration. Without a locale
+the decimal marker is the configuration default.
+
+Input bytes are kept exactly. Input that is not valid UTF-8 is recorded as
+bytes, not repaired.
+
+=== 2. Preprocessing
+
+`Plurimath::UnicodeMath::Parser.new(text)` rewrites the input before the
+grammar sees it. In order, as written in `parser.rb`:
+
+. Labels. A `#` inside `✎(...)` is protected; the text is then split on
+  `#`. The first part is kept as the formula and, when there was more than
+  one part, the last part becomes the label (`@splitted`). Parts between the
+  first and the last are dropped.
+. The formula text is HTML-entity encoded with hexadecimal references, then
+  `&` and `"` are turned back into `&` and `"`.
+. Comments: everything from the first `⫷` (`⫷`) to the last
+  `⫸` (`⫸`) is removed.
+. `\\` becomes `\`.
+. `\uXXXX` escapes (one to five hex digits) become `&#xXXXX;`.
+. Leading and trailing whitespace is stripped.
+
+The grammar consumes this encoded text, not the caller's text. Positions in
+grammar errors and in raw-tree slices are positions in the encoded text.
+
+The gem's public `parse` owns this preprocessing and reports refusal
+positions in the caller's text (see the model contract). The low-level parse
+of already preprocessed text is an internal boundary.
+
+=== 3. Raw grammar tree
+
+`Plurimath::UnicodeMath::Parse.new.parse(text)` runs the grammar with
+Parsanol in `compatibility: :parslet` mode. The result is a tree of hashes
+and arrays whose leaves are `Parsanol::Slice` objects. When preprocessing
+found a label, the facade wraps the tree as
+`{labeled_tr_value: tree, labeled_tr_id: label}` before the transform; the
+capture records the grammar tree and the label separately.
+
+The grammar refuses input by raising `Parsanol::ParseFailed`. The error
+message names the expected alternatives and a line and character position
+in the encoded text.
+
+The grammar is not self-contained: `ConstantsRules` builds alternations from
+`Constants` when the grammar loads, and `Constants.wrapper_symbols` reads
+Plurimath's symbol catalog (`Utility.symbols_hash(:umath)`). The
+decimal-marker rule reads `Plurimath.configuration`. A gem grammar must take
+these inputs from committed data, not from a live Plurimath.
+
+=== 4. Facade formula and writers
+
+`Plurimath::Math.parse` applies the transform and returns a
+`Plurimath::Math::Formula`. Grammar refusals and transform failures both
+surface as `Plurimath::Math::ParseError`. The capture records the formula
+object graph and the exact output of the `to_asciimath`, `to_latex`,
+`to_mathml`, `to_umath`, `to_omml` and `to_html` writers.
+
+== What the gem must keep (syntax)
+
+* Every input the legacy grammar accepts at the pinned commit is accepted,
+  and every input it refuses is refused.
+* For accepted input, the internal raw tree, with the captures added by
+  the shape normalization removed (`scripts/unwrap.rb`, see below), matches
+  the legacy raw tree in the canonical tagged form below: same keys, same
+  order, same slice text and offsets.
+* Preprocessing matches the legacy preprocessing for the same input and
+  locale, including label splitting and comment removal.
+* A refusal reports an error category and a position. The position is part
+  of the contract; the wording of the grammar's error message is not, unless
+  a corpus row depends on it.
+
+== Grammar and artifacts
+
+`data/umath-grammar.json` is a one-time export of the legacy grammar
+at the pinned commit, written by `scripts/generate_grammar.rb --export`.
+`scripts/generate_grammar.rb` turns it into `grammars/umath.parg`;
+rules that read the decimal marker are emitted once per marker, giving the
+entries `dot`, `comma` and `arabic`.
+
+=== Capture-shape normalization
+
+Under Parslet's flattening rules the legacy grammar gives one capture name
+text, a hash or a list depending on which alternative matched. Before
+writing the grammar, `scripts/umath_shapes.rb` rewrites the rules so
+that every capture always holds one kind. It only adds captures, which never
+changes what the grammar accepts. Rules N1 to N7 are the shared ones in
+`scripts/normalize_shapes.rb`, the same file in every stage gem; N8 and the
+checks below are UnicodeMath's, in `scripts/umath_shapes.rb`:
+
+N1 text alternative:: a text-only alternative of a node slot becomes
+`(x) as literal`.
+N2 empty alternative:: an optional part of a node slot becomes
+`x / "" as empty`.
+N3 list in node slot:: a list in a node slot becomes `(x) as list`.
+N4 list slot:: a capture that can repeat always holds a list.
+N5 text sibling:: text next to captures in a sequence that legacy could
+fold to text becomes `literal` (or `literal_2`, ...).
+N6 forced rule:: a rule that can return text or a node is rewritten once,
+for all its call sites.
+N7 presence capture:: a capture of a lookahead (always nil) becomes
+`(x "" as present) as name`.
+N8 symbol spelling:: the text of a symbol capture (one the legacy
+transform looks up in the symbol catalog, listed in
+`scripts/generate_grammar.rb`) is captured as `spelling`; in the symbol
+captures that hold a node (`symbol`, `open_paren`, `close_paren`,
+`prime_accent_symbols`) N1 uses `spelling` instead of `literal`.
+
+Generation fails when a capture still holds more than one kind, or nil, when
+one sequence can carry the same added capture twice, or when a sequence can
+have more than one hash part after a `list` capture (legacy appends each of
+them to the list separately, and the unwrap could not tell them apart). The
+generator writes, beside the grammar, `grammars/umath.wrappers.json`
+(the added capture names, how to remove each one, and how often each rule
+applied) and `grammars/umath.normalized-rules.json` (the normalized
+rules, from which `scripts/generate_model.rb` writes the model classes).
+`ruby scripts/generate_grammar.rb --check` and
+`ruby scripts/generate_model.rb --check` fail when any of these files is
+stale; CI runs both.
+
+`scripts/unwrap.rb` is the test-only inverse. It replaces each added capture
+with the value legacy gave there and repeats Parslet's own folding
+(`Parsanol::Atoms::CanFlatten#merge_fold` and `#flatten_repetition`). The
+parity replay, the model replay and the stage-cases generator use it; the
+gem does not.
+
+`scripts/compile.rb` compiles the grammar with
+`parsanol parg compile ... --bindings all` (Parsanol 1.3.131 or later). That
+writes `artifacts/umath.json` and one bindings artifact per Parsanol
+binding target, `artifacts/umath.bindings.<target>.json`. With
+`--determinism` the script compiles a second time and requires identical
+bytes for every file. It then writes `artifacts/umath.checksums.txt`,
+which records the compiler version, the envelope and per-target binding
+checksums and the sha256 of every file. CI compiles again with the compiler
+and Ruby version that the committed artifact records, and fails if any file
+differs.
+
+Each entry binds its root capture twice: `umath_tree` as `raw`, the
+normalized tree with slices and offsets, and as `any` under the path `model`,
+the plain tree that the typed model is built from. At load time the gem
+checks the bindings artifact's kind, checksum, target and version, pairs it
+with the parse artifact and requires both bindings. It never falls back to
+another artifact.
+
+== Symbol catalog
+
+`data/umath-symbols.json` is the symbol catalog the legacy transform
+uses: every UnicodeMath spelling of Plurimath's `Math::Symbols::Symbol` and
+`Math::Symbols::Paren` classes, as `Plurimath::Utility.all_symbols_classes(:umath)`
+returns them at the pinned commit. For each spelling it records the Unicode
+text the class writes as UnicodeMath (`value`) and whether the class is a
+symbol or a paren (`kind`). It has 3282 spellings, 64 of them parens.
+
+`scripts/export_symbols.rb` writes the file. It needs a clean checkout of
+the pinned commit and the oracle bundle:
+
+[source,sh]
+----
+BUNDLE_GEMFILE=scripts/oracle/Gemfile BUNDLE_PATH=vendor/oracle-bundle \
+  bundle exec ruby scripts/export_symbols.rb ORACLE_CHECKOUT
+----
+
+Like the grammar export, this is a provenance step. The build and the gem
+read only the committed file and never load Plurimath.
+
+== Capture harness
+
+`scripts/oracle/capture.rb` records the legacy behaviour for rows of an
+inventory.
+
+=== Inventory
+
+Inventory files live in `corpora/oracle/inventory/` and use schema
+`umath-oracle-inventory/1`. Each row has an `id`, exactly one of
+`input` (text) or `input_base64` (exact bytes) and optional `options`
+(currently `locale`). Ids must be unique across all files and safe as file
+names. The harness stops before running anything if an id is duplicated or
+a requested id is missing.
+
+=== Processes and deadlines
+
+Each row runs in a fresh worker process, so class-level caches in the legacy
+grammar never leak from one locale to another. The worker reads its request
+on standard input and writes one JSON result to file descriptor 3. Its
+standard output and error go to log files, so stray output from Plurimath
+cannot corrupt the result.
+
+The supervisor measures a deadline with the monotonic clock. Reaching
+end-of-file on the result pipe does not end the wait: the supervisor keeps
+polling the process status until the process exits or the same deadline
+passes. Only the deadline produces a timeout. A timed-out worker gets
+`TERM`, then `KILL` after a grace period, sent to its process group.
+
+=== Outcomes
+
+[cols="1,3"]
+|===
+| Outcome | Meaning
+
+| `succeeded` | The grammar produced a raw tree.
+| `refused` | The grammar raised `Parsanol::ParseFailed`. This is data.
+| `errored` | Preprocessing or the grammar raised another exception. This is data.
+| `timed_out` | No exit before the deadline. The capture is incomplete.
+| `crashed` | The worker died, exited non-zero, or exited without a complete, valid result. The capture is incomplete.
+|===
+
+Only the worker can report `succeeded`, `refused` or `errored`; a worker
+result that claims any other outcome is treated as a crash. A timeout or
+crash is never counted as a refusal.
+
+Exceptions at the formula and writer boundaries are recorded inside the row
+and do not change the row's outcome.
+
+=== Checkpoints and resume
+
+Each finished row is written as its own file, `rows/<id>.json`, through a
+temporary file and a rename, so a row file is either complete or absent.
+Each row stores a digest of its request and of the harness identity (oracle
+commit, harness file checksums, Ruby version). On a rerun, rows with the same
+digest and a data outcome are reused; timed-out, crashed and changed rows are
+run again. `--max-attempts N` stops after N new rows, which the specs use to
+test resuming.
+
+=== Report
+
+The report (schema `umath-oracle-report/1`) lists the expected ids,
+counts per outcome, missing ids, the checksum of every row file, the
+inventory files with their checksums and the harness identity. It is
+`complete` only when every expected id has a row and no row timed out or
+crashed. The capture exits 0 when the report is complete, 2 when it is not
+and 1 when it could not start.
+
+=== Canonical tagged form
+
+Values are stored in a tagged form that keeps apart everything the parity
+comparison needs to see:
+
+* hash (`"t": "hash"`, entries in insertion order) and array (`"t": "array"`);
+* symbol (`"sym"`) and string (`"str"`), for keys and values alike;
+* a nil value (`"nil"`) and a missing key (no entry);
+* the empty string and the empty array;
+* parser slices (`"slice"`), with their text, character offset and byte
+  offset;
+* any other object (`"obj"`), with its class and its instance variables in
+  definition order;
+* strings that are not valid UTF-8 (`"bytes"`, base64);
+* numbers as their Ruby string form, never converted.
+
+Object addresses in exception messages are replaced with `0xADDR`, because
+they change on every run.
+
+== Running the capture
+
+The oracle runs in its own bundle, separate from the gem's development
+bundle. `scripts/oracle/Gemfile` lists the runtime dependencies of the
+pinned Plurimath commit and `scripts/oracle/Gemfile.lock` fixes their
+versions.
+
+[source,sh]
+----
+git clone https://github.com/plurimath/plurimath ORACLE_CHECKOUT
+git -C ORACLE_CHECKOUT checkout e7ab4a5186c9be96f62d670574bc2c72f9365ad4
+
+BUNDLE_GEMFILE=scripts/oracle/Gemfile BUNDLE_PATH=vendor/oracle-bundle \
+  bundle install
+
+bundle exec ruby scripts/oracle/capture.rb --oracle ORACLE_CHECKOUT \
+  --ids smoke-success,smoke-refusal --out tmp/oracle-capture
+----
+
+Set `ORACLE_BUNDLE_PATH` when the oracle bundle is installed somewhere other
+than `vendor/oracle-bundle`.
+
+`corpora/oracle/smoke/rows/` holds the rows of the smoke capture: one
+accepted input and one refused input, run before any bulk capture. CI runs
+the smoke capture again and compares each row's outcome and boundaries with
+these files. To regenerate them, run the capture above with
+`--out corpora/oracle/smoke` and delete the `logs/` and `report.json` it also
+writes there.
+
+The harness specs in `spec/oracle/` use a fake worker instead of Plurimath.
+They cover success, refusal, a real deadline, crashes, end-of-file before
+exit and resuming after a stop or a killed supervisor.

data/docs/testsuite-pin.adoc

@@ -0,0 +1,95 @@
+= The plurimath-testsuite pin
+
+The UnicodeMath stage corpus lives in
+https://github.com/plurimath/plurimath-testsuite[plurimath-testsuite], in
+immutable numbered generations under `stages/unicode/generations/`. Each
+generation is listed, with the digest of its files, in the append-only ledger
+`generations/unicode.yaml`. This gem replays exactly one of those
+generations, named in `spec/fixtures/testsuite-pin.yml`:
+
+[source,yaml]
+----
+schema: plurimath-stage/pin/1
+repository: https://github.com/plurimath/plurimath-testsuite
+commit: 2a18ff43a26716a60c1c4330d5107c26dc73963c
+language: unicode
+generation: 2
+snapshot_digest: sha256:0a6f19b9a7836e71544d1a027e3f3ea2397dc5e9a1aba8b7834079b1dc91593d
+----
+
+Generation 2 keeps every case of generation 1 (the legacy corpus, produced by
+Plurimath 0.11.12) byte for byte and adds a raw-tree row for each of its 72
+cases, rejections included, written by `scripts/stage_cases.rb` at umath
+1929f69.
+
+== What CI checks
+
+The `corpus` workflow checks out the testsuite at the pinned commit and then:
+
+. runs the testsuite's own validator,
+  `ruby scripts/validate-stages.rb <checkout> --pin spec/fixtures/testsuite-pin.yml`,
+  which checks that the checkout is at the pinned commit, that the ledger has
+  the pinned generation with the pinned digest, and that the generation's
+  files hash to that digest;
+. runs the same validator on copies of the pin with a wrong commit, a wrong
+  digest and a wrong generation, and fails if any of them is accepted;
+. runs `spec/corpus`, which makes the same checks in
+  `scripts/replay_corpus.rb` fail on a small generated testsuite;
+. runs `scripts/replay_corpus.rb --testsuite`, which repeats the pin checks
+  and then parses every row of every raw-tree payload of the generation with
+  `Umath::RawParser.parse` on the Ruby and the native engine. Each
+  engine's result, the preprocessed text and the tree in the legacy grammar's
+  shape or the refusal and its byte offset, must equal the row's expected
+  value in the canonical form of the testsuite's
+  `docs/tree-parity-contract.adoc`.
+
+The rows whose source id is listed in
+`spec/parser/fixtures/oracle/native-divergences.yml` are the exception on the
+native engine: Parsanol's native engine takes another alternative of the same
+grammar for those inputs, so their native result must still differ from the
+expected value, and the replay fails once one of them matches, so the list
+can be shortened. None of the listed inputs is in generation 2, whose rows
+were written only where both engines agreed.
+
+To run the replay locally:
+
+[source,sh]
+----
+git clone https://github.com/plurimath/plurimath-testsuite tmp/testsuite
+git -C tmp/testsuite checkout 2a18ff43a26716a60c1c4330d5107c26dc73963c
+bundle exec ruby scripts/replay_corpus.rb --testsuite tmp/testsuite --report tmp/replay.json
+----
+
+The `replay` workflow checks out the testsuite at the same commit and fails if
+its commit and the pin's differ. The `stage-cases` workflow writes the
+generation after the latest one of its source commit, so it still reads
+generation 1 at b4520d8; it also checks out the pinned commit and requires the
+raw-tree payloads it writes to equal the pinned generation's byte for byte.
+The same workflow also writes a typed-model generation (`scripts/stage_cases.rb
+--boundary typed-model`) from the latest generation at 5e32851: every payload
+copied byte for byte plus a `model-<group>.yaml` payload of typed models from
+`Umath.parse` for each legacy payload, uploaded as the
+`stage-generation-model` artifact.
+
+== Parity with the legacy parser
+
+The pinned generation's raw-tree rows were generated from the generation 1
+inputs, so replaying them checks the current grammar against the trees this
+gem produced when they were written. The comparison with the legacy parser
+itself is made by the `replay` workflow: it captures the parse tree of the
+pinned Plurimath for every UnicodeMath case of the testsuite's
+`fixtures/unicode-expectations.yaml` and for the rows authored in this
+repository, and requires the raw tree and the typed model built by
+`Umath.parse` to match it on both engines, apart from the listed native
+divergences. The legacy corpus itself is not changed by this gem: generation 1
+stays as it was imported.
+
+== Moving the pin
+
+Commit, generation and digest change together, in one change, and only to a
+generation that is already merged in the testsuite. Take the values from the
+ledger entry, update `TESTSUITE_COMMIT` in the `corpus` and `replay` workflows
+to the same commit, and let CI replay the new generation. A generation whose
+rows this gem no longer reproduces is a regression to fix here, or a reason
+for a new generation in the testsuite; the pin never points at a generation
+the gem does not match.