latexmath 0.1.4 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (59) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +50 -0
  3. data/LICENSE.md +33 -0
  4. data/README.adoc +133 -14
  5. data/artifacts/latex.bindings.python.json +1 -0
  6. data/artifacts/latex.bindings.rs.json +1 -0
  7. data/artifacts/latex.bindings.ruby.json +1 -0
  8. data/artifacts/latex.bindings.ts.json +1 -0
  9. data/artifacts/latex.checksums.txt +19 -0
  10. data/artifacts/latex.json +1 -0
  11. data/data/latex-symbols.json +3301 -0
  12. data/docs/model-contract.adoc +125 -0
  13. data/docs/parser-contract.adoc +349 -0
  14. data/docs/testsuite-pin.adoc +76 -0
  15. data/grammars/latex.wrappers.json +17 -0
  16. data/lib/latexmath/model/nodes.rb +918 -0
  17. data/lib/latexmath/model.rb +44 -0
  18. data/lib/latexmath/parser/artifact.rb +80 -0
  19. data/lib/latexmath/parser/configuration.rb +35 -0
  20. data/lib/latexmath/parser/engine.rb +161 -0
  21. data/lib/latexmath/parser/errors.rb +51 -0
  22. data/lib/latexmath/parser/native_recursion_guard.rb +43 -0
  23. data/lib/latexmath/parser/preprocessor.rb +77 -0
  24. data/lib/latexmath/parser/raw_parser.rb +84 -0
  25. data/lib/latexmath/parser/source_map.rb +121 -0
  26. data/lib/latexmath/parser/unwrap.rb +91 -0
  27. data/lib/latexmath/parser.rb +97 -0
  28. data/lib/latexmath/symbols.rb +74 -0
  29. data/lib/latexmath/version.rb +1 -1
  30. data/lib/latexmath.rb +5 -87
  31. data/model/latex.lml.checksums.txt +36 -0
  32. data/model/latex.lml.json +3350 -0
  33. metadata +60 -71
  34. data/.editorconfig +0 -14
  35. data/.github/workflows/test.yml +0 -80
  36. data/.gitignore +0 -22
  37. data/.rspec +0 -3
  38. data/.rubocop.yml +0 -129
  39. data/CODE_OF_CONDUCT.md +0 -74
  40. data/Gemfile +0 -10
  41. data/Gemfile.lock +0 -79
  42. data/Rakefile +0 -95
  43. data/bin/console +0 -14
  44. data/bin/setup +0 -8
  45. data/exe/latexmath +0 -8
  46. data/latexmath.gemspec +0 -34
  47. data/lib/latexmath/aggregator.rb +0 -351
  48. data/lib/latexmath/converter.rb +0 -424
  49. data/lib/latexmath/equation.rb +0 -13
  50. data/lib/latexmath/ext.rb +0 -9
  51. data/lib/latexmath/symbol.rb +0 -21
  52. data/lib/latexmath/tokenizer.rb +0 -82
  53. data/lib/latexmath/xml/builder.rb +0 -4
  54. data/lib/latexmath/xml/element.rb +0 -25
  55. data/lib/unimathsymbols.js.erb +0 -4
  56. data/lib/unimathsymbols.txt +0 -2864
  57. data/opal/latexmath-opal.rb +0 -40
  58. data/opal/ox.rb +0 -64
  59. data/opal/pseudoenumerator.rb +0 -19
@@ -0,0 +1,125 @@
1
+ = LaTeX model contract
2
+ :toc:
3
+
4
+ == Scope
5
+
6
+ `Latexmath.parse(input, decimal: ".", engine: :auto)` returns a
7
+ `Latexmath::Model::Document`. This document says how that model is derived
8
+ from the grammar, how it is built, and how it is checked. The syntax side,
9
+ including preprocessing, errors and the legacy raw tree, is in
10
+ link:parser-contract.adoc[the parser contract].
11
+
12
+ == Shape normalization of the grammar
13
+
14
+ Under Parslet's flattening rules one capture name of the legacy grammar can
15
+ hold text, a hash, a list or nil, depending on which alternative matched. A
16
+ typed model needs one type per field. So `scripts/generate_grammar.rb` first
17
+ rewrites the exported grammar so that every capture holds one kind of value,
18
+ and then emits `grammars/latex.parg`. The rewrite only adds captures or spells
19
+ a construct in an equivalent way, so the grammar accepts exactly the same
20
+ inputs.
21
+
22
+ The rules N1 to N7 come from `scripts/normalize_shapes.rb`, shared with the
23
+ other stage gems. LaTeX needs four more, applied before them:
24
+
25
+ [cols="1,4"]
26
+ |===
27
+ |Rule |Rewrite
28
+
29
+ |N8 catalog text
30
+ |The text of a `symbols` or `unicode_symbols` capture, which the legacy
31
+ transform looks up in the symbol table, is captured as `spelling`. The model
32
+ holds a `MathSymbol` there.
33
+
34
+ |N9 empty list
35
+ |A capture of text that the legacy grammar writes as an empty list when
36
+ nothing matched (`(*x) as c`, also through a rule) holds an empty string
37
+ instead: `*x` is written `*x &*x`. Its text is captured as `text_list` (or is
38
+ the `spelling` of N8).
39
+
40
+ |N10 list site
41
+ |A capture that holds a hash at some places and a list at others is captured
42
+ as `items` where it holds a list.
43
+
44
+ |N11 named option
45
+ |`[x] as c`, which is nil when `x` is absent, is written
46
+ `(x / "" as absent) as c`.
47
+ |===
48
+
49
+ Generation stops if any capture is still multi-kind or can hold nil. It also
50
+ stops if a rule that is shared by every decimal marker would normalize
51
+ differently under one of them.
52
+
53
+ `grammars/latex.wrappers.json` lists every added capture with how it unwraps:
54
+
55
+ [cols="1,1,3"]
56
+ |===
57
+ |Capture |Kind |Legacy value
58
+
59
+ |`literal`, `literal_<n>` |text |the wrapped text
60
+ |`spelling`, `text_list` |text_list |the text, or `[]` when it is empty
61
+ |`items` |items |the list
62
+ |`absent` |nil |nil
63
+ |===
64
+
65
+ `Latexmath::Unwrap` replaces each added capture by that value and folds the
66
+ parts with Parsanol's own `merge_fold` and `flatten_repetition`, so it gives
67
+ back the legacy tree exactly. It serves the raw-tree boundary and parity
68
+ replay only.
69
+
70
+ `grammars/latex.normalized-rules.json` holds the normalized rules of the
71
+ `dot` entry, from which the model classes are generated.
72
+
73
+ == Model classes
74
+
75
+ `scripts/generate_model.rb` writes `lib/latexmath/model/nodes.rb` from the
76
+ capture shapes of the normalized rules:
77
+
78
+ * a capture that holds a hash becomes a class named after it, with one
79
+ attribute per key;
80
+ * a capture that holds a list becomes a class `<Capture>Item` for its
81
+ elements, and the attribute is a collection of it;
82
+ * a capture that holds text is a `:string` attribute;
83
+ * a capture whose only key is `spelling` is a `MathSymbol`.
84
+
85
+ An attribute has the same name and type wherever it occurs. The matrix row
86
+ break, which the legacy tree keys as `"\\"`, is the attribute `row_break`.
87
+ The classes are plain LutaML Model classes: no union types, no cast hooks.
88
+ `Document` has `expression`, the formula's tree, and `decimal`, the decimal
89
+ marker the input was parsed with. Numbers keep their digits and marker as
90
+ written.
91
+
92
+ `MathSymbol` has the attribute `spelling`, the text the grammar matched. Its
93
+ `value` (Unicode text) and `kind` (`symbol`, `open` or `close`) are read from
94
+ `data/latex-symbols.json` for the spelling, stripped, as in the legacy
95
+ transform. A spelling the table lacks stands for itself, kind `symbol`.
96
+
97
+ == Construction
98
+
99
+ Each grammar entry declares two bindings on the tree capture:
100
+
101
+ ----
102
+ latex_tree -> latex_tree (raw)
103
+ latex_tree -> model (any)
104
+ ----
105
+
106
+ The raw binding gives the tree with slices and offsets. The model binding
107
+ gives the same tree as plain data, and `Document.new(expression: ...)` builds
108
+ the whole typed tree from the declared attribute types. No code walks the
109
+ parse tree.
110
+
111
+ Input with no formula (empty, or only spaces and newlines, which the
112
+ preprocessing removes) returns a `Document` whose `expression` is nil.
113
+
114
+ == Checks
115
+
116
+ * `ruby scripts/generate_grammar.rb --check` and
117
+ `ruby scripts/generate_model.rb --check` require the committed grammar,
118
+ wrappers, normalized rules and model classes to equal a fresh generation.
119
+ * `bundle exec ruby scripts/verify_lml.rb` exports every model class with
120
+ `Lutaml::Model::Lml.export`, requires the result to equal
121
+ `model/latex.lml.json` and its checksums, and verifies every document.
122
+ * The parity replay (`spec/parser/oracle_parity_spec.rb`) requires, for every
123
+ parsed row and on both engines, that the unwrapped tree equals the legacy
124
+ tree, that the model as data equals the bound data, and that the unwrapped
125
+ model gives the legacy tree's text.
@@ -0,0 +1,349 @@
1
+ = LaTeX parser contract (syntax)
2
+ :toc:
3
+
4
+ == Status and scope
5
+
6
+ This document fixes the LaTeX syntax boundary that the latexmath stage gem
7
+ must reproduce, and the way the legacy behaviour is captured as evidence. It
8
+ covers input preprocessing, the raw grammar tree, decimal markers and errors.
9
+
10
+ It does not define the typed language model. The gem's one public entry point
11
+ is `Latexmath.parse(input, ...)`, and it returns the typed LaTeX model. The
12
+ fields, types and class names of that model are defined in
13
+ link:model-contract.adoc[the model contract]. The raw grammar tree described here is an internal and test
14
+ boundary. It is not public API.
15
+
16
+ The shared stage profile is amath
17
+ https://github.com/plurimath/amath/blob/main/docs/stage-contract.adoc[`docs/stage-contract.adoc`].
18
+ It defines the common shape of the stage gems: one public `parse`, the error
19
+ classes and their offsets, artifacts and replay. This document keeps the rules
20
+ that are specific to LaTeX.
21
+
22
+ == Implementation in this gem
23
+
24
+ The raw-tree boundary already exists in this gem:
25
+
26
+ * `grammars/latex.parg`, generated by `scripts/generate_grammar.rb` from the
27
+ pinned Plurimath grammar, and its compiled parse artifact
28
+ `artifacts/latex.json` with one bindings artifact per Parsanol target,
29
+ `artifacts/latex.bindings.<target>.json`. `scripts/compile.rb` builds them
30
+ with `parsanol parg compile --bindings all` (Parsanol 1.3.131 or later),
31
+ verifies and pairs every bindings artifact, and with `--determinism`
32
+ requires a second compilation to give the same bytes;
33
+ * `Latexmath::Preprocessor`, the legacy preprocessing described below;
34
+ * `Latexmath::RawParser`, which runs preprocessing, the grammar and the
35
+ declared bindings on the Ruby or the native Parsanol engine, and returns
36
+ the preprocessed text and the legacy raw tree. It is internal;
37
+ * `data/latex-symbols.json`, the legacy symbol table, and
38
+ `Latexmath::Symbols`, which reads it (see <<Symbols>>).
39
+
40
+ `Latexmath::ParseError` carries the offsets described under <<Errors>>.
41
+
42
+ == Reference implementation
43
+
44
+ The behaviour to reproduce is the LaTeX path of Plurimath at one fixed commit.
45
+ `scripts/oracle/pins.yml` records it:
46
+
47
+ * Plurimath `e7ab4a5186c9be96f62d670574bc2c72f9365ad4`
48
+ * plurimath-testsuite `281059b213026a54a193cb020a5816f57cabf3a8`, the source
49
+ of the fixture rows the full replay captures (see <<Replay>>)
50
+
51
+ The legacy LaTeX path at that commit consists of these files:
52
+
53
+ [cols="3,1,1,4"]
54
+ |===
55
+ |File |Bytes |Rules |Role
56
+
57
+ |`lib/plurimath/latex.rb`
58
+ |459
59
+ |0
60
+ |Entry class. `to_formula` runs the parser.
61
+
62
+ |`lib/plurimath/latex/parser.rb`
63
+ |1033
64
+ |0
65
+ |Preprocessing, then grammar, then transform.
66
+
67
+ |`lib/plurimath/latex/parse.rb`
68
+ |11744
69
+ |35
70
+ |Parsanol grammar. It also builds rules from symbol and constant tables at
71
+ run time.
72
+
73
+ |`lib/plurimath/base_number_prefix.rb`
74
+ |1622
75
+ |6
76
+ |Hexadecimal, binary and octal number rules shared with other languages.
77
+
78
+ |`lib/plurimath/latex/constants.rb`
79
+ |5301
80
+ |0
81
+ |Tables the grammar reads: environments, parentheses, operators.
82
+
83
+ |`lib/plurimath/latex/utility.rb`
84
+ |4359
85
+ |0
86
+ |Helpers, including HTML entity encoding of the decimal marker.
87
+
88
+ |`lib/plurimath/latex/transform.rb`
89
+ |29673
90
+ |116
91
+ |Turns the raw tree into Plurimath formula objects.
92
+ |===
93
+
94
+ Rule counts are the number of `rule(` lines in each file. `pins.yml` also
95
+ records the SHA-256 of each file. The grammar is data-driven: many of its
96
+ alternatives come from Plurimath symbol tables, so a stage grammar must be
97
+ generated from the pinned commit rather than written by hand.
98
+
99
+ == Input preprocessing
100
+
101
+ The public entry point owns preprocessing. The legacy steps, in order, are:
102
+
103
+ . Remember every `\mbox{...}` and `\text{...}` group in the original input.
104
+ . Decode HTML entities, then encode the whole text again with hexadecimal
105
+ entities. Every non-ASCII character, every newline and the characters
106
+ `<`, `>`, `&`, `"` and `'` become `&#x...;`.
107
+ . Remove every space that does not follow a backslash, and every run of raw
108
+ newlines.
109
+ . Replace a row break followed by a space (`\\ `) with a plain row break `\\`.
110
+ . Turn `&#x26;` back into `&` and `&#x22;` back into `"`. The other entities
111
+ stay encoded.
112
+ . Turn a backslash followed by an encoded newline (`\&#xa;`) into an escaped
113
+ space `\ `, unless that backslash ends a row break. Remove the remaining
114
+ `&#xa;`.
115
+ . Put the remembered `\mbox` and `\text` groups back exactly as the caller
116
+ wrote them, including spaces and non-ASCII characters.
117
+
118
+ For example, `a < b \\ c` followed by a newline and `\text{é x} é` becomes
119
+ `a&#x3c;b\\c\text{é x}&#xe9;`.
120
+
121
+ The grammar parses this preprocessed text, not the caller's input. Slice
122
+ offsets in the raw tree point into the preprocessed text. Outside restored
123
+ `\mbox` and `\text` groups the preprocessed text is ASCII.
124
+
125
+ == Raw grammar tree
126
+
127
+ The raw tree is what `Plurimath::Latex::Parse#parse` returns for the
128
+ preprocessed text. It is made of hashes, arrays and slices:
129
+
130
+ * A hash maps capture names to values. Capture names are symbols. Insertion
131
+ order is part of the tree.
132
+ * An array keeps repeated captures in input order.
133
+ * A slice is matched text with its offset in the preprocessed text. Parsanol
134
+ counts that offset (`Slice#offset`) in bytes, on both engines. The capture
135
+ records it as the byte offset and also records the character offset it
136
+ falls at. The two differ only after non-ASCII text inside a restored
137
+ `\mbox` or `\text` group. A stage gem that reports positions in the
138
+ caller's text must map offsets back explicitly.
139
+
140
+ One key is special. The grammar captures a matrix row break as the symbol
141
+ `:"\\\\"`, and `Parse#parse` then renames that key to the string `"\\\\"`.
142
+ This string key is part of the tree. A grammar format that cannot spell the
143
+ key must emit another name and rename it back.
144
+
145
+ The grammar in this gem adds captures so that every capture holds one kind
146
+ of value (see link:model-contract.adoc[the model contract]). The tree it
147
+ produces is therefore not the legacy tree itself. `Latexmath::Unwrap` removes
148
+ the added captures, using `grammars/latex.wrappers.json`, and gives the
149
+ legacy tree back; `Latexmath::RawParser` returns both. Parity is checked on
150
+ the unwrapped tree.
151
+
152
+ == Decimal marker and locale
153
+
154
+ The caller chooses a locale with the `locale:` option. The locale selects the
155
+ decimal marker: `.` by default, `,` for locales such as `fr`, and the Arabic
156
+ decimal separator `٫` (U+066B) for `ar`. The grammar matches the marker in its encoded
157
+ form, because preprocessing has already turned non-ASCII characters into
158
+ hexadecimal entities.
159
+
160
+ The legacy grammar caches generated rules in class variables. After the first
161
+ parse in a process, those rules keep the decimal marker of that first parse.
162
+ The stage gem must instead behave like a fresh process for every call: one
163
+ call, one marker, no state carried between calls. For the same reason the
164
+ capture harness runs every row in a new process.
165
+
166
+ == Errors
167
+
168
+ A refusal is an input the legacy path does not accept. It is data, not a
169
+ failure of the capture. A refusal happens at one of three stages:
170
+
171
+ preprocess:: preprocessing raised an error.
172
+ grammar:: the grammar did not match the preprocessed text.
173
+ transform:: the grammar matched, but the transform could not build a formula.
174
+
175
+ For grammar refusals the error position is part of the contract: the byte
176
+ offset in the preprocessed text where matching failed (Parsanol's
177
+ `Cause#pos`, which counts bytes), and the character offset it falls at. The
178
+ class of the error is recorded. The wording of grammar messages is not part
179
+ of the contract; the capture keeps only its first line and a SHA-256 of the
180
+ full message.
181
+
182
+ `Latexmath::ParseError` reports the position in the caller's text, as the
183
+ stage contract requires: `input` is the caller's input as normalized UTF-8,
184
+ and `byte_offset` and `char_offset` count UTF-8 bytes and characters in it.
185
+ `Latexmath::SourceMap` maps the grammar position back through the
186
+ preprocessing. It runs the preprocessing steps again, recording for every
187
+ character written the caller character it came from, and maps only when it
188
+ writes exactly the bytes the preprocessing wrote (otherwise the offsets are
189
+ nil). A character the preprocessing wrote in place of other text (an encoded
190
+ or decoded entity, the escaped space for a backslash before a newline) maps
191
+ to the start of that text; a restored `\text` or `\mbox` group maps
192
+ character for character; removed spaces and newlines leave nothing behind,
193
+ so a position after them maps to the caller character that follows.
194
+
195
+ The grammar's own position stays on the error for replay and diagnostics:
196
+ `grammar_input` is the preprocessed text and `grammar_byte_offset` and
197
+ `grammar_char_offset` are the recorded position in it. Raw replay compares
198
+ those with the capture, and checks that the public position lies in the
199
+ caller's text. `NestingTooDeep` sets both inputs and leaves every offset
200
+ nil.
201
+
202
+ The mapping does not change the preprocessed text itself: the raw boundary
203
+ still returns it as the legacy preprocessing does, in US-ASCII when nothing
204
+ outside a restored group is non-ASCII.
205
+
206
+ == Symbols
207
+
208
+ The legacy transform turns the text of a `:symbols` or `:unicode_symbols`
209
+ capture, stripped, into a Plurimath symbol:
210
+ `Plurimath::Utility.all_symbols_classes(:latex)[text]&.new`, or a plain
211
+ `Plurimath::Math::Symbols::Symbol` of that text when the table lacks it.
212
+
213
+ `scripts/export_symbols.rb` writes that table from the pinned Plurimath into
214
+ `data/latex-symbols.json`, so the gem resolves symbols without Plurimath. It
215
+ runs in the oracle bundle (see <<Oracle environment>>):
216
+
217
+ [source,sh]
218
+ ----
219
+ BUNDLE_GEMFILE=<oracle>/.latexmath-oracle/Gemfile \
220
+ bundle exec ruby scripts/export_symbols.rb --oracle <oracle> [--check]
221
+ ----
222
+
223
+ The file records the oracle repository, commit, `lib` tree and oracle lock,
224
+ counts, and one record per spelling, sorted by spelling:
225
+
226
+ `spelling`:: the captured text the table keys: a command name without its
227
+ backslash (`alpha`), a hexadecimal entity (`&#x3b1;`) or another spelling
228
+ Plurimath lists for LaTeX;
229
+ `value`:: the Unicode text of the MathML the legacy class writes;
230
+ `mathml`:: that MathML element, `mi` or `mo`;
231
+ `kind`:: `symbol`, or `open`, `close` or `paren` for a Plurimath paren;
232
+ `class`:: the legacy class, relative to `Plurimath::Math::Symbols`, for the
233
+ facade mapper.
234
+
235
+ At the pinned commit the table has 3,280 spellings of 1,421 classes: 3,218
236
+ symbols, 41 opening and 21 closing parens. `Latexmath::Symbols.lookup` gives
237
+ the record for a captured text, or nil; `Latexmath::Symbols.resolve` gives a
238
+ plain symbol of the text instead of nil, as the legacy transform does. The
239
+ CI parity job runs the export with `--check` against the pinned checkout.
240
+
241
+ == Capture of legacy behaviour
242
+
243
+ `scripts/oracle/capture.rb` records the legacy behaviour row by row. Each row
244
+ names an input and its configuration. For each row it records four
245
+ boundaries:
246
+
247
+ . the caller's input and options;
248
+ . the preprocessed text;
249
+ . the raw grammar tree, or the grammar error;
250
+ . the facade outcome of `Plurimath::Math.parse`: the formula objects and the
251
+ output of every writer (AsciiMath, LaTeX, MathML, OMML, HTML, UnicodeMath),
252
+ or the refusal.
253
+
254
+ The formula objects are recorded by class and instance variables. That record
255
+ is the baseline to compare against later. It does not define the stage model.
256
+
257
+ === Canonical tagged form
258
+
259
+ Every captured value is written in a tagged form that keeps the differences
260
+ the comparison needs: hash versus array, symbol versus string, key order, nil
261
+ versus a missing key, empty string versus empty list, object class, slice text
262
+ and offsets. Numbers keep their spelling and are never converted to floats.
263
+ Text that is not valid UTF-8 is kept as hexadecimal bytes. Nothing is sorted.
264
+
265
+ A slice is written as `{"t": "slice", "v": text, "byte_offset": n,
266
+ "char_offset": m}`. `byte_offset` is Parsanol's offset as it is;
267
+ `char_offset` is the number of characters in the first `byte_offset` bytes of
268
+ the preprocessed text, read as UTF-8. A recorded error has `byte_offset` and
269
+ `char_offset` in the same sense. An error message's first line is a plain
270
+ string when it is valid UTF-8. A message that quotes text which is not valid
271
+ UTF-8 keeps its first line, cut to 160 bytes, in the tagged bytes form
272
+ `{"t": "bytes", "hex": ..., "enc": ...}`, and the SHA-256 covers its bytes.
273
+
274
+ Raw trees and formulas of nested input go deeper than the JSON library's
275
+ default nesting limit of 100. The worker, the checkpoint and the report write
276
+ and read JSON without that limit.
277
+
278
+ === Row outcomes
279
+
280
+ Every row ends in exactly one of four states:
281
+
282
+ succeeded:: the child exited with status 0 and replied with a capture.
283
+ refused:: the child exited with status 0 and replied with a refusal.
284
+ timed_out:: the child was still running when the deadline passed.
285
+ crashed:: the child exited with a non-zero status, died from a signal, or
286
+ wrote a reply that is not a valid capture.
287
+
288
+ A timeout or crash is an incomplete capture, never a refusal. The deadline is
289
+ measured on a monotonic clock. The end of the child's output is not the end
290
+ of the child: the harness keeps waiting for the process to exit, and only a
291
+ passed deadline with a live process counts as a timeout. A timed-out child and
292
+ its process group are stopped.
293
+
294
+ === Inventory, report and resume
295
+
296
+ Rows come from an inventory file such as `corpora/oracle/smoke.yaml`. Row
297
+ identifiers must be unique. The report lists the selected identifiers and the
298
+ count of rows in each state, and it is complete only when every selected row
299
+ succeeded or was refused, with no missing or duplicate rows. The capture exits
300
+ with status 0 only for a complete report.
301
+
302
+ With `--checkpoint`, every finished row is appended to a JSON Lines file and
303
+ flushed to disk before the next row starts. A later run with the same oracle,
304
+ lock, worker, inventory and selection skips rows that succeeded or were
305
+ refused, and runs timed-out and crashed rows again. A checkpoint from a
306
+ different capture is rejected. A partly written last line is dropped.
307
+
308
+ === Oracle environment
309
+
310
+ The oracle is a clean checkout of the pinned Plurimath commit.
311
+ `scripts/oracle/setup.rb` installs its dependencies from the committed
312
+ `scripts/oracle/oracle.gemfile.lock` into a bundle inside that checkout, kept
313
+ apart from the latexmath bundle. The capture refuses to run when the checkout
314
+ is at another commit, has local changes, or has a different lock. The report
315
+ records the commit, the lock checksum and the worker checksum, and every row
316
+ records the Ruby, Plurimath and Parsanol versions that produced it.
317
+
318
+ The CI job `oracle-smoke` runs the harness smoke capture against the pinned
319
+ checkout and keeps the report as a build artifact:
320
+
321
+ [source,sh]
322
+ ----
323
+ ruby scripts/oracle/setup.rb --oracle "$ORACLE_CHECKOUT"
324
+ bundle exec ruby scripts/oracle/capture.rb --oracle "$ORACLE_CHECKOUT" \
325
+ --ids smoke-success,smoke-refusal --report smoke.json
326
+ ----
327
+
328
+ [[Replay]]
329
+ === Full replay
330
+
331
+ The `replay` workflow (`.github/workflows/replay.yml`) captures the legacy
332
+ oracle for every LaTeX case of the pinned testsuite's
333
+ `fixtures/parser-expectations.yaml`, in shards, and for the authored rows of
334
+ `spec/parser/fixtures/oracle-sample.yaml` and `corpora/oracle/smoke.yaml`.
335
+ It replays each capture through `spec/parser/oracle_parity_spec.rb` on the
336
+ Ruby and the native engine. It runs on pull requests that change the grammar,
337
+ the artifacts, the parser, the harness or the parity spec, every week and on
338
+ demand. Every shard keeps its inventory, report, checkpoint and capture log as
339
+ a build artifact. A mismatch is a parity finding to fix in the grammar,
340
+ preprocessing or runtime; tolerances, row selection and expected data are not
341
+ changed to make it pass.
342
+
343
+ == Open points for the model contract
344
+
345
+ The model contract settles which raw tree constructs become which typed model
346
+ fields. These remain open:
347
+
348
+ * how preprocessing source maps are exposed on the model, if at all;
349
+ * how the selected engine is reported on the model.
@@ -0,0 +1,76 @@
1
+ = The plurimath-testsuite pin
2
+
3
+ The LaTeX stage corpus lives in
4
+ https://github.com/plurimath/plurimath-testsuite[plurimath-testsuite], in
5
+ immutable numbered generations under `stages/latex/generations/`. Each
6
+ generation is listed, with the digest of its files, in the append-only ledger
7
+ `generations/latex.yaml`. This gem replays exactly one of those generations,
8
+ named in `spec/fixtures/testsuite-pin.yml`:
9
+
10
+ [source,yaml]
11
+ ----
12
+ schema: plurimath-stage/pin/1
13
+ repository: https://github.com/plurimath/plurimath-testsuite
14
+ commit: fac69501e826d6bc140ca02e37bd70333cefd988
15
+ language: latex
16
+ generation: 2
17
+ snapshot_digest: sha256:bad80c8a703e0761e96434779ff3826506be3702025b1bd77eb4c0e74bc58dde
18
+ ----
19
+
20
+ Generation 2 keeps every case of generation 1 (the legacy corpus, produced by
21
+ Plurimath 0.11.12) byte for byte and adds a raw-tree row for each of its 139
22
+ cases, rejections included, written by `scripts/stage_cases.rb`.
23
+
24
+ == What CI checks
25
+
26
+ The `corpus` workflow checks out the testsuite at the pinned commit and then:
27
+
28
+ . runs the testsuite's own validator,
29
+ `ruby scripts/validate-stages.rb <checkout> --pin spec/fixtures/testsuite-pin.yml`,
30
+ which checks that the checkout is at the pinned commit, that the ledger has
31
+ the pinned generation with the pinned digest, and that the generation's
32
+ files hash to that digest;
33
+ . runs the same validator on copies of the pin with a wrong commit, a wrong
34
+ digest and a wrong generation, and fails if any of them is accepted;
35
+ . runs `spec/corpus`, which makes the same checks in
36
+ `scripts/replay_corpus.rb` fail on a small generated testsuite;
37
+ . runs `scripts/replay_corpus.rb`, which repeats the pin checks and then
38
+ parses every row of every raw-tree payload of the generation with
39
+ `Latexmath::RawParser.parse` on the Ruby and the native engine. Each
40
+ engine's result, the preprocessed text and tree or the refusal and its byte
41
+ offset, must equal the row's expected value in the canonical form of the
42
+ testsuite's `docs/tree-parity-contract.adoc`.
43
+
44
+ To run the replay locally:
45
+
46
+ [source,sh]
47
+ ----
48
+ git clone https://github.com/plurimath/plurimath-testsuite tmp/testsuite
49
+ git -C tmp/testsuite checkout fac69501e826d6bc140ca02e37bd70333cefd988
50
+ bundle exec ruby scripts/replay_corpus.rb --testsuite tmp/testsuite --report tmp/replay.json
51
+ ----
52
+
53
+ The `replay` and `stage-cases` workflows check out the testsuite at the same
54
+ commit and fail if their commit and the pin's differ.
55
+
56
+ == Parity with the legacy parser
57
+
58
+ The pinned generation's raw-tree rows were generated from the generation 1
59
+ inputs, so replaying them checks the current grammar against the trees this
60
+ gem produced when they were written. The comparison with the legacy parser
61
+ itself is made by the `replay` workflow: it captures the parse tree of the
62
+ pinned Plurimath (`scripts/oracle/pins.yml`) for every LaTeX case of the
63
+ testsuite's `fixtures/parser-expectations.yaml` and for the rows authored in
64
+ this repository, and requires the raw tree and the typed model built by
65
+ `Latexmath.parse` to match it on both engines. The legacy corpus itself is
66
+ not changed by this gem: generation 1 stays as it was imported.
67
+
68
+ == Moving the pin
69
+
70
+ Commit, generation and digest change together, in one change, and only to a
71
+ generation that is already merged in the testsuite. Take the values from the
72
+ ledger entry, update `TESTSUITE_COMMIT` in the `corpus`, `replay` and
73
+ `stage-cases` workflows to the same commit, and let CI replay the new
74
+ generation. A generation whose rows this gem no longer reproduces is a
75
+ regression to fix here, or a reason for a new generation in the testsuite;
76
+ the pin never points at a generation the gem does not match.
@@ -0,0 +1,17 @@
1
+ {
2
+ "format": "latexmath-wrappers/1",
3
+ "wrappers": {
4
+ "absent": "nil",
5
+ "items": "items",
6
+ "literal": "text",
7
+ "spelling": "text_list",
8
+ "text_list": "text_list"
9
+ },
10
+ "rules": {
11
+ "N1 text alternative": 144,
12
+ "N10 list site": 5,
13
+ "N11 named option": 8,
14
+ "N8 catalog text": 3246,
15
+ "N9 empty list": 4
16
+ }
17
+ }