@tikoci/centrs 0.1.4 → 0.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (68) hide show
  1. package/CHANGELOG.md +309 -0
  2. package/GLOSSARY.txt +40 -1
  3. package/README.md +8 -4
  4. package/commands/AGENTS.md +7 -4
  5. package/commands/api/AGENTS.md +13 -1
  6. package/commands/api/README.md +37 -8
  7. package/commands/api/examples.md +31 -1
  8. package/commands/execute/README.md +39 -10
  9. package/commands/execute/examples.md +79 -7
  10. package/commands/explain/README.md +1195 -155
  11. package/commands/explain/examples.md +99 -3
  12. package/commands/mcp/README.md +7 -4
  13. package/commands/mcp/examples.md +2 -2
  14. package/commands/terminal/examples.md +7 -1
  15. package/docs/CLI.md +2 -2
  16. package/docs/CONSTITUTION.md +72 -15
  17. package/docs/MATRIX.md +89 -52
  18. package/docs/errors/validation/syntax.md +29 -4
  19. package/package.json +13 -2
  20. package/src/api-fanout.ts +17 -4
  21. package/src/api.ts +254 -67
  22. package/src/cli/api.ts +1 -1
  23. package/src/cli/btest.ts +1 -1
  24. package/src/cli/explain.ts +1 -1
  25. package/src/cli/terminal.ts +1 -1
  26. package/src/core/envelope.ts +53 -0
  27. package/src/core/error-catalog.ts +2 -1
  28. package/src/core/execute-command.ts +148 -0
  29. package/src/core/inspect.ts +36 -4
  30. package/src/core/version.ts +4 -0
  31. package/src/core/yaml.ts +86 -0
  32. package/src/execute-fanout.ts +18 -4
  33. package/src/execute.ts +225 -244
  34. package/src/explain/arg-tokens.ts +62 -25
  35. package/src/explain/args.ts +398 -52
  36. package/src/explain/blocks.ts +54 -11
  37. package/src/explain/brace-tokens.ts +4 -3
  38. package/src/explain/catalog.ts +42 -44
  39. package/src/explain/coordinates.ts +159 -0
  40. package/src/explain/defects.ts +2 -1
  41. package/src/explain/escape-tokens.ts +77 -0
  42. package/src/explain/highlight-projection.ts +308 -0
  43. package/src/explain/menus.ts +9 -5
  44. package/src/explain/operator-tokens.ts +13 -7
  45. package/src/explain/operators.ts +27 -20
  46. package/src/explain/path-tokens.ts +1 -1
  47. package/src/explain/pathresolve.ts +358 -113
  48. package/src/explain/quoted-string.ts +110 -20
  49. package/src/explain/scope-brace.ts +175 -22
  50. package/src/explain/segment.ts +429 -21
  51. package/src/explain/separator.ts +187 -0
  52. package/src/explain/split-owner.ts +194 -0
  53. package/src/explain/string-tokens.ts +14 -4
  54. package/src/explain/symbol-values.ts +209 -173
  55. package/src/explain/symbols.ts +21 -1
  56. package/src/explain/transport.ts +17 -0
  57. package/src/explain/value-tokens.ts +15 -12
  58. package/src/explain/verbsplit.ts +165 -23
  59. package/src/explain/write.ts +47 -6
  60. package/src/explain.ts +522 -72
  61. package/src/index.ts +19 -2
  62. package/src/mcp/server.ts +2 -1
  63. package/src/offline-gate.ts +148 -0
  64. package/src/retrieve-fanout.ts +1 -1
  65. package/src/retrieve.ts +1 -59
  66. package/src/settings.ts +2 -1
  67. package/src/transfer-fanout.ts +1 -1
  68. package/src/transfer.ts +1 -1
@@ -9,20 +9,13 @@ a path, with completion-style candidates for building the right command.
9
9
  validation): the cheap, safe knowledge tier in front of the runners
10
10
  (`execute` / `api` / `retrieve`).
11
11
 
12
- Status: `designed` over `rest-api` and `native-api` — the transports the live
13
- inspection probes ride; every other cell is `—`. The offline mode is
14
- protocol-connection-less, and **`centrs explain '<input>'` runs today** (issues #202b
15
- and #202c): the row stays `designed` because the grid tracks protocol cells and
16
- offline occupies none. See `docs/MATRIX.md` for the row. A first design round
17
- (2026-07-19, recorded in #90) settled the surface shape and the offline model;
18
- the **phase-0 canonicalization grounding lab (#185) is complete and this spec
19
- is now ratified** — every ratification-gating question was answered with cited
20
- evidence (see [Phase-0 ratification](#phase-0-ratification-185)). The findings
21
- folded in below; a few of them **amend** the surface (tristate `containsWrite`,
22
- an `ambiguous` canonicalizer verdict, a defect-region diagnostics contract, a
23
- byte-count-preserving coordinate rule, the native-api `:parse` readout). Flag
24
- names and sizing thresholds stay provisional — those are implementation opens,
25
- not ratification blockers.
12
+ Status: offline baseline `verified`; live probes over `rest-api` and `native-api`
13
+ `designed`. **`centrs explain '<input>'` runs today**, without a router or CDB.
14
+ [`docs/MATRIX.md`](../../docs/MATRIX.md#offline-analysis) tracks these separately;
15
+ [#90](https://github.com/tikoci/centrs/issues/90) indexes the remaining tasks.
16
+ The spec is ratified on the [phase-0 grounding lab](#phase-0-ratification-185).
17
+ The current finish line is [Offline baseline acceptance](#offline-baseline-acceptance),
18
+ not completion of the live protocol cells.
26
19
  Load-bearing rules — envelope, errors, settings precedence, identity,
27
20
  validation, protocol selection — live in
28
21
  [`docs/CONSTITUTION.md`](../../docs/CONSTITUTION.md).
@@ -62,6 +55,10 @@ validation, protocol selection — live in
62
55
 
63
56
  ## Where `explain` sits
64
57
 
58
+ - **Shared analysis across transports.** The offline result is independently
59
+ useful to libraries, agents, and editors. Its facts can later accompany
60
+ other commands' envelopes, irrespective of their transport; the rich
61
+ analysis is not currently applied as a runner policy gate.
65
62
  - **Never executes, never mutates.** All live probes (`/console/inspect`
66
63
  highlight/completion/syntax/child and `:put [:parse …]`) are read-only
67
64
  inspection. Execution-based probes (required-argument discovery via
@@ -84,6 +81,16 @@ validation, protocol selection — live in
84
81
  to **support one**: hover, diagnostics, completion, and semantic-token needs
85
82
  of `lsp-routeros-ts`/tikbook are first-class consumers of the export shape.
86
83
 
84
+ Normalized source structure is the foundation for possible configuration
85
+ checks. Preserve command order, selectors, source locations, literal/computed
86
+ value distinctions, and unknowns; an input fragment does not establish final
87
+ device state. Script-variable bindings are not yet an object-reference model
88
+ for interfaces, bridges, or other configuration entities. Security advice,
89
+ rule collections, and a browser/Monaco inspector are possible consumers, not
90
+ new requirements to implement those products during the offline baseline.
91
+ quickchr owns the CHR provisioning/test machinery; rosetta supplies docs and
92
+ reference knowledge. This direction adds no runtime dependency on either.
93
+
87
94
  ## Canonicalization is the core
88
95
 
89
96
  The engine is a **grown `canonicalizeExecuteCommand`** (decision: grow the
@@ -165,16 +172,16 @@ today's path/verb/args split and script-vs-structured gate:
165
172
  The *container* half of that pair is since decidable. `src/explain/menus.ts`
166
173
  (#207) bakes only the menu-vs-command node type — not a schema — and it is
167
174
  version-less because the pinned trees show no `dir`↔`cmd` flip across
168
- 7.10.2 → 7.24rc2 or across architectures. So `/ip/route` now resolves as
175
+ 7.10.2 → 7.25beta3 or across architectures. So `/ip/route` now resolves as
169
176
  `navigation` (#210). The floor is unmoved: the table is deliberately
170
177
  incomplete, absence from it decides nothing, and every rule above it still
171
178
  refuses rather than guesses.
172
179
 
173
180
  The *command* half is what #228 addressed, by adding a second first-order
174
181
  source rather than a schema. MikroTik's CLI Reference is generated from the
175
- definition structs, and measured against the same four pinned trees it has
176
- **zero navigation-vs-command contradictions** across 968 exactly-matching
177
- paths, three RouterOS versions and two architectures; all but one of the 102
182
+ definition structs, and measured against the same six pinned trees it has
183
+ **zero navigation-vs-command contradictions** across 959 exactly-matching
184
+ paths, four RouterOS versions and two architectures; all but one of the 94
178
185
  published paths absent from every tree carry a `package`/`conditions`/`syscap`
179
186
  gate — their own or an ancestor's — that predicts the absence.
180
187
  `src/explain/catalog.ts` unions the two with per-entry provenance, which is
@@ -198,6 +205,22 @@ today's path/verb/args split and script-vs-structured gate:
198
205
  inventory reshape fails loudly instead of shrinking the table (#285,
199
206
  tikoci/rosetta#137).
200
207
 
208
+ **Source refresh (#335, accepted 2026-09-13).** The inspect pins retain the
209
+ 7.10.2 and 7.16 history anchors, then pair x86 and arm64 for both current
210
+ stable 7.24.2 and prerelease 7.25beta3. Pairing architectures makes additions
211
+ attributable to version rather than to an architecture swap while preserving
212
+ the hardware-only menu floor. Reviewed independently, the inspect union moves
213
+ from 615 to 617 menus: 7.24.2 removes `/ip/ipsec/key/qkd`, and 7.25beta3 adds
214
+ `/interface/ethernet/pon`, `/interface/xfrm`, and `/ip/ssh/known-hosts`.
215
+ Separately, the current CLI Reference moves from 1,070 pages / 1,077 entries
216
+ to 1,053 / 1,060: it adds
217
+ `/interface/ethernet/switch/rule/reset-counters`, removes the five
218
+ `/interface/bridge/msrp` paths plus `/root`, moves 12 paths from `both` to
219
+ `inspect`, and changes 16 published gate strings. Those current-only
220
+ publication changes are accepted: none introduces a kind contradiction. The
221
+ coherent generated catalog has 1,120 paths: 959 `both`, 67 `inspect`, and 94
222
+ `published`; it still reports zero navigation-vs-command contradictions.
223
+
201
224
  There are real unknowns in how far offline parsing can go (expression
202
225
  grammar, scope fidelity vs `:parse`, `[]`-nesting corner cases). Grounding
203
226
  those unknowns was **phase 0** (#185); its findings are folded in throughout
@@ -262,7 +285,10 @@ command accepts.
262
285
  Offline mode does ship two generated **structure** tables, baked at build time
263
286
  from pinned sources and read as ordinary closed lists. They may say whether a
264
287
  path is navigation or a command; they never describe what a command accepts.
265
- Absence from either abstains and never rejects.
288
+ Absence from either abstains and never rejects. Regenerate with
289
+ `bun run explain:menus` / `bun run explain:catalog`; `--check` drift-gates both
290
+ in QA. Do not hand-edit them — each file's header carries its source pin and
291
+ rationale.
266
292
 
267
293
  - `src/explain/menus.ts` (#207) — container paths from pinned restraml
268
294
  `/console/inspect` trees. **Read today** to confirm navigation, on its own by
@@ -499,7 +525,8 @@ Standard envelope (constitution: result envelope); `data` sketch:
499
525
  severity channels. **Malformed input carries a defect *region*** (Q14): each
500
526
  diagnostic points at the byte span of its defect, and the detectable classes
501
527
  are {unbalanced delimiter, unterminated string, invalid escape, invalid sigil,
502
- invalid unquoted hash, BOM, non-ASCII, over-depth nesting}. **Two classes named
528
+ invalid unquoted hash, invalid apostrophe (#355), BOM, non-ASCII,
529
+ over-depth nesting}. **Two classes named
503
530
  in the phase-0 draft
504
531
  of this list are deliberately deferred** (maintainer decision, #192), because
505
532
  each would require offline to assert something it cannot prove:
@@ -570,19 +597,24 @@ and its phase is named below.
570
597
  all: a `[…]`/`(…)`/`$x`/`{…}` value, an escape this phase does not decode, a
571
598
  left-hand side that is not a RouterOS name, a token split by a continuation.
572
599
  Never partially read, because a dropped argument silently changes what a
573
- rendered `curl` DOES. Two refusals are not lexical at all, but
574
- places where centrs's two readers disagree about an **unquoted** value: a
575
- `'`, which RouterOS treats as an ordinary character and the locked execute
576
- gate treats as a quote (`comment=it's` is `it's` here and `its` to the
577
- gate); and a `\f`/`\v`, which the gate splits tokens on (JavaScript `\s`)
578
- and every explain module does not (ASCII whitespace). The device-correct
579
- reading is the analysis's in both, but the gate cannot be corrected — so
580
- phase 1 publishes neither rather than putting two confident values in one
581
- result. Inside a `"…"` run the two agree and nothing is refused. The
582
- advisory value-anchor view is intentionally wider only for exact array
600
+ rendered `curl` DOES. One refusal is not lexical at all, but a place where
601
+ centrs's two readers disagree about an **unquoted** value: a `\f`/`\v`,
602
+ which the retired execute gate split tokens on (JavaScript `\s`) and every
603
+ explain module does not (ASCII whitespace). The device-correct reading is
604
+ the analysis's, but that gate could not be corrected — so phase 1 publishes
605
+ neither rather than putting two confident values in one result. Inside a
606
+ `"…"` run the two agree and nothing is refused.
607
+
608
+ The advisory value-anchor view is intentionally wider only for exact array
583
609
  literals: a non-empty `{…}` value or a parenthesized run with a depth-zero
584
610
  comma is locatable and hints `array`, while the strict REST view continues
585
611
  to refuse the whole list.
612
+
613
+ An unquoted `'` used to be a second such disagreement, described here as a
614
+ character RouterOS treats as ordinary and the gate treats as a quote. That
615
+ reading was wrong on both halves and #355 settled it: `'` is not a RouterOS
616
+ token at all, so `comment=it's` is REJECTED by the device rather than
617
+ carrying either value, and the quote-counting gate that disagreed is gone.
586
618
  - **A token is read but carries no `value`** — the list still reads. The token
587
619
  is delimited and classified; only its literal value is unknowable, as for
588
620
  the positional in `:log info "result: $[…]"`. **`value` absent means there
@@ -882,11 +914,11 @@ and its phase is named below.
882
914
  `bun run explain:value-census:readme` and gated against it by
883
915
  `bun run explain:value-census:readme:check`; the fixture itself is gated
884
916
  against a fresh corpus run by `bun run explain:value-census:check`. Of the
885
- 13,168 emitted values in a statement the strict argument lexer ALSO read, 0
917
+ 13,161 emitted values in a statement the strict argument lexer ALSO read, 0
886
918
  disagree with it on half-open byte span or decoded text, while the
887
- prefix-safe scan retains a further 6,540 values across 569 statements whose
888
- strict REST reading abstains. Of 19,708 emitted occurrences, 5,636 are array
889
- members (529 keyed, 1,147 nested inside another member) and 813 are arrays;
919
+ prefix-safe scan retains a further 6,538 values across 568 statements whose
920
+ strict REST reading abstains. Of 19,699 emitted occurrences, 5,636 are array
921
+ members (529 keyed, 1,147 nested inside another member) and 808 are arrays;
890
922
  the corpus holds no source-literal `id` example. The three structural
891
923
  counters — spans addressing bytes outside their own source, members naming a
892
924
  container that does not exist, members escaping the container they name —
@@ -1003,7 +1035,9 @@ navigated, so the context stays as unknown as it already was.
1003
1035
  (string-internal unknown/lowercase-hex/truncated — CHR `highlight` `error`
1004
1036
  `:parse` `expected message value`, grounded stable+testing #247),
1005
1037
  `bad-sigil`,
1006
- `invalid-hash`. Seven classes the device itself rejects.
1038
+ `invalid-hash`, `invalid-apostrophe` (#355, below),
1039
+ `missing-statement-separator` (#311, below). Nine classes
1040
+ the device itself rejects.
1007
1041
  - `warning` — `over-depth`, because it is centrs's own resource bound and says
1008
1042
  nothing about whether the input is legal; and an `ambiguous`/`unknown`
1009
1043
  resolution, never an error, so the default `--fail-on error` cannot fail a
@@ -1013,6 +1047,386 @@ navigated, so the context stays as unknown as it already was.
1013
1047
  fail), and `context-lost`, which reports a reading that is correct while the
1014
1048
  document's menu context was already gone.
1015
1049
 
1050
+ ### The missing statement separator (#311)
1051
+
1052
+ Two commands written with no `;` or newline between them used to read as one
1053
+ command with extra operands, and offline reported `pass`: `/ip/address add
1054
+ interface=ether1 /ip/route add gateway=192.168.88.1` published `gateway` as an
1055
+ attribute of `/ip/address add`. The rule that closes it is a known **menu path**
1056
+ in operand position followed by a console **verb** — never "a `/` after the
1057
+ verb", because a slash-shaped operand is legal and `/file remove
1058
+ /flash/skins/foo.html` lowers cleanly to `numbers=/flash/skins/foo.html`. The
1059
+ menu spelling may be slashed, spaced, or mixed (Q3 R4); the device rejects
1060
+ `/ip route add` exactly as it rejects `/ip/route add`.
1061
+
1062
+ **The device evidence is not what the report said, and the correction widened
1063
+ the rule.** #311 gave the error byte as the `=` of the trailing attribute
1064
+ (`expected end of command (line 1 column 55)`). That byte is an artefact of that
1065
+ head: `add` abbreviates `/ip/address/add`'s own `address=` argument, so the
1066
+ console reads `add gateway` as `address=gateway` and only complains at the `=`
1067
+ after it. Change the head so the second verb abbreviates nothing and the answer
1068
+ changes shape — `/system/note set note=hello /ip/route add gateway=1.2.3.4` is
1069
+ no hard reject at all; `:parse` returns IL carrying `bad parameter /ip/route
1070
+ (line 1 column 38)`, naming the second **path**. Both are refusals, and which
1071
+ one appears depends on per-menu argument names, which offline does not have. So
1072
+ the grounded claim is the negative one: across every probed spelling on CHR
1073
+ 7.24.2 and 7.23.5, RouterOS answers `bad parameter <path>` or `expected end of
1074
+ command`, and never accepts the shape. A trailing attribute is therefore **not**
1075
+ required, which #311's own example had made it look like.
1076
+
1077
+ The head-dependence goes one step further than "which wording", and it bounds
1078
+ what the diagnostic may say. On a head whose own value slot takes a bare
1079
+ positional, the second path is absorbed into that slot and the device refuses
1080
+ the **verb** instead: `:put /ip/route print` lowers to `/putmessage=/ip/route`
1081
+ and answers `bad parameter print`, and `:local a /ip/route add` lowers to
1082
+ `/localname=$a;value=/ip/route` and answers `bad parameter add`. Both are still
1083
+ refusals — `/system/script/run /ip/route`, the same shape with no verb after
1084
+ the path, lowers cleanly to `number=/ip/route` — so the offline `error` stands.
1085
+ But the message says the bytes **read as** a second command, which is centrs's
1086
+ heuristic reading, and never that the device starts one there or which token it
1087
+ will name. Every row above is pinned in
1088
+ `test/integration/explain-separator.test.ts`.
1089
+
1090
+ **The contradicted reading is withdrawn, not patched.** The statement stays
1091
+ `resolved` — `/ip/address add` really is the head, and `highlight` agrees. What
1092
+ is withdrawn is withdrawn by three separate gates, each with its own trigger:
1093
+
1094
+ 1. **The argument list.** `withoutSeparatedArguments` drops `command.args` and
1095
+ sets `arguments.read` to `false`. This is the same all-or-nothing move
1096
+ `enforceGateParity` makes for a gate/analysis disagreement, but it is a
1097
+ distinct gate on a distinct trigger — the two never consult each other. The
1098
+ device says the move is right rather than merely cautious:
1099
+ `… /ip/route add placeholder` lowers to
1100
+ `address=placeholder;;interface=ether1`, a phantom attribute the source never
1101
+ wrote, because the second run's words can abbreviate the HEAD's argument
1102
+ names.
1103
+ 2. **The invocation.** `rejectedBySeparator(runs)` sets the new
1104
+ `ExplainTransportInput.rejected`, which `classifyExplainTransport` honours
1105
+ ahead of every mapping rule and answers `unknown`. Without it the degraded
1106
+ `arguments` would fall through to `execute` and print a ready-to-paste
1107
+ `centrs execute '<router>' '<the malformed text>'` for bytes the same result
1108
+ calls a syntax error. **Unread is not rejected** — an unread argument list
1109
+ still legitimately rides the raw CLI surface, and that path is unchanged.
1110
+ 3. **The value facts from the run onward.** `values.occurrences` is an axis of
1111
+ its own, not a projection of `arguments` — the value census above pins how
1112
+ many values it retains in statements whose strict REST reading abstains — so
1113
+ gate 1 does not reach it and a cut is needed. The cut is the run's first byte, and
1114
+ the device draws the line in the same place: `:parse` reads `gateway` as the
1115
+ VALUE of `/ip/address/add`'s own `address=`, never as an attribute NAME,
1116
+ while the head's `interface=ether1` survives into the IL (as `note=hello`
1117
+ does in the `/system/note set` row). Values before the cut therefore stay;
1118
+ values reaching into or past it are dropped, and with them the `value` token
1119
+ class over those bytes.
1120
+
1121
+ **Abstentions, stated.** A menu path with no verb after it (`… /ip/route`) and a
1122
+ bare operand a menu does not accept (`… placeholder`, `/file remove a b`) also draw
1123
+ `bad parameter`, but that is the generic "no such positional" complaint;
1124
+ deciding it needs the per-menu positional list, which `catalog.ts` deliberately
1125
+ is not, and a separator is not necessarily the fix. Both abstain. The shape
1126
+ filter over a candidate path does admit a leading digit, because
1127
+ `/interface/6to4` is a real menu in both structure tables; presence in those
1128
+ tables, never the filter, is what decides. Over the 948-script pinned corpus the
1129
+ rule fires **0 times**, so no published census figure moves; the 38 corpus
1130
+ scripts the device answers `expected end of command` for are all other causes
1131
+ (line continuations, `<tab>`/`<VALUE>` placeholders, pasted shell and Python,
1132
+ stray parens), which is #203's genre bias showing.
1133
+
1134
+ **The reach limit is closed (#316).** This rule used to read
1135
+ `statement.arguments.tokens`, the all-or-nothing strict reading, so a statement
1136
+ holding a variable carried no tokens and the run in it was missed:
1137
+ `/ip/address add interface=ether1 /ip/route add gateway=$g` reported `pass`
1138
+ where its literal twin reported the separator. It now reads the skip-tolerant
1139
+ token stream described below, and both spellings report the separator — as does
1140
+ `/ip/address add interface=$x /ip/route add gateway=1.1.1.1`, where the
1141
+ undecodable token comes FIRST and a prefix-only reading would still have missed
1142
+ it. The corpus count is unchanged at **0** for the same reason it always was
1143
+ (#203's genre bias), so this buys reach rather than a number. What still stops
1144
+ the rule is an unterminated string or an unbalanced delimiter, because neither
1145
+ walk can find a boundary to resume from.
1146
+
1147
+ ### The token stream a rule reads (#316)
1148
+
1149
+ `lexArguments` is all-or-nothing by design: its consumer renders a runnable REST
1150
+ request, and a partially-read argument list changes what the rendered command
1151
+ DOES. That is the right reading for a renderer and the wrong one for a rule —
1152
+ the first token it declines discards every token already decoded, which put
1153
+ **48.2%** of the corpus's argument-bearing statements out of reach of every
1154
+ token-level rule and of the `arg` token fill.
1155
+
1156
+ `lexArgumentTokens` is a third reading over the same walk — alongside strict
1157
+ `lexArguments` and the prefix-safe `lexValueAnchors` values already use. Same
1158
+ text, same `from`, the same boundary rules and the same refusal strings,
1159
+ differing only in what a refusal does. Where the strict reading discards the list, this one publishes the token
1160
+ it could not decode with its bytes located, `undecided` set to that same reason
1161
+ and **no `value`**, then carries on to the next token. `statement.arguments` is
1162
+ unchanged and still strict; nothing that renders a command gained reach.
1163
+
1164
+ **Skip-and-continue, measured against stop-at-first-refusal.** #316 named two
1165
+ designs and said the fork was load-bearing rather than cosmetic. It is. Both are
1166
+ priced by `bun run explain:arg-reach` over the 948-script pinned corpus, from
1167
+ one implementation — design A is the tolerant token list truncated at its first
1168
+ `undecided` token, which is exactly the prefix the strict walk discards:
1169
+
1170
+ | | strict | A — prefix | B — skip |
1171
+ | --- | ---: | ---: | ---: |
1172
+ | statements reached, of 14,267 | 7,385 (51.8%) | 10,754 (75.4%) | 14,267 (100.0%) |
1173
+ | argument tokens | 16,652 | 20,481 | 31,310 |
1174
+ | missing-separator runs found | 0 | 0 | 0 |
1175
+
1176
+ A further 58 statements sit outside every column because their bytes are not
1177
+ addressable (normalized, or a span widened to the enclosing statement); the
1178
+ sweep counts them rather than dropping them silently, since `explain.ts` refuses
1179
+ those too and lexing them here would measure a reading the product does not
1180
+ offer.
1181
+
1182
+ A recovers 3,369 statements; the 3,513 statements whose FIRST interesting token
1183
+ is the undecodable one have an empty prefix and are reached only by B — so the
1184
+ cheaper design leaves slightly more behind than it recovers. The separator row
1185
+ is 0 across all three and is reported rather than buried: on this corpus neither
1186
+ design finds a run the strict reading missed, which is what #203's genre bias
1187
+ predicts for a typo class published scripts do not carry. The sweep also asserts
1188
+ what makes the A column trustworthy — that the strict tokens are the tolerant
1189
+ walk's prefix byte for byte, and that both walks name the same refusal reason —
1190
+ and fails if any statement disagrees (0 do).
1191
+
1192
+ **Where both walks still stop.** An unterminated string and an unbalanced
1193
+ `(`/`[`/`{` have no knowable end, so there is no boundary to resume from. A `;`
1194
+ is a third and deliberately so: it is a statement boundary the segmenter owns,
1195
+ and resuming after it would read the next statement's bytes as this one's
1196
+ arguments. No corpus statement reaches any of the three, and that is a property
1197
+ of the pipeline rather than luck: an unterminated string and an unbalanced
1198
+ delimiter are document defects the segmenter and resolver answer first, so no
1199
+ addressable statement with an argument list is produced at all, and a `;` is
1200
+ split into two statements before any argument is lexed. The unbalanced case is
1201
+ also the one place the two readings name a different reason, and it follows from
1202
+ what each needed: the strict walk refuses a `[…]` on sight and never asks where
1203
+ it ends, while the tolerant walk asks — because the answer is what it would
1204
+ resume from — and says `an unclosed structured argument value` when there is
1205
+ none.
1206
+
1207
+ **What it moved, including the cost.** The `arg` fill now paints the `name=` run
1208
+ of a token whose value only the device knows, so classified bytes go 69.77% →
1209
+ **71.73%** (`arg` tokens 11,438 → 16,526). On the device-agreement slice the
1210
+ bytes where the device decided and centrs abstained drop 5,540 → 4,158, and two
1211
+ whole cells (`unclassified` → `arg-dot`, `unclassified` → `arg-scope`) empty
1212
+ out. Against that, projection disagreements rise 143 → 360, and **every one of
1213
+ the 217 new bytes is the `=` of an attribute**: `arg` → `syntax-meta`, one cell,
1214
+ 358 runs. That is not a new wrong assertion. It is #293's provisional
1215
+ vocabulary — one `arg` class for both the name bytes and the `=` — meeting the
1216
+ device's context-dependent answer over 2.5x as many bytes as before, and it
1217
+ gave #264 B5 a concrete, counted reason to split the `=` into its own class.
1218
+ It did, and the relabel moved no byte coverage at all: disagreements fell
1219
+ 360 → 2 and the classified-byte census did not change by a digit.
1220
+
1221
+ **The one residue, sized so it is not mistaken for a defect.** A `\<newline>`
1222
+ continuation inside an argument leaves its token `undecided`, because the bytes
1223
+ of a continued value are not contiguous and a `valueSpan` over them would be a
1224
+ lie (`args.ts` → *What it refuses*). The conservative case is a token whose
1225
+ value is already complete BEFORE the continuation, which is then withheld
1226
+ needlessly. Measured on the 948-script corpus: 109 statements make the strict
1227
+ lexer refuse for this reason, the tolerant walk reaches 114 statements carrying
1228
+ 138 such tokens, and **all 138 put the continuation immediately after the `=`**
1229
+ — the value really does live on the next line. Exactly **one** token
1230
+ (`mode=none\<newline>`) has a complete value in front of its continuation. So
1231
+ narrowing the rule would recover one token, and deciding it correctly needs the
1232
+ device-grounded answer for what a continuation does to a value's INTERIOR, which
1233
+ is #225's probe-matrix territory rather than a lexer tweak. Recorded there; not
1234
+ carried as pending work here.
1235
+
1236
+ **What the public surface gained, and what it did not.** The package root
1237
+ exports the reading itself — `lexExplainArgumentTokens` and
1238
+ `ExplainArgumentTokenReading` — for the same reason `lexExplainValueAnchors` is
1239
+ exported: a consumer building its own token-level rule needs the reading a rule
1240
+ wants, not the one a renderer wants. What did NOT change is the *envelope*:
1241
+ `explainCommand`'s `structure.statements[].arguments` is still the strict
1242
+ reading, `ExplainArgumentToken` carries no `undecided` field, and `undecided`
1243
+ never appears on an `ExplainArguments` reading, because the strict lexer
1244
+ publishes no token it could not decide. Whether the tolerant stream should reach
1245
+ the envelope in its own right — so a browser consumer reading `explainCommand`'s
1246
+ result can see a located `gateway=` whose value is a runtime expression — was
1247
+ left to #264 B5, which decided it reaches the envelope **as the token
1248
+ partition** and nowhere else: naming the `=` `arg-sep` locates the argument
1249
+ without loosening `arguments`, and `undecided` stays off the envelope. See
1250
+ [The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5).
1251
+
1252
+ ### The menu path with arguments (#324)
1253
+
1254
+ A run that is nothing but a **known menu**, with arguments after it and no
1255
+ verb, used to come back `resolved` with a verb the analyzer made up.
1256
+ `/ip/firewall/address-list name=ytkids timeout=1h` reported path
1257
+ `/ip/firewall`, verb `address-list`; the spaced spelling
1258
+ `/ip firewall address-list name=x` promoted `firewall`, one level higher
1259
+ still. The punctuation half of the ratified Q6 rule is what fires — "no space
1260
+ token, arguments follow, so the last run token is the verb", or "the first
1261
+ space token is the verb" — and neither premise it needs was ever checked:
1262
+
1263
+ ```text
1264
+ isKnownMenuPath(/ip/firewall/address-list) -> true (it is a menu)
1265
+ VERBS.has("address-list") -> false (it is not a verb)
1266
+ ```
1267
+
1268
+ This is the **symmetric half of R9**. R9 (#211 B1) made `pathresolve`'s
1269
+ `menuNavPath` consult `VERBS` so it could not claim navigation where
1270
+ `verbsplit` had decided a verb; this makes `resolveVerb` consult the menu
1271
+ tables so it cannot claim a verb where the whole run names a menu and the
1272
+ segment it would promote is not a verb. The reading degrades to **`ambiguous`**
1273
+ with the standard located `explain/canonicalizer/ambiguous-statement` warning,
1274
+ naming both premises. The adjacent readings are untouched —
1275
+ `/ip/firewall/address-list add name=x` resolves, the bare path is navigation,
1276
+ `/system/reboot` is a command — which is what makes this a seam rather than a
1277
+ parser limit.
1278
+
1279
+ **It abstains; it does not reject.** Offline knows it cannot name a verb here,
1280
+ and that needs no device evidence. Whether RouterOS refuses the shape outright
1281
+ is a *device* claim, so unlike #311's CHR-grounded `error` this stays a
1282
+ warning and the statement carries no verdict about acceptance.
1283
+
1284
+ **Four exclusions, each for its own reason.** A frozen-vocabulary hit is
1285
+ excluded by the `VERBS` premise itself (a vocabulary split promotes a `VERBS`
1286
+ member by construction). A **published command** is excluded by `catalogAt`:
1287
+ first-order evidence about that exact path outranks a container listing of its
1288
+ prefix, which is what keeps `/system/gps/monitor once` resolved. A scripting
1289
+ **directive** is excluded by R10, which is measured. And a `{` after the run is
1290
+ excluded by shape: that is a menu *container* body (`/system {identity print}`,
1291
+ H7) where the run legitimately names a menu and no verb is missing —
1292
+ abstaining there would drop the `invalid-command-brace` error the statement
1293
+ after it carries.
1294
+
1295
+ **Priced before landing, as #211 B2 asks.** Over the frozen dev/holdout
1296
+ partition (948-script corpus, 17,212 statements) **10 statements flip, all
1297
+ `resolved` → `ambiguous`**: 1 dev, 9 holdout. **None is a genuine no-verb
1298
+ command.** Three are real RouterOS missing a verb —
1299
+ `/ip/firewall/address-list name=ytkids …`, `/interface bridge [find]
1300
+ vlan-filtering=yes`, `/user-manager user [find …] attributes=…` — and seven
1301
+ are pasted non-RouterOS prose: five Python traceback `File "…"` lines,
1302
+ `User ID: 506`, and `port 80`. Those seven reach the rule at all because
1303
+ `/file`, `/user` and `/port` are real menus and the table lookup folds case,
1304
+ which is #203's genre bias arriving through a second door. Every one of the
1305
+ ten was a fabrication, so the abstention has no measured false-positive cost
1306
+ on this corpus.
1307
+
1308
+ **The execute gate is unmoved**, as the constitution requires: `canonical`
1309
+ still reports `mode: "structured"`, path `/ip/firewall`, verb `address-list`
1310
+ for this input, because that is what `canonicalizeExecuteCommand` decides and
1311
+ an explain-only abstention may not reinterpret it. `enforceGateParity` is
1312
+ about the argument *list* of a `kind: "command"` reading, so it does not fire
1313
+ on a statement that carries no reading at all — the gate and the analysis are
1314
+ two answers to two questions, and only one of them changed.
1315
+
1316
+ The invariant this closes — `pathresolve` and `verbsplit` never contradict
1317
+ each other on one statement — is asserted in
1318
+ `test/unit/explain-pathresolve.test.ts` beside R9's own cases. Presence in the
1319
+ shipped structure tables still narrows a reading and absence still decides
1320
+ nothing (#207/#228), so an unlisted menu spelled `…/address-list` is read
1321
+ exactly as before.
1322
+
1323
+ ### Two spellings the device accepts and offline used to reject (#347, #348)
1324
+
1325
+ Both were found the same way, and the method is worth keeping: run the analyzer
1326
+ over all 948 corpus scripts and compare its verdict to that build's own `:parse`
1327
+ verdict, which the corpus already stores. The number that matters is the **false
1328
+ reject** — a script the device ACCEPTS that offline analysis fails — because
1329
+ that is validation blocking input RouterOS is happy with. It was 4, identical on
1330
+ 7.23.5, 7.24.2 and 7.25beta3; it is now 1.
1331
+
1332
+ The converse count is not a parser score and is not quoted as one: the corpus is
1333
+ whole forum posts including prose output, so most of what the device rejects and
1334
+ offline passes is #203's genre bias rather than a gap.
1335
+
1336
+ **The `=` is optional on a scope slot (#347).** `:if (…) do={…} else={…}`,
1337
+ `:if (…) do={…} else {…}` and `:if (…) do {…} else {…}` are the same construct:
1338
+ `:parse` lowers all three to byte-identical IL (`…;do=;(evl …);else=;(evl …)`)
1339
+ on CHR 7.21.5, 7.23.5, 7.24.2 and 7.25beta3. The MikroTik manual documents only
1340
+ the `else={…}` spelling, so this is a real-but-undocumented form. The rule is
1341
+ deliberately narrower than "any word before a brace": only the four
1342
+ `SCOPE_ARG_NAMES` members may drop the `=`, and the word may not be the
1343
+ statement's own leading directive — otherwise `:local z {1;2}` would stop
1344
+ reading its brace as an array, and `:do {` would have its slot renamed away from
1345
+ the `command` the IL uses. A COLON also disqualifies the word: the device parses
1346
+ `:if (1=1) do {…}` but refuses `:if (1=1) :do {…}` with `expected end of command`
1347
+ on 7.21.5 and 7.24.2, because a colon makes it a directive and a directive
1348
+ cannot occupy an argument slot. The control is `elsy {`, rejected on all four
1349
+ builds, which is what makes the acceptance a device fact rather than a probe
1350
+ artefact.
1351
+
1352
+ **A relative menu-scope block composes the path (#348).** `/ip { address {
1353
+ print } }` is `(evl (evl (evl /ip/address/print)))` on device and
1354
+ `/interface { ethernet { print } }` the same way; the ABSOLUTE spelling
1355
+ `/ip/address { print }` already passed offline, and only the nested relative
1356
+ form failed. A menu block's body is a statement list, so it may open with a
1357
+ submenu, with one of the thirteen frozen CRUD verbs (`verbs.ts`), with a
1358
+ catalog-named command, with a `:` scripting directive, or with an absolute
1359
+ command — corpus 521 is the directive case, `ethernet { :put … }`.
1360
+
1361
+ That last rule is bounded by what a STATEMENT can start with, which is what
1362
+ keeps `{1;2}` an array literal: an array member opens with a digit or a quote
1363
+ and never reaches the identifier branch. It is also deliberately not widened
1364
+ into a claim of existence. Menu-scope resolution is schema-dependent —
1365
+ `/ip { zzznotamenu { print } }` is a `syntax error` on device — and offline has
1366
+ no schema, so a catalog MISS abstains rather than being decided either way.
1367
+ That is the catalog's own "a hit is decisive, a miss says nothing" contract, and
1368
+ turning such a miss into an acceptance would be the mirror-image defect of the
1369
+ one being fixed here.
1370
+
1371
+ ### `'` is not a RouterOS token (#355)
1372
+
1373
+ `'` (U+0027) is not a string delimiter and not a legal bare byte. Only `"` opens
1374
+ a string. `:parse` rejects every spelling of a bare apostrophe byte-identically
1375
+ on CHR 7.21.5, 7.23.5, 7.24.2 and 7.25beta3:
1376
+
1377
+ | input | device `:parse` | offline span |
1378
+ | --- | --- | --- |
1379
+ | `/system/identity/set name=don't` | `expected end of command (line 1 column 30)` | 29 |
1380
+ | `/system/identity/set name=don't'` | `expected end of command (line 1 column 30)` | 29 |
1381
+ | `/system/identity/set name='abc'` | `syntax error (line 1 column 27)` | 26 |
1382
+ | `:put 'abc'` | `syntax error (line 1 column 6)` | 5 |
1383
+
1384
+ The device's column is 1-based, so each offline span is the device's own byte.
1385
+ The accepting controls — `comment="it's here"` and a `# don't` comment line —
1386
+ parse cleanly on all four builds, which is what makes the rejection a device
1387
+ fact rather than a probe artefact. Rows 2–4 carry BALANCED apostrophes and are
1388
+ rejected anyway: this is a lexical rule, not a quote-balance one, and the remedy
1389
+ is always `"`, never a matching `'`.
1390
+
1391
+ Only the FIRST apostrophe is recorded, **document-wide**, the same way the hard
1392
+ `#` is: the device reports one column and it is the first one (row 2 reports
1393
+ column 30, not the trailing apostrophe), because classification stops there. A
1394
+ nested scope body is a separate scan (`segmentScopeBody`), so per-scan
1395
+ suppression is not enough on its own and the collapse to the earliest occurrence
1396
+ happens where the diagnostics are composed — which keeps each segmenter call
1397
+ pure. Apostrophes inside a `"`-string or a comment never reach the rule at all —
1398
+ both runs are consumed whole by the scan before it — so no special case is
1399
+ needed for either.
1400
+
1401
+ Symbol resolution stops at the apostrophe too, exactly as it stops at a
1402
+ malformed escape: the device classes nothing after a hard reject, so
1403
+ `:put 'abc'; :put $after` must not publish an occurrence for `$after`.
1404
+ Occurrences BEFORE the defect stand, which is the lab's X1 rule.
1405
+
1406
+ **This was a false ACCEPT, which the corpus join cannot see.** The false-reject
1407
+ measurement scores one direction only, so the analyzer passing all four forms
1408
+ with zero diagnostics never appeared in it. Over the pinned corpus the rule
1409
+ fires on 40 of the 7.24.2 scripts and **every one of them is a script the device
1410
+ rejects** — none of the 617 it accepts. It moves the false-reject count not at
1411
+ all (still 1, the extra `}` of corpus 228) while taking the caught count from 62
1412
+ to 93.
1413
+
1414
+ The token census moves, and only in the direction the fail-closed symbol stop
1415
+ predicts: 68 fewer `variable-*` tokens and their 306 bytes returning to
1416
+ `unclassified` (classified 1,023,413 → 1,023,107). Those are variable references
1417
+ sitting after a bare apostrophe, which the device never reaches and centrs
1418
+ therefore stops naming. The value, operator and bracket-equals censuses and
1419
+ `explain:arg-reach` are all unchanged.
1420
+
1421
+ **It also retires a wrong rule.** `execute` carried a private
1422
+ `hasUnbalancedQuotes` preflight that modelled `'` as a delimiter, moved into the
1423
+ offline gate by #354. It had no comment awareness, so `# don't` opened a quote
1424
+ it never closed: over the same corpus it false-rejected **51** of the 617
1425
+ device-accepted scripts. Deleting it costs 21 catches — all of them scripts the
1426
+ device rejects anyway, where the device stage is still the backstop — and takes
1427
+ the gate's false rejects from 52 to 1. A false reject blocks input RouterOS
1428
+ accepts, which is the figure that matters.
1429
+
1016
1430
  ### String escape validation (#247, #252)
1017
1431
 
1018
1432
  The accepted set is the **device's**, not the manual's. The
@@ -1061,8 +1475,9 @@ for the three issue rows `:put "\q"` / `:put "\x0a"` / `:put "\0a"` vs
1061
1475
  - Scoring a defect rule against the corpus needs the right oracle.
1062
1476
  `parseil_results.ok` is **not** one: `:parse` never throws, it returns a
1063
1477
  diagnostic *value*, so the capture records `ok=1 / status="ok"` for scripts
1064
- the device rejected (only 3 of 2739 rows are `ok=0`, and those are capture
1065
- failures). The verdict is in `il_text` — plain prose such as
1478
+ the device rejected (only 5 of 4635 rows are `ok=0`, and those are capture
1479
+ failures — one `edge-cases/oversize-32k.rsc` REST `413` per captured
1480
+ version). The verdict is in `il_text` — plain prose such as
1066
1481
  `expected input value (line 7 column 49)` means rejected, IL means accepted.
1067
1482
 
1068
1483
  ### The operator surface (#255)
@@ -1071,14 +1486,62 @@ Grounded, not transcribed, for the same reason the escape set above is: the
1071
1486
  [manual's operator list](https://manual.mikrotik.com/docs/developer-guides/scripting/index.md#operators)
1072
1487
  is a lower bound. `bun run explain:probe:operators` sweeps the manual's list
1073
1488
  **plus** plausible non-operators (`not`, `..`, `xor`, `mod`, `is`, `div`, …)
1074
- **plus** every IL head the corpus census saw, on CHR 7.23.3 stable, 7.24rc4
1075
- testing and 7.21.5 long-term — one build per release channel, so a claim about
1076
- any channel is evidence rather than an assumption. Re-cut the fixture with
1489
+ **plus** every IL head the corpus census saw, on CHR 7.24.2 stable, 7.23.5
1490
+ long-term and 7.25beta3 development — the same three builds the pinned corpus
1491
+ classes `complete`, so the operator axis and the token partition are finally
1492
+ scored on the same firmware (#342) — plus 7.21.5 as an **age anchor**, which is
1493
+ not a channel claim but the only live evidence that the surface behaves the same
1494
+ on a pre-7.23 branch. Re-cut the fixture with
1077
1495
  `bun run explain:operator-slice`; the table
1078
1496
  below is generated from `src/explain/operators.ts` by
1079
1497
  `bun run explain:operator-readme` and gated by
1080
1498
  `bun run explain:operator-readme:check`.
1081
1499
 
1500
+ **The census is a candidate generator, and boilerplate defeats its own
1501
+ tie-breaker.** `scripts/explain-operator-census.ts` reports every head-position
1502
+ token in the device IL, tagged by shape, because IL is a debug rendering that
1503
+ does not quote strings: a string literal is emitted bare, so string content is
1504
+ indistinguishable from a node head. Its stated honesty rule is that
1505
+ `distinctScripts` beats frequency — "a head seen in one script is one author's
1506
+ string, a head seen in three hundred is a language feature". Only the complete
1507
+ builds carry parseIL for the 35-script `tangentsoft` device-`/export` stratum,
1508
+ and that stratum showed the rule has a blind spot, because a stock string
1509
+ replicated across a whole stratum of device exports looks exactly like a
1510
+ language feature by script count:
1511
+
1512
+ | word-shaped head | scripts, partial builds → complete | what it actually is |
1513
+ | ---------------- | ---------------------------------: | ------------------- |
1514
+ | `for` | 2 → 22 | the defconf comment `accept to local loopback (for CAPsMAN)` — in 20 of the 35 device exports, plus the 2 forum scripts that already carried it |
1515
+ | `line` | 53 → 84 | device diagnostic prose, `bad parameter hw-offload (line 33 column 147)` |
1516
+
1517
+ Those arrows are a **coverage** step, not a version step: the left column is
1518
+ what a 913-script capture could see, the right what a 948-script one sees.
1519
+
1520
+ Neither is an operator, and both are `word`-shaped — the same bucket #255's
1521
+ `not` would have appeared in. So the sweep, not the census, still decides; a
1522
+ head that arrives with a whole stratum at once is boilerplate until the device
1523
+ says otherwise. The two heads that stratum added outright,
1524
+ `configuration.mode=` and `[system`, are `other`-shaped and are the documented
1525
+ string-noise class: a dotted wifi property name emitted bare, and text inside
1526
+ an embedded `/system script add source=` value.
1527
+
1528
+ **The census counts only the complete builds; a separate axis dates heads.**
1529
+ Occurrences are summed across versions, so admitting a partial capture would
1530
+ scale every figure by what happened to be captured — on this snapshot, letting
1531
+ all six builds in inflated the headline by ~20% while `distinctScripts`, the
1532
+ column that actually carries the argument, did not move at all. So
1533
+ `headOccurrences` / `headVersions` / `headArities` read
1534
+ `coverage_class = 'complete'` only, and the census states its version set in
1535
+ `versions` so a gate cannot silently compare two different universes.
1536
+
1537
+ Presence, unlike frequency, *is* monotone in coverage — a 913-of-948 capture
1538
+ that shows a head proves that build had it — so the partial builds keep one
1539
+ job: `headFirstSeen` dates each head against every captured build. That is what
1540
+ still grounds the `any` claim below at 7.20.8, a build the counting side no
1541
+ longer reads. Read a *late* first-seen as "not captured before here", never as
1542
+ "new in": the only two heads that do not reach 7.20.8 are the `tangentsoft`
1543
+ pair above, which no partial build could have seen.
1544
+
1082
1545
  **`:parse` IL is the oracle, and `highlight` cannot be.** IL is prefix form with
1083
1546
  the operator as its node's head, so it names the operator and shows its
1084
1547
  operands. `highlight`'s `syntax-meta` is the device's *residual structure*
@@ -1103,7 +1566,7 @@ Five things the device says that the manual does not:
1103
1566
  3. **`$`, `[`, `]` are syntax, not operator heads.** `$x` stays an atom in the
1104
1567
  IL and `[:tostr 1]` lowers to an `evl` node. Those bytes belong to the
1105
1568
  substitution axis.
1106
- 4. **`any` is an operator** (prefix, arity 1) and is not in the manual's list — a **nil-check**: `:typeof (any x)` is `bool`, `false` only for `nil`/`nothing` (the value of an undefined `:local` and of `[:nothing]`), `true` for everything else including `0`, `""` and `false`. It is the idiom `:if (any $x) ...` to test a variable that may be `nil`; `(true any false)` is not infix at all but juxtaposition `( true (any false))`, and `(1 . any [:nothing])` is concat `1`+`false`. Present since at least 7.20.8 (corpus `any|7.20.8:2`, and swept live on 7.21.5 long-term, 7.23.3 stable and 7.24rc4 testing with no difference between them).
1569
+ 4. **`any` is an operator** (prefix, arity 1) and is not in the manual's list — a **nil-check**: `:typeof (any x)` is `bool`, `false` only for `nil`/`nothing` (the value of an undefined `:local` and of `[:nothing]`), `true` for everything else including `0`, `""` and `false`. It is the idiom `:if (any $x) ...` to test a variable that may be `nil`; `(true any false)` is not infix at all but juxtaposition `( true (any false))`, and `(1 . any [:nothing])` is concat `1`+`false`. Present since at least 7.20.8 (corpus `headFirstSeen[any] = 7.20.8`, the oldest captured build, and swept live on 7.21.5 and 7.23.5 long-term, 7.24.2 stable and 7.25beta3 development with no difference between them).
1107
1570
  5. **`&&` and `||` are spellings**, lowering to the `and` and `or` nodes.
1108
1571
 
1109
1572
  The `(>…)` and `<%%` forms are in the table on the same footing as `+`.
@@ -1119,7 +1582,7 @@ their `precedence` is honestly `null`. That is "unmeasured", not "unknown": the
1119
1582
  sweep asks them separately, every `(U 1 B 2)` and `(1 B U 2)` for
1120
1583
  `U`∈`!`,`any`,`~`,`-`,`>` against all 24 binaries — 240 probes, recorded in the
1121
1584
  fixture's `unary` block. All 240 are accepted and the **binary is outer in every
1122
- one**, on all three versions, so each prefix operator binds tighter than every
1585
+ one**, on all four swept builds, so each prefix operator binds tighter than every
1123
1586
  binary including `->` (14) and `<%%` (13). The table stores one `precedence` per
1124
1587
  spelling, which is the binary level; `~`, `-` and `>` are here because their
1125
1588
  unary reading has no other record.
@@ -1131,8 +1594,14 @@ not concat at all** — `1.` lexes as a *variable name* and the row comes back a
1131
1594
  juxtaposition, `( $1. 2)`. A rule that claims every `.` byte for the operator
1132
1595
  emits a span the device does not have.
1133
1596
 
1134
- Everything in the sweep is identical across 7.21.5, 7.23.3 and 7.24rc4 except
1135
- one runtime row: `:put ({2;1} > {1;2;3})` evaluates to `true` only on 7.24rc4.
1597
+ Everything in the sweep is identical across all four builds except one runtime
1598
+ row: `:put ({2;1} > {1;2;3})` evaluates to `true` on 7.24.2 and 7.25beta3 and
1599
+ errors on both long-term builds. **Array comparison is not fixed on current
1600
+ long-term** — 7.23.5 is newer than the 7.23.3 this was first measured on and
1601
+ still errors — so the change rode the 7.24 line and was not backported. The two
1602
+ errors also differ: 7.23.5 says `cannot compare if array is more than array`,
1603
+ 7.21.5 says `cannot compare if nothing is more than nothing`, so on 7.21.5 the
1604
+ operands did not survive to the comparison at all.
1136
1605
  Identical is measured, not assumed — the slice diffs verdict, arity, precedence,
1137
1606
  associativity, `highlight` run, unary placement and op-axis per version. So the
1138
1607
  operator table needs no version gate; anything reporting what a comparison
@@ -1231,49 +1700,64 @@ And the grounded complement — asked, and refused:
1231
1700
  `bun run explain:token-census:readme` and gated against it by
1232
1701
  `bun run explain:token-census:readme:check`; the fixture itself is gated
1233
1702
  against a fresh corpus run by `bun run explain:token-census:check`. Of
1234
- 1,426,731 analyzed bytes, 990,840 are classified (69.45%), the remaining
1235
- 435,891 are `unclassified`. The census emits 205,072 tokens (avg 216.3 per
1703
+ 1,426,731 analyzed bytes, 1,023,107 are classified (71.71%), the remaining
1704
+ 403,624 are `unclassified`. The census emits 241,539 tokens (avg 254.8 per
1236
1705
  script). Every byte belongs to exactly one token — sorted by `start`, no
1237
- gaps, no overlaps, `join(slice) === input` — and the `class` field is
1238
- provisional until #264 B5. Each B2 fill should move the classified
1706
+ gaps, no overlaps, `join(slice) === input` — and the `class` field is the
1707
+ vocabulary #264 B5 settled. Each B2 fill should move the classified
1239
1708
  percentage.
1240
1709
  <!-- END GENERATED token-census -->
1241
1710
 
1242
1711
  B1's `data.tokens[]` is live behind `--tokens` — a total, gapless byte
1243
- partition whose `class` is provisional until #264 B5 (every unclaimed byte is
1244
- `unclassified`). Since B1 its only fill source was `data.spans[]` (comment runs
1712
+ partition whose `class` draws on the vocabulary #264 B5 settled (every unclaimed
1713
+ byte is `unclassified`). Since B1 its only fill source was `data.spans[]` (comment runs
1245
1714
  and resolved variable occurrences); **the path fill claims resolved menu and
1246
1715
  command-name bytes as provisional `dir` / `cmd` tokens**, including valid path
1247
1716
  slashes and nested command substitutions, while ambiguous, malformed, and
1248
- source-unmapped runs stay unclassified. **`#293`'s arg fill** in
1249
- `src/explain/arg-tokens.ts` then claims argument names and
1250
- their `=` (name run
1251
- `[span.start, valueSpan.start - 1)` plus the single `=` byte at
1252
- `valueSpan.start - 1`) on the residual left by spans, before operators see it
1717
+ source-unmapped runs stay unclassified. A substitution's bytes are located by
1718
+ the resolution's published **`innerSpan`**, never by arithmetic on its `span`:
1719
+ `span` covers `[…]` for a bare substitution but `$[…]` for one interpolated
1720
+ inside a string (the sigil is part of it), and `inner` is trimmed on top of
1721
+ that, so `span.start + 1` was one byte early in a string and two bytes wrong
1722
+ for `[ … ]`. Both refused the fill's slice check and fell through to the
1723
+ string fill, which is how a command `structure.subcommands` had already
1724
+ resolved came back as opaque `string` bytes (#325). The check itself stays:
1725
+ where `Loc` widens the resolution to its statement, the slice no longer equals
1726
+ `inner` and the bytes are left unclaimed rather than landed on the wrong
1727
+ place. **`#293`'s arg fill** in
1728
+ `src/explain/arg-tokens.ts` then claims argument names as `arg` (name run
1729
+ `[span.start, valueSpan.start - 1)`) and the single `=` byte at
1730
+ `valueSpan.start - 1` as `arg-sep` (#264 B5) on the residual left by spans,
1731
+ before operators see it
1253
1732
  — and **`#295`'s value fill follows it** — `src/explain/value-tokens.ts`
1254
1733
  claims argument value bytes and leaf array-literal members
1255
1734
  (`data.values.occurrences`, leaves only via `parent` containment,
1256
- quotes-included) on the residual left by `spans`+`path`+`arg`, before the
1735
+ quotes-included) on the residual left by `spans`+`path`+`arg`+`escaped`, before the
1257
1736
  **B3 string/brace fills** see it — `src/explain/string-tokens.ts` claims
1258
- quoted-string runs (delimiters included) and `src/explain/brace-tokens.ts`
1737
+ quoted-string runs (delimiters included, escapes already taken) and `src/explain/brace-tokens.ts`
1259
1738
  claims every residual brace delimiter (`{`/`}`), including delimiters from
1260
1739
  invalid braced references, on the residual left by
1261
- `spans`+`path`+`arg`+`value`, and finally `operator` sees only what none of
1740
+ `spans`+`path`+`arg`+`escaped`+`value`, and finally `operator` sees only what none of
1262
1741
  them wanted — which is why the operator fill can abstain on `, / = -` outside
1263
1742
  `( )`, on bytes glued after an argument `=`, and on bytes glued into an
1264
- argument name without a smarter operator scanner. Fill order is enforced by
1743
+ argument name without a smarter operator scanner. **`src/explain/escape-tokens.ts`
1744
+ claims valid string-internal escapes as `escaped`** between `arg` and `value`:
1745
+ an escape is interior to a quoted run, so it has to claim before the two fills
1746
+ that would otherwise swallow it whole, and it splits their runs into fragments
1747
+ around it the same way a `variable-*` span already did. Fill order is enforced by
1265
1748
  argument order to `buildTokens` ([#290 design decision 1](../../src/explain.ts)):
1266
- `spans` claims first, then `path`, `arg`, `value`, `string`, `brace`, and
1267
- finally `operator`.
1749
+ `spans` claims first, then `path`, `arg`, `escaped`, `value`, `string`, `brace`,
1750
+ and finally `operator`.
1268
1751
  `ExplainTokenClass` is
1269
- `ExplainSpanClass | "dir" | "cmd" | "operator" | "arg" | "value" | "string" |
1270
- "brace" | "unclassified"` — one
1271
- provisional `operator` class for all 26 spellings + 2 aliases, one provisional
1272
- `arg` class for both the name bytes and the `=` separator, one provisional
1273
- `value` class for every leaf value span, plus provisional `string` (quoted
1274
- runs) and `brace` (scope delimiters) — emit first, name later (whether the `=`
1275
- or per-shape `value` later deserve their own classes is #264 B5 and does not
1276
- move the byte coverage), not per-operator/per-category/shape (`#264` B5).
1752
+ `ExplainSpanClass | "dir" | "cmd" | "operator" | "arg" | "arg-sep" | "value" |
1753
+ "string" | "escaped" | "brace" | "unclassified"` — one `operator` class for all 26
1754
+ spellings + 2 aliases, `arg` for an argument's name bytes and `arg-sep` for the
1755
+ single `=` that binds it to its value, one `value` class for every leaf value
1756
+ span, plus `string` (quoted runs minus their escapes), `escaped` (one valid
1757
+ string-internal escape sequence) and `brace` (scope delimiters). #264
1758
+ settled which of those merges keep their name and which do not, on a stated
1759
+ rule rather than taste — see
1760
+ [The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5).
1277
1761
  `ExplainSpanClass` and `data.spans[]` stay proof-only; `src/explain/operators.ts`
1278
1762
  remains data plus accessors, and the operator table above is its source. The
1279
1763
  `centrs → highlight` projection is B4 and reads both.
@@ -1289,11 +1773,413 @@ each grounded on the corpus device oracle (`parseil_results.il_text`):
1289
1773
  | A spelling glued immediately after an argument `=`. | `in-interface-list=!LAN`, `.id=*2`, `oid=.1.3.6.1.2.1` — the byte after an argument `=` starts the value. |
1290
1774
  | A spelling glued into an argument **name**. | `:foreach x in=$list` is `/foreach counter=$x` with no `(in …)` node anywhere; the IL keeps `security.authentication-types=wpa2-psk` as one name and renders `.id` as the single symbol `$.id`, never `(. …)`. |
1291
1775
 
1292
- The first abstention has a measured cost: `find where name="x"` inside `[ … ]`
1293
- *does* lower to a real `(= $name x)` node, so 224 of the corpus's 1,259
1294
- bracket-interior `=` bytes are genuine comparisons that now stay
1295
- `unclassified` — against 1,035 that were plain `arg=value`. A `where`-aware
1296
- fill can take them back later; claiming all 1,259 would be 82% wrong.
1776
+ The first abstention has a measured cost, and it is now generated rather than
1777
+ asserted. The figures below were prose with no script behind them and no gate
1778
+ on them for three releases, quoting a `224` / `1,259` / `1,035` split that no
1779
+ universe reproduces (#341). The abstention itself is as wide as it ever was —
1780
+ what was wrong was the accounting of it. They are re-derived by
1781
+ `bun run explain:bracket-equals` and gated by
1782
+ `bun run explain:bracket-equals:readme:check`.
1783
+
1784
+ <!-- BEGIN GENERATED bracket-equals — regenerate with `bun run explain:bracket-equals:readme` -->
1785
+ Measured by `bun run explain:bracket-equals` over the pinned corpus:
1786
+ **1,079** `=` bytes sit inside `[ … ]`, across
1787
+ 263 of 948 scripts, and **1,067** of them come back
1788
+ `unclassified` — the operator fill abstains on essentially all of it. No
1789
+ later fill rescues them: the `arg` fill claims **0**, because it offers only
1790
+ the `=` its own located argument token names and a `[ … ]` substitution is
1791
+ not where it locates arguments.
1792
+
1793
+ Of those 1,067, **448** are governed by a `find`/`where` query and are
1794
+ genuine comparisons — the device lowers `[find address=$IP]` to
1795
+ `(= $address $IP)` — against **619** plain `arg=value`. So abstaining is
1796
+ 42.0% wrong and claiming the lot would be 58.0% wrong; a `where`-aware
1797
+ fill can take the query half later. For contrast the fill does claim
1798
+ **1,402** `=` bytes whose innermost opener is `(`.
1799
+
1800
+ The query reading is the device's, not a guess: on 7.24.2, 93 of the
1801
+ 137 scripts carrying one have IL with at least as many `(= …)` nodes.
1802
+ Of the rest, 28 are scripts the console refused outright — their IL is prose,
1803
+ with no nodes of any kind — and 16 use a menu the capture build could not
1804
+ resolve (a forum snippet on `/interface wireless` answers `bad command name
1805
+ wireless`), so the device never reached the query to lower it. Neither is a
1806
+ counter-example to the reading. Whole-corpus `(= …)` head nodes, which also
1807
+ count expression comparisons the fill already claims:
1808
+ 7.23.5 1,233, 7.24.2 1,240, 7.25beta3 1,240.
1809
+ <!-- END GENERATED bracket-equals -->
1810
+
1811
+ ### Device agreement for the token partition (#264 B4, #263)
1812
+
1813
+ The census above counts what offline analysis *claimed*. This counts what the
1814
+ device *agreed with*. The oracle is the committed stratified slice of the Q13
1815
+ `/console/inspect request=highlight` captures
1816
+ ([`test/fixtures/explain/highlight-streams.slice.json`](../../test/fixtures/explain/highlight-streams.slice.json)),
1817
+ so the scorer needs no router, no corpus and no network, and gates in the unit
1818
+ suite rather than in the corpus job. Both oracles must see the same bytes, so
1819
+ the source text is reconstructed from the device's own run-length pairs — never
1820
+ from the corpus row, which still holds the non-ASCII the capture replaced
1821
+ (the #269 rule, one layer up).
1822
+
1823
+ **`highlight` is not one oracle; it is four kinds of answer.** Scoring a byte
1824
+ means first knowing what kind of answer came back, because three of the four
1825
+ are not judgments about syntax at all:
1826
+
1827
+ | Kind | Classes | Why it cannot be scored as agreement |
1828
+ | ---- | ------- | ------------------------------------ |
1829
+ | syntax | `cmd` `dir` `arg` `arg-dot` `arg-scope` `comment` `escaped` `syntax-meta` `variable-local` `variable-global` `variable-auto` `variable-parameter` | — this is the region that can be scored |
1830
+ | non-syntax | `obj-inactive` `obj-dynamic` `obj-disabled` `variable-undefined` `syntax-obsolete` | The source text alone does not decide these. centrs has no class for most of them at all — `ExplainSymbolClass` deliberately excludes `undefined` — and where it *does* claim the byte (an `arg` the device calls `obj-inactive`) it is the **scorer** that abstains, because the device was not making a syntax judgement there. *Why* each one is out of reach is a second question, answered per byte rather than per class; see applicability below. |
1831
+ | silence | `none` | The device's abstention, not a class named "none". Agreeing with silence is not agreement. |
1832
+ | parser-stop | `error` | The device gives up here — see below. |
1833
+
1834
+ **Applicability is a second axis, and it is not the class name.** #263's four
1835
+ categories — offline-decidable, schema-dependent, state-dependent,
1836
+ version-dependent — are assigned from the *pair* (what centrs read the byte as,
1837
+ what the device called it) plus whether the two captures even agreed, because
1838
+ one highlight class carries more than one category. Version disagreement is
1839
+ checked first, ahead even of the device's own silence: a byte one build
1840
+ classifies and the next leaves `none` is a version difference, not an absence.
1841
+ `obj-inactive` on an argument name means the menu's schema has no such field
1842
+ (**schema** — `default-vlan-id`, under an `/interface ethernet switch port` a
1843
+ CHR does not have); the same class on a menu or command name means the package
1844
+ is not installed on this box (**state**: `/zerotier`, `/user-manager`,
1845
+ `/container`). `obj-dynamic` on the routing table `main` is a live object
1846
+ (**state**).
1847
+ `syntax-obsolete` is this version's deprecation table talking, landing on the
1848
+ whitespace where modern syntax wants `=` (`} else {`, `:foreach conn in
1849
+ $conns`) — **version**. And `variable-undefined` is deliberately left
1850
+ **uncategorized**: an undefined reference may be an external global, a
1851
+ query/filter field name or a lexical spelling, nothing committed here separates
1852
+ the three, and an absent binding in *this* document is not evidence of a
1853
+ runtime undefined variable either. Leaving it explicit is the answer, not a gap
1854
+ to be filled by assumption.
1855
+
1856
+ **The device stops classifying at its first `error`, and mostly does not
1857
+ recover.** Measured over the slice, `error` appears **at most once per script
1858
+ and is always exactly one byte wide**, and on long-term 7.23.5 every byte after
1859
+ it comes back `none`. Most of the slice sits in that tail. A percentage computed
1860
+ over the whole stream therefore measures the device giving up, not the parser,
1861
+ which is why the tail is its own outcome and is never folded into agreement. The
1862
+ rule is not absolute across versions, so the report does not assert it: stable
1863
+ 7.24.2 and development 7.25beta3 classify again after the `error` in one run and
1864
+ stop on fewer scripts, and
1865
+ `parser-recovered` in the table below is the figure that says by how much —
1866
+ zero would have been the claim, and it is not zero. Both counts are generated,
1867
+ not quoted here, so neither can go stale against the fixture.
1868
+
1869
+ **The projection is declared, never fitted.** `src/explain/highlight-projection.ts`
1870
+ maps each centrs token class to the device classes it predicts, justified by
1871
+ what the class was *defined* to mean rather than by where it happened to land.
1872
+ A projection tuned to the measurement would make the measurement circular. So a
1873
+ class whose intent has no single device counterpart — `string`, which spans a
1874
+ delimiter the device calls `syntax-meta`, escapes it calls `escaped` and an
1875
+ interior it declines to classify, and `value`, for which the device has no class
1876
+ at all — projects nowhere, and its bytes are counted as `unprojected` with the
1877
+ device's own answers tabulated beside them. That table is the evidence #264 B5
1878
+ needs in order to decide which provisional merge deserves to be split; it is not
1879
+ a score.
1880
+
1881
+ <!-- cspell:disable — the example table quotes corpus script paths verbatim, forum-thread typos included -->
1882
+ <!-- BEGIN GENERATED highlight-agreement — regenerate with `bun run explain:highlight-agreement:readme` -->
1883
+ The agreement report is re-derivable with
1884
+ `bun run explain:highlight-agreement` and needs no router and no corpus: it
1885
+ scores the token partition against
1886
+ `test/fixtures/explain/highlight-streams.slice.json`, 70 stratified scripts of
1887
+ committed `/console/inspect request=highlight` output. The figures below are
1888
+ generated from `test/fixtures/explain/highlight-agreement.json` by
1889
+ `bun run explain:highlight-agreement:readme`, gated against it by
1890
+ `bun run explain:highlight-agreement:readme:check`, and the fixture itself is
1891
+ gated against a fresh measurement by
1892
+ `test/unit/explain-highlight-agreement.test.ts`. Of 85,529 bytes at 7.24.2,
1893
+ 27,974 are bytes **both** sides decided a syntax class for, and 99.99% of
1894
+ those agree (dev 99.99%, holdout 100.00%). Set `comment` aside — it is 65.63%
1895
+ of that decided region and mostly the corpus's harness-injected `# Source:`
1896
+ banner (#203) — and the remaining 9,615 bytes agree 99.98%. The device stops
1897
+ classifying at its one-byte `error`: 36 of 70 scripts carry one, and the
1898
+ 41,968 bytes from there on are not a judgment about anything. The oracle
1899
+ itself moves between captures: 36 stop at 7.24.2 and 43 stop at 7.23.5 and 36
1900
+ stop at 7.25beta3, and 27 of the 70 streams differ between them. **This is a
1901
+ trend line, not a pass gate** — the slice is a per-(split, class) quota over a
1902
+ corpus that is 96.8% two authors, so the percentage describes this slice.
1903
+
1904
+ | centrs class | projects to | agree | disagree | unprojected | of decided |
1905
+ | ------------ | ----------- | ----: | -------: | ----------: | ---------: |
1906
+ | `comment` | `comment` | 18,359 | 0 | 0 | 100.00% |
1907
+ | `variable-local` | `variable-local` | 1,123 | 0 | 0 | 100.00% |
1908
+ | `variable-global` | `variable-global` | 582 | 0 | 0 | 100.00% |
1909
+ | `variable-auto` | `variable-auto` | 92 | 0 | 0 | 100.00% |
1910
+ | `variable-parameter` | `variable-parameter` | 459 | 0 | 0 | 100.00% |
1911
+ | `dir` | `dir` | 1,677 | 0 | 0 | 100.00% |
1912
+ | `cmd` | `cmd` | 2,244 | 0 | 0 | 100.00% |
1913
+ | `arg` | `arg`, `arg-dot`, `arg-scope` | 2,188 | 0 | 0 | 100.00% |
1914
+ | `arg-sep` | `syntax-meta` | 382 | 0 | 0 | 100.00% |
1915
+ | `operator` | `syntax-meta` | 237 | 2 | 0 | 99.16% |
1916
+ | `brace` | `syntax-meta` | 211 | 0 | 0 | 100.00% |
1917
+ | `string` | *abstains* | 0 | 0 | 499 | — |
1918
+ | `escaped` | `escaped` | 418 | 0 | 0 | 100.00% |
1919
+ | `value` | *abstains* | 0 | 0 | 144 | — |
1920
+
1921
+ | outcome | bytes @7.24.2 | bytes @7.23.5 | bytes @7.25beta3 |
1922
+ | ------- | ----: | ----: | ----: |
1923
+ | agree — both decided, projection accepts | 27,972 | 27,218 | 27,972 |
1924
+ | disagree — both decided, projection rejects | 2 | 2 | 2 |
1925
+ | unprojected — no declared projection covers the pair | 643 | 633 | 643 |
1926
+ | offline-silent — device decided, centrs abstained | 4,300 | 4,150 | 4,300 |
1927
+ | non-syntax — the device answered something syntax cannot decide | 847 | 733 | 847 |
1928
+ | device-silent — centrs decided, device said `none` | 5,815 | 5,674 | 5,815 |
1929
+ | both-silent | 3,886 | 3,702 | 3,886 |
1930
+ | parser-stopped — at/after the device's `error` byte, and silent from there | 41,968 | 43,417 | 41,968 |
1931
+ | parser-recovered — past that `error`, and the device classified anyway | 96 | 0 | 96 |
1932
+
1933
+ Where the device decided and offline analysis did not (4,300 bytes at 7.24.2,
1934
+ the next fill's target list): `syntax-meta` 3,122, `comment` 355,
1935
+ `variable-local` 326, `arg` 197, `escaped` 145, `dir` 66, 89 across the rest.
1936
+ Most of the `syntax-meta` share is whitespace the device merged into an
1937
+ adjacent structure run rather than a token centrs missed.
1938
+
1939
+ **Applicability** is the second axis (#263): of the bytes the device did
1940
+ answer, what kind of fact was it answering? Assigned from the pair — what
1941
+ centrs read the byte as and what the device called it — plus whether the
1942
+ captures agreed, never by assigning a whole class to one category. It covers
1943
+ every measured byte, the post-`error` tail included: where one capture stops
1944
+ parsing and the next does not, that tail IS the version disagreement, which is
1945
+ why it dominates the category.
1946
+
1947
+ | applicability @7.24.2 | bytes | leading cells |
1948
+ | ---------------------------- | ----: | ------------- |
1949
+ | offline-decidable | 32,002 | `comment` → `comment` 18,006, `unclassified` → `syntax-meta` 2,999, `cmd` → `cmd` 2,132 |
1950
+ | schema-dependent | 326 | `arg` → `obj-inactive` 326 |
1951
+ | state-dependent | 168 | `dir` → `obj-inactive` 120, `cmd` → `obj-inactive` 21, `unclassified` → `obj-dynamic` 20 |
1952
+ | version-dependent | 1,158 | `comment` → `comment` 353, `unclassified` → `obj-inactive` 159, `unclassified` → `syntax-meta` 123 |
1953
+ | uncategorized | 213 | `unclassified` → `obj-inactive` 90, `unclassified` → `variable-undefined` 75, `value` → `variable-undefined` 21 |
1954
+ | no-device-answer | 51,662 | `none` where the captures agree, wherever it falls |
1955
+
1956
+ One representative run per cell, so B5 reads a fragment rather than a count.
1957
+ `runs` is how often that exact fragment occurs; the location is a slice script
1958
+ and a byte offset into the stream the device saw.
1959
+
1960
+ | outcome | cell | fragment | runs | first at |
1961
+ | ------- | ---- | -------- | ---: | -------- |
1962
+ | disagree | `operator` → `arg` | `"in"` | 1 | `forum/amm0/topic-169456-having-the-where-filter-in-scripting-signifantly-increases-the-execution-time-an/post-0006-snippet-01.rsc` @615 |
1963
+ | unprojected | `string` → `syntax-meta` | `"\""` | 311 | `eworm/ppp-on-up.rsc` @435 |
1964
+ | unprojected | `value` → `syntax-meta` | `"\""` | 88 | `forum/amm0/topic-153357-using-wifiwave2-to-bridge-two-audience-wirelessly-thoughts-4-address-mode/post-0001-snippet-01.rsc` @1436 |
1965
+ | unprojected | `string` → `arg` | `"as-value"` | 1 | `forum/amm0/topic-266499-how-to-condense-variables-passed-through-to-function/post-0020-snippet-01.rsc` @279 |
1966
+ | unprojected | `string` → `cmd` | `"len"` | 1 | `forum/amm0/topic-155978-zerotier-on-mikrotik-a-rosetta-stone-v7-1-1/post-0001-snippet-01.rsc` @1794 |
1967
+ | unprojected | `string` → `dir` | `":"` | 1 | `forum/amm0/topic-155978-zerotier-on-mikrotik-a-rosetta-stone-v7-1-1/post-0001-snippet-01.rsc` @1793 |
1968
+ <!-- END GENERATED highlight-agreement -->
1969
+ <!-- cspell:enable -->
1970
+
1971
+ **What the disagreements are.** Two cells carry all of them, and neither is a
1972
+ lexing accident:
1973
+
1974
+ - `arg` → `syntax-meta`: the `=` byte the `arg` class merges in. The device
1975
+ reads a separator as structure. This is the measured cost of the merge the
1976
+ token-census block already flags, and it is the single largest reason `arg`
1977
+ scores below the other name-coincident classes.
1978
+ - `operator` → `arg`: the space-separated `in` of `:foreach conn in $conns`,
1979
+ which the device reads as the argument **name** `in`. The two oracles
1980
+ genuinely differ here — `:parse` IL carries `in` in the operator set, and the
1981
+ operator-abstention table above already records that `:foreach x in=$list`
1982
+ lowers with no `(in …)` node.
1983
+
1984
+ The third cell is gone. `cmd` → `dir` was the whole of #324 — a known menu path
1985
+ carrying arguments and no verb, where `verbsplit` promoted the last segment;
1986
+ those 12 bytes now abstain instead of disagreeing, and the 14 `dir` bytes the
1987
+ same statements contributed to `agree` abstain with them. That is the price,
1988
+ and [the fix states it](#the-menu-path-with-arguments-324): withdrawing a
1989
+ fabricated verb also withdraws the menu reading it was built on.
1990
+
1991
+ The `unprojected` residue is what #264 B5 read to decide which merges to
1992
+ split — [The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5) records
1993
+ the verdict on each. It was also where #325 lived:
1994
+ a quoted string claimed any `$[…]` substitution inside it, so bytes the device
1995
+ resolves to `cmd` or `dir` came back as one opaque `string` run even though
1996
+ `structure.subcommands` had already resolved that same span. Those cells have
1997
+ largely emptied (`string` → `cmd` 42 → 3, `string` → `dir` 19 → 1, with the
1998
+ same fix recovering padded bare brackets `[ … ]` from `offline-silent`), and
1999
+ the two residues left are stated rather than fixed:
2000
+
2001
+ - `$(…)` — a **parenthesized** interpolation carrying a bracket, as in
2002
+ `apikey-len=$([:len $apikey])`. Only `$[` is scanned inside a string, so the
2003
+ inner command is not resolved there in the first place; this is a parser
2004
+ reach, not a fill-order question (4 bytes on the slice).
2005
+ - a bare-word **argument** inside a substitution — `as-value` in
2006
+ `:put "$[/ip address print as-value]"`. The path fill claims the path and
2007
+ verb; the argument fill reads `statements[].arguments`, and a substitution is
2008
+ not a statement, so it has no candidate to claim (8 bytes).
2009
+
2010
+ The `$` sigil and the `[` / `]` delimiters stay `string`, and `value` bytes
2011
+ are, to the device, simply unclassified — both settled by
2012
+ [The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5) below.
2013
+
2014
+ ### The vocabulary, and what earns a class (#264 B5)
2015
+
2016
+ `data.tokens[]` shipped with its `class` marked provisional from B1 onward —
2017
+ emit first, name later (#293 design decision 2). This is the "name later", and
2018
+ the vocabulary below is what `@tikoci/centrs/explain` offers a consumer. It is
2019
+ no longer provisional.
2020
+
2021
+ **The rule a class has to pass.** A merged class is split when both hold, and
2022
+ stays merged otherwise:
2023
+
2024
+ 1. **A consumer needs the distinction** — it changes what that consumer renders
2025
+ or decides, not just how a token dump reads.
2026
+ 2. **The envelope cannot already supply it** — the distinction is not
2027
+ recoverable by joining `tokens[]` to another published surface on byte
2028
+ offsets.
2029
+
2030
+ The second test is the one that does the work. A class that restates a fact
2031
+ `structure`, `symbols` or `values` already publishes is a second place for that
2032
+ fact to go stale, and the two copies can then disagree about the same bytes. So
2033
+ the question is never "would a name here be nice"; it is "what would a consumer
2034
+ have to do without it". Applied to the five merges the fills carried, exactly
2035
+ one passes:
2036
+
2037
+ | merge | what a consumer would do without it | verdict |
2038
+ | ----- | ----------------------------------- | ------- |
2039
+ | `arg` — the name bytes **and** the `=` | subtract one from `arguments[].valueSpan.start`, which exists only where the STRICT reading read. **5,088 of the 16,526 `=` runs the fill paints (30.8%, in 436 of 948 corpus scripts) sit in a statement whose `arguments` refused** — led by `do=` (2,393), `else=` (328) and `in=` (212), the block binder of every `:if`, `:foreach` and `:while`. For those there is nothing to subtract from. | **split** → `arg-sep` |
2040
+ | `operator` — 26 spellings + 2 aliases | slice the input at the token's span: the spelling *is* the bytes, and `src/explain/operators.ts` maps it to its IL node | keep |
2041
+ | `value` — every leaf, whatever its shape | join `data.values.occurrences[]` on the offset; #225's three axes (shape, observed, schema) live there, and a class per shape would be a fourth copy free to contradict them | keep |
2042
+ | `brace` — scope and array delimiters | `structure.blocks[]` locates every scope block; an array container is the `values.occurrences[]` entry that is some other occurrence's `parent` | keep |
2043
+ | `string` — delimiters, escapes, interior | the opening delimiter is the run's first byte and, where the run is closed, the last is its partner (an unterminated one is already a located defect). The **escapes were a different answer** — see below | **partly split** → `escaped`; delimiters and interior stay merged |
2044
+
2045
+ Re-derive the `arg-sep` figures with `bun run explain:arg-reach`, which prices
2046
+ the same gap it prices the reach fork with.
2047
+
2048
+ **What the split cost.** Nothing, which was the claim and is now the
2049
+ measurement. Classified bytes are identical — 1,023,405 of 1,426,731 (71.73%),
2050
+ equal to the last recorded decimal before and after — and `arg` gave up exactly
2051
+ the 16,526 bytes `arg-sep` claims, one per attribute. Token count rises 214,503
2052
+ → 231,026 because one run became two. **This is a retag.**
2053
+
2054
+ (Those totals are the corpus as it stood for THIS measurement. The classified
2055
+ total is 1,023,107 today: #347/#348 later classified 8 further bytes, and #355's
2056
+ fail-closed symbol stop then unclaimed 306 of them — variable bytes sitting
2057
+ after a bare apostrophe, which the device never reaches. The identity being
2058
+ claimed here is between the two sides of the split, not with any later total —
2059
+ restating it against today's figure would credit this retag with a movement a
2060
+ different change made. The generated block above carries the current numbers.)
2061
+
2062
+ One count is not a clean split: `arg`'s own run count falls 16,526 → 16,523.
2063
+ Three attributes lost their name run entirely, because an earlier `variable-*`
2064
+ span had already claimed the name bytes — `set $ifcId ssid=$ssid`, and two
2065
+ `on-error=` in one snippet — leaving only the `=` on the residual. Before the
2066
+ split those three emitted a token *called* `arg` that contained nothing but an
2067
+ `=`. That is the merge in miniature, and it is why the number goes down.
2068
+
2069
+ **The agreement number moved a long way, and it is still not a score.** Against
2070
+ the committed device slice, projection disagreements fall **360 → 2** and the
2071
+ decided region goes 98.66% → 99.99% (`arg` alone 85.50% → 100.00%). Read that
2072
+ as the withdrawal of a wrong assertion, not as a parser that got better: centrs
2073
+ stopped calling a separator a name over the 358 runs the device had been
2074
+ reading as `syntax-meta` all along, and not one byte changed which analyzer
2075
+ found it. The two survivors are the `operator` → `arg` `in` of
2076
+ `:foreach conn in $conns`, where the two oracles genuinely differ.
2077
+
2078
+ The projection entry was **declared before the measurement, never fitted to
2079
+ it**: `arg-sep` predicts `syntax-meta` because a separator is structure
2080
+ punctuation, the same reason `brace` and `operator` predict it. The slice then
2081
+ put 358 of 358 there.
2082
+
2083
+ **Where the tolerant reading reaches the envelope (#316's open decision).** It
2084
+ reaches it *as the token partition*, and that is the only place it should.
2085
+ `structure.statements[].arguments` stays the strict all-or-nothing reading,
2086
+ because its consumer renders a runnable request and a half-read argument list
2087
+ changes what the rendered command does. But with the `=` named, `tokens[]`
2088
+ locates the argument on its own:
2089
+
2090
+ ```text
2091
+ /ip route add dst-address=1.1.1.1 gateway=$gw
2092
+ arguments: { read: false, why: "a variable value" }
2093
+ tokens: … arg("gateway") arg-sep("=") unclassified("$") variable-parameter("gw")
2094
+
2095
+ :if ($x = 1) do={ :put "hi" }
2096
+ arguments: { read: false, why: "a substitution or expression value" }
2097
+ tokens: … operator("=") … arg("do") arg-sep("=") brace("{") …
2098
+ ```
2099
+
2100
+ Neither statement publishes a single argument, and in both a browser consumer
2101
+ can still see a located `gateway=` / `do=`. The second line is the
2102
+ discrimination working as well: the same `=` byte is `operator` inside
2103
+ `($x = 1)` and `arg-sep` in `do=`, decided by fill order and meaning rather
2104
+ than by a byte scan.
2105
+
2106
+ What `tokens[]` still does **not** carry is the *reason* a value was not
2107
+ decoded. `undecided` stays off `ExplainArgumentToken` and out of the
2108
+ envelope: the envelope's value-fact surface is `data.values.occurrences[]`,
2109
+ where absence at an offset already *is* the abstention, and a second spelling
2110
+ of the same abstention would be free to disagree with it. A consumer that needs
2111
+ the reason imports `lexExplainArgumentTokens` from the package root — which is
2112
+ why that export exists.
2113
+
2114
+ **The second split the rule admits, now made: `escaped`.** A *valid* string
2115
+ escape is recoverable from nothing. Only INVALID ones surface elsewhere, as
2116
+ `bad-string-escape` diagnostics, so a consumer holding `tokens[]` has no
2117
+ published surface to join against to find the valid ones — and the device does
2118
+ class them, as one of the 19 highlight classes. Both halves of the rule hold,
2119
+ so the class is earned.
2120
+
2121
+ Unlike `arg-sep` this is a **fill change** rather than a pure relabel of one
2122
+ analyzer's own output: the escape grammar (uppercase-hex `\XX`, the whitespace
2123
+ continuation, the twelve single-character forms) has to be walked, and the
2124
+ bytes come out of two fills at once rather than one.
2125
+ `src/explain/escape-tokens.ts` runs between `arg` and `value`.
2126
+
2127
+ **The walk is not new, and deliberately not a second one.**
2128
+ [String escape validation](#string-escape-validation-247-252) already owned this
2129
+ grammar for `bad-string-escape`. `walkStringEscapes` now returns *both* halves
2130
+ of that one reading — the valid escapes and the first invalid one — so an
2131
+ `escaped` token and a `bad-string-escape` diagnostic cannot disagree about which
2132
+ bytes are an escape. Deriving them from two walks would have made that
2133
+ disagreement possible, and the document is walked once rather than twice. The
2134
+ walk stops at the first invalid escape, so `escaped` is the prefix the validator
2135
+ vouched for: past a malformed escape the string boundaries are themselves in
2136
+ doubt, and painting there would be a claim the walk cannot support.
2137
+
2138
+ **What the split cost: nothing, and the census proves it.** On the corpus
2139
+ `escaped` claims **19,960 bytes over 7,408 tokens**, and every one of those
2140
+ bytes came from a fill that already held it — `string` gives up 17,672 and
2141
+ `value` 2,288, which is exactly 19,960. Classified bytes are **identical to the
2142
+ last decimal**, 1,023,405 of 1,426,731 (71.73076073906013%), so in byte terms
2143
+ this is a retag even though it took a new walk to make — again, the identity is
2144
+ between this change's own before and after, and the corpus total has since moved
2145
+ to 1,023,107 / 241,539 tokens under #347/#348 and #355. Only the token count
2146
+ moves, 231,026 → 241,663: the 7,408 new `escaped` tokens plus the 3,229
2147
+ fragments they leave behind when they split a run they sit inside (`string`
2148
+ +2,643, `value` +586). That second number is well under one per escape because
2149
+ escapes cluster — adjacent ones like `\C3\BD`, and ones flush against a
2150
+ delimiter, add no fragment at all.
2151
+
2152
+ **What it bought.** Against the committed device slice `escaped` claims **418
2153
+ bytes on 100.00% agreement, 0 disagreements** — the 53 `string` → `escaped`
2154
+ runs and the 1 `value` → `escaped` that were `unprojected` before, scored for
2155
+ the first time. Total `unprojected` falls **1,051 → 633 bytes** (−39.8%) and the
2156
+ both-decided region grows 26,794 → 27,212. No byte changed which analyzer found
2157
+ it and no disagreement was introduced.
2158
+
2159
+ The projection entry was again **declared before it was measured**: `escaped`
2160
+ predicts `escaped` because the class means one valid string-internal escape and
2161
+ that is the run the device gives the same name. `escaped` was already listed in
2162
+ `DEVICE_CLASS_KIND` as a syntax class; the agreement report's `unprojected`
2163
+ table showed the bytes were being merged away, it did not pick the target.
2164
+
2165
+ **The residue this does not claim, sized.** 145 bytes across 24 runs in 4 slice
2166
+ scripts stay `unclassified` where the device says `escaped`, and every one of
2167
+ them is the same shape: a `\`-newline **line continuation in code**, outside any
2168
+ string (`… != true) \<newline> do={`). The device's run there swallows the
2169
+ next line's indentation too, so the 145 bytes are 48 of `\`+terminator and 97 of
2170
+ trailing whitespace. That is a different claim from a string escape — centrs has
2171
+ no continuation token, and matching the device would mean deciding how much
2172
+ following whitespace a continuation owns — so it is **not** folded in here. It
2173
+ belongs with the continuation question #225 already owns (what a continuation
2174
+ does to a value's interior); this paragraph is its size, not a plan.
2175
+
2176
+ Delimiters are the near miss that shows the rule discriminates rather than
2177
+ rubber-stamps: `"` bytes are 311 `string` → `syntax-meta` runs and 88
2178
+ `value` → `syntax-meta`, the device classes them too, and they stay merged
2179
+ anyway, because a consumer can find them by looking at the run's own first and
2180
+ last byte. The `$` sigil and the `[` / `]` of a substitution stay `string` on
2181
+ the same ground — `structure.subcommands` already publishes the resolution, and
2182
+ its `innerSpan` locates the brackets around it.
1297
2183
 
1298
2184
  ### Designed, not implemented (the CLI surface, #202b)
1299
2185
 
@@ -1314,12 +2200,11 @@ That is the difference — an accepted flag must do something observable.
1314
2200
 
1315
2201
  ## MCP and library surfaces
1316
2202
 
1317
- - **The cross-project handoff is the structure layer, not the token layer.**
1318
- `explain`'s data is three layers: byte ranges and their classes
1319
- (`spans`, and `tokens` when #264 lands) — centrs-owned, and nothing outside
1320
- centrs consumes them; the resolved reading (`structure`, `symbols`,
1321
- `values`) — path, verb, argument names, value spans; and doc/version
1322
- enrichment, which is rosetta's (#175). A consumer that wants enrichment takes
2203
+ - **Consumers use the layer that answers their question.** Byte ranges and
2204
+ their classes (`spans` and the shipped `tokens` facet) serve editor/token
2205
+ consumers; the resolved reading (`structure`, `symbols`, `values`) carries
2206
+ path, verb, arguments, and value facts; doc/version enrichment belongs to
2207
+ rosetta (#175). A consumer that wants enrichment takes
1323
2208
  the **middle** layer: given `{path, verb, args}` plus a version it does its
1324
2209
  own lookups, whereas handing it tokens would make it re-derive from bytes a
1325
2210
  path centrs already resolved. So #223 is a decision about the structure
@@ -1350,8 +2235,102 @@ That is the difference — an accepted flag must do something observable.
1350
2235
  shapes. `lsp-routeros-ts` and
1351
2236
  tikbook are the intended external consumers (hover/diagnostics/completion/
1352
2237
  semantic tokens over these calls); an LSP *protocol* surface on centrs
1353
- stays out of scope (#90) — but the export shape is validated against a real
1354
- LSP consumption spike before it hardens (staging phase 4).
2238
+ stays out of scope (#90). A bounded offline browser/editor consumption check
2239
+ now precedes vocabulary stabilization (#264 B5); full live LSP integration
2240
+ remains later work. The entry is `@tikoci/centrs/explain` and it is browser-verified —
2241
+ see [The offline package entry](#the-offline-package-entry-312).
2242
+
2243
+ ### The offline package entry (#312)
2244
+
2245
+ **`@tikoci/centrs/explain` -> `src/explain.ts`** is the documented public entry for
2246
+ offline analysis. It is a real boundary, not a naming convention: the module
2247
+ graph reachable from it opens no connection, reads no CDB, and touches no
2248
+ filesystem, so it bundles and runs in a browser, a Worker, or an editor host.
2249
+
2250
+ ```sh
2251
+ bun run explain:browser-consumer # boundary + executed consumer report
2252
+ bun run explain:browser-consumer --json # machine-readable
2253
+ ```
2254
+
2255
+ Both halves are gated in the unit suite by
2256
+ `test/unit/explain-browser-entry.test.ts`, so neither needs a router or the
2257
+ corpus.
2258
+
2259
+ **Why a gate and not a note.** The entry reached the transport stack until this
2260
+ landed, for two *pure* helpers: `toYaml` (from `retrieve.ts`) and
2261
+ `canonicalizeExecuteCommand`/`isWriteShaped` (from `execute.ts`). Importing
2262
+ either dragged in `mac-telnet` (`node:dgram`, `node:os`), `native-api`
2263
+ (`node:crypto`) and `ssh`. Both now live in `src/core/` — `src/core/yaml.ts` and
2264
+ `src/core/execute-command.ts` — moved character for character, with `execute.ts`
2265
+ re-exporting the gate because it remains that gate's documented public home.
2266
+ **Nothing about the script-vs-structured gate moved with them**; the locked
2267
+ contract table in `test/unit/execute-canonicalize-contract.test.ts` still imports
2268
+ from `execute` and still passes.
2269
+
2270
+ That coupling was re-introduced once already by a one-line import, which is why
2271
+ the static half of the check names the offending **edge** (`explain.ts ->
2272
+ execute.ts -> mac-telnet-console.ts -> mac-telnet.ts`) rather than leaving a
2273
+ reader with the bundler's polyfill complaint four modules downstream.
2274
+
2275
+ **What "runs in a browser" is proven by.** Compilation is not the proof. A
2276
+ bundler silently substitutes browser polyfills, so the check bundles a worker
2277
+ entry around the module with `--target browser`, wraps the output in a function
2278
+ that shadows every host global (`Bun`, `process`, `require`, `module`,
2279
+ `exports`, `__dirname`, `__filename`), and **spawns that bundle as the Worker**.
2280
+ `Bun` is a non-configurable global — `delete globalThis.Bun` throws — so
2281
+ shadowing, not deletion, is what makes the absence real, and doing it in the
2282
+ bundler's banner keeps the harness free of `eval`/`new Function` while making
2283
+ the bundle text on disk the artifact that carries the claim.
2284
+
2285
+ It then compares the whole `ExplainData` against the Bun-native run, field for
2286
+ field: structure, diagnostics, tokens, spans, `canonical`, and
2287
+ `runtimeAcceptance`. Fourteen cases, covering normal and write-shaped commands,
2288
+ multi-statement and nested input, three malformed classes, Unicode, astral, a
2289
+ leading BOM, IPv6, empty input, and the facets switched off. All fourteen are
2290
+ byte-identical today.
2291
+
2292
+ Two things the bundle deliberately exercises rather than bypasses. It imports
2293
+ through the **published subpath** `@tikoci/centrs/explain`, not
2294
+ `../src/explain.ts`, so a typo in `package.json`'s `exports` map fails the gate
2295
+ instead of leaving it green while every real consumer breaks (the graph walk
2296
+ additionally asserts that the `exports` entry and the measured file are the same
2297
+ path). And it calls `resolveExplainFormat(undefined)`, whose `env` default is
2298
+ the `typeof`-guarded `hostEnv()` — nothing else reaches that parameter, so
2299
+ without this a regression to a bare `Bun.env` would pass every case here while a
2300
+ browser caller of that exported function got a `TypeError`.
2301
+
2302
+ The shadowing is **measured from inside the bundle**, not assumed: the worker
2303
+ reports which host globals it could still see, and an empty list is part of the
2304
+ gate. Without it, a wrapper that silently stopped being emitted would leave
2305
+ every "identical" above produced with host APIs in reach, proving nothing. Only
2306
+ `Bun` and `process` are probed, because they are the only two of the seven a
2307
+ bundle can honestly observe — Bun constant-folds `typeof require` to
2308
+ `"function"` at build time, mentioning `__dirname`/`__filename` makes it inject
2309
+ the build machine's absolute paths as local constants (the probe would cause
2310
+ what it reports), and mentioning `module`/`exports` reclassifies the entry as
2311
+ CommonJS, whose top-level `export default` is a syntax error inside the wrapper.
2312
+ All seven stay shadowed regardless.
2313
+
2314
+ The one tolerated host builtin is **`node:net`**, for `isIP` in
2315
+ `src/explain/values.ts` — a pure predicate with no socket in it. A bundler
2316
+ replaces it with a regex pair, so the browser build genuinely runs a *different*
2317
+ `isIP` than Bun does; the two IPv6 cases exist to price that substitution, which
2318
+ is the only place its verdict is observable. A builtin that polyfills to a stub
2319
+ (`node:fs`) or throws (`node:dgram`) could not be checked this way, so the
2320
+ allowlist is a boundary, not a list to extend.
2321
+
2322
+ **What it does not claim.** Not that the `centrs` package works in a browser —
2323
+ it does not, and is not meant to. Not DOM integration, no real browser engine.
2324
+ And these cases are a boundary corpus, not a parser-coverage corpus; measured
2325
+ parser evidence stays in the censuses and
2326
+ [Device agreement](#device-agreement-for-the-token-partition-264-b4-263).
2327
+
2328
+ **What this hands #264 B5.** The comparison serializes the entire public result,
2329
+ so it pins the vocabulary a consumer actually depends on: every `tokens[]` class
2330
+ name, every span pair, and the `ExplainData` key set are now load-bearing across
2331
+ a process boundary. A B5 rename is therefore a visible break here rather than a
2332
+ silent one — which is the point of settling the vocabulary before an external
2333
+ consumer commits to it.
1355
2334
 
1356
2335
  ## Non-goals
1357
2336
 
@@ -1418,7 +2397,8 @@ of 913). The ratified answers hold for what was measured; they are **not**
1418
2397
  evidence of coverage over device configuration, which is the genre most likely
1419
2398
  to be handed to `explain` and the one that concentrates Q4 path context and
1420
2399
  Q6's bare-path ambiguity. Quote phase-0 numbers per genre, never as one blended
1421
- figure. #203 tracks the capture and the re-score.
2400
+ figure. #203 records the capture decision and caveat; #336 owns the current
2401
+ corpus repin and remeasurement.
1422
2402
 
1423
2403
  Those figures are re-derivable, not eyeballed: `bun run explain:corpus-census`
1424
2404
  (`scripts/explain-corpus-census.ts`) reproduces the whole genre table from
@@ -1430,8 +2410,33 @@ literals, and DHCP client-ids as `:` directives, which under-counts exactly the
1430
2410
  config genre in question). Since phase 0 the corpus has grown to 948 with a
1431
2411
  `tangentsoft` stratum of 35 genuine `/export … terse` device captures, taking
1432
2412
  export-banner share to 4.6%; **that stratum is not part of any phase-0 figure
1433
- above**, and no promoted module has been scored against it yet (#203
1434
- deliverable 2).
2413
+ above**. Current whole-corpus offline censuses include its source text.
2414
+
2415
+ The pinned snapshot carries **three complete builds** — long-term `7.23.5`,
2416
+ stable `7.24.2` and development `7.25beta3` — each with *both* oracles over all
2417
+ 948 scripts, alongside six partial captures (parseIL at 7.20.8, 7.22.1,
2418
+ 7.23rc1; highlight at 7.9.2, 7.23.2, 7.24rc2) that cover only 913. Schema v4
2419
+ names that split rather than leaving it to be discovered by hand: the
2420
+ `v_version_coverage` view classifies each build, and
2421
+ `coverage_class = 'complete'` is the filter every version-comparable query uses.
2422
+ `source_scripts` is byte-identical across both repins that reached this
2423
+ snapshot, so every text-derived report — the token and value censuses,
2424
+ `explain:arg-reach`, the browser-consumer proof — is unchanged by construction,
2425
+ and only the IL-derived operator census moved.
2426
+
2427
+ **The three complete builds do not agree with each other, and that is the
2428
+ finding.** Scored over all 948 scripts:
2429
+
2430
+ | oracle | long-term 7.23.5 ↔ stable 7.24.2 | stable 7.24.2 ↔ development 7.25beta3 |
2431
+ | ------ | -------------------------------: | ------------------------------------: |
2432
+ | `:parse` IL (`il_sha256`) | 87 scripts differ | 11 |
2433
+ | `highlight` (count + class set) | 33 | 0 |
2434
+
2435
+ Stable and development are the same build for these purposes; long-term is the
2436
+ one that moves. So an operator-axis or token-partition figure is a claim *about
2437
+ a channel*, not about RouterOS — which is why the census reports its builds as
2438
+ separate columns and states its version set, rather than collapsing them to one
2439
+ number. Long-term is also the channel a real device most likely runs.
1435
2440
 
1436
2441
  The corpus itself is not in this repo and is not moving here — `lsp-routeros-ts`
1437
2442
  owns producing snapshots, centrs owns which snapshot it measures against.
@@ -1553,75 +2558,106 @@ validity check: `.scratch/q203-blocker-validity.ts`.
1553
2558
  **Non-blocking / deferred (not ratification-gating):** Q5 expression depth, Q7
1554
2559
  tokenizer corners, Q9 potential-command taxonomy, Q12 span-vocabulary draft —
1555
2560
  these refine the surface during implementation but did not gate ratification.
1556
- They keep their homes in staging: **Q7** is now concrete as #201 (lexical
1557
- boundaries — sigil spellings, escape validity, statement-start eligibility)
1558
- and #199 (the shared scanner's `$[…]`-in-string blind spot), both raised by the
1559
- Q13 promotion and both spanning already-promoted modules; **Q5** and **Q9** are
1560
- phase-1 measurements; **Q12** is spec open item 2 and hardens in phase 4. The
2561
+ Two of the four have since closed. **Q7** became #201 (lexical boundaries —
2562
+ sigil spellings, escape validity, statement-start eligibility) and #199 (the
2563
+ shared scanner's `$[…]`-in-string blind spot), both raised by the Q13 promotion
2564
+ and both now closed. **#199**'s fix is the one shared scanner,
2565
+ `src/explain/quoted-string.ts`, which is what keeps the segmenter, the brace
2566
+ classifier and the block scan from drifting on where a string ends; **#201** is
2567
+ the wider one, spanning statement context, sigil/escape reading and defect
2568
+ detection across several already-promoted modules.
2569
+ **Q12** closed with #264: the vocabulary is settled on a stated rule and is no
2570
+ longer provisional — see
2571
+ [The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5). **Q5** and
2572
+ **Q9** remain phase-1 measurements. The
1561
2573
  probe framework's disposition (reuse of the mutation suite, coordinate
1562
2574
  fixtures, and adversarial generators as product test fixtures) is tracked in
1563
2575
  issue #186 rather than carried into mainline.
1564
2576
 
1565
2577
  ## Definition of done and staging
1566
2578
 
1567
- `designed` on the strength of this README. When implementation starts, offline
1568
- examples gate via unit/fixture tests and each live cell advances to
1569
- `CHR-passed` only when its `examples.md` entries run green via
1570
- `bun run test:integration` (constitution: done definition). Suggested staging
1571
- (sequence, not schedule):
1572
-
1573
- - **Phase 0 — canonicalization grounding lab — COMPLETE (#185).** Experiments (corpus +
1574
- CHR cross-checks against `:parse`/highlight) established what offline parsing
1575
- can actually achieve: statement segmentation, block/scope resolution,
1576
- sub-command path re-constitution, stateful-per-document path context, symbol
1577
- scope, transport classification, write-shape, malformed-input recovery,
1578
- coordinates, and stress invariants. Every ratification-gating question was
1579
- answered with cited evidence; the findings are folded into the surface above
1580
- and summarized in [Phase-0 ratification](#phase-0-ratification-185). This
1581
- spec is ratified on that basis.
1582
- - **Phase 0.5 — product contract fixtures — SUBSTANTIALLY COMPLETE (#186).**
1583
- Promote a minimal, reviewed subset of the lab into product-owned tests; do
1584
- not import or execute `.scratch/` code. One vertical-slice PR per lab suite,
1585
- each landing an exported module under `src/explain/` plus frozen fixtures
1586
- under `test/fixtures/explain/`. Shipped: Q15 coordinates (#188 →
1587
- `coordinates.ts`), Q1 segmentation (#189 → `segment.ts`), Q2–Q4 path
1588
- resolution (#191 → `pathresolve.ts`), Q17 single-pass container walker
1589
- (#194 → `blocks.ts`, closing #190), Q6 verb/menu boundary (#193 →
1590
- `verbsplit.ts`), Q16 write-shape tristate (#195 → `write.ts`), Q14 cascade /
1591
- context-certainty (#197), Q13 symbol scope (#200 → `symbols.ts`). These pin
1592
- the ratified thresholds: no confident command invented after a defect, zero
1593
- false negatives on statically obvious writes, 100% coordinate invariants,
1594
- deterministic/well-formed spans, bounded depth, no throw, and roughly linear
1595
- scaling. **Remaining:** #192 (Q14 defect-*region* spans and the
1596
- `ambiguous`/`unknown` verdict vocabulary — the verdict half wants the phase-1
1597
- envelope). The lab framework and its two ~7.5 MB per-character highlight
1598
- captures stay out of mainline (#186). What durable files CITE does not stay
1599
- out: the frozen dev/holdout split, the Q8 REST probe capture, a stratified
1600
- slice of the highlight streams, and the three device re-derivation probes
1601
- are committed under `test/fixtures/explain/` and `scripts/probes/`.
1602
- - **Phase 1 — offline core** — the grown canonicalizer: structure + gate
1603
- verdict + per-statement resolution and transport classification +
1604
- diagnostics (+ `--curl` rendering), and the CLI/MCP surface over it. Phase
1605
- 0.5 shipped the analysis modules as library exports only; phase 1 composes
1606
- them behind one entry point, adds the envelope and `src/cli/explain.ts`, and
1607
- turns `examples.md` green. **Landing in four PRs:** #192a defect regions
1608
- (#222), #202a composition + envelope (#224), **#202b the CLI surface + the
1609
- offline examples**, then #202c-1 argument reading and #202c-2 transport
1610
- classification + `--curl`, which close #202. Carried in with it: the lexical-boundary
1611
- hardening (#201) and the shared-scanner substitution blind spot (#199), both
1612
- found during Q13 and both spanning modules already promoted; plus the
1613
- deferred lab questions Q5 (expression depth) and Q9 (potential-command
1614
- taxonomy), which become focused tests as this surface hardens.
1615
- - **Phase 2 — live probes** — highlight + `:parse` (+
1616
- completion/child/syntax facets) over rest-api/native-api with the safety rules
1617
- above, including the
1618
- broad-query describe ladder, smart sizing, and CHR-backed anchors for Q6,
1619
- Q8, Q10, and Q11.
1620
- - **Phase 3 — facet polish** — `--complete`/`--schema` ergonomics, truncation
1621
- counts/hints, `--full`.
1622
- - **Phase 4 — library/LSP alignment** — export shape hardened against a real
1623
- lsp-routeros-ts consumption spike (semantic tokens via the centrs span
1624
- vocabulary + color map).
2579
+ Verification follows the [constitution](../../docs/CONSTITUTION.md#done-definition).
2580
+ The phase-0 grounding and promoted phase-1 CLI/library core (#185/#202) are
2581
+ already in the product. The remaining task graph is maintained in #90; issue
2582
+ closure for an earlier slice does not establish the baseline below.
2583
+
2584
+ ### Offline baseline acceptance
2585
+
2586
+ The offline capability can advance from `coded` to `verified` when:
2587
+
2588
+ - **Examples and diagnostics:** the offline examples and focused analysis/CLI
2589
+ contract tests pass. Known false confident readings in the baseline task
2590
+ list, starting with #311's missing separator, are fixed with source-located
2591
+ diagnostics and accepted/malformed controls. Unknown or ambiguous input
2592
+ remains explicit; no fabricated statements, paths, values, or runtime acceptance.
2593
+ - **Bounded execution:** the existing public `explainCommand` performance,
2594
+ coordinate, depth, no-throw, and determinism contracts pass. A reproduced
2595
+ performance failure is investigated before changing its threshold (#313). Cost
2596
+ must grow with input size and no faster: a wall-clock budget on a thermally
2597
+ throttled machine measures the machine, so the growth **shape** is pinned by a
2598
+ ratio between two sizes timed back to back, and an absolute threshold is never
2599
+ raised in place of finding the growing term. One input shape is not enough —
2600
+ #313's separator-free input carries no symbols, so it never reaches the
2601
+ symbol/value machinery where the next growth term was hiding (#317), and
2602
+ neither of those carries a `do={…}`, which is where the third one was (#320).
2603
+ A guarded shape is only guarded for the analyzers it actually runs, so a new
2604
+ growth guard states which analyzers its input reaches and is mutation-tested
2605
+ against the term it removed. **Met:** #322's fixed-size depth guard grows
2606
+ nesting 16× (depth 4 → 64) under the public token/curl path and stays below
2607
+ the 6× bound; deterministic companion guards cover colon-led, bare-directive,
2608
+ retry, array/substitution, bracket, and string-interpolation shapes and fail
2609
+ when child-scope traversal is restored.
2610
+ - **Measured token surface:** the total, gapless `--tokens` partition and
2611
+ generated censuses remain reproducible; #264 B4 / #263's projection and
2612
+ device-agreement report are in
2613
+ [Device agreement](#device-agreement-for-the-token-partition-264-b4-263),
2614
+ re-derivable without a router or the corpus. Report decided errors,
2615
+ abstentions, version/state differences, and corpus bias separately — the
2616
+ report's outcome buckets and its four applicability categories are that
2617
+ separation. Classified-byte percentage is not syntax coverage, and agreement
2618
+ is a trend, not a new numerical pass gate. A token-level rule and a token fill
2619
+ read the skip-tolerant argument stream, never the strict REST reading (#316);
2620
+ the two share one boundary walk, the strict reading's reach is unchanged, and
2621
+ the skip-vs-prefix fork is priced rather than assumed —
2622
+ [The token stream a rule reads](#the-token-stream-a-rule-reads-316). That
2623
+ stream reaches the envelope as the token partition and nowhere else, which is
2624
+ what makes the `=` boundary visible to a consumer whose statement published
2625
+ no arguments at all (#264 B5).
2626
+ - **Usable library contract:** one bounded browser/editor consumer imports a
2627
+ documented public offline entry point, runs analysis without transport/CDB
2628
+ dependencies, and verifies structure, diagnostics, tokens, and source-position
2629
+ mapping against the CLI/library result (#312). **Met** by `@tikoci/centrs/explain` and
2630
+ `bun run explain:browser-consumer` — see
2631
+ [The offline package entry](#the-offline-package-entry-312); the boundary is a
2632
+ module-graph gate and the consumer proof executes the browser bundle rather
2633
+ than only compiling it. The token vocabulary that consumer reads is settled by
2634
+ #264 — [The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5) —
2635
+ on a stated rule, with the two splits it makes (`arg-sep`, then `escaped`)
2636
+ and the merges it keeps each recorded there with the surface that already
2637
+ carries the distinction. A complete editor, LSP server, palette
2638
+ reproduction, or SCIP implementation is not required to prove this contract.
2639
+ - **Reachable evidence and explicit residue:** changed RouterOS semantic claims
2640
+ have committed CHR-grounded controls and a reproducible capture/replay path.
2641
+ Every remaining issue has a stated scope and dependency; #225 distinguishes
2642
+ offline value work from live types, #272 owns evidence reachability, and #211's
2643
+ unlisted-path decision retains the current abstention until separately decided.
2644
+
2645
+ Retain the existing checks: `bun run lint`, `bun run test`, `bun run build`, and
2646
+ `bun run lint:ci`, plus the affected CHR grounding/integration procedure from
2647
+ the constitution. Add the consumer and agreement checks with their implementation.
2648
+ No new rules engine, schema snapshot, final-state interpreter, or network-behavior
2649
+ test framework is part of this baseline.
2650
+
2651
+ ### Current and following capabilities
2652
+
2653
+ - **Current — live probes (#236, phase 2):** highlight + `:parse` and
2654
+ completion/child/syntax over rest-api/native-api. B4's projection/agreement
2655
+ informs this work; each live cell requires its own CHR-passing examples.
2656
+ - **MCP alignment (#223):** the deliberate flat-to-rich response migration,
2657
+ coordinated with rosetta. Independent of the offline token vocabulary.
2658
+ - **Facet polish (phase 3):** live sizing, `--full`, completion/schema ergonomics.
2659
+ - **Full LSP integration (phase 4):** consume live and offline results in
2660
+ `lsp-routeros-ts`, building on the earlier offline consumer check.
1625
2661
 
1626
2662
  ## Decisions (2026-07-19 round) and remaining opens
1627
2663
 
@@ -1689,13 +2725,17 @@ Still open (implementation opens — none block ratification):
1689
2725
 
1690
2726
  1. Final flag names (`--full`, `--curl`, facet names) and the smart-sizing
1691
2727
  thresholds (what counts as "cheap", what triggers truncation).
1692
- 2. The centrs span vocabulary itself: class list, mapping table from RouterOS
1693
- highlight classes (and their colors), and how unknown/new upstream classes
1694
- degrade. Phase 0 pinned the raw inputs (Q13's per-occurrence highlight-class
1695
- corpus; the observed 19 classes on 7.23.2 plus drift on 7.24rc2) but the
1696
- centrs-owned class list and color map are a **draft** (lab question Q12,
1697
- non-ratification-gating); they harden in staging phase 4 against the real
1698
- lsp-routeros-ts consumption spike.
2728
+ 2. The mapping table from RouterOS highlight classes to **colors**, and how
2729
+ unknown/new upstream classes degrade. Phase 0 pinned the raw inputs (Q13's
2730
+ per-occurrence highlight-class corpus; the observed 19 classes on 7.23.2 plus
2731
+ drift on 7.24rc2). The centrs-owned **class list is no longer open**: #264 B5
2732
+ settled it on a stated rule and `data.tokens[].class` is no longer provisional
2733
+ ([The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5)), and the
2734
+ projection onto the device's own classes is reported by
2735
+ `bun run explain:highlight-agreement`
2736
+ ([Device agreement](#device-agreement-for-the-token-partition-264-b4-263)).
2737
+ What stays open here is the color map and the drift policy. Live mappings
2738
+ and full LSP integration follow separately.
1699
2739
  3. Whether the live describe ladder needs result caching per target+version
1700
2740
  (probe cost vs freshness) — deferred to implementation evidence.
1701
2741