@tikoci/centrs 0.1.3 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +348 -0
- package/GLOSSARY.txt +252 -0
- package/LICENSE +21 -0
- package/README.md +44 -12
- package/commands/AGENTS.md +14 -5
- package/commands/api/AGENTS.md +13 -1
- package/commands/api/README.md +56 -26
- package/commands/api/examples.md +46 -1
- package/commands/btest/README.md +21 -32
- package/commands/check/README.md +6 -1
- package/commands/discover/README.md +5 -11
- package/commands/execute/README.md +77 -36
- package/commands/execute/examples.md +120 -10
- package/commands/explain/README.md +2756 -0
- package/commands/explain/examples.md +702 -0
- package/commands/mcp/README.md +7 -4
- package/commands/mcp/examples.md +2 -2
- package/commands/retrieve/README.md +25 -27
- package/commands/retrieve/examples.md +16 -0
- package/commands/terminal/examples.md +7 -1
- package/commands/transfer/README.md +28 -20
- package/commands/transfer/examples.md +29 -0
- package/docs/CLI.md +353 -0
- package/docs/CONSTITUTION.md +138 -30
- package/docs/MATRIX.md +92 -18
- package/docs/errors/README.md +3 -0
- package/docs/errors/quickchr/machine-not-found.md +22 -0
- package/docs/errors/quickchr/machine-stopped.md +20 -0
- package/docs/errors/quickchr/package-unavailable.md +27 -0
- package/docs/errors/quickchr/unsupported-via.md +35 -0
- package/docs/errors/quickchr/unsupported.md +35 -0
- package/docs/errors/routeros/version-unsupported.md +26 -0
- package/docs/errors/transport/provider-trust.md +29 -0
- package/docs/errors/usage/stdin-ignored.md +28 -0
- package/docs/errors/validation/syntax.md +29 -4
- package/docs/errors/validation/unknown-attribute.md +37 -4
- package/package.json +56 -10
- package/src/api-fanout.ts +46 -18
- package/src/api.ts +351 -121
- package/src/cli/api.ts +26 -7
- package/src/cli/btest.ts +19 -3
- package/src/cli/execute.ts +25 -6
- package/src/cli/explain.ts +503 -0
- package/src/cli/retrieve.ts +33 -1
- package/src/cli/selection.ts +71 -11
- package/src/cli/settings.ts +2 -2
- package/src/cli/terminal.ts +1 -1
- package/src/cli/transfer.ts +23 -6
- package/src/cli.ts +15 -2
- package/src/core/envelope.ts +58 -2
- package/src/core/error-catalog.ts +42 -1
- package/src/core/execute-command.ts +148 -0
- package/src/core/inspect.ts +36 -4
- package/src/core/preflight.ts +306 -0
- package/src/core/routeros-string.ts +24 -0
- package/src/core/version.ts +4 -0
- package/src/core/yaml.ts +86 -0
- package/src/errors.ts +1 -0
- package/src/execute-fanout.ts +47 -18
- package/src/execute.ts +339 -307
- package/src/explain/arg-tokens.ts +151 -0
- package/src/explain/args.ts +1749 -0
- package/src/explain/blocks.ts +175 -0
- package/src/explain/brace-slots.ts +143 -0
- package/src/explain/brace-tokens.ts +39 -0
- package/src/explain/catalog.ts +1343 -0
- package/src/explain/coordinates.ts +488 -0
- package/src/explain/defects.ts +218 -0
- package/src/explain/escape-tokens.ts +77 -0
- package/src/explain/highlight-projection.ts +308 -0
- package/src/explain/is-known-menu.ts +17 -0
- package/src/explain/menus.ts +668 -0
- package/src/explain/operator-tokens.ts +316 -0
- package/src/explain/operators.ts +465 -0
- package/src/explain/path-tokens.ts +150 -0
- package/src/explain/pathresolve.ts +1569 -0
- package/src/explain/quoted-string.ts +331 -0
- package/src/explain/scope-brace.ts +461 -0
- package/src/explain/segment.ts +1165 -0
- package/src/explain/separator.ts +187 -0
- package/src/explain/split-owner.ts +194 -0
- package/src/explain/string-tokens.ts +77 -0
- package/src/explain/symbol-values.ts +556 -0
- package/src/explain/symbols.ts +1617 -0
- package/src/explain/token-ranges.ts +72 -0
- package/src/explain/transport.ts +514 -0
- package/src/explain/value-tokens.ts +83 -0
- package/src/explain/values.ts +357 -0
- package/src/explain/verbs.ts +64 -0
- package/src/explain/verbsplit.ts +955 -0
- package/src/explain/write.ts +856 -0
- package/src/explain.ts +2602 -0
- package/src/index.ts +225 -6
- package/src/mcp/server.ts +2 -1
- package/src/offline-gate.ts +148 -0
- package/src/protocols/mac-telnet-console.ts +61 -17
- package/src/resolver/facts.ts +1 -1
- package/src/resolver/index.ts +23 -4
- package/src/resolver/quickchr-provider.ts +597 -0
- package/src/resolver/selection.ts +99 -12
- package/src/resolver/service-endpoint.ts +155 -0
- package/src/resolver/target.ts +1 -1
- package/src/retrieve-fanout.ts +27 -13
- package/src/retrieve.ts +82 -77
- package/src/settings.ts +2 -1
- package/src/transfer-fanout.ts +43 -26
- package/src/transfer.ts +88 -45
|
@@ -0,0 +1,2756 @@
|
|
|
1
|
+
# explain
|
|
2
|
+
|
|
3
|
+
Analyze a RouterOS command **before** anything runs it: canonical form and
|
|
4
|
+
structure, LSP-like token spans and diagnostics, how the command would actually
|
|
5
|
+
run (`api`-able vs `execute`-required, including a rendered REST/`curl` form),
|
|
6
|
+
and — against a live device — what commands, arguments, and values are valid at
|
|
7
|
+
a path, with completion-style candidates for building the right command.
|
|
8
|
+
`explain` is the "E" of the explain → validate → run split (constitution:
|
|
9
|
+
validation): the cheap, safe knowledge tier in front of the runners
|
|
10
|
+
(`execute` / `api` / `retrieve`).
|
|
11
|
+
|
|
12
|
+
Status: offline baseline `verified`; live probes over `rest-api` and `native-api`
|
|
13
|
+
`designed`. **`centrs explain '<input>'` runs today**, without a router or CDB.
|
|
14
|
+
[`docs/MATRIX.md`](../../docs/MATRIX.md#offline-analysis) tracks these separately;
|
|
15
|
+
[#90](https://github.com/tikoci/centrs/issues/90) indexes the remaining tasks.
|
|
16
|
+
The spec is ratified on the [phase-0 grounding lab](#phase-0-ratification-185).
|
|
17
|
+
The current finish line is [Offline baseline acceptance](#offline-baseline-acceptance),
|
|
18
|
+
not completion of the live protocol cells.
|
|
19
|
+
Load-bearing rules — envelope, errors, settings precedence, identity,
|
|
20
|
+
validation, protocol selection — live in
|
|
21
|
+
[`docs/CONSTITUTION.md`](../../docs/CONSTITUTION.md).
|
|
22
|
+
|
|
23
|
+
## Why (grounding)
|
|
24
|
+
|
|
25
|
+
- `~/GitHub/bench-routeros-tools` (`REPORT.md`) grounds the split empirically:
|
|
26
|
+
agents need a cheap, safe knowledge/validation tier gating a thin runner, and
|
|
27
|
+
a handful of verbs over canonical `{path, verb, args}` beats per-command
|
|
28
|
+
tools. It also measured the **inspect-vs-runtime gap** (`/console/inspect`
|
|
29
|
+
accepted `blackhole=yes` where the runtime wanted the bare flag): schema
|
|
30
|
+
acceptance is necessary, never sufficient.
|
|
31
|
+
- The wire-format research is done and public:
|
|
32
|
+
[`routeros-syntax-inspection`](https://github.com/tikoci/routeros-skills/tree/main/routeros-syntax-inspection)
|
|
33
|
+
(probe selection, highlight/parseIL/completion/child/syntax semantics,
|
|
34
|
+
hazards, provenance rules) and
|
|
35
|
+
[`routeros-command-tree`](https://github.com/tikoci/routeros-skills/tree/main/routeros-command-tree)
|
|
36
|
+
(tree traversal, schema generation), both grounded by full-corpus captures in
|
|
37
|
+
[tikoci/lsp-routeros-ts `docs/`](https://github.com/tikoci/lsp-routeros-ts/tree/main/docs)
|
|
38
|
+
(913 scripts × 7.9.2/7.23.2/7.24rc2). This spec cites those references
|
|
39
|
+
rather than restating them; they are the canonical wire truth.
|
|
40
|
+
- Decisions locked in #90: `validate` stays **boolean** (the dry-run gate
|
|
41
|
+
inside `execute`/`api` and `centrs_validate`); `explain` is the surface that
|
|
42
|
+
carries richness; centrs is the **canonicalizer/explain owner** and
|
|
43
|
+
`lsp-routeros-ts` (and tikbook) eventually consume the centrs library
|
|
44
|
+
instead of owning their own analysis core.
|
|
45
|
+
- Boundary from rosetta B-0013 (#175), sharpened by the 2026-07-19 round:
|
|
46
|
+
rosetta owns *static, docs-grounded* explanation — prose, URLs, version
|
|
47
|
+
history, "what version supports what" — and never grows validate/run; centrs
|
|
48
|
+
owns canonicalization and anything device-aware. **centrs does not call
|
|
49
|
+
rosetta, and does not consume restraml/rosetta schema data, for now**: the
|
|
50
|
+
projects stay decoupled until centrs's final shape is known. At most,
|
|
51
|
+
`explain` output may *steer* toward rosetta in a tip (e.g. "for docs/version
|
|
52
|
+
history, ask rosetta"). Deeper integration (for example rosetta publishing
|
|
53
|
+
stable GitHub-Pages links centrs help could cite) is future rosetta-side
|
|
54
|
+
work, tracked there, not here.
|
|
55
|
+
|
|
56
|
+
## Where `explain` sits
|
|
57
|
+
|
|
58
|
+
- **Shared analysis across transports.** The offline result is independently
|
|
59
|
+
useful to libraries, agents, and editors. Its facts can later accompany
|
|
60
|
+
other commands' envelopes, irrespective of their transport; the rich
|
|
61
|
+
analysis is not currently applied as a runner policy gate.
|
|
62
|
+
- **Never executes, never mutates.** All live probes (`/console/inspect`
|
|
63
|
+
highlight/completion/syntax/child and `:put [:parse …]`) are read-only
|
|
64
|
+
inspection. Execution-based probes (required-argument discovery via
|
|
65
|
+
deliberate errors) are out of scope — they belong to research tooling, not
|
|
66
|
+
this verb.
|
|
67
|
+
- **`validate` is unchanged.** `--validate` remains the boolean gate on the
|
|
68
|
+
runners; `explain` is what you call when you want to know *why*, or to build
|
|
69
|
+
the command in the first place. Internally they share machinery; externally
|
|
70
|
+
they stay distinct surfaces (constitution: validation).
|
|
71
|
+
- **The structured-mode gate stays authoritative.** `explain` exposes the
|
|
72
|
+
canonicalizer's verdict (`mode`, `writeShaped`) but must not widen what
|
|
73
|
+
counts as `structured` (constitution: canonicalizer ownership).
|
|
74
|
+
- **Passing `explain` is not a promise the router will accept the command.**
|
|
75
|
+
Every result carries this asymmetry explicitly (see
|
|
76
|
+
[Result shape](#result-shape-sketch)); only `execute`/`api` against a real
|
|
77
|
+
target proves runtime acceptance.
|
|
78
|
+
- **Library-first.** The deliverable is exported functions on `@tikoci/centrs`
|
|
79
|
+
that need no CLI and (offline) no CDB; the CLI verb and the MCP tool are
|
|
80
|
+
thin adapters. centrs ships no LSP server, but the library API must be able
|
|
81
|
+
to **support one**: hover, diagnostics, completion, and semantic-token needs
|
|
82
|
+
of `lsp-routeros-ts`/tikbook are first-class consumers of the export shape.
|
|
83
|
+
|
|
84
|
+
Normalized source structure is the foundation for possible configuration
|
|
85
|
+
checks. Preserve command order, selectors, source locations, literal/computed
|
|
86
|
+
value distinctions, and unknowns; an input fragment does not establish final
|
|
87
|
+
device state. Script-variable bindings are not yet an object-reference model
|
|
88
|
+
for interfaces, bridges, or other configuration entities. Security advice,
|
|
89
|
+
rule collections, and a browser/Monaco inspector are possible consumers, not
|
|
90
|
+
new requirements to implement those products during the offline baseline.
|
|
91
|
+
quickchr owns the CHR provisioning/test machinery; rosetta supplies docs and
|
|
92
|
+
reference knowledge. This direction adds no runtime dependency on either.
|
|
93
|
+
|
|
94
|
+
## Canonicalization is the core
|
|
95
|
+
|
|
96
|
+
The engine is a **grown `canonicalizeExecuteCommand`** (decision: grow the
|
|
97
|
+
in-repo canonicalizer; rosetta's pure `canonicalize.ts` is reference material
|
|
98
|
+
to borrow from as needed, with **no sync obligation** — centrs solves a
|
|
99
|
+
different problem and is expected to diverge). Target capabilities, beyond
|
|
100
|
+
today's path/verb/args split and script-vs-structured gate:
|
|
101
|
+
|
|
102
|
+
- **Structure, not just segmentation.** Parse RouterOS input into an AST-like
|
|
103
|
+
structure so complex expressions, blocks, and scopes can be identified —
|
|
104
|
+
including multi-statement `.rsc` scripts.
|
|
105
|
+
- **Sub-command resolution with re-constituted paths.** In
|
|
106
|
+
`/ip/address remove [find comment=defconf]`, the inner `find` carries the
|
|
107
|
+
enclosing path (`ip,address`); the canonical structure exposes that inner
|
|
108
|
+
command with its resolved path rather than treating `[…]` as an opaque blob.
|
|
109
|
+
- **Symbol scopes follow RouterOS scope identity, not brace depth alone.** Names
|
|
110
|
+
are case-sensitive. Control-flow `do={…}` bodies share the enclosing scope,
|
|
111
|
+
but a named-function definition (`:local F do={…}` / `:global F do={…}`)
|
|
112
|
+
starts a closure: outer locals become parameters there, and a global must be
|
|
113
|
+
re-imported inside the body. If the parser cannot establish which body owns a
|
|
114
|
+
binding, it abstains instead of assigning a confident symbol class (Q13).
|
|
115
|
+
- **Transport classification — fail-closed.** For each statement, say how it
|
|
116
|
+
would actually run: `api-candidate` (representable as a structured `api`
|
|
117
|
+
operation, with the REST method/path/body), `execute` (script mode,
|
|
118
|
+
expressions, sub-commands), or `unknown` — each with a stated basis. The
|
|
119
|
+
canonicalizer alone cannot *prove* a REST mapping (the syntax-inspection map
|
|
120
|
+
keeps CLI→REST conversion separate from source parsing), so a ready-to-run
|
|
121
|
+
`curl` renders **only** where an explicit, tested mapping rule covers the
|
|
122
|
+
command family; anything else classifies `unknown` rather than promoting a
|
|
123
|
+
heuristic into executable output. The `centrs execute`/`centrs api`
|
|
124
|
+
invocation renders for either classified case. In particular, selector-based
|
|
125
|
+
writes route to `execute`, while selector-less `set` is `unknown` offline:
|
|
126
|
+
without a schema, `/ip/dns set …` (a singleton) and
|
|
127
|
+
`/ip/firewall/filter set …` (an id-bearing table) have the same shape. Live
|
|
128
|
+
evidence may lift the singleton case; offline never guesses it.
|
|
129
|
+
- **Mutation analysis is separate from the gate.** The execute gate's verdict
|
|
130
|
+
(`mode`, `writeShaped`) is reproduced **unchanged** — a bracketed
|
|
131
|
+
sub-command selector stays `mode: "script"` / `writeShaped: false`, exactly
|
|
132
|
+
as `canonicalizeExecuteCommand` and its anchor tests
|
|
133
|
+
(`test/unit/execute-canonicalize-contract.test.ts`) pin it. Richer
|
|
134
|
+
explain-only inference (an inner `remove` detected in structure) lands in a
|
|
135
|
+
**distinct** field, `structure.containsWrite` (basis `heuristic`) — never by
|
|
136
|
+
widening *or* reinterpreting `writeShaped`, so agents cannot mistake an
|
|
137
|
+
explain inference for a guard `execute` actually applies. **`containsWrite`
|
|
138
|
+
is three-valued `true | false | unknown`, not a boolean** (Q16, ratified): a
|
|
139
|
+
curated write-verb table covers ~93% of command nodes and is near-immune to
|
|
140
|
+
version drift (~0.07% of nodes per minor version), but ~6–7% of nodes carry a
|
|
141
|
+
write-shaped verb *outside* any small table (`reboot`, `format`, `delete`,
|
|
142
|
+
`reset-configuration`, …), so a two-valued field would have to call
|
|
143
|
+
`/disk format-drive` a non-write. The field therefore **fails closed to
|
|
144
|
+
`unknown`** for any verb it cannot confirm, with a **hard requirement of zero
|
|
145
|
+
false negatives on statically obvious writes** and `unknown` reported
|
|
146
|
+
separately from `false`. Dynamic execution (`:parse`, stored scripts,
|
|
147
|
+
variable/function invocation) is `unknown` by construction.
|
|
148
|
+
- **Heuristics owned and labeled.** Offline conclusions are canonicalizer
|
|
149
|
+
facts (provenance `canonicalizer`, basis `heuristic`/`derived`) — never
|
|
150
|
+
presented as device or schema truth.
|
|
151
|
+
- **Fail-closed with an `ambiguous` verdict.** Schema-free analysis has a hard
|
|
152
|
+
floor: a statement that is *nothing but a path* (`/ip/route` is a directory,
|
|
153
|
+
`/system/reboot` is a command, and the text is identical) cannot be resolved
|
|
154
|
+
by a text rule alone. The canonicalizer must abstain wherever nothing but the
|
|
155
|
+
text is available to decide — a distinct **`ambiguous`** verdict, separate
|
|
156
|
+
from "unresolved" — rather than guess. Since #210 the baked container floor
|
|
157
|
+
answers this one pair where it can confirm the path, and since #228 the
|
|
158
|
+
published command axis answers it from the other side, so the requirement
|
|
159
|
+
binds on everything *neither* table carries — still most of it; the two
|
|
160
|
+
paragraphs below that scope it are normative, not a footnote. Four
|
|
161
|
+
ratification questions reached this floor independently (Q6 verb/menu
|
|
162
|
+
boundary, Q3 absolute inner paths, Q13 `variable-undefined`, Q14 bare-word
|
|
163
|
+
recovery), so it is a **structural property of schema-free canonicalization**,
|
|
164
|
+
not a per-case wart; the fail-closed rule is promoted to the constitution's
|
|
165
|
+
canonicalizer-ownership section. The measured cost is small: the shipped
|
|
166
|
+
no-schema verb/menu rule scores **99.9%
|
|
167
|
+
precision on decided at ~3% abstention**, and a static snapshot would recover
|
|
168
|
+
only ~2.3pp of that abstention while churning ~9% of tree nodes per minor
|
|
169
|
+
version — which is why **decision 3 (no offline schema snapshot) ratifies
|
|
170
|
+
unchanged**.
|
|
171
|
+
|
|
172
|
+
The *container* half of that pair is since decidable. `src/explain/menus.ts`
|
|
173
|
+
(#207) bakes only the menu-vs-command node type — not a schema — and it is
|
|
174
|
+
version-less because the pinned trees show no `dir`↔`cmd` flip across
|
|
175
|
+
7.10.2 → 7.25beta3 or across architectures. So `/ip/route` now resolves as
|
|
176
|
+
`navigation` (#210). The floor is unmoved: the table is deliberately
|
|
177
|
+
incomplete, absence from it decides nothing, and every rule above it still
|
|
178
|
+
refuses rather than guesses.
|
|
179
|
+
|
|
180
|
+
The *command* half is what #228 addressed, by adding a second first-order
|
|
181
|
+
source rather than a schema. MikroTik's CLI Reference is generated from the
|
|
182
|
+
definition structs, and measured against the same six pinned trees it has
|
|
183
|
+
**zero navigation-vs-command contradictions** across 959 exactly-matching
|
|
184
|
+
paths, four RouterOS versions and two architectures; all but one of the 94
|
|
185
|
+
published paths absent from every tree carry a `package`/`conditions`/`syscap`
|
|
186
|
+
gate — their own or an ancestor's — that predicts the absence.
|
|
187
|
+
`src/explain/catalog.ts` unions the two with per-entry provenance, which is
|
|
188
|
+
what keeps `menus.ts`'s device-confirmed floor intact rather than diluting it.
|
|
189
|
+
The publication is still not device truth, but as of #285 it is at least
|
|
190
|
+
spelled like the CLI: MikroTik reshaped it from module pages into per-command
|
|
191
|
+
leaf pages whose slug *is* the CLI path, so the definition-module spellings
|
|
192
|
+
(`caps-man/acl/access-list`) are gone and the alias allowlist that recovered
|
|
193
|
+
them is empty — kept, and kept asserted, to refuse the next one loudly. So
|
|
194
|
+
`/system/reboot` now resolves as a `command` at `/system`, and the floor is
|
|
195
|
+
again unmoved — the residual is a path in *neither* table, such as
|
|
196
|
+
`/disk/format-drive`, which 7.23.2 spells `/disk format` and MikroTik does not
|
|
197
|
+
publish at all.
|
|
198
|
+
|
|
199
|
+
Discovery is the union of MikroTik's two published inventories, not the
|
|
200
|
+
sitemap alone. A *branching* menu is served as a trailing-slash category URL
|
|
201
|
+
with no `.md` of its own, while that menu's own entry is published at
|
|
202
|
+
`<dir>/<basename>.md` and listed only in `llms.txt` — so sitemap-only
|
|
203
|
+
discovery dropped 256 of 1,070 pages, silently and field-heavily. Generation
|
|
204
|
+
asserts that every category dir still contributes its leaf, so the next
|
|
205
|
+
inventory reshape fails loudly instead of shrinking the table (#285,
|
|
206
|
+
tikoci/rosetta#137).
|
|
207
|
+
|
|
208
|
+
**Source refresh (#335, accepted 2026-09-13).** The inspect pins retain the
|
|
209
|
+
7.10.2 and 7.16 history anchors, then pair x86 and arm64 for both current
|
|
210
|
+
stable 7.24.2 and prerelease 7.25beta3. Pairing architectures makes additions
|
|
211
|
+
attributable to version rather than to an architecture swap while preserving
|
|
212
|
+
the hardware-only menu floor. Reviewed independently, the inspect union moves
|
|
213
|
+
from 615 to 617 menus: 7.24.2 removes `/ip/ipsec/key/qkd`, and 7.25beta3 adds
|
|
214
|
+
`/interface/ethernet/pon`, `/interface/xfrm`, and `/ip/ssh/known-hosts`.
|
|
215
|
+
Separately, the current CLI Reference moves from 1,070 pages / 1,077 entries
|
|
216
|
+
to 1,053 / 1,060: it adds
|
|
217
|
+
`/interface/ethernet/switch/rule/reset-counters`, removes the five
|
|
218
|
+
`/interface/bridge/msrp` paths plus `/root`, moves 12 paths from `both` to
|
|
219
|
+
`inspect`, and changes 16 published gate strings. Those current-only
|
|
220
|
+
publication changes are accepted: none introduces a kind contradiction. The
|
|
221
|
+
coherent generated catalog has 1,120 paths: 959 `both`, 67 `inspect`, and 94
|
|
222
|
+
`published`; it still reports zero navigation-vs-command contradictions.
|
|
223
|
+
|
|
224
|
+
There are real unknowns in how far offline parsing can go (expression
|
|
225
|
+
grammar, scope fidelity vs `:parse`, `[]`-nesting corner cases). Grounding
|
|
226
|
+
those unknowns was **phase 0** (#185); its findings are folded in throughout
|
|
227
|
+
this spec and summarized in [Phase-0 ratification](#phase-0-ratification-185).
|
|
228
|
+
|
|
229
|
+
## Three questions, one pipeline
|
|
230
|
+
|
|
231
|
+
An agent (the primary caller) asks `explain` three kinds of question before
|
|
232
|
+
touching `execute` / `api` / `retrieve`:
|
|
233
|
+
|
|
234
|
+
1. **Check** — *"is this command right, and how does it run?"*
|
|
235
|
+
`/ip/route add dst-address=10.9.0.0/16 blackhole=yes` → canonical
|
|
236
|
+
`{path, verb, args, mode, writeShaped}`, spans, diagnostics, transport
|
|
237
|
+
classification (+ `curl`/runner rendering on request).
|
|
238
|
+
2. **Assist** — *"what can come next?"*
|
|
239
|
+
`/ip/firewall/filter add ch` + cursor → completion candidates with
|
|
240
|
+
provenance, so the caller composes a correct command instead of guessing
|
|
241
|
+
and retrying.
|
|
242
|
+
3. **Describe** — *"what exists at this path?"* (live)
|
|
243
|
+
A broad query returns broader, **bounded** results that invite drill-down:
|
|
244
|
+
- `/ip/address` → the verbs available there (with arg counts or a hint to
|
|
245
|
+
name a verb for its args);
|
|
246
|
+
- `/ip/address/set` → the settable arguments;
|
|
247
|
+
- `/ip/address/print` → special-cased to the `.proplist` value set, i.e.
|
|
248
|
+
what the output *can* contain. **The probe recipe is grounded** (Q10,
|
|
249
|
+
version-flat 7.23.2/7.24rc2): `completion input="<menu>/print proplist="`
|
|
250
|
+
with rows filtered `show === "true"` yields the property-name set (a
|
|
251
|
+
superset of the keys a `GET` returns). Two device traps are pinned:
|
|
252
|
+
(1) the console word is the dot-free `proplist=` — the REST body key
|
|
253
|
+
`.proplist` returns generic value metadata, not the set; and (2) the rows
|
|
254
|
+
are selected by `show=true`, not by `style` (whose value here is `none`,
|
|
255
|
+
not `arg`). The `child`/`syntax` fallback is needed only for singleton
|
|
256
|
+
settings menus that expose no `.proplist`.
|
|
257
|
+
|
|
258
|
+
Positions matter more than tokens: the same word can be a verb or an argument
|
|
259
|
+
(`/ip/address comment …` is the `comment` *verb*; `/ip/address/add comment=…`
|
|
260
|
+
is the `comment` *argument* of `add`), and a probe at a value position (input
|
|
261
|
+
ending in `=`) asks for value candidates or type info rather than names. The
|
|
262
|
+
canonicalizer resolves the role offline where it can and labels the basis;
|
|
263
|
+
live completion is the authority.
|
|
264
|
+
|
|
265
|
+
Offline, `explain` answers with canonicalizer facts only (structure, gate,
|
|
266
|
+
transport shape). Enumerating what a device accepts — verbs, args, enums,
|
|
267
|
+
`.proplist` sets — is **live** evidence; a no-router run that would benefit says so
|
|
268
|
+
with a tip ("richer data is available if you name a router").
|
|
269
|
+
|
|
270
|
+
## Evidence modes
|
|
271
|
+
|
|
272
|
+
| Mode | Trigger | Sources | Provenance label |
|
|
273
|
+
| ---- | ------- | ------- | ---------------- |
|
|
274
|
+
| **Offline** (default) | no `<router>` given | the centrs canonicalizer only: segmentation, AST-like structure, path/verb/args split, script-vs-structured, write-shape, transport classification, `curl`/runner rendering | `canonicalizer` (basis `heuristic`/`derived`) |
|
|
275
|
+
| **Live** | `<router>` given | `request=highlight` (byte-classified spans, first hard error), `:parse` (structure, error line/column), `request=completion` / `syntax` / `child` (candidates, structured help, children/args/`.proplist` sets) | `live-inspect`, stamped with the device's version |
|
|
276
|
+
|
|
277
|
+
There is **no static schema snapshot** (decision, 2026-07-19, amended 2026-08-07
|
|
278
|
+
by #228): offline mode makes no call at analysis time — not to a router, not to
|
|
279
|
+
rosetta, not to the network — and ships no argument names, types, enums,
|
|
280
|
+
per-menu verb lists or `.proplist`. Version and schema questions are steered to
|
|
281
|
+
a live router (or to rosetta, by tip). Offline diagnostics are structural
|
|
282
|
+
("this does not parse", "this is script-shaped"), never claims about what a
|
|
283
|
+
command accepts.
|
|
284
|
+
|
|
285
|
+
Offline mode does ship two generated **structure** tables, baked at build time
|
|
286
|
+
from pinned sources and read as ordinary closed lists. They may say whether a
|
|
287
|
+
path is navigation or a command; they never describe what a command accepts.
|
|
288
|
+
Absence from either abstains and never rejects. Regenerate with
|
|
289
|
+
`bun run explain:menus` / `bun run explain:catalog`; `--check` drift-gates both
|
|
290
|
+
in QA. Do not hand-edit them — each file's header carries its source pin and
|
|
291
|
+
rationale.
|
|
292
|
+
|
|
293
|
+
- `src/explain/menus.ts` (#207) — container paths from pinned restraml
|
|
294
|
+
`/console/inspect` trees. **Read today** to confirm navigation, on its own by
|
|
295
|
+
`write.ts` and `symbols.ts` and unioned by `is-known-menu.ts` elsewhere.
|
|
296
|
+
- `src/explain/catalog.ts` (#228) — path → kind, per-entry provenance and
|
|
297
|
+
MikroTik's published applicability gate, unioned from those same trees and
|
|
298
|
+
CLI Reference. Its **command axis is read today** by `pathresolve.ts` (R12),
|
|
299
|
+
`verbsplit.ts` and `write.ts`. Since #235 its **menu/settings kind is read
|
|
300
|
+
too**, unioned with `menus.ts` behind one `isKnownMenuPath` helper
|
|
301
|
+
(`src/explain/is-known-menu.ts`): it confirms a relative bare path once the
|
|
302
|
+
inherited context is applied (`filter` or `firewall filter` under `/ip`), and
|
|
303
|
+
it is the same union `verbsplit.ts` consults for an absolute bare path, which
|
|
304
|
+
now reads `/interface/ethernet/poe` as navigation instead of `ambiguous`. An
|
|
305
|
+
unknown joined path abstains and poisons following relative statements. A gate
|
|
306
|
+
never decides anything offline — it explains why a published path may be
|
|
307
|
+
missing from a given router, and no router was consulted to build the table.
|
|
308
|
+
|
|
309
|
+
`write.ts` deliberately stays on `menus.ts` alone, and that asymmetry is the
|
|
310
|
+
safety rule rather than an oversight: a published command misread as a menu
|
|
311
|
+
would drop a WRITE as navigation, so a `published`-only entry is decisive for
|
|
312
|
+
`command` and only a tie-breaker for `menu`. The #235 union is safe where it is
|
|
313
|
+
read because both consumers there answer "where does the context move?", and a
|
|
314
|
+
wrong menu costs a path, not a missed write.
|
|
315
|
+
|
|
316
|
+
A live target arrives through the **same resolver as every other command**
|
|
317
|
+
(CDB name/MAC/group keys, `--quickchr`, future TikTOML — #134/#174): a
|
|
318
|
+
target-aware `explain` gets provider targets for free and adds no bespoke
|
|
319
|
+
resolution path. Fan-out does not apply — `explain` takes at most one router
|
|
320
|
+
and rejects selectors with `usage/fanout-not-supported`, like `terminal`.
|
|
321
|
+
|
|
322
|
+
Implementation-normative safety rules (evidence in the skill/LSP references):
|
|
323
|
+
per-probe timeouts always; skip scripting-keyword paths for `syntax`/
|
|
324
|
+
`completion` on versions **before 7.21.4** — the REST-server deadlock is
|
|
325
|
+
measured through 7.20.8 and 7.21.4 is the first version proven fixed
|
|
326
|
+
(SUP-127641), so the conservative skip covers the untested gap between;
|
|
327
|
+
route input past the 32,767-byte highlight cap to `:parse` or report
|
|
328
|
+
truncation — never analyze a silent prefix; offsets are **UTF-8 byte** offsets
|
|
329
|
+
over ASCII-normalized input and the normalization is recorded (the exact
|
|
330
|
+
normalization rule is pinned under [Result shape](#result-shape-sketch), Q15);
|
|
331
|
+
`[]`, timeout, and transport failure stay distinct outcomes.
|
|
332
|
+
|
|
333
|
+
The **native-api `:parse` readout is grounded, and the cell is *not* degraded**
|
|
334
|
+
(Q11, version-flat 7.23.2/7.24rc2): `:put [:parse "<input>"]` invoked over the
|
|
335
|
+
native API's `/execute` **with `as-string=""`** returns the parser text —
|
|
336
|
+
including the `(line N column M)` diagnostic on a syntax error — byte-identical
|
|
337
|
+
to what a console transport prints. The opaque `*NN` value seen otherwise is the
|
|
338
|
+
**no-`as-string` job handle**, not a `:parse` limitation; the readout rule is
|
|
339
|
+
"pass `as-string` and interpret `ret`". The existing execute gate already passes
|
|
340
|
+
`as-string` but currently discards `ret`; phase 2 must extract/fix that shared
|
|
341
|
+
readout rather than copy the omission. `/console/inspect`
|
|
342
|
+
(highlight/syntax/completion) works over the native API's `talk` framing too. So
|
|
343
|
+
the line/column diagnostic is available on both live cells — the earlier caveat
|
|
344
|
+
that only a console transport surfaces the parser text is retired.
|
|
345
|
+
|
|
346
|
+
## Surface (Option A — decided)
|
|
347
|
+
|
|
348
|
+
```bash
|
|
349
|
+
centrs explain '<input>' [flags] # offline
|
|
350
|
+
centrs explain <router> '<input>' [flags] # live — target-first
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
`<input>` is a command, a script fragment (`--file` / stdin for scripts), or a
|
|
354
|
+
bare menu path. The positional grammar stays **target-first like every other
|
|
355
|
+
router-taking command** (amended in PR review — the shared resolver/selection
|
|
356
|
+
helpers assume it): one positional means offline and it *is* the input; two
|
|
357
|
+
positionals mean live with the router first; `--` is accepted before the
|
|
358
|
+
input; **`--file` replaces the input positional in either form**, so with it
|
|
359
|
+
every positional is a target. Adding a
|
|
360
|
+
router therefore never reinterprets a previously valid offline invocation.
|
|
361
|
+
|
|
362
|
+
**Ambient stdin is read only when no positional could be the input** — that is,
|
|
363
|
+
the offline form with nothing else to analyze. A positional always wins, and
|
|
364
|
+
when fd 0 is redirected anyway the result carries a
|
|
365
|
+
[`usage/stdin-ignored`](../../docs/errors/usage/stdin-ignored.md) warning rather
|
|
366
|
+
than quietly dropping the pipe. `--file -` names stdin explicitly; it is the
|
|
367
|
+
offline fix today, since the live form it also spells
|
|
368
|
+
(`… | centrs explain edge1 --file -`) is parsed and then refused with
|
|
369
|
+
`usage/not-implemented` until phase 2. The narrower rule is
|
|
370
|
+
deliberate and measured: `runCli` is called in-process by tests, so reading fd 0
|
|
371
|
+
on any path a positional invocation reaches consumes the invoking shell's
|
|
372
|
+
stdin — it fed those bytes to the analyzer and broke 21 unrelated tests once,
|
|
373
|
+
then passed on the next run because the read had drained the fd. Detecting the
|
|
374
|
+
collision is a stat on fd 0, never a read, which is why it is reported instead
|
|
375
|
+
of resolved.
|
|
376
|
+
One verb serves all three intents; the accepted risk is that
|
|
377
|
+
the refined "broad query → broader results" scheme adds complexity to one
|
|
378
|
+
surface rather than splitting into sub-verbs. (Option B — sub-verbs
|
|
379
|
+
`explain check|complete|path` — and Option C — folding into the runners — were
|
|
380
|
+
considered and set aside; C is rejected outright: the bench grounds a
|
|
381
|
+
*separable* cheap tier, offline explain has no runner to ride, and MCP already
|
|
382
|
+
serves `centrs_explain` with no CDB.)
|
|
383
|
+
|
|
384
|
+
**Smart sizing, not depth knobs.** There is no `--depth`. `explain` returns a
|
|
385
|
+
*reasonable* amount by default: when more detail is cheap it is included; when
|
|
386
|
+
a subtree (paths × verbs × args) is large, it returns **counts plus a warning
|
|
387
|
+
that the tree was truncated**, with hints for the narrower query that expands
|
|
388
|
+
the interesting part. A single override flag — working name `--full`, final
|
|
389
|
+
name TBD — lifts the size limits regardless of result size.
|
|
390
|
+
|
|
391
|
+
| Flag | Adds | Notes |
|
|
392
|
+
| ---- | ---- | ----- |
|
|
393
|
+
| *(base)* | canonical structure, spans, diagnostics, transport classification | offline: pure canonicalizer; live: + highlight + `:parse` |
|
|
394
|
+
| `--complete [--cursor <byte>]` | continuation candidates at the cursor (default: end of input) | live evidence; offline emits the live-target tip |
|
|
395
|
+
| `--schema` | path enumeration: verbs, args, types, enums, `.proplist` | live evidence; offline emits the live-target tip |
|
|
396
|
+
| `--curl` | rendered REST call (`curl …`) for `api-candidate` statements covered by a tested mapping rule, plus the equivalent `centrs api`/`centrs execute` invocation | offline-capable; placeholder host + elided credentials when no router is given; `unknown` classifications render no curl |
|
|
397
|
+
| `--full` *(name TBD)* | lift smart-sizing limits | — |
|
|
398
|
+
|
|
399
|
+
Rendering stays separate: `--format`/`--json` remain the settings-vocabulary
|
|
400
|
+
output switch. The longer-term wish behind "format levels" — a colorized
|
|
401
|
+
breakdown of the command (e.g. unset attributes in grey) — is a **rendering**
|
|
402
|
+
concern for the CLI/TUI layer over the same data, not a richness level.
|
|
403
|
+
|
|
404
|
+
Exit codes follow the `check` pattern: `0` clean, `2` when diagnostics meet
|
|
405
|
+
`--fail-on <error|warning|never>` (default `error`), `1` on command failure.
|
|
406
|
+
|
|
407
|
+
## Result shape (sketch)
|
|
408
|
+
|
|
409
|
+
Standard envelope (constitution: result envelope); `data` sketch:
|
|
410
|
+
|
|
411
|
+
```json
|
|
412
|
+
{
|
|
413
|
+
"input": {
|
|
414
|
+
"bytes": 51,
|
|
415
|
+
"normalized": false,
|
|
416
|
+
"truncated": false,
|
|
417
|
+
"positionMap": [ { "analyzed": { "start": 0, "end": 51 }, "originalUtf16": { "start": 0, "end": 51 } } ]
|
|
418
|
+
},
|
|
419
|
+
"verdict": "pass",
|
|
420
|
+
"canonical": { "path": "/ip/route", "verb": "add", "args": { "dst-address": "10.9.0.0/16", "blackhole": "yes" }, "mode": "structured", "writeShaped": true },
|
|
421
|
+
"structure": {
|
|
422
|
+
"statementCount": 1,
|
|
423
|
+
"statements": [
|
|
424
|
+
{
|
|
425
|
+
"span": { "start": 0, "end": 51 },
|
|
426
|
+
"resolution": "resolved",
|
|
427
|
+
"kind": "command",
|
|
428
|
+
"command": { "path": "/ip/route", "verb": "add", "args": { "dst-address": "10.9.0.0/16", "blackhole": "yes" } },
|
|
429
|
+
"transport": { "classification": "api-candidate", "rest": { "method": "PUT", "path": "/rest/ip/route" }, "curl": "curl -u … https://<router>/rest/ip/route …", "centrs": "centrs api <router> …" },
|
|
430
|
+
"ev": "e0"
|
|
431
|
+
}
|
|
432
|
+
],
|
|
433
|
+
"blocks": [],
|
|
434
|
+
"containsWrite": true,
|
|
435
|
+
"subcommands": []
|
|
436
|
+
},
|
|
437
|
+
"symbols": { "occurrences": [] },
|
|
438
|
+
"values": { "occurrences": [ { "id": "v0", "span": { "start": 26, "end": 37 }, "tokenSpan": { "start": 14, "end": 37 }, "kind": "attribute", "name": "dst-address", "quoted": false, "facts": { "shapeHints": { "values": ["ip-prefix"], "ev": "e9" } } } ] },
|
|
439
|
+
"spans": [ { "start": 0, "end": 9, "class": "path", "ev": "e0" } ],
|
|
440
|
+
"diagnostics": [],
|
|
441
|
+
"evidence": [
|
|
442
|
+
{ "id": "e0", "source": "canonicalizer", "probe": "resolveVerbs", "basis": "heuristic", "outcome": "ok" },
|
|
443
|
+
{ "id": "e9", "source": "canonicalizer", "probe": "valueShapeHints", "basis": "heuristic", "outcome": "ok" }
|
|
444
|
+
],
|
|
445
|
+
"runtimeAcceptance": "not-proven"
|
|
446
|
+
}
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
- **The two verdicts are separate.** `data.verdict` is the maximum diagnostic
|
|
450
|
+
severity (`pass` / `warn` / `fail`) and drives `--fail-on` only. Each entry in
|
|
451
|
+
`structure.statements[]` has a canonicalizer `resolution` of `resolved`,
|
|
452
|
+
`ambiguous`, or `unknown`. `ambiguous` means the input has multiple valid
|
|
453
|
+
schema-free readings (for example a bare path may be a menu or a no-argument
|
|
454
|
+
command); `unknown` means analysis cannot safely recover a reading. Neither is
|
|
455
|
+
silently collapsed to a diagnostic severity or to a guessed command. A
|
|
456
|
+
resolved statement also carries `kind: "menu" | "command"`, and the three
|
|
457
|
+
shapes are a **discriminated union**, not four independent optional fields:
|
|
458
|
+
a menu carries `command: { path }` and no verb (navigation names a menu and
|
|
459
|
+
nothing else), a command carries `command: { path, verb }`, and a refusal
|
|
460
|
+
carries `unresolved` and no `command`. Narrowing on `resolution`/`kind` yields
|
|
461
|
+
the fields that exist, which is what the LSP and MCP consumers need from the
|
|
462
|
+
exported types. The library reports the two kinds as *separate
|
|
463
|
+
resolutions* — `src/explain/verbsplit.ts` returns `navigation` (with
|
|
464
|
+
`kind: "menu"`) beside `resolved` (with `kind: "command"`), so `resolved`
|
|
465
|
+
keeps meaning "a verb was decided" for its callers. Phase 1 folds `navigation`
|
|
466
|
+
into this envelope's `resolved` + `kind: "menu"`; the envelope vocabulary here
|
|
467
|
+
is unchanged.
|
|
468
|
+
- **Whole-input gate and per-statement analysis stay distinct.** `canonical`
|
|
469
|
+
reproduces the existing execute gate's whole-input fields unchanged, including
|
|
470
|
+
its `mode` and `writeShaped` decision. `structure.statements[]` is the
|
|
471
|
+
addressable rich surface for script-scale input: every statement carries its
|
|
472
|
+
source span, resolution, resolved command when available, transport
|
|
473
|
+
classification, and evidence id. This is where multi-statement scripts and
|
|
474
|
+
partial recovery live; a single top-level transport object cannot represent
|
|
475
|
+
them.
|
|
476
|
+
- **Every derived fact keeps provenance via `evidence[]`** — facts reference
|
|
477
|
+
a stable evidence id (`ev`), and the evidence entry carries source
|
|
478
|
+
(`canonicalizer` vs `live-inspect`), probe, basis (direct response vs
|
|
479
|
+
derived vs heuristic), outcome (`ok`/`empty`/`timeout`/`transport-error`),
|
|
480
|
+
and the RouterOS version stamp. A single global version field is *not*
|
|
481
|
+
enough: facts in one result can come from different probes with partial
|
|
482
|
+
failures. The `RouterOsSyntaxEvidence` contract in lsp-routeros-ts
|
|
483
|
+
`docs/syntax-inspection-map.md` is the adaptation source. Resolved target
|
|
484
|
+
identity lives in the envelope's `meta.target` (constitution) — `data`
|
|
485
|
+
grows no competing identity surface.
|
|
486
|
+
- **Envelope semantics mirror `check`**: an analysis that ran is `ok: true`
|
|
487
|
+
even when the input is riddled with errors — the diagnostics *are* the
|
|
488
|
+
data, summarized by `data.verdict` (max severity after analysis). Exit
|
|
489
|
+
codes derive from the verdict vs `--fail-on`; `ok: false` is reserved for
|
|
490
|
+
genuine command failure (unresolvable target, usage error, probe transport
|
|
491
|
+
failure). This lets the library and MCP reproduce the same decision without
|
|
492
|
+
a process exit code.
|
|
493
|
+
- **Coordinates are contracted, not implied**: probe offsets are UTF-8 byte
|
|
494
|
+
offsets over the *analyzed* (ASCII-normalized) input with `end` exclusive,
|
|
495
|
+
and `input.positionMap[]` maps half-open analyzed-byte runs to half-open
|
|
496
|
+
original UTF-16 runs. Identity runs are retained, so consumers never need a
|
|
497
|
+
separate coordinate contract for normalized and unnormalized input.
|
|
498
|
+
The library surface additionally provides original-document positions
|
|
499
|
+
(line + UTF-16 character) for LSP consumers; `--cursor <byte>` stays a
|
|
500
|
+
wire-level CLI convenience. **The ASCII-normalization is byte-count-preserving**
|
|
501
|
+
(Q15, ratified): each non-ASCII byte is replaced by a single placeholder byte
|
|
502
|
+
one-for-one (a multi-byte character becomes N placeholder bytes, never
|
|
503
|
+
collapsed to one), so an analyzed byte offset **equals** the device's UTF-8 /
|
|
504
|
+
`highlight` byte offset identically and offline spans and highlight spans share
|
|
505
|
+
one coordinate system with no fixup. A reference mapper validated the required
|
|
506
|
+
invariants at 100% over ASCII / BMP / astral-surrogate / combining-mark / tab /
|
|
507
|
+
LF-CRLF / lone-surrogate fixtures: every span in bounds; original↔analyzed
|
|
508
|
+
round-trips at boundaries; slicing the original range yields the intended
|
|
509
|
+
token; `end === len` is the legal end-of-input cursor; a cursor interior to a
|
|
510
|
+
multi-byte character snaps to that character's boundary. centrs does not
|
|
511
|
+
NFC-fold — a combining mark is its own position — and `\r\n` counts as one line
|
|
512
|
+
advance while a lone `\r` is not a line break. The production parser runs on the
|
|
513
|
+
analyzed string directly (it is pure ASCII, so its JS index *is* the byte
|
|
514
|
+
offset) and this mapper converts to original line/UTF-16 positions.
|
|
515
|
+
- **Spans use a centrs-owned vocabulary** (decision): raw RouterOS highlight
|
|
516
|
+
classes are not the default surface. The centrs vocabulary must be at least
|
|
517
|
+
as rich as the RouterOS token classes, and centrs maintains the mapping —
|
|
518
|
+
including color mappings derived from the original RouterOS highlight
|
|
519
|
+
colors — so an LSP consuming centrs can reproduce the current RouterOS
|
|
520
|
+
color scheme faithfully. The indirection decouples consumers from
|
|
521
|
+
cross-version drift in the raw classes; raw classes stay available in
|
|
522
|
+
provenance/debug output.
|
|
523
|
+
- Diagnostics are slash-namespaced (`explain/<source>/<slug>`), carry byte
|
|
524
|
+
spans (or line/column from `:parse`), and follow the standard error-model
|
|
525
|
+
severity channels. **Malformed input carries a defect *region*** (Q14): each
|
|
526
|
+
diagnostic points at the byte span of its defect, and the detectable classes
|
|
527
|
+
are {unbalanced delimiter, unterminated string, invalid escape, invalid sigil,
|
|
528
|
+
invalid unquoted hash, invalid apostrophe (#355), BOM, non-ASCII,
|
|
529
|
+
over-depth nesting}. **Two classes named
|
|
530
|
+
in the phase-0 draft
|
|
531
|
+
of this list are deliberately deferred** (maintainer decision, #192), because
|
|
532
|
+
each would require offline to assert something it cannot prove:
|
|
533
|
+
- **truncation** — offline cannot distinguish a truncated *complete-looking*
|
|
534
|
+
token from a finished one without a schema, as the `--complete` paragraph
|
|
535
|
+
below already states. What is detectable is the *continuation state*
|
|
536
|
+
(trailing `=`, open delimiter), which is a `--complete` concern reported
|
|
537
|
+
against a live target, not a defect region. Emitting it as a defect would be
|
|
538
|
+
the confident-claim-on-ambiguous-input posture Q14 forbids.
|
|
539
|
+
- **stray mid-token delimiter** — plausible but ungrounded. It needs a probe
|
|
540
|
+
matrix (spellings × positions × accepted-form neighbors) before a detector
|
|
541
|
+
is worth having; the #201 round is the standing evidence that a lexical rule
|
|
542
|
+
curated from intuition produces defects the corpus cannot reach.
|
|
543
|
+
|
|
544
|
+
Both remain candidates; re-opening either means grounding it first and
|
|
545
|
+
amending this list. `BOM` and `non-ASCII` are **positional facts, never
|
|
546
|
+
errors** — they record where the byte-count-preserving normalization stood in
|
|
547
|
+
for unreadable bytes, and a renderer must not give them error severity (a
|
|
548
|
+
non-ASCII value such as `name="router-🚀"` is a legal command, and non-ASCII
|
|
549
|
+
occurs in ~12% of the phase-0 corpus). Two hard
|
|
550
|
+
recovery rules follow from the phase-0 mutation suite, both **fail-closed**:
|
|
551
|
+
(a) corruption may only degrade a command to `ambiguous`/`unknown`, **never
|
|
552
|
+
invent a confident one** after a defect region — the same fail-closed floor as
|
|
553
|
+
the canonicalizer's `ambiguous` verdict, and the direct fix for the measured
|
|
554
|
+
case where a stray `;` inside a value splits off a garbage fragment that
|
|
555
|
+
offline would otherwise resolve to a bogus path; (b) a **bare-word-headed
|
|
556
|
+
statement** resolved against document context degrades to `unknown` when the
|
|
557
|
+
context is uncertain (an upstream defect can poison it), because a bare word is
|
|
558
|
+
Q4's cascade residual. Deeply nested delimiters emit an over-depth diagnostic
|
|
559
|
+
rather than recursing (Q17 measured stack-overflow and O(n²) blow-up in a naive
|
|
560
|
+
recursive walker; the product parser bounds depth and scans single-pass).
|
|
561
|
+
Offline `--complete` detects trailing-`=` and open-delimiter continuation but
|
|
562
|
+
cannot distinguish a truncated *complete-looking* token from a finished one
|
|
563
|
+
without a schema — so it surfaces the detectable continuation state and defers
|
|
564
|
+
candidate generation to a live target.
|
|
565
|
+
- `runtimeAcceptance: "not-proven"` is structural, not boilerplate: it is the
|
|
566
|
+
inspect-vs-runtime gap made machine-readable.
|
|
567
|
+
- Live-state token classes (disabled/dynamic/inactive object references)
|
|
568
|
+
surface as facts, not errors; severity policy belongs to the caller.
|
|
569
|
+
|
|
570
|
+
### What phase 1 offline actually emits (#202a)
|
|
571
|
+
|
|
572
|
+
The sketch above is the whole shape, and `examples.md` describes the FINISHED
|
|
573
|
+
command — both stay as they are. This table says which parts `src/explain.ts`
|
|
574
|
+
ships today, so a reader can tell a decision from an oversight. Every gap closes
|
|
575
|
+
by ADDING a field, never by changing one, which is why the examples do not need
|
|
576
|
+
to move: an example asserting a field this phase omits is simply not green yet,
|
|
577
|
+
and its phase is named below.
|
|
578
|
+
|
|
579
|
+
| Part | Phase 1 offline |
|
|
580
|
+
| ---- | --------------- |
|
|
581
|
+
| `input`, `verdict`, `canonical`, `structure`, `diagnostics`, `runtimeAcceptance` | complete |
|
|
582
|
+
| `evidence[]` | the **offline subset**: `source` is always `canonicalizer` and there is no RouterOS version stamp |
|
|
583
|
+
| `structure.statements[].command` | `path` + `verb`, plus **`args` where the argument list was read** (#202c-1); the ordered token list is `statements[].arguments.tokens` |
|
|
584
|
+
| `structure.statements[].transport` | complete for commands: `api-candidate`, `execute`, or fail-closed `unknown`, with an equivalent `centrs` invocation and opt-in `curl` for tested REST mappings (#202c-2). The `basis` names the evidence for the route it chose or refused |
|
|
585
|
+
| `symbols.occurrences[]` | Q13 name/span/class plus semantic `role`, result-local `bindingIds`, sigil spelling, and an abstention note where needed; **plus** (#239 S2) `valueId` on declarations/assignments with a literal RHS in the same statement, and `reachingValueIds` + `reachingUnknown` on references (flow-sensitive, branch → set, loop/non-literal → `unknown`, no RHS evaluation) |
|
|
586
|
+
| `values.occurrences[]` | result-local value id/span plus three separate fact homes: offline `shapeHints`; live `observedType` and `schemaType` remain absent until phase 2. `kind: "element"` rows are array members, carrying `parent` and, for a keyed member, `name` |
|
|
587
|
+
| `spans` | comment runs and resolved variable classes only; **no value shape or type** |
|
|
588
|
+
| `schema`, `completion` | absent — live evidence, phase 2 |
|
|
589
|
+
|
|
590
|
+
- **Per-statement arguments are lexed, and the list is all-or-nothing** (#202c-1,
|
|
591
|
+
`src/explain/args.ts`). The ordered token list with spans is
|
|
592
|
+
`statements[].arguments.tokens` (the phase-0 normal form); `command.args` is
|
|
593
|
+
its derived object view, present only when the list was read. Two different
|
|
594
|
+
outcomes, which a reader must not conflate:
|
|
595
|
+
- **The whole list is refused** — `arguments.read` is `false` with a reason,
|
|
596
|
+
and `command.args` is absent — when a token cannot be delimited or named at
|
|
597
|
+
all: a `[…]`/`(…)`/`$x`/`{…}` value, an escape this phase does not decode, a
|
|
598
|
+
left-hand side that is not a RouterOS name, a token split by a continuation.
|
|
599
|
+
Never partially read, because a dropped argument silently changes what a
|
|
600
|
+
rendered `curl` DOES. One refusal is not lexical at all, but a place where
|
|
601
|
+
centrs's two readers disagree about an **unquoted** value: a `\f`/`\v`,
|
|
602
|
+
which the retired execute gate split tokens on (JavaScript `\s`) and every
|
|
603
|
+
explain module does not (ASCII whitespace). The device-correct reading is
|
|
604
|
+
the analysis's, but that gate could not be corrected — so phase 1 publishes
|
|
605
|
+
neither rather than putting two confident values in one result. Inside a
|
|
606
|
+
`"…"` run the two agree and nothing is refused.
|
|
607
|
+
|
|
608
|
+
The advisory value-anchor view is intentionally wider only for exact array
|
|
609
|
+
literals: a non-empty `{…}` value or a parenthesized run with a depth-zero
|
|
610
|
+
comma is locatable and hints `array`, while the strict REST view continues
|
|
611
|
+
to refuse the whole list.
|
|
612
|
+
|
|
613
|
+
An unquoted `'` used to be a second such disagreement, described here as a
|
|
614
|
+
character RouterOS treats as ordinary and the gate treats as a quote. That
|
|
615
|
+
reading was wrong on both halves and #355 settled it: `'` is not a RouterOS
|
|
616
|
+
token at all, so `comment=it's` is REJECTED by the device rather than
|
|
617
|
+
carrying either value, and the quote-counting gate that disagreed is gone.
|
|
618
|
+
- **A token is read but carries no `value`** — the list still reads. The token
|
|
619
|
+
is delimited and classified; only its literal value is unknowable, as for
|
|
620
|
+
the positional in `:log info "result: $[…]"`. **`value` absent means there
|
|
621
|
+
is no literal value**, whatever the token's kind, so a consumer rendering a
|
|
622
|
+
runnable command reads `value`, never the source `text`, and treats absence
|
|
623
|
+
as not-renderable.
|
|
624
|
+
|
|
625
|
+
Measured on the frozen corpus: 40.0% of CRUD-verb commands read, 0 arguments
|
|
626
|
+
dropped against the IL oracle, and 0 contradictions with `canonical.args`
|
|
627
|
+
where both decided. That last one holds *because* of the fail-closed rules
|
|
628
|
+
above, not by accident — a trailing `;`, a `'`, and `\f`/`\v` each
|
|
629
|
+
contradicted until review found them. The `;` case is enforced at the
|
|
630
|
+
composition boundary rather than in the lexer, because segmentation strips the
|
|
631
|
+
delimiter before the lexer ever sees it: where the gate read the whole input
|
|
632
|
+
as `structured` and the single statement's reading differs, the analysis
|
|
633
|
+
abstains.
|
|
634
|
+
|
|
635
|
+
What offline still cannot do is NAME a positional operand — RouterOS binds
|
|
636
|
+
`:log info "x"` to `message=x` from its schema, and offline reports the
|
|
637
|
+
located positional instead of inventing the name.
|
|
638
|
+
- **Transport is classified per resolved command and fails closed.** Only the
|
|
639
|
+
Q8 rules exercised on CHR 7.23.2 and 7.24rc2 become `api-candidate`;
|
|
640
|
+
script-shaped inputs become `execute`, and untested or ambiguous mappings
|
|
641
|
+
become `unknown` with no runnable REST rendering. Four rules are keyed on a
|
|
642
|
+
literal verb (`add`/`get`/`set`/`remove`) and `print` has its three shapes;
|
|
643
|
+
the fifth was recorded as the RULE `action → POST /rest/<path>/<command>`, so
|
|
644
|
+
every other verb rides it. That rule maps a URL and does **not** claim the
|
|
645
|
+
verb exists — a `catalog.ts` miss narrows the basis string, never the
|
|
646
|
+
classification, because "a MISS says nothing" is that table's contract and
|
|
647
|
+
existence is the phase-2 `/console/inspect` answer. An action operand offline
|
|
648
|
+
cannot NAME (`enable *A`, `monitor 0`) fails closed, since RouterOS binds it
|
|
649
|
+
from per-menu schema. Curl is opt-in and uses placeholders offline. Menus and
|
|
650
|
+
refused readings have no transport because there is no command to classify.
|
|
651
|
+
- **`spans` carries what offline can prove.** Comment runs, and the variable
|
|
652
|
+
classes Q13 scored at 100% precision on resolved bindings; an abstention is
|
|
653
|
+
omitted rather than rendered as a guess. The Q12 vocabulary over
|
|
654
|
+
path/verb/argument bytes wants device `highlight` as its oracle. A subset is
|
|
655
|
+
not a claim that the vocabulary is closed.
|
|
656
|
+
- **`symbols` is the semantic Q13 projection plus flow-sensitive value refs
|
|
657
|
+
(#239 S2).** Each occurrence carries its source name/span, resolved class
|
|
658
|
+
(or `null` when offline abstains), role (`declaration`, `binding`,
|
|
659
|
+
`assignment`, `reference`, or filter `field`), and result-local binding
|
|
660
|
+
identities. Those identities distinguish shadowed names and connect `:set`/
|
|
661
|
+
reference sites to their declaration; they are stable only within one result.
|
|
662
|
+
When `:onerror` establishes two bindings, the list is ordered outer-scope
|
|
663
|
+
binding first, statement-scope binding second. A declaration/binding/
|
|
664
|
+
assignment with a literal RHS in the same statement carries `valueId`
|
|
665
|
+
pointing at its `values.occurrences[]` row; a reference carries
|
|
666
|
+
`reachingValueIds` (the set of literal ids that may reach it) and
|
|
667
|
+
`reachingUnknown` when the set is incomplete. Linear code is last-write-wins;
|
|
668
|
+
a branch merge is a set; a loop merge or a non-literal RHS (`$x`,
|
|
669
|
+
`[find …]`, an expression) is `unknown` — no RHS evaluation, so a
|
|
670
|
+
`reachingUnknown` reference must not be read as a type assertion. The three
|
|
671
|
+
value facts themselves stay on `values.occurrences[].facts`; symbols only
|
|
672
|
+
*refer* to them. A hyphen always terminates a bare `$name`: `$set-dns` reads
|
|
673
|
+
`$set`, while `$"set-dns"` is the spelling that reaches a hyphenated
|
|
674
|
+
declaration. Braced forms such as `${set-dns}` and `${"set-dns"}` are
|
|
675
|
+
rejected by RouterOS at `{` and resolve no reference. In an expression the
|
|
676
|
+
remaining `-dns` is scanned as subtraction plus its right operand; inside a
|
|
677
|
+
string it is literal text.
|
|
678
|
+
- **Evidence is offline-shaped, not the whole contract.** The bullet above says
|
|
679
|
+
an evidence entry carries `source` (`canonicalizer` vs `live-inspect`) and a
|
|
680
|
+
RouterOS version stamp; phase 1 emits neither the live source nor the stamp,
|
|
681
|
+
because no probe ran and there is no device to stamp. `ExplainEvidence` is
|
|
682
|
+
typed to what offline produces rather than pre-declaring variants nothing can
|
|
683
|
+
emit, so phase 2 widens the union and adds the stamp — an addition, not a
|
|
684
|
+
change to what a caller reads today.
|
|
685
|
+
- **Value facts are three axes, never one `type` string (#225).** RouterOS
|
|
686
|
+
types values at parse time while `highlight` classes every value byte `none`,
|
|
687
|
+
so type evidence is not a subset of Q12 spans. Each result-local
|
|
688
|
+
`values.occurrences[]` row owns one value span and a `facts` object:
|
|
689
|
+
1. `shapeHints: { values, ev }` is offline and heuristic — a list, because the
|
|
690
|
+
axis admits overlap;
|
|
691
|
+
2. `observedType?: { value, ev }` is what a live parser/IL reading supports;
|
|
692
|
+
3. `schemaType?: { value, ev }` is the argument's declared live-schema type.
|
|
693
|
+
|
|
694
|
+
The stable/testing CHR matrix fixes the offline lexicon: `num`, `ip`,
|
|
695
|
+
`ip-prefix`, `ip6`, `ip6-prefix`, `id`, `time`, `array`, `mac`, `bool`, and
|
|
696
|
+
`str` — `op` is deliberately not a member: a deferred `(> …)` abstains
|
|
697
|
+
offline and is reported only as a live `observedType` (#288); quoted literals
|
|
698
|
+
hint only `str`, while malformed and out-of-range address-like controls
|
|
699
|
+
abstain even in named attributes. Most vocabulary borrows RouterOS scripting
|
|
700
|
+
type names (with the explicit `macAddr` schema exception below), so a hint
|
|
701
|
+
must never spell a scripting type the device contradicts: `num` is
|
|
702
|
+
**integer-only**, since RouterOS numbers are integers and a dotted decimal is
|
|
703
|
+
an IPv4 shortcut — bare `2.2` is observed as `ip` (`2.0.0.2`), so it hints
|
|
704
|
+
`ip` alone. On that grounded lexicon each spelling reaches at most one shape.
|
|
705
|
+
Colon-form time accepts RouterOS's displayed H:M / H:M:S spelling plus
|
|
706
|
+
week/day prefixes and fractional tails; `00:11:22` is therefore `time`, not a
|
|
707
|
+
short MAC. `mac` deliberately names the CLI Reference `macAddr` schema
|
|
708
|
+
spelling rather than a scripting type: a full six-octet MAC hints `mac` while
|
|
709
|
+
scalar `:typeof` reports `str`. `*1` hints the documented `id` type. A single
|
|
710
|
+
non-time colon is not an address attempt — `comment=foo:bar` hints `str` —
|
|
711
|
+
while malformed multi-colon runs stay fail-closed.
|
|
712
|
+
|
|
713
|
+
Arrays are the one structured literal admitted by offline hint anchoring.
|
|
714
|
+
Non-empty `{…}` values and parenthesized expressions with a depth-zero comma
|
|
715
|
+
hint `array`; grouping `(1)`, empty groups, and scope braces do not. The
|
|
716
|
+
strict REST argument reader still refuses every structured value. RouterOS
|
|
717
|
+
has no empty-array literal (`{}` is a syntax error); `[:toarray ""]` produces
|
|
718
|
+
an empty array, but remains an expression and therefore an offline
|
|
719
|
+
abstention. Indexing that empty array produces `nothing`. `.` remains an
|
|
720
|
+
expression operator rather than part of a literal: it produces `str` for
|
|
721
|
+
`1 . 2`, but distributes over arrays in either operand order, so offline
|
|
722
|
+
abstains on the whole concat expression and leaves its result type to #236.
|
|
723
|
+
`[:parse "…"]` similarly produces the observed but undocumented `code` type,
|
|
724
|
+
and `[:nothing]` produces `nil`; neither has a standalone literal spelling,
|
|
725
|
+
so neither becomes an offline shape member. `(> …)` deferred expressions
|
|
726
|
+
are the same case with a different outcome: `(>[:return 1])` is `op` and
|
|
727
|
+
`(>{"a"=1})` is `array` at runtime, but both lower to `(> …)` with arity
|
|
728
|
+
1 and identical `:parse` IL — only `:typeof` distinguishes them
|
|
729
|
+
(`test/fixtures/explain/operators.json` → `sweep.runtime`
|
|
730
|
+
`typeof-deferred`/`deferred-array-typeof` and `sweep.opAxis` `defer-*` on
|
|
731
|
+
7.21.5, 7.23.3, 7.24rc4). Offline abstains on every such form; `op` lives
|
|
732
|
+
only as a live `observedType` (#288, a bounded #225 decision).
|
|
733
|
+
Shape remains non-authoritative: `100000w` is time-shaped but observed as
|
|
734
|
+
`str`. Argument context remains separate: on 7.23.3 and 7.24rc3, both
|
|
735
|
+
`/ip/address address=2.2` and firewall `src-address=2.2` canonicalize it to
|
|
736
|
+
`2.0.0.2`, while `comment=2.2` stays text, netwatch `interval=2.2`
|
|
737
|
+
normalizes to `00:00:02.200`, a boolean slot rejects it, and `:parse` still
|
|
738
|
+
accepts `src-address=not-an-ip`. Boolean grounding has the same separation:
|
|
739
|
+
bare `true`/`false` and `yes`/`no` are scalar `bool`, while their quoted forms
|
|
740
|
+
are `str`; a CLI `disabled=` slot accepts bare or quoted `yes`/`no` but rejects
|
|
741
|
+
`true`/`false`, while REST accepts JSON booleans and the string forms
|
|
742
|
+
`yes`/`no`/`true`/`false`. `:tobool` conversion is a separate runtime operation
|
|
743
|
+
and does not widen the hint. Thus a hint never validates a value or becomes a
|
|
744
|
+
diagnostic, and never changes `runtimeAcceptance`. Quoting likewise never
|
|
745
|
+
earns a wrong-type diagnostic: commands may cast quoted strings at runtime.
|
|
746
|
+
Offline emits only `shapeHints`; phase 2 adds the two live facts into their
|
|
747
|
+
existing homes when its safe IL/schema producers land.
|
|
748
|
+
|
|
749
|
+
**A located array is read member by member (#225 V1).** Each member of a
|
|
750
|
+
`{…}`/`(…)` literal is its own `values.occurrences[]` row with `kind:
|
|
751
|
+
"element"`, its own span and hints, a `parent` naming the container's
|
|
752
|
+
result-local id, and `name` when the brace form spells a key. The separator is
|
|
753
|
+
the delimiter's and not a choice: a brace splits on `;` while a depth-zero `,`
|
|
754
|
+
inside one builds a nested array (`{1,2}` is a ONE-member array whose member
|
|
755
|
+
is `(, 1 2)`), a paren splits on `,`, and `(1;2)` does not parse.
|
|
756
|
+
|
|
757
|
+
**`=` does three different things inside a literal, and a byte of whitespace
|
|
758
|
+
picks between them (#258).** It binds a key only in the brace form, and only
|
|
759
|
+
where the NAME TOUCHES the sign: `{a=1}` is the key `a`, but `{a =1}` lowers
|
|
760
|
+
to `(= $a 1)` and `(a=1,b=2)` is two comparisons, so keys are read in braces
|
|
761
|
+
only. A key whose right side is EMPTY is not an empty-valued key — the device
|
|
762
|
+
drops the sign and keeps the name as a POSITIONAL `str`, so `{a=}` is the
|
|
763
|
+
one-member array `a`, `{"a b"=}` decodes to `a b`, and `{1.1=}` is the string
|
|
764
|
+
`1.1` rather than the address (no member lexicon runs on a name). Everywhere
|
|
765
|
+
else the sign compares, which returns `bool` whatever it compares, so
|
|
766
|
+
`{$a=1}`, `{(a)=1}`, `{[:timestamp]=1}` and `{{1;2}=1}` are each one `bool`
|
|
767
|
+
member. Every key is a `str` however it is spelled: `{1=1}` keys on the string
|
|
768
|
+
`1`, and `{1.1=1.1}` keys on `1.1` while its VALUE goes through the address
|
|
769
|
+
shortcut to `1.0.0.1`.
|
|
770
|
+
|
|
771
|
+
**A member key's name is not the identifier grammar**, and was swept
|
|
772
|
+
character by character rather than borrowed. `.`, `-` and `/` are ordinary
|
|
773
|
+
name bytes anywhere in it, including alone and repeated — `{.id=1}`,
|
|
774
|
+
`{..id=1}`, `{.=1}`, `{-=1}`, `{--=1}`, `{/=1}`, `{a/b=1}` and `{-.1=1}` each
|
|
775
|
+
bind the key spelled exactly that way — while `_` does NOT, even though
|
|
776
|
+
`$a_b` is a good variable: `{_a=1}` and `{a_b=1}` lower to comparisons.
|
|
777
|
+
Neither do `: * ~ % > < + ! @ #`. `@` is stronger still — not a member
|
|
778
|
+
character at ANY position, so `{a@b}`, `{@a}`, `{1;a@b}` and `(a@b,2)` are
|
|
779
|
+
withdrawn outright, though the same bytes are fine one slot out
|
|
780
|
+
(`:local z a@b` parses) or inside quotes (`{"a@b"}`).
|
|
781
|
+
|
|
782
|
+
`bool` is claimed only where the `=` is the member's TOP operator, which
|
|
783
|
+
offline reads as a left side that is one complete operand — a name, `$name`,
|
|
784
|
+
or a fully-enclosing `(…)`/`[…]`/`{…}`. `{a b=1}` lowers to `( $a (= $b 1))`
|
|
785
|
+
and `{$a=1,2}` to `(, (= $a 1) 2)`; in both the sign is nested under an
|
|
786
|
+
operator whose type is its operands', so both abstain. That operand class is
|
|
787
|
+
**narrower than RouterOS's**, deliberately: `{a_b=1}`, `{:a=1}`, `{1+1=2}`
|
|
788
|
+
and `{$a/b=1}` are `bool` on the device and silent here, because widening it
|
|
789
|
+
means first proving the `=` stays on top for the new spelling. Operand
|
|
790
|
+
position is also stricter than KEY position — `{a-=1}` binds the key `a-`
|
|
791
|
+
while `{a- =1}` does not parse at all — so both classes end on an
|
|
792
|
+
alphanumeric there.
|
|
793
|
+
|
|
794
|
+
An empty side is a
|
|
795
|
+
syntax error in every spelling asked — `{=}`, `{=1}`, `(=1,2)`, `{$a=}`,
|
|
796
|
+
`{(a)=}`, `{a =}`, `{a b=}`, `{a,b=}`, `{[:len 1]=}`, `{{1;2}=}`, `{1+1=}`,
|
|
797
|
+
`{a<=}`, `{a==}`, `(a=,2)` — so it withdraws the literal rather than
|
|
798
|
+
abstaining on the member. That withdrawal needs POSITIVE evidence that the
|
|
799
|
+
left side is an expression (a byte no key may hold — `NOT_IN_MEMBER_NAME` in
|
|
800
|
+
`src/explain/args.ts`, the complement of `MEMBER_NAME`), never the key
|
|
801
|
+
reader's silence: a name spelling this phase does not recognize abstains
|
|
802
|
+
instead, so the next `.id` costs a dropped member rather than a withdrawn
|
|
803
|
+
literal.
|
|
804
|
+
|
|
805
|
+
**Two member faults have nothing to do with `=` and are checked before it.**
|
|
806
|
+
A `$` takes an alphanumeric or a quote and nothing else — `$"a b"` is a good
|
|
807
|
+
variable name while `$.id`, `$_a`, `$:a`, `$/a`, `$$a`, `$(x)` and a bare `$`
|
|
808
|
+
are syntax errors — and no run may END on a dangling operator: `{-}`, `{--}`,
|
|
809
|
+
`{a-}`, `{$a-}` and `{$a.}` are refused, though `{a.}`, `{.}`, `{/}` and
|
|
810
|
+
`{a/}` parse, because a bare name absorbs a trailing dot and a `$` name does
|
|
811
|
+
not. Both hold anywhere in a member, and both reach the `=` axis only through
|
|
812
|
+
a left operand (`{a- =1}`, `{$a-=1}`, `{$.id=1}`). The 186
|
|
813
|
+
rows are in
|
|
814
|
+
`test/fixtures/explain/values.json` → `interiorGrounding.keyBinding`, and the
|
|
815
|
+
value census below is unchanged by all of it — no statement in the 948-script
|
|
816
|
+
corpus reaches any of these branches, so the fixture is the only falsifier
|
|
817
|
+
they have.
|
|
818
|
+
|
|
819
|
+
**A member is not an argument value, because the device does not parse it as
|
|
820
|
+
one.** Inside a literal RouterOS reads an expression, so the member lexicon is
|
|
821
|
+
its own: a bare word is a variable reference (`{abc}` lowers to `$abc`), and
|
|
822
|
+
so is a full MAC (`{00:11:22:33:44:55}`) and any time literal past the signed
|
|
823
|
+
64-bit nanosecond range, whose cliff sits between `15250w` and `15251w`,
|
|
824
|
+
`106751d` and `106752d`, `9223372036s` and `9223372037s`. `*1` does not parse
|
|
825
|
+
at all there, though `:local x *1` is `id`. Hexadecimal is added — `0x10` is
|
|
826
|
+
16, while `0X10` is a variable — and `str`-by-fallback, `mac` and `id` are
|
|
827
|
+
dropped. Everything else abstains: expressions, substitutions, `$x`, and the
|
|
828
|
+
dotted-decimal-to-seconds fallback described below.
|
|
829
|
+
|
|
830
|
+
**A `{…}` array literal is legal only in a root scripting directive's value
|
|
831
|
+
slot, and only in a slot the device proved.** `/console/inspect` classes the
|
|
832
|
+
`{` byte `error` and `:parse` refuses the statement for
|
|
833
|
+
`/ip/route/add comment={1;2}`, `/ip/dns/set servers={1.1.1.1;8.8.8.8}` — a
|
|
834
|
+
LIST-typed attribute, which is what rules out a schema-shaped reading —
|
|
835
|
+
`/interface/print .proplist={name;comment}`,
|
|
836
|
+
`/ip/route/print where comment={1;2}`, the relative spelling
|
|
837
|
+
`ip route add comment={1;2}`, and `:log info message={1;2}`; while
|
|
838
|
+
`:local z {1;2}`, `:put {1;2}`, `:len {1;2}` and `:foreach i in={1;2}` all
|
|
839
|
+
parse.
|
|
840
|
+
|
|
841
|
+
Being a directive is necessary but not sufficient, and the path does not
|
|
842
|
+
decide: `:delay {1;2}`, `:beep {1;2}`, `:resolve {1;2}`,
|
|
843
|
+
`:if condition={1;2}` and `:local name={1;2}` are syntax errors at a `/` path
|
|
844
|
+
too. The gate is keyed on **(verb, slot)** from a device sweep of every root
|
|
845
|
+
builtin and every slot it completes — 222 rows in
|
|
846
|
+
`test/fixtures/explain/values.json` → `interiorGrounding.braceSlots`, baked
|
|
847
|
+
into `src/explain/brace-slots.ts` — and a slot has four outcomes, not two:
|
|
848
|
+
|
|
849
|
+
| outcome | what the slot does with `{(1,2)}` | example |
|
|
850
|
+
| ------- | --------------------------------- | ------- |
|
|
851
|
+
| `array` | evaluates it — IL `(, 1 2)` | `:local z`, `:put`, `:foreach in=`, `:for from=` |
|
|
852
|
+
| `code` | runs it as a program | `:onerror in=`, `:retry command=`, `:if do=` |
|
|
853
|
+
| `text` | keeps it verbatim as a script | `:execute script=`, `:grep script=` |
|
|
854
|
+
| `error` | refuses the statement | `:delay`, `:beep`, `:local name=` |
|
|
855
|
+
|
|
856
|
+
Only `array` slots are read as arrays; the other three and the 30 rows the
|
|
857
|
+
sweep could not ask are refused, which costs coverage and never correctness.
|
|
858
|
+
A positional is keyed by its index, because position decides — `:local {1;2}`
|
|
859
|
+
puts the literal in the NAME slot and does not parse, while `:local z {1;2}`
|
|
860
|
+
puts it in the VALUE slot and does. `{1;2}` alone cannot build this table:
|
|
861
|
+
`:local z {1;2}` and `:execute script={1;2}` both lower to `…=1;2`, so the
|
|
862
|
+
array test has to be a literal whose evaluated and verbatim readings differ. The `(…)` spelling is never rejected at the delimiter in any of those
|
|
863
|
+
positions — `.proplist=(name,comment)` still draws `bad parameter .proplist`,
|
|
864
|
+
but that is a name-level diagnostic and the plain `name,comment` spelling
|
|
865
|
+
draws it too — and a `(…)` or `[…]` around a brace restores the expression
|
|
866
|
+
context. The braces a
|
|
867
|
+
command argument does take are script bodies (`source=`, `on-event=`), and
|
|
868
|
+
those are not arrays either — they are refused earlier as scope blocks. A
|
|
869
|
+
top-level brace value in any other command argument emits
|
|
870
|
+
`explain/canonicalizer/invalid-command-brace` at the `{`; parentheses or a
|
|
871
|
+
`[…]` substitution restore the expression context where the same literal is
|
|
872
|
+
legal. An empty member is a
|
|
873
|
+
syntax error (`{;}`, `{;1}`, `{1;;2}`, `(1,)`) and withdraws the enclosing
|
|
874
|
+
`array` shape with it; a single trailing separator is legal in the brace form
|
|
875
|
+
only (`{1;}` and `{1;2;}` parse, `{1;2,}` and `{2,}` do not).
|
|
876
|
+
|
|
877
|
+
**A nested member the device rejects withdraws its container too.** A `(…)`
|
|
878
|
+
inside a literal is a GROUP as often as an array — `{(1)}` is the one-member
|
|
879
|
+
array `1` — so an empty group (`{()}`, `{a=()}`) and an empty comma member
|
|
880
|
+
(`{(1,)}`, `{(,)}`, `{1;(2,)}`, `(1,(2,))`) are named as faults rather than
|
|
881
|
+
abstained on: `:parse` rejects each of them at every nesting depth, while
|
|
882
|
+
`{(1)}`, `{a=(1)}` and `{1;(2)}` parse. `highlight` is not the oracle for this
|
|
883
|
+
family — it accepts `{1;2,}`, `{2,}` and `{(1,2),}`, all of which `:parse`
|
|
884
|
+
rejects. Member descent stops after eight frames, and the bound WITHDRAWS
|
|
885
|
+
rather than keeping an unverified shape: a fault below it still makes the
|
|
886
|
+
whole statement a syntax error — `:parse` rejects a `(1,)` buried nine levels
|
|
887
|
+
deep exactly as it rejects a shallow one — so a literal nested deeper than
|
|
888
|
+
eight is refused instead of called an array. The deepest member in the
|
|
889
|
+
948-script corpus sits at depth 6.
|
|
890
|
+
|
|
891
|
+
**The bare comma spelling is the one place a hint is genuinely plural.**
|
|
892
|
+
`=1,2,3` is not a syntax error anywhere, and whether the device SPLITS it is
|
|
893
|
+
decided by the argument's type, which offline does not have:
|
|
894
|
+
`servers=1.1.1.1,8.8.8.8` lowers to `servers=1.1.1.1;8.8.8.8` and
|
|
895
|
+
`dst-port=80,443` to `dst-port=;80;443`, while `comment=a,b` and
|
|
896
|
+
`interface=ether1,ether2` keep the whole run as one string. A named attribute
|
|
897
|
+
therefore carries `["array", "str"]` — both readings, neither validated. In an
|
|
898
|
+
expression position there is no second reading (`:local x 1,2` is a
|
|
899
|
+
two-member `array`, and so is `a,b`), so the hint is singular there. No
|
|
900
|
+
members are located for a bare comma run: whether it splits at all is the
|
|
901
|
+
schema's answer, and a member span would be a guess. The `(1,2)` spelling,
|
|
902
|
+
where the delimiters prove it, is anchored and descended into as usual.
|
|
903
|
+
|
|
904
|
+
The short IPv4 spelling fills the LOW octets and every field is one octet:
|
|
905
|
+
`1.255` is `1.0.0.255` but `1.256` is `time` `00:00:01.256`, `1.16777215` is
|
|
906
|
+
`time` and not `1.255.255.255`, and `1.1.256` is plain text. Offline abstains
|
|
907
|
+
on the seconds fallback rather than encoding a chain that depends on a failed
|
|
908
|
+
address attempt.
|
|
909
|
+
|
|
910
|
+
<!-- BEGIN GENERATED value-census — regenerate with `bun run explain:value-census:readme` -->
|
|
911
|
+
The corpus census is re-derivable with `bun run explain:value-census` and
|
|
912
|
+
covers 948 source scripts. The figures below are generated from
|
|
913
|
+
`test/fixtures/explain/values.json` → `corpus` by
|
|
914
|
+
`bun run explain:value-census:readme` and gated against it by
|
|
915
|
+
`bun run explain:value-census:readme:check`; the fixture itself is gated
|
|
916
|
+
against a fresh corpus run by `bun run explain:value-census:check`. Of the
|
|
917
|
+
13,161 emitted values in a statement the strict argument lexer ALSO read, 0
|
|
918
|
+
disagree with it on half-open byte span or decoded text, while the
|
|
919
|
+
prefix-safe scan retains a further 6,538 values across 568 statements whose
|
|
920
|
+
strict REST reading abstains. Of 19,699 emitted occurrences, 5,636 are array
|
|
921
|
+
members (529 keyed, 1,147 nested inside another member) and 808 are arrays;
|
|
922
|
+
the corpus holds no source-literal `id` example. The three structural
|
|
923
|
+
counters — spans addressing bytes outside their own source, members naming a
|
|
924
|
+
container that does not exist, members escaping the container they name —
|
|
925
|
+
read 0, 0 and 0, and each must stay 0.
|
|
926
|
+
<!-- END GENERATED value-census -->
|
|
927
|
+
|
|
928
|
+
The strict lexer remains all-or-nothing; only non-authoritative hints use the
|
|
929
|
+
wider view.
|
|
930
|
+
|
|
931
|
+
### Offline comment placement (#245)
|
|
932
|
+
|
|
933
|
+
Comment recognition is contextual, not a rule that every unquoted `#` begins a
|
|
934
|
+
comment. The shared offline walkers follow this table, grounded with
|
|
935
|
+
`/console/inspect request=highlight`, `:parse` IL, and `:typeof` on CHR 7.23.3
|
|
936
|
+
(stable) and 7.24rc3 (testing):
|
|
937
|
+
|
|
938
|
+
This intentionally follows the live parser rather than the scripting manual's
|
|
939
|
+
broader prose that a `#` starts a comment. The same live probes distinguish a
|
|
940
|
+
comment, a literal hash value, and a hard error at the exact byte.
|
|
941
|
+
|
|
942
|
+
| Position | Device reading | Offline rule |
|
|
943
|
+
| -------- | -------------- | ------------ |
|
|
944
|
+
| Statement-leading at the document root, after a real statement separator, in a `do`/`else`/`foreach do` body, or immediately inside a `[…]` command substitution | `comment` through newline | Mask it and preserve byte offsets. A command first consumes that lead, so `[:put #test]` keeps `#test` as the value. |
|
|
945
|
+
| First non-space content inside a stored-script brace such as `on-event={ # c` | `comment` | Treat exact `source`/`script` and `on-*`-named brace values as comment-bearing script bodies; a suffix such as `myScript` is not one. |
|
|
946
|
+
| Immediate line start after `\` + newline | `comment`; the pending statement survives | Mask it in both argument views; the continuation-reach rules remain H5/#215. |
|
|
947
|
+
| Inside an array (`{#test}`, `{1;#test}`, `{a=1;#b=2}`) or parenthesized expression (`(1,#test)`) | hard `error` at `#`; `:parse` reports `syntax error` | Do not mask it or emit an array value hint; semantic resolution stops there. |
|
|
948
|
+
| A scope-named brace (`do=`/`else=`/`command=`/`script=`/`source=`/`on-*=`) nested inside an array (`:local z {do={ # c⏎:put 1}}`, `:local z {script={ # c⏎…}}`, `:local z { { # c…}}`) — and likewise a bare `{` or `/menu {` inside an array | hard `error` at `#`; `/console/inspect` request `highlight` classes that byte `error` on CHR 7.23.3 (grounded in #249 on the four `#249` rows plus `bare {`/`source`/`on-event` corners) | A brace bears statements only when its enclosing context already does — `enclosing && braceStartsStatements(…)` on each walker's stack (`contexts`/`delimStack`/`frames`/`stack`). Inside an array the name is an array key and its `{…}` is a nested array. A `[…]` substitution nested in that array does re-enter a statement context, so `[:do { # c…}]` stays a real comment and a bare `{` under a bracket does too. |
|
|
949
|
+
| Inside a `[…]` substitution that is itself nested in an array (`{[:put #test]}`, `{1;[:put #test]}`) | literal value (`none`) — the bracket restores the statement role the array dropped | Keep it as content and keep the enclosing array readable. The role is per frame, so an array or group opened again inside that bracket (`{[:put {#test}]}`, `{[:put (1,#test)]}`) is `error` once more; skipping whole `[…]` regions would wrongly accept those. |
|
|
950
|
+
| In an attribute or bare value (`comment=#test`, `comment=a#b`, `:global y #test`) | literal value (`none` highlight class) | Keep it as content. |
|
|
951
|
+
| Inside a quoted run | string content | Keep it as content. |
|
|
952
|
+
| After a closing scope brace (`} # c`) | hard `error` at `#` | Do not mask it. |
|
|
953
|
+
| After a complete fixed-arity scripting directive (`:local x 2 #`, `:set x 3 #`, `:put 2 #`) | hard `error` at `#`; `:parse` reports `expected end of command` | Emit `explain/canonicalizer/invalid-hash` at the first such byte and fail closed after it. A hash in the still-open value slot (`:local x #test`, `:set x #test`, `:put #test`) remains content. |
|
|
954
|
+
|
|
955
|
+
An otherwise unrecognized `name={…}` is structurally an expression/array, not
|
|
956
|
+
a stored-script body. Whether the target command's schema accepts an array at
|
|
957
|
+
that argument is a separate live schema/runtime question; comment classification
|
|
958
|
+
does not imply type acceptance.
|
|
959
|
+
|
|
960
|
+
This distinction is why brace role lives in one shared primitive below the
|
|
961
|
+
segmenter, block reader, and symbol resolver. The `#249` widening adds the
|
|
962
|
+
single conjunct `enclosing && braceStartsStatements(…)` at the four walker
|
|
963
|
+
sites so a scope-named or bare brace inside an array is another array.
|
|
964
|
+
The strict argument lexer now uses the same comment-masked structural view as value anchoring, so a continuation
|
|
965
|
+
comment cannot fabricate positional operands or downgrade a tested REST shape.
|
|
966
|
+
The 948-script corpus contains no such continuation-comment argument case. The same holds for #249:
|
|
967
|
+
the corpus re-measurement below shows no readable/abstention movement — no
|
|
968
|
+
`do=`/`else=`/`command=`/`script=` brace sits inside an array in the 948
|
|
969
|
+
scripts, so the fix closes a device-grounded false `pass` with zero corpus
|
|
970
|
+
blast radius. The readable set stayed exactly 7,385 statements. Recognizing a bracket-leading hash
|
|
971
|
+
as a comment conservatively folds two previously separate abstentions inside
|
|
972
|
+
one foreign kernel-panic transcript (`[#1]` / `[#2]`) into its already-unknown
|
|
973
|
+
outer statement, leaving 14,329 argument-bearing candidates and 6,944
|
|
974
|
+
abstentions (48.4612%). That two-statement movement is a blast-radius
|
|
975
|
+
measurement, not the grounding evidence; the two CHR versions above are the
|
|
976
|
+
oracle. (Those are the #249-era figures; the #285 catalog correction has since
|
|
977
|
+
moved the readable set to 7,392 — see the re-measurement below.)
|
|
978
|
+
|
|
979
|
+
The same corpus run finds five new `invalid-hash` diagnostics, all in pasted
|
|
980
|
+
non-RouterOS material: one NGINX server block, one JavaScript highlighter, one
|
|
981
|
+
JSON-like transcript, and two shell-shaped snippets. No RouterOS command reading
|
|
982
|
+
moved; these diagnostics say those bytes are not valid RouterOS, not that the
|
|
983
|
+
foreign snippets were expected to parse as RouterOS.
|
|
984
|
+
|
|
985
|
+
The default text renderer surfaces comment spans in a `comments:` section; JSON
|
|
986
|
+
continues to carry the same ranges as `spans[]` entries with `class: "comment"`.
|
|
987
|
+
|
|
988
|
+
Corpus re-measurement for #249 — `bun -e 'explainCommand'` over the 948-script
|
|
989
|
+
`lsp-routeros-ts/test-data/corpus.sqlite` (same harness as § *What `offline` actually emits*):
|
|
990
|
+
|
|
991
|
+
- `stmts` 18,648 · `argCandidates` 14,329 · `argReadable` 7,385 · `abstentions`
|
|
992
|
+
6,944 (48.4612%) — **identical** before and after the `enclosing &&`
|
|
993
|
+
conjunct (stashed vs patched, verified by `git stash --keep-index`).
|
|
994
|
+
- `invalid-hash` diagnostics: 5 files / 5 diagnostics both before and after —
|
|
995
|
+
the widening has no corpus hit, because the corpus contains no `do=`/`else=`/
|
|
996
|
+
`command=`/`script=` brace nested in an array. Zero blast radius is the
|
|
997
|
+
expected outcome; the grounding remains the 15-input CHR probe below
|
|
998
|
+
(CHR 7.23.3 `highlight` at the `#` byte, 0/15 mismatches), not a corpus delta.
|
|
999
|
+
|
|
1000
|
+
The #285 catalog correction moved those figures, and only upward. Same harness,
|
|
1001
|
+
old table vs new:
|
|
1002
|
+
|
|
1003
|
+
| | #249 | #285 |
|
|
1004
|
+
| --- | ---: | ---: |
|
|
1005
|
+
| `stmts` | 18,648 | 18,648 |
|
|
1006
|
+
| `argCandidates` | 14,329 | **14,336** |
|
|
1007
|
+
| `argReadable` | 7,385 | **7,392** |
|
|
1008
|
+
| `abstentions` | 6,944 | 6,944 (**48.4375%**) |
|
|
1009
|
+
| `ambiguous-statement` | 91 | **85** |
|
|
1010
|
+
| `unresolved-statement` | 3,911 | **3,910** |
|
|
1011
|
+
| `context-lost` | 69 | **83** |
|
|
1012
|
+
|
|
1013
|
+
Six scripts move, and every statement in them is a correction. Seven statements
|
|
1014
|
+
that read as `ambiguous`/`unknown` now resolve and their arguments read —
|
|
1015
|
+
`/interface/monitor-traffic`, `/system/package/apply-changes`,
|
|
1016
|
+
`/system/package/update/{check-for-updates,download,install}` — which is the
|
|
1017
|
+
whole of the `argCandidates`/`argReadable` gain, with `abstentions` flat. Five
|
|
1018
|
+
more read the same but with the right verb: `/ip hotspot user reset-counters
|
|
1019
|
+
[find …]` took `hotspot` as the verb and now takes `reset-counters`, which also
|
|
1020
|
+
reclassifies it as a write. No document changes `containsWrite` verdict.
|
|
1021
|
+
|
|
1022
|
+
`context-lost` rising is the resolver **withdrawing** a certainty claim, not
|
|
1023
|
+
losing ground. It is `info` severity and marks a statement that resolved
|
|
1024
|
+
correctly while the document context was already unknown. All 14 are in one
|
|
1025
|
+
script — a bare list of command paths, one per line, with no navigation
|
|
1026
|
+
anywhere. Previously `/interface/monitor-traffic` was unreadable and so *might*
|
|
1027
|
+
have been an absolute navigation, which R4 has REPLACE the context; certainty
|
|
1028
|
+
was restored on that maybe. Knowing it is a command, the resolver knows it never
|
|
1029
|
+
navigated, so the context stays as unknown as it already was.
|
|
1030
|
+
|
|
1031
|
+
- **Severity is fixed here, because it drives `--fail-on`.** Three buckets, and
|
|
1032
|
+
the split is not "structural vs not":
|
|
1033
|
+
- `error` — `unclosed`, `unbalanced-close`, `unterminated-string`,
|
|
1034
|
+
`bad-escape` (code-position `\` + non-whitespace), `bad-string-escape`
|
|
1035
|
+
(string-internal unknown/lowercase-hex/truncated — CHR `highlight` `error`
|
|
1036
|
+
`:parse` `expected message value`, grounded stable+testing #247),
|
|
1037
|
+
`bad-sigil`,
|
|
1038
|
+
`invalid-hash`, `invalid-apostrophe` (#355, below),
|
|
1039
|
+
`missing-statement-separator` (#311, below). Nine classes
|
|
1040
|
+
the device itself rejects.
|
|
1041
|
+
- `warning` — `over-depth`, because it is centrs's own resource bound and says
|
|
1042
|
+
nothing about whether the input is legal; and an `ambiguous`/`unknown`
|
|
1043
|
+
resolution, never an error, so the default `--fail-on error` cannot fail a
|
|
1044
|
+
document whose only sin is being unreadable without a schema — which is most
|
|
1045
|
+
of RouterOS scripting.
|
|
1046
|
+
- `info` — `bom`/`non-ascii` (positional facts: a legal command must not
|
|
1047
|
+
fail), and `context-lost`, which reports a reading that is correct while the
|
|
1048
|
+
document's menu context was already gone.
|
|
1049
|
+
|
|
1050
|
+
### The missing statement separator (#311)
|
|
1051
|
+
|
|
1052
|
+
Two commands written with no `;` or newline between them used to read as one
|
|
1053
|
+
command with extra operands, and offline reported `pass`: `/ip/address add
|
|
1054
|
+
interface=ether1 /ip/route add gateway=192.168.88.1` published `gateway` as an
|
|
1055
|
+
attribute of `/ip/address add`. The rule that closes it is a known **menu path**
|
|
1056
|
+
in operand position followed by a console **verb** — never "a `/` after the
|
|
1057
|
+
verb", because a slash-shaped operand is legal and `/file remove
|
|
1058
|
+
/flash/skins/foo.html` lowers cleanly to `numbers=/flash/skins/foo.html`. The
|
|
1059
|
+
menu spelling may be slashed, spaced, or mixed (Q3 R4); the device rejects
|
|
1060
|
+
`/ip route add` exactly as it rejects `/ip/route add`.
|
|
1061
|
+
|
|
1062
|
+
**The device evidence is not what the report said, and the correction widened
|
|
1063
|
+
the rule.** #311 gave the error byte as the `=` of the trailing attribute
|
|
1064
|
+
(`expected end of command (line 1 column 55)`). That byte is an artefact of that
|
|
1065
|
+
head: `add` abbreviates `/ip/address/add`'s own `address=` argument, so the
|
|
1066
|
+
console reads `add gateway` as `address=gateway` and only complains at the `=`
|
|
1067
|
+
after it. Change the head so the second verb abbreviates nothing and the answer
|
|
1068
|
+
changes shape — `/system/note set note=hello /ip/route add gateway=1.2.3.4` is
|
|
1069
|
+
no hard reject at all; `:parse` returns IL carrying `bad parameter /ip/route
|
|
1070
|
+
(line 1 column 38)`, naming the second **path**. Both are refusals, and which
|
|
1071
|
+
one appears depends on per-menu argument names, which offline does not have. So
|
|
1072
|
+
the grounded claim is the negative one: across every probed spelling on CHR
|
|
1073
|
+
7.24.2 and 7.23.5, RouterOS answers `bad parameter <path>` or `expected end of
|
|
1074
|
+
command`, and never accepts the shape. A trailing attribute is therefore **not**
|
|
1075
|
+
required, which #311's own example had made it look like.
|
|
1076
|
+
|
|
1077
|
+
The head-dependence goes one step further than "which wording", and it bounds
|
|
1078
|
+
what the diagnostic may say. On a head whose own value slot takes a bare
|
|
1079
|
+
positional, the second path is absorbed into that slot and the device refuses
|
|
1080
|
+
the **verb** instead: `:put /ip/route print` lowers to `/putmessage=/ip/route`
|
|
1081
|
+
and answers `bad parameter print`, and `:local a /ip/route add` lowers to
|
|
1082
|
+
`/localname=$a;value=/ip/route` and answers `bad parameter add`. Both are still
|
|
1083
|
+
refusals — `/system/script/run /ip/route`, the same shape with no verb after
|
|
1084
|
+
the path, lowers cleanly to `number=/ip/route` — so the offline `error` stands.
|
|
1085
|
+
But the message says the bytes **read as** a second command, which is centrs's
|
|
1086
|
+
heuristic reading, and never that the device starts one there or which token it
|
|
1087
|
+
will name. Every row above is pinned in
|
|
1088
|
+
`test/integration/explain-separator.test.ts`.
|
|
1089
|
+
|
|
1090
|
+
**The contradicted reading is withdrawn, not patched.** The statement stays
|
|
1091
|
+
`resolved` — `/ip/address add` really is the head, and `highlight` agrees. What
|
|
1092
|
+
is withdrawn is withdrawn by three separate gates, each with its own trigger:
|
|
1093
|
+
|
|
1094
|
+
1. **The argument list.** `withoutSeparatedArguments` drops `command.args` and
|
|
1095
|
+
sets `arguments.read` to `false`. This is the same all-or-nothing move
|
|
1096
|
+
`enforceGateParity` makes for a gate/analysis disagreement, but it is a
|
|
1097
|
+
distinct gate on a distinct trigger — the two never consult each other. The
|
|
1098
|
+
device says the move is right rather than merely cautious:
|
|
1099
|
+
`… /ip/route add placeholder` lowers to
|
|
1100
|
+
`address=placeholder;;interface=ether1`, a phantom attribute the source never
|
|
1101
|
+
wrote, because the second run's words can abbreviate the HEAD's argument
|
|
1102
|
+
names.
|
|
1103
|
+
2. **The invocation.** `rejectedBySeparator(runs)` sets the new
|
|
1104
|
+
`ExplainTransportInput.rejected`, which `classifyExplainTransport` honours
|
|
1105
|
+
ahead of every mapping rule and answers `unknown`. Without it the degraded
|
|
1106
|
+
`arguments` would fall through to `execute` and print a ready-to-paste
|
|
1107
|
+
`centrs execute '<router>' '<the malformed text>'` for bytes the same result
|
|
1108
|
+
calls a syntax error. **Unread is not rejected** — an unread argument list
|
|
1109
|
+
still legitimately rides the raw CLI surface, and that path is unchanged.
|
|
1110
|
+
3. **The value facts from the run onward.** `values.occurrences` is an axis of
|
|
1111
|
+
its own, not a projection of `arguments` — the value census above pins how
|
|
1112
|
+
many values it retains in statements whose strict REST reading abstains — so
|
|
1113
|
+
gate 1 does not reach it and a cut is needed. The cut is the run's first byte, and
|
|
1114
|
+
the device draws the line in the same place: `:parse` reads `gateway` as the
|
|
1115
|
+
VALUE of `/ip/address/add`'s own `address=`, never as an attribute NAME,
|
|
1116
|
+
while the head's `interface=ether1` survives into the IL (as `note=hello`
|
|
1117
|
+
does in the `/system/note set` row). Values before the cut therefore stay;
|
|
1118
|
+
values reaching into or past it are dropped, and with them the `value` token
|
|
1119
|
+
class over those bytes.
|
|
1120
|
+
|
|
1121
|
+
**Abstentions, stated.** A menu path with no verb after it (`… /ip/route`) and a
|
|
1122
|
+
bare operand a menu does not accept (`… placeholder`, `/file remove a b`) also draw
|
|
1123
|
+
`bad parameter`, but that is the generic "no such positional" complaint;
|
|
1124
|
+
deciding it needs the per-menu positional list, which `catalog.ts` deliberately
|
|
1125
|
+
is not, and a separator is not necessarily the fix. Both abstain. The shape
|
|
1126
|
+
filter over a candidate path does admit a leading digit, because
|
|
1127
|
+
`/interface/6to4` is a real menu in both structure tables; presence in those
|
|
1128
|
+
tables, never the filter, is what decides. Over the 948-script pinned corpus the
|
|
1129
|
+
rule fires **0 times**, so no published census figure moves; the 38 corpus
|
|
1130
|
+
scripts the device answers `expected end of command` for are all other causes
|
|
1131
|
+
(line continuations, `<tab>`/`<VALUE>` placeholders, pasted shell and Python,
|
|
1132
|
+
stray parens), which is #203's genre bias showing.
|
|
1133
|
+
|
|
1134
|
+
**The reach limit is closed (#316).** This rule used to read
|
|
1135
|
+
`statement.arguments.tokens`, the all-or-nothing strict reading, so a statement
|
|
1136
|
+
holding a variable carried no tokens and the run in it was missed:
|
|
1137
|
+
`/ip/address add interface=ether1 /ip/route add gateway=$g` reported `pass`
|
|
1138
|
+
where its literal twin reported the separator. It now reads the skip-tolerant
|
|
1139
|
+
token stream described below, and both spellings report the separator — as does
|
|
1140
|
+
`/ip/address add interface=$x /ip/route add gateway=1.1.1.1`, where the
|
|
1141
|
+
undecodable token comes FIRST and a prefix-only reading would still have missed
|
|
1142
|
+
it. The corpus count is unchanged at **0** for the same reason it always was
|
|
1143
|
+
(#203's genre bias), so this buys reach rather than a number. What still stops
|
|
1144
|
+
the rule is an unterminated string or an unbalanced delimiter, because neither
|
|
1145
|
+
walk can find a boundary to resume from.
|
|
1146
|
+
|
|
1147
|
+
### The token stream a rule reads (#316)
|
|
1148
|
+
|
|
1149
|
+
`lexArguments` is all-or-nothing by design: its consumer renders a runnable REST
|
|
1150
|
+
request, and a partially-read argument list changes what the rendered command
|
|
1151
|
+
DOES. That is the right reading for a renderer and the wrong one for a rule —
|
|
1152
|
+
the first token it declines discards every token already decoded, which put
|
|
1153
|
+
**48.2%** of the corpus's argument-bearing statements out of reach of every
|
|
1154
|
+
token-level rule and of the `arg` token fill.
|
|
1155
|
+
|
|
1156
|
+
`lexArgumentTokens` is a third reading over the same walk — alongside strict
|
|
1157
|
+
`lexArguments` and the prefix-safe `lexValueAnchors` values already use. Same
|
|
1158
|
+
text, same `from`, the same boundary rules and the same refusal strings,
|
|
1159
|
+
differing only in what a refusal does. Where the strict reading discards the list, this one publishes the token
|
|
1160
|
+
it could not decode with its bytes located, `undecided` set to that same reason
|
|
1161
|
+
and **no `value`**, then carries on to the next token. `statement.arguments` is
|
|
1162
|
+
unchanged and still strict; nothing that renders a command gained reach.
|
|
1163
|
+
|
|
1164
|
+
**Skip-and-continue, measured against stop-at-first-refusal.** #316 named two
|
|
1165
|
+
designs and said the fork was load-bearing rather than cosmetic. It is. Both are
|
|
1166
|
+
priced by `bun run explain:arg-reach` over the 948-script pinned corpus, from
|
|
1167
|
+
one implementation — design A is the tolerant token list truncated at its first
|
|
1168
|
+
`undecided` token, which is exactly the prefix the strict walk discards:
|
|
1169
|
+
|
|
1170
|
+
| | strict | A — prefix | B — skip |
|
|
1171
|
+
| --- | ---: | ---: | ---: |
|
|
1172
|
+
| statements reached, of 14,267 | 7,385 (51.8%) | 10,754 (75.4%) | 14,267 (100.0%) |
|
|
1173
|
+
| argument tokens | 16,652 | 20,481 | 31,310 |
|
|
1174
|
+
| missing-separator runs found | 0 | 0 | 0 |
|
|
1175
|
+
|
|
1176
|
+
A further 58 statements sit outside every column because their bytes are not
|
|
1177
|
+
addressable (normalized, or a span widened to the enclosing statement); the
|
|
1178
|
+
sweep counts them rather than dropping them silently, since `explain.ts` refuses
|
|
1179
|
+
those too and lexing them here would measure a reading the product does not
|
|
1180
|
+
offer.
|
|
1181
|
+
|
|
1182
|
+
A recovers 3,369 statements; the 3,513 statements whose FIRST interesting token
|
|
1183
|
+
is the undecodable one have an empty prefix and are reached only by B — so the
|
|
1184
|
+
cheaper design leaves slightly more behind than it recovers. The separator row
|
|
1185
|
+
is 0 across all three and is reported rather than buried: on this corpus neither
|
|
1186
|
+
design finds a run the strict reading missed, which is what #203's genre bias
|
|
1187
|
+
predicts for a typo class published scripts do not carry. The sweep also asserts
|
|
1188
|
+
what makes the A column trustworthy — that the strict tokens are the tolerant
|
|
1189
|
+
walk's prefix byte for byte, and that both walks name the same refusal reason —
|
|
1190
|
+
and fails if any statement disagrees (0 do).
|
|
1191
|
+
|
|
1192
|
+
**Where both walks still stop.** An unterminated string and an unbalanced
|
|
1193
|
+
`(`/`[`/`{` have no knowable end, so there is no boundary to resume from. A `;`
|
|
1194
|
+
is a third and deliberately so: it is a statement boundary the segmenter owns,
|
|
1195
|
+
and resuming after it would read the next statement's bytes as this one's
|
|
1196
|
+
arguments. No corpus statement reaches any of the three, and that is a property
|
|
1197
|
+
of the pipeline rather than luck: an unterminated string and an unbalanced
|
|
1198
|
+
delimiter are document defects the segmenter and resolver answer first, so no
|
|
1199
|
+
addressable statement with an argument list is produced at all, and a `;` is
|
|
1200
|
+
split into two statements before any argument is lexed. The unbalanced case is
|
|
1201
|
+
also the one place the two readings name a different reason, and it follows from
|
|
1202
|
+
what each needed: the strict walk refuses a `[…]` on sight and never asks where
|
|
1203
|
+
it ends, while the tolerant walk asks — because the answer is what it would
|
|
1204
|
+
resume from — and says `an unclosed structured argument value` when there is
|
|
1205
|
+
none.
|
|
1206
|
+
|
|
1207
|
+
**What it moved, including the cost.** The `arg` fill now paints the `name=` run
|
|
1208
|
+
of a token whose value only the device knows, so classified bytes go 69.77% →
|
|
1209
|
+
**71.73%** (`arg` tokens 11,438 → 16,526). On the device-agreement slice the
|
|
1210
|
+
bytes where the device decided and centrs abstained drop 5,540 → 4,158, and two
|
|
1211
|
+
whole cells (`unclassified` → `arg-dot`, `unclassified` → `arg-scope`) empty
|
|
1212
|
+
out. Against that, projection disagreements rise 143 → 360, and **every one of
|
|
1213
|
+
the 217 new bytes is the `=` of an attribute**: `arg` → `syntax-meta`, one cell,
|
|
1214
|
+
358 runs. That is not a new wrong assertion. It is #293's provisional
|
|
1215
|
+
vocabulary — one `arg` class for both the name bytes and the `=` — meeting the
|
|
1216
|
+
device's context-dependent answer over 2.5x as many bytes as before, and it
|
|
1217
|
+
gave #264 B5 a concrete, counted reason to split the `=` into its own class.
|
|
1218
|
+
It did, and the relabel moved no byte coverage at all: disagreements fell
|
|
1219
|
+
360 → 2 and the classified-byte census did not change by a digit.
|
|
1220
|
+
|
|
1221
|
+
**The one residue, sized so it is not mistaken for a defect.** A `\<newline>`
|
|
1222
|
+
continuation inside an argument leaves its token `undecided`, because the bytes
|
|
1223
|
+
of a continued value are not contiguous and a `valueSpan` over them would be a
|
|
1224
|
+
lie (`args.ts` → *What it refuses*). The conservative case is a token whose
|
|
1225
|
+
value is already complete BEFORE the continuation, which is then withheld
|
|
1226
|
+
needlessly. Measured on the 948-script corpus: 109 statements make the strict
|
|
1227
|
+
lexer refuse for this reason, the tolerant walk reaches 114 statements carrying
|
|
1228
|
+
138 such tokens, and **all 138 put the continuation immediately after the `=`**
|
|
1229
|
+
— the value really does live on the next line. Exactly **one** token
|
|
1230
|
+
(`mode=none\<newline>`) has a complete value in front of its continuation. So
|
|
1231
|
+
narrowing the rule would recover one token, and deciding it correctly needs the
|
|
1232
|
+
device-grounded answer for what a continuation does to a value's INTERIOR, which
|
|
1233
|
+
is #225's probe-matrix territory rather than a lexer tweak. Recorded there; not
|
|
1234
|
+
carried as pending work here.
|
|
1235
|
+
|
|
1236
|
+
**What the public surface gained, and what it did not.** The package root
|
|
1237
|
+
exports the reading itself — `lexExplainArgumentTokens` and
|
|
1238
|
+
`ExplainArgumentTokenReading` — for the same reason `lexExplainValueAnchors` is
|
|
1239
|
+
exported: a consumer building its own token-level rule needs the reading a rule
|
|
1240
|
+
wants, not the one a renderer wants. What did NOT change is the *envelope*:
|
|
1241
|
+
`explainCommand`'s `structure.statements[].arguments` is still the strict
|
|
1242
|
+
reading, `ExplainArgumentToken` carries no `undecided` field, and `undecided`
|
|
1243
|
+
never appears on an `ExplainArguments` reading, because the strict lexer
|
|
1244
|
+
publishes no token it could not decide. Whether the tolerant stream should reach
|
|
1245
|
+
the envelope in its own right — so a browser consumer reading `explainCommand`'s
|
|
1246
|
+
result can see a located `gateway=` whose value is a runtime expression — was
|
|
1247
|
+
left to #264 B5, which decided it reaches the envelope **as the token
|
|
1248
|
+
partition** and nowhere else: naming the `=` `arg-sep` locates the argument
|
|
1249
|
+
without loosening `arguments`, and `undecided` stays off the envelope. See
|
|
1250
|
+
[The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5).
|
|
1251
|
+
|
|
1252
|
+
### The menu path with arguments (#324)
|
|
1253
|
+
|
|
1254
|
+
A run that is nothing but a **known menu**, with arguments after it and no
|
|
1255
|
+
verb, used to come back `resolved` with a verb the analyzer made up.
|
|
1256
|
+
`/ip/firewall/address-list name=ytkids timeout=1h` reported path
|
|
1257
|
+
`/ip/firewall`, verb `address-list`; the spaced spelling
|
|
1258
|
+
`/ip firewall address-list name=x` promoted `firewall`, one level higher
|
|
1259
|
+
still. The punctuation half of the ratified Q6 rule is what fires — "no space
|
|
1260
|
+
token, arguments follow, so the last run token is the verb", or "the first
|
|
1261
|
+
space token is the verb" — and neither premise it needs was ever checked:
|
|
1262
|
+
|
|
1263
|
+
```text
|
|
1264
|
+
isKnownMenuPath(/ip/firewall/address-list) -> true (it is a menu)
|
|
1265
|
+
VERBS.has("address-list") -> false (it is not a verb)
|
|
1266
|
+
```
|
|
1267
|
+
|
|
1268
|
+
This is the **symmetric half of R9**. R9 (#211 B1) made `pathresolve`'s
|
|
1269
|
+
`menuNavPath` consult `VERBS` so it could not claim navigation where
|
|
1270
|
+
`verbsplit` had decided a verb; this makes `resolveVerb` consult the menu
|
|
1271
|
+
tables so it cannot claim a verb where the whole run names a menu and the
|
|
1272
|
+
segment it would promote is not a verb. The reading degrades to **`ambiguous`**
|
|
1273
|
+
with the standard located `explain/canonicalizer/ambiguous-statement` warning,
|
|
1274
|
+
naming both premises. The adjacent readings are untouched —
|
|
1275
|
+
`/ip/firewall/address-list add name=x` resolves, the bare path is navigation,
|
|
1276
|
+
`/system/reboot` is a command — which is what makes this a seam rather than a
|
|
1277
|
+
parser limit.
|
|
1278
|
+
|
|
1279
|
+
**It abstains; it does not reject.** Offline knows it cannot name a verb here,
|
|
1280
|
+
and that needs no device evidence. Whether RouterOS refuses the shape outright
|
|
1281
|
+
is a *device* claim, so unlike #311's CHR-grounded `error` this stays a
|
|
1282
|
+
warning and the statement carries no verdict about acceptance.
|
|
1283
|
+
|
|
1284
|
+
**Four exclusions, each for its own reason.** A frozen-vocabulary hit is
|
|
1285
|
+
excluded by the `VERBS` premise itself (a vocabulary split promotes a `VERBS`
|
|
1286
|
+
member by construction). A **published command** is excluded by `catalogAt`:
|
|
1287
|
+
first-order evidence about that exact path outranks a container listing of its
|
|
1288
|
+
prefix, which is what keeps `/system/gps/monitor once` resolved. A scripting
|
|
1289
|
+
**directive** is excluded by R10, which is measured. And a `{` after the run is
|
|
1290
|
+
excluded by shape: that is a menu *container* body (`/system {identity print}`,
|
|
1291
|
+
H7) where the run legitimately names a menu and no verb is missing —
|
|
1292
|
+
abstaining there would drop the `invalid-command-brace` error the statement
|
|
1293
|
+
after it carries.
|
|
1294
|
+
|
|
1295
|
+
**Priced before landing, as #211 B2 asks.** Over the frozen dev/holdout
|
|
1296
|
+
partition (948-script corpus, 17,212 statements) **10 statements flip, all
|
|
1297
|
+
`resolved` → `ambiguous`**: 1 dev, 9 holdout. **None is a genuine no-verb
|
|
1298
|
+
command.** Three are real RouterOS missing a verb —
|
|
1299
|
+
`/ip/firewall/address-list name=ytkids …`, `/interface bridge [find]
|
|
1300
|
+
vlan-filtering=yes`, `/user-manager user [find …] attributes=…` — and seven
|
|
1301
|
+
are pasted non-RouterOS prose: five Python traceback `File "…"` lines,
|
|
1302
|
+
`User ID: 506`, and `port 80`. Those seven reach the rule at all because
|
|
1303
|
+
`/file`, `/user` and `/port` are real menus and the table lookup folds case,
|
|
1304
|
+
which is #203's genre bias arriving through a second door. Every one of the
|
|
1305
|
+
ten was a fabrication, so the abstention has no measured false-positive cost
|
|
1306
|
+
on this corpus.
|
|
1307
|
+
|
|
1308
|
+
**The execute gate is unmoved**, as the constitution requires: `canonical`
|
|
1309
|
+
still reports `mode: "structured"`, path `/ip/firewall`, verb `address-list`
|
|
1310
|
+
for this input, because that is what `canonicalizeExecuteCommand` decides and
|
|
1311
|
+
an explain-only abstention may not reinterpret it. `enforceGateParity` is
|
|
1312
|
+
about the argument *list* of a `kind: "command"` reading, so it does not fire
|
|
1313
|
+
on a statement that carries no reading at all — the gate and the analysis are
|
|
1314
|
+
two answers to two questions, and only one of them changed.
|
|
1315
|
+
|
|
1316
|
+
The invariant this closes — `pathresolve` and `verbsplit` never contradict
|
|
1317
|
+
each other on one statement — is asserted in
|
|
1318
|
+
`test/unit/explain-pathresolve.test.ts` beside R9's own cases. Presence in the
|
|
1319
|
+
shipped structure tables still narrows a reading and absence still decides
|
|
1320
|
+
nothing (#207/#228), so an unlisted menu spelled `…/address-list` is read
|
|
1321
|
+
exactly as before.
|
|
1322
|
+
|
|
1323
|
+
### Two spellings the device accepts and offline used to reject (#347, #348)
|
|
1324
|
+
|
|
1325
|
+
Both were found the same way, and the method is worth keeping: run the analyzer
|
|
1326
|
+
over all 948 corpus scripts and compare its verdict to that build's own `:parse`
|
|
1327
|
+
verdict, which the corpus already stores. The number that matters is the **false
|
|
1328
|
+
reject** — a script the device ACCEPTS that offline analysis fails — because
|
|
1329
|
+
that is validation blocking input RouterOS is happy with. It was 4, identical on
|
|
1330
|
+
7.23.5, 7.24.2 and 7.25beta3; it is now 1.
|
|
1331
|
+
|
|
1332
|
+
The converse count is not a parser score and is not quoted as one: the corpus is
|
|
1333
|
+
whole forum posts including prose output, so most of what the device rejects and
|
|
1334
|
+
offline passes is #203's genre bias rather than a gap.
|
|
1335
|
+
|
|
1336
|
+
**The `=` is optional on a scope slot (#347).** `:if (…) do={…} else={…}`,
|
|
1337
|
+
`:if (…) do={…} else {…}` and `:if (…) do {…} else {…}` are the same construct:
|
|
1338
|
+
`:parse` lowers all three to byte-identical IL (`…;do=;(evl …);else=;(evl …)`)
|
|
1339
|
+
on CHR 7.21.5, 7.23.5, 7.24.2 and 7.25beta3. The MikroTik manual documents only
|
|
1340
|
+
the `else={…}` spelling, so this is a real-but-undocumented form. The rule is
|
|
1341
|
+
deliberately narrower than "any word before a brace": only the four
|
|
1342
|
+
`SCOPE_ARG_NAMES` members may drop the `=`, and the word may not be the
|
|
1343
|
+
statement's own leading directive — otherwise `:local z {1;2}` would stop
|
|
1344
|
+
reading its brace as an array, and `:do {` would have its slot renamed away from
|
|
1345
|
+
the `command` the IL uses. A COLON also disqualifies the word: the device parses
|
|
1346
|
+
`:if (1=1) do {…}` but refuses `:if (1=1) :do {…}` with `expected end of command`
|
|
1347
|
+
on 7.21.5 and 7.24.2, because a colon makes it a directive and a directive
|
|
1348
|
+
cannot occupy an argument slot. The control is `elsy {`, rejected on all four
|
|
1349
|
+
builds, which is what makes the acceptance a device fact rather than a probe
|
|
1350
|
+
artefact.
|
|
1351
|
+
|
|
1352
|
+
**A relative menu-scope block composes the path (#348).** `/ip { address {
|
|
1353
|
+
print } }` is `(evl (evl (evl /ip/address/print)))` on device and
|
|
1354
|
+
`/interface { ethernet { print } }` the same way; the ABSOLUTE spelling
|
|
1355
|
+
`/ip/address { print }` already passed offline, and only the nested relative
|
|
1356
|
+
form failed. A menu block's body is a statement list, so it may open with a
|
|
1357
|
+
submenu, with one of the thirteen frozen CRUD verbs (`verbs.ts`), with a
|
|
1358
|
+
catalog-named command, with a `:` scripting directive, or with an absolute
|
|
1359
|
+
command — corpus 521 is the directive case, `ethernet { :put … }`.
|
|
1360
|
+
|
|
1361
|
+
That last rule is bounded by what a STATEMENT can start with, which is what
|
|
1362
|
+
keeps `{1;2}` an array literal: an array member opens with a digit or a quote
|
|
1363
|
+
and never reaches the identifier branch. It is also deliberately not widened
|
|
1364
|
+
into a claim of existence. Menu-scope resolution is schema-dependent —
|
|
1365
|
+
`/ip { zzznotamenu { print } }` is a `syntax error` on device — and offline has
|
|
1366
|
+
no schema, so a catalog MISS abstains rather than being decided either way.
|
|
1367
|
+
That is the catalog's own "a hit is decisive, a miss says nothing" contract, and
|
|
1368
|
+
turning such a miss into an acceptance would be the mirror-image defect of the
|
|
1369
|
+
one being fixed here.
|
|
1370
|
+
|
|
1371
|
+
### `'` is not a RouterOS token (#355)
|
|
1372
|
+
|
|
1373
|
+
`'` (U+0027) is not a string delimiter and not a legal bare byte. Only `"` opens
|
|
1374
|
+
a string. `:parse` rejects every spelling of a bare apostrophe byte-identically
|
|
1375
|
+
on CHR 7.21.5, 7.23.5, 7.24.2 and 7.25beta3:
|
|
1376
|
+
|
|
1377
|
+
| input | device `:parse` | offline span |
|
|
1378
|
+
| --- | --- | --- |
|
|
1379
|
+
| `/system/identity/set name=don't` | `expected end of command (line 1 column 30)` | 29 |
|
|
1380
|
+
| `/system/identity/set name=don't'` | `expected end of command (line 1 column 30)` | 29 |
|
|
1381
|
+
| `/system/identity/set name='abc'` | `syntax error (line 1 column 27)` | 26 |
|
|
1382
|
+
| `:put 'abc'` | `syntax error (line 1 column 6)` | 5 |
|
|
1383
|
+
|
|
1384
|
+
The device's column is 1-based, so each offline span is the device's own byte.
|
|
1385
|
+
The accepting controls — `comment="it's here"` and a `# don't` comment line —
|
|
1386
|
+
parse cleanly on all four builds, which is what makes the rejection a device
|
|
1387
|
+
fact rather than a probe artefact. Rows 2–4 carry BALANCED apostrophes and are
|
|
1388
|
+
rejected anyway: this is a lexical rule, not a quote-balance one, and the remedy
|
|
1389
|
+
is always `"`, never a matching `'`.
|
|
1390
|
+
|
|
1391
|
+
Only the FIRST apostrophe is recorded, **document-wide**, the same way the hard
|
|
1392
|
+
`#` is: the device reports one column and it is the first one (row 2 reports
|
|
1393
|
+
column 30, not the trailing apostrophe), because classification stops there. A
|
|
1394
|
+
nested scope body is a separate scan (`segmentScopeBody`), so per-scan
|
|
1395
|
+
suppression is not enough on its own and the collapse to the earliest occurrence
|
|
1396
|
+
happens where the diagnostics are composed — which keeps each segmenter call
|
|
1397
|
+
pure. Apostrophes inside a `"`-string or a comment never reach the rule at all —
|
|
1398
|
+
both runs are consumed whole by the scan before it — so no special case is
|
|
1399
|
+
needed for either.
|
|
1400
|
+
|
|
1401
|
+
Symbol resolution stops at the apostrophe too, exactly as it stops at a
|
|
1402
|
+
malformed escape: the device classes nothing after a hard reject, so
|
|
1403
|
+
`:put 'abc'; :put $after` must not publish an occurrence for `$after`.
|
|
1404
|
+
Occurrences BEFORE the defect stand, which is the lab's X1 rule.
|
|
1405
|
+
|
|
1406
|
+
**This was a false ACCEPT, which the corpus join cannot see.** The false-reject
|
|
1407
|
+
measurement scores one direction only, so the analyzer passing all four forms
|
|
1408
|
+
with zero diagnostics never appeared in it. Over the pinned corpus the rule
|
|
1409
|
+
fires on 40 of the 7.24.2 scripts and **every one of them is a script the device
|
|
1410
|
+
rejects** — none of the 617 it accepts. It moves the false-reject count not at
|
|
1411
|
+
all (still 1, the extra `}` of corpus 228) while taking the caught count from 62
|
|
1412
|
+
to 93.
|
|
1413
|
+
|
|
1414
|
+
The token census moves, and only in the direction the fail-closed symbol stop
|
|
1415
|
+
predicts: 68 fewer `variable-*` tokens and their 306 bytes returning to
|
|
1416
|
+
`unclassified` (classified 1,023,413 → 1,023,107). Those are variable references
|
|
1417
|
+
sitting after a bare apostrophe, which the device never reaches and centrs
|
|
1418
|
+
therefore stops naming. The value, operator and bracket-equals censuses and
|
|
1419
|
+
`explain:arg-reach` are all unchanged.
|
|
1420
|
+
|
|
1421
|
+
**It also retires a wrong rule.** `execute` carried a private
|
|
1422
|
+
`hasUnbalancedQuotes` preflight that modelled `'` as a delimiter, moved into the
|
|
1423
|
+
offline gate by #354. It had no comment awareness, so `# don't` opened a quote
|
|
1424
|
+
it never closed: over the same corpus it false-rejected **51** of the 617
|
|
1425
|
+
device-accepted scripts. Deleting it costs 21 catches — all of them scripts the
|
|
1426
|
+
device rejects anyway, where the device stage is still the backstop — and takes
|
|
1427
|
+
the gate's false rejects from 52 to 1. A false reject blocks input RouterOS
|
|
1428
|
+
accepts, which is the figure that matters.
|
|
1429
|
+
|
|
1430
|
+
### String escape validation (#247, #252)
|
|
1431
|
+
|
|
1432
|
+
The accepted set is the **device's**, not the manual's. The
|
|
1433
|
+
[Scripting manual's constant-escape table](https://manual.mikrotik.com/docs/developer-guides/scripting/#constant-escape-sequences)
|
|
1434
|
+
is a *lower bound*: it omits `\?`, and it documents the whitespace continuation
|
|
1435
|
+
only under a separate "Line joining" section. Treating that table as a closed
|
|
1436
|
+
allow-list is what #251 did and what #252 had to undo — it made 44 device-valid
|
|
1437
|
+
corpus scripts read `fail`. The set below comes from sweeping `:put "\<c>"` over
|
|
1438
|
+
every byte `0x20`–`0x7E` plus TAB/LF/CR/CRLF on CHR 7.23.3, scored on both the
|
|
1439
|
+
`/console/inspect request=highlight` class and the runtime result:
|
|
1440
|
+
|
|
1441
|
+
- `\"` `\\` `\n` `\r` `\t` `\$` `\_` `\?` `\a` `\b` `\f` `\v`
|
|
1442
|
+
- `\XX` where `XX` is **uppercase** hex
|
|
1443
|
+
- `\` followed by whitespace (space, TAB, CR, LF, CRLF) — a **line
|
|
1444
|
+
continuation**, valid inside a string and not just in code. The device
|
|
1445
|
+
swallows the pair and emits nothing (`:put "a\ b"` → `ab`). CRLF counts as one
|
|
1446
|
+
continuation, so it is the only non-hex three-byte escape.
|
|
1447
|
+
|
|
1448
|
+
Everything else is rejected, including `\` before a non-ASCII byte. `\$` is
|
|
1449
|
+
grounded on `highlight` (`escaped,escaped`, no `error` on CHR 7.23.3/7.24rc3)
|
|
1450
|
+
and the manual; the `:parse` wrapper (`:put [:parse "..."]`) is not used for
|
|
1451
|
+
that escape because outer-string escaping makes the inner `\$` appear as `$` to
|
|
1452
|
+
the parser — for the same reason #252's rows are grounded by executing the input
|
|
1453
|
+
directly rather than through the wrapper. Grounded on CHR 7.23.3 stable and
|
|
1454
|
+
7.24rc3 testing (`highlight` hard `error` + `:parse` `expected message value`
|
|
1455
|
+
for the three issue rows `:put "\q"` / `:put "\x0a"` / `:put "\0a"` vs
|
|
1456
|
+
`":put "\0A"` valid):
|
|
1457
|
+
|
|
1458
|
+
- A malformed escape (`\q`, `\x`, `\x0a`, truncated single-hex `\0`, lowercase
|
|
1459
|
+
hex `\0a`/`\5f`) is a located `bad-string-escape` structural defect
|
|
1460
|
+
(severity `error`) at the byte the device marks `error` — the unknown char or
|
|
1461
|
+
the lowercase second hex digit. For a truncated single-hex escape the device
|
|
1462
|
+
marks the closing quote; centrs reports at the backslash instead, because that
|
|
1463
|
+
byte is stable when the string is unterminated.
|
|
1464
|
+
- `\ff` is `\f` + literal `f` and `\aF` is `\a` + `F`; neither is invalid hex.
|
|
1465
|
+
- The one shared `scanQuotedString` below every walker still **recovers the
|
|
1466
|
+
closing quote** past a malformed escape, so no #199 nested-substitution
|
|
1467
|
+
quote-phase bug is reintroduced. Validation is a shared
|
|
1468
|
+
`collectStringEscapeDefects` walk with the same substitution frames; only the
|
|
1469
|
+
first invalid escape per document is reported and it fails closed after it.
|
|
1470
|
+
- Corpus blast radius (948 scripts): **4** `bad-string-escape` diagnostics (vs
|
|
1471
|
+
15 code-position `bad-escape`), and the device rejects all 4. Under #251's
|
|
1472
|
+
closed allow-list this was 62 flags of which **44 were false positives on
|
|
1473
|
+
valid RouterOS** — mostly multi-line `source=`/`on-event=` strings (`\`+LF)
|
|
1474
|
+
and `\?`.
|
|
1475
|
+
- Scoring a defect rule against the corpus needs the right oracle.
|
|
1476
|
+
`parseil_results.ok` is **not** one: `:parse` never throws, it returns a
|
|
1477
|
+
diagnostic *value*, so the capture records `ok=1 / status="ok"` for scripts
|
|
1478
|
+
the device rejected (only 5 of 4635 rows are `ok=0`, and those are capture
|
|
1479
|
+
failures — one `edge-cases/oversize-32k.rsc` REST `413` per captured
|
|
1480
|
+
version). The verdict is in `il_text` — plain prose such as
|
|
1481
|
+
`expected input value (line 7 column 49)` means rejected, IL means accepted.
|
|
1482
|
+
|
|
1483
|
+
### The operator surface (#255)
|
|
1484
|
+
|
|
1485
|
+
Grounded, not transcribed, for the same reason the escape set above is: the
|
|
1486
|
+
[manual's operator list](https://manual.mikrotik.com/docs/developer-guides/scripting/index.md#operators)
|
|
1487
|
+
is a lower bound. `bun run explain:probe:operators` sweeps the manual's list
|
|
1488
|
+
**plus** plausible non-operators (`not`, `..`, `xor`, `mod`, `is`, `div`, …)
|
|
1489
|
+
**plus** every IL head the corpus census saw, on CHR 7.24.2 stable, 7.23.5
|
|
1490
|
+
long-term and 7.25beta3 development — the same three builds the pinned corpus
|
|
1491
|
+
classes `complete`, so the operator axis and the token partition are finally
|
|
1492
|
+
scored on the same firmware (#342) — plus 7.21.5 as an **age anchor**, which is
|
|
1493
|
+
not a channel claim but the only live evidence that the surface behaves the same
|
|
1494
|
+
on a pre-7.23 branch. Re-cut the fixture with
|
|
1495
|
+
`bun run explain:operator-slice`; the table
|
|
1496
|
+
below is generated from `src/explain/operators.ts` by
|
|
1497
|
+
`bun run explain:operator-readme` and gated by
|
|
1498
|
+
`bun run explain:operator-readme:check`.
|
|
1499
|
+
|
|
1500
|
+
**The census is a candidate generator, and boilerplate defeats its own
|
|
1501
|
+
tie-breaker.** `scripts/explain-operator-census.ts` reports every head-position
|
|
1502
|
+
token in the device IL, tagged by shape, because IL is a debug rendering that
|
|
1503
|
+
does not quote strings: a string literal is emitted bare, so string content is
|
|
1504
|
+
indistinguishable from a node head. Its stated honesty rule is that
|
|
1505
|
+
`distinctScripts` beats frequency — "a head seen in one script is one author's
|
|
1506
|
+
string, a head seen in three hundred is a language feature". Only the complete
|
|
1507
|
+
builds carry parseIL for the 35-script `tangentsoft` device-`/export` stratum,
|
|
1508
|
+
and that stratum showed the rule has a blind spot, because a stock string
|
|
1509
|
+
replicated across a whole stratum of device exports looks exactly like a
|
|
1510
|
+
language feature by script count:
|
|
1511
|
+
|
|
1512
|
+
| word-shaped head | scripts, partial builds → complete | what it actually is |
|
|
1513
|
+
| ---------------- | ---------------------------------: | ------------------- |
|
|
1514
|
+
| `for` | 2 → 22 | the defconf comment `accept to local loopback (for CAPsMAN)` — in 20 of the 35 device exports, plus the 2 forum scripts that already carried it |
|
|
1515
|
+
| `line` | 53 → 84 | device diagnostic prose, `bad parameter hw-offload (line 33 column 147)` |
|
|
1516
|
+
|
|
1517
|
+
Those arrows are a **coverage** step, not a version step: the left column is
|
|
1518
|
+
what a 913-script capture could see, the right what a 948-script one sees.
|
|
1519
|
+
|
|
1520
|
+
Neither is an operator, and both are `word`-shaped — the same bucket #255's
|
|
1521
|
+
`not` would have appeared in. So the sweep, not the census, still decides; a
|
|
1522
|
+
head that arrives with a whole stratum at once is boilerplate until the device
|
|
1523
|
+
says otherwise. The two heads that stratum added outright,
|
|
1524
|
+
`configuration.mode=` and `[system`, are `other`-shaped and are the documented
|
|
1525
|
+
string-noise class: a dotted wifi property name emitted bare, and text inside
|
|
1526
|
+
an embedded `/system script add source=` value.
|
|
1527
|
+
|
|
1528
|
+
**The census counts only the complete builds; a separate axis dates heads.**
|
|
1529
|
+
Occurrences are summed across versions, so admitting a partial capture would
|
|
1530
|
+
scale every figure by what happened to be captured — on this snapshot, letting
|
|
1531
|
+
all six builds in inflated the headline by ~20% while `distinctScripts`, the
|
|
1532
|
+
column that actually carries the argument, did not move at all. So
|
|
1533
|
+
`headOccurrences` / `headVersions` / `headArities` read
|
|
1534
|
+
`coverage_class = 'complete'` only, and the census states its version set in
|
|
1535
|
+
`versions` so a gate cannot silently compare two different universes.
|
|
1536
|
+
|
|
1537
|
+
Presence, unlike frequency, *is* monotone in coverage — a 913-of-948 capture
|
|
1538
|
+
that shows a head proves that build had it — so the partial builds keep one
|
|
1539
|
+
job: `headFirstSeen` dates each head against every captured build. That is what
|
|
1540
|
+
still grounds the `any` claim below at 7.20.8, a build the counting side no
|
|
1541
|
+
longer reads. Read a *late* first-seen as "not captured before here", never as
|
|
1542
|
+
"new in": the only two heads that do not reach 7.20.8 are the `tangentsoft`
|
|
1543
|
+
pair above, which no partial build could have seen.
|
|
1544
|
+
|
|
1545
|
+
**`:parse` IL is the oracle, and `highlight` cannot be.** IL is prefix form with
|
|
1546
|
+
the operator as its node's head, so it names the operator and shows its
|
|
1547
|
+
operands. `highlight`'s `syntax-meta` is the device's *residual structure*
|
|
1548
|
+
class — measured over `test/fixtures/explain/highlight-streams.slice.json` it
|
|
1549
|
+
covers `=`, `"`, `$`, brackets, braces, parens, `;`, `,`, `/` and whitespace
|
|
1550
|
+
runs as well as `||`, `&&`, `!=`, `.` and `~` — and adjacent runs are **merged**
|
|
1551
|
+
(`="`, `($`, `")+` each arrive as one token; the run around `and`, `or` and `in`
|
|
1552
|
+
swallows the preceding space). An operator boundary is not recoverable from it.
|
|
1553
|
+
What it does decide is *structure versus word*, which is what rules `not` out:
|
|
1554
|
+
it comes back `variable-undefined`.
|
|
1555
|
+
|
|
1556
|
+
Five things the device says that the manual does not:
|
|
1557
|
+
|
|
1558
|
+
1. **`not` is not an operator.** Nor are `xor`, `mod`, `is`, `div`, `band`,
|
|
1559
|
+
`bor`, `shl`, `shr`, `eq`, `ne`. Inside a paren group a bare word is a
|
|
1560
|
+
**variable reference**, so `(1 not 2)` parses — as `( 1 $not 2)`, with an
|
|
1561
|
+
unnamed juxtaposition node. "It parsed" is therefore not evidence of an
|
|
1562
|
+
operator, which is what the sweep's `(1 zzz 2)` control exists to prove.
|
|
1563
|
+
2. **`..` is not a range operator** — `(1 .. 2)` is `( (. 1 $.) 2)`, concat
|
|
1564
|
+
applied to a variable named `.`. Same for `//`. And `<>` is not "not equal":
|
|
1565
|
+
`(1 <> 2)` is `(< 1 (> 2))`, less-than applied to a *deferred* `2`.
|
|
1566
|
+
3. **`$`, `[`, `]` are syntax, not operator heads.** `$x` stays an atom in the
|
|
1567
|
+
IL and `[:tostr 1]` lowers to an `evl` node. Those bytes belong to the
|
|
1568
|
+
substitution axis.
|
|
1569
|
+
4. **`any` is an operator** (prefix, arity 1) and is not in the manual's list — a **nil-check**: `:typeof (any x)` is `bool`, `false` only for `nil`/`nothing` (the value of an undefined `:local` and of `[:nothing]`), `true` for everything else including `0`, `""` and `false`. It is the idiom `:if (any $x) ...` to test a variable that may be `nil`; `(true any false)` is not infix at all but juxtaposition `( true (any false))`, and `(1 . any [:nothing])` is concat `1`+`false`. Present since at least 7.20.8 (corpus `headFirstSeen[any] = 7.20.8`, the oldest captured build, and swept live on 7.21.5 and 7.23.5 long-term, 7.24.2 stable and 7.25beta3 development with no difference between them).
|
|
1570
|
+
5. **`&&` and `||` are spellings**, lowering to the `and` and `or` nodes.
|
|
1571
|
+
|
|
1572
|
+
The `(>…)` and `<%%` forms are in the table on the same footing as `+`.
|
|
1573
|
+
`(> x)` at arity 1 is the deferred-expression form — `[:typeof (>[:return 1])]`
|
|
1574
|
+
is `op` — while `(2 > 1)` at arity 2 is the comparison; **arity is the only
|
|
1575
|
+
thing that separates them**, so a table keyed on spelling alone gets one wrong.
|
|
1576
|
+
`<%%` applies a deferred expression to an argument array, binding positionals
|
|
1577
|
+
from `$0`; a `do={…}` function binds the same arguments from `$1`, because
|
|
1578
|
+
there `$0` is the function's own name.
|
|
1579
|
+
|
|
1580
|
+
Both `!` and `any` are prefix-only, so the *pair* sweep cannot carry them and
|
|
1581
|
+
their `precedence` is honestly `null`. That is "unmeasured", not "unknown": the
|
|
1582
|
+
sweep asks them separately, every `(U 1 B 2)` and `(1 B U 2)` for
|
|
1583
|
+
`U`∈`!`,`any`,`~`,`-`,`>` against all 24 binaries — 240 probes, recorded in the
|
|
1584
|
+
fixture's `unary` block. All 240 are accepted and the **binary is outer in every
|
|
1585
|
+
one**, on all four swept builds, so each prefix operator binds tighter than every
|
|
1586
|
+
binary including `->` (14) and `<%%` (13). The table stores one `precedence` per
|
|
1587
|
+
spelling, which is the binary level; `~`, `-` and `>` are here because their
|
|
1588
|
+
unary reading has no other record.
|
|
1589
|
+
|
|
1590
|
+
Spacing decides the *tokens*, not just the tree. `(1.2)` is an IP literal and
|
|
1591
|
+
`(.1)` a time literal, but the rest of the matrix is where a tokenizer goes
|
|
1592
|
+
wrong: `(1 . 2)` and `(1 .2)` are both concat (`(. 1 2)`), while **`(1. 2)` is
|
|
1593
|
+
not concat at all** — `1.` lexes as a *variable name* and the row comes back as
|
|
1594
|
+
juxtaposition, `( $1. 2)`. A rule that claims every `.` byte for the operator
|
|
1595
|
+
emits a span the device does not have.
|
|
1596
|
+
|
|
1597
|
+
Everything in the sweep is identical across all four builds except one runtime
|
|
1598
|
+
row: `:put ({2;1} > {1;2;3})` evaluates to `true` on 7.24.2 and 7.25beta3 and
|
|
1599
|
+
errors on both long-term builds. **Array comparison is not fixed on current
|
|
1600
|
+
long-term** — 7.23.5 is newer than the 7.23.3 this was first measured on and
|
|
1601
|
+
still errors — so the change rode the 7.24 line and was not backported. The two
|
|
1602
|
+
errors also differ: 7.23.5 says `cannot compare if array is more than array`,
|
|
1603
|
+
7.21.5 says `cannot compare if nothing is more than nothing`, so on 7.21.5 the
|
|
1604
|
+
operands did not survive to the comparison at all.
|
|
1605
|
+
Identical is measured, not assumed — the slice diffs verdict, arity, precedence,
|
|
1606
|
+
associativity, `highlight` run, unary placement and op-axis per version. So the
|
|
1607
|
+
operator table needs no version gate; anything reporting what a comparison
|
|
1608
|
+
*means* does.
|
|
1609
|
+
|
|
1610
|
+
<!-- BEGIN GENERATED operator-table — regenerate with `bun run explain:operator-readme` -->
|
|
1611
|
+
The device builds a node for **26** spellings, reads **3**
|
|
1612
|
+
as something else, and refuses the other **33** the sweep asked about.
|
|
1613
|
+
|
|
1614
|
+
| operator | arity | precedence | associativity | category |
|
|
1615
|
+
| -------- | ----- | ---------- | ------------- | -------- |
|
|
1616
|
+
| `,` | 2 | 1 | variadic | concatenation |
|
|
1617
|
+
| `or` | 2 | 2 | variadic | logical |
|
|
1618
|
+
| `and` | 2 | 3 | variadic | logical |
|
|
1619
|
+
| `in` | 2 | 4 | left | logical |
|
|
1620
|
+
| `<` | 2 | 5 | left | relational |
|
|
1621
|
+
| `>` | 1, 2 | 5 | left | relational |
|
|
1622
|
+
| `=` | 2 | 5 | left | relational |
|
|
1623
|
+
| `<=` | 2 | 5 | left | relational |
|
|
1624
|
+
| `>=` | 2 | 5 | left | relational |
|
|
1625
|
+
| `!=` | 2 | 5 | left | relational |
|
|
1626
|
+
| `~` | 1, 2 | 5 | left | relational |
|
|
1627
|
+
| `.` | 2 | 6 | variadic | concatenation |
|
|
1628
|
+
| `<<` | 2 | 7 | right | bitwise |
|
|
1629
|
+
| `>>` | 2 | 7 | right | bitwise |
|
|
1630
|
+
| `\|` | 2 | 8 | variadic | bitwise |
|
|
1631
|
+
| `^` | 2 | 9 | variadic | bitwise |
|
|
1632
|
+
| `&` | 2 | 10 | variadic | bitwise |
|
|
1633
|
+
| `+` | 2 | 11 | variadic | arithmetic |
|
|
1634
|
+
| `-` | 1, 2 | 11 | left | arithmetic |
|
|
1635
|
+
| `*` | 2 | 12 | variadic | arithmetic |
|
|
1636
|
+
| `/` | 2 | 12 | left | arithmetic |
|
|
1637
|
+
| `%` | 2 | 12 | left | arithmetic |
|
|
1638
|
+
| `<%%` | 2 | 13 | right | apply |
|
|
1639
|
+
| `->` | 2 | 14 | left | access |
|
|
1640
|
+
| `!` | 1 | not measured | not measured | logical |
|
|
1641
|
+
| `any` | 1 | not measured | not measured | logical |
|
|
1642
|
+
|
|
1643
|
+
Precedence runs 1 (loosest) to 14 (tightest), measured over every
|
|
1644
|
+
ordered pair rather than transcribed. `variadic` means the device FLATTENS
|
|
1645
|
+
the operator — `(1 + 2 + 3)` is one node with three children, not two nested
|
|
1646
|
+
ones. The 2 prefix-only operators never appear in a pair and so
|
|
1647
|
+
carry no measured level.
|
|
1648
|
+
|
|
1649
|
+
Spellings the device reads as something else:
|
|
1650
|
+
|
|
1651
|
+
| spelling | reads as | kind |
|
|
1652
|
+
| -------- | -------- | ---- |
|
|
1653
|
+
| `&&` | `(<%% (and 1 2) )` | alias |
|
|
1654
|
+
| `\|\|` | `(<%% (or 1 2) )` | alias |
|
|
1655
|
+
| `<>` | `(<%% (< 1 (> 2)) )` | re-lexed |
|
|
1656
|
+
|
|
1657
|
+
And the grounded complement — asked, and refused:
|
|
1658
|
+
|
|
1659
|
+
| spelling | why not |
|
|
1660
|
+
| -------- | ------- |
|
|
1661
|
+
| `not` | reads-as-variable |
|
|
1662
|
+
| `xor` | reads-as-variable |
|
|
1663
|
+
| `mod` | reads-as-variable |
|
|
1664
|
+
| `is` | reads-as-variable |
|
|
1665
|
+
| `div` | reads-as-variable |
|
|
1666
|
+
| `band` | reads-as-variable |
|
|
1667
|
+
| `bor` | reads-as-variable |
|
|
1668
|
+
| `shl` | reads-as-variable |
|
|
1669
|
+
| `shr` | reads-as-variable |
|
|
1670
|
+
| `eq` | reads-as-variable |
|
|
1671
|
+
| `ne` | reads-as-variable |
|
|
1672
|
+
| `line` | reads-as-variable |
|
|
1673
|
+
| `array` | reads-as-variable |
|
|
1674
|
+
| `as` | reads-as-variable |
|
|
1675
|
+
| `at` | reads-as-variable |
|
|
1676
|
+
| `for` | reads-as-variable |
|
|
1677
|
+
| `none` | reads-as-variable |
|
|
1678
|
+
| `outside` | reads-as-variable |
|
|
1679
|
+
| `evl` | reads-as-variable |
|
|
1680
|
+
| `..` | reads-as-variable |
|
|
1681
|
+
| `//` | reads-as-variable |
|
|
1682
|
+
| `**` | rejected |
|
|
1683
|
+
| `===` | rejected |
|
|
1684
|
+
| `+=` | rejected |
|
|
1685
|
+
| `++` | rejected |
|
|
1686
|
+
| `?` | rejected |
|
|
1687
|
+
| `?:` | rejected |
|
|
1688
|
+
| `<%` | rejected |
|
|
1689
|
+
| `%%` | rejected |
|
|
1690
|
+
| `;` | rejected |
|
|
1691
|
+
| `$` | juxtaposition-only |
|
|
1692
|
+
| `[` | rejected |
|
|
1693
|
+
| `]` | rejected |
|
|
1694
|
+
<!-- END GENERATED operator-table -->
|
|
1695
|
+
|
|
1696
|
+
<!-- BEGIN GENERATED token-census — regenerate with `bun run explain:token-census:readme` -->
|
|
1697
|
+
The token census is re-derivable with `bun run explain:token-census` and
|
|
1698
|
+
covers 948 source scripts. The figures below are generated from
|
|
1699
|
+
`test/fixtures/explain/tokens.json` → `corpus` by
|
|
1700
|
+
`bun run explain:token-census:readme` and gated against it by
|
|
1701
|
+
`bun run explain:token-census:readme:check`; the fixture itself is gated
|
|
1702
|
+
against a fresh corpus run by `bun run explain:token-census:check`. Of
|
|
1703
|
+
1,426,731 analyzed bytes, 1,023,107 are classified (71.71%), the remaining
|
|
1704
|
+
403,624 are `unclassified`. The census emits 241,539 tokens (avg 254.8 per
|
|
1705
|
+
script). Every byte belongs to exactly one token — sorted by `start`, no
|
|
1706
|
+
gaps, no overlaps, `join(slice) === input` — and the `class` field is the
|
|
1707
|
+
vocabulary #264 B5 settled. Each B2 fill should move the classified
|
|
1708
|
+
percentage.
|
|
1709
|
+
<!-- END GENERATED token-census -->
|
|
1710
|
+
|
|
1711
|
+
B1's `data.tokens[]` is live behind `--tokens` — a total, gapless byte
|
|
1712
|
+
partition whose `class` draws on the vocabulary #264 B5 settled (every unclaimed
|
|
1713
|
+
byte is `unclassified`). Since B1 its only fill source was `data.spans[]` (comment runs
|
|
1714
|
+
and resolved variable occurrences); **the path fill claims resolved menu and
|
|
1715
|
+
command-name bytes as provisional `dir` / `cmd` tokens**, including valid path
|
|
1716
|
+
slashes and nested command substitutions, while ambiguous, malformed, and
|
|
1717
|
+
source-unmapped runs stay unclassified. A substitution's bytes are located by
|
|
1718
|
+
the resolution's published **`innerSpan`**, never by arithmetic on its `span`:
|
|
1719
|
+
`span` covers `[…]` for a bare substitution but `$[…]` for one interpolated
|
|
1720
|
+
inside a string (the sigil is part of it), and `inner` is trimmed on top of
|
|
1721
|
+
that, so `span.start + 1` was one byte early in a string and two bytes wrong
|
|
1722
|
+
for `[ … ]`. Both refused the fill's slice check and fell through to the
|
|
1723
|
+
string fill, which is how a command `structure.subcommands` had already
|
|
1724
|
+
resolved came back as opaque `string` bytes (#325). The check itself stays:
|
|
1725
|
+
where `Loc` widens the resolution to its statement, the slice no longer equals
|
|
1726
|
+
`inner` and the bytes are left unclaimed rather than landed on the wrong
|
|
1727
|
+
place. **`#293`'s arg fill** in
|
|
1728
|
+
`src/explain/arg-tokens.ts` then claims argument names as `arg` (name run
|
|
1729
|
+
`[span.start, valueSpan.start - 1)`) and the single `=` byte at
|
|
1730
|
+
`valueSpan.start - 1` as `arg-sep` (#264 B5) on the residual left by spans,
|
|
1731
|
+
before operators see it
|
|
1732
|
+
— and **`#295`'s value fill follows it** — `src/explain/value-tokens.ts`
|
|
1733
|
+
claims argument value bytes and leaf array-literal members
|
|
1734
|
+
(`data.values.occurrences`, leaves only via `parent` containment,
|
|
1735
|
+
quotes-included) on the residual left by `spans`+`path`+`arg`+`escaped`, before the
|
|
1736
|
+
**B3 string/brace fills** see it — `src/explain/string-tokens.ts` claims
|
|
1737
|
+
quoted-string runs (delimiters included, escapes already taken) and `src/explain/brace-tokens.ts`
|
|
1738
|
+
claims every residual brace delimiter (`{`/`}`), including delimiters from
|
|
1739
|
+
invalid braced references, on the residual left by
|
|
1740
|
+
`spans`+`path`+`arg`+`escaped`+`value`, and finally `operator` sees only what none of
|
|
1741
|
+
them wanted — which is why the operator fill can abstain on `, / = -` outside
|
|
1742
|
+
`( )`, on bytes glued after an argument `=`, and on bytes glued into an
|
|
1743
|
+
argument name without a smarter operator scanner. **`src/explain/escape-tokens.ts`
|
|
1744
|
+
claims valid string-internal escapes as `escaped`** between `arg` and `value`:
|
|
1745
|
+
an escape is interior to a quoted run, so it has to claim before the two fills
|
|
1746
|
+
that would otherwise swallow it whole, and it splits their runs into fragments
|
|
1747
|
+
around it the same way a `variable-*` span already did. Fill order is enforced by
|
|
1748
|
+
argument order to `buildTokens` ([#290 design decision 1](../../src/explain.ts)):
|
|
1749
|
+
`spans` claims first, then `path`, `arg`, `escaped`, `value`, `string`, `brace`,
|
|
1750
|
+
and finally `operator`.
|
|
1751
|
+
`ExplainTokenClass` is
|
|
1752
|
+
`ExplainSpanClass | "dir" | "cmd" | "operator" | "arg" | "arg-sep" | "value" |
|
|
1753
|
+
"string" | "escaped" | "brace" | "unclassified"` — one `operator` class for all 26
|
|
1754
|
+
spellings + 2 aliases, `arg` for an argument's name bytes and `arg-sep` for the
|
|
1755
|
+
single `=` that binds it to its value, one `value` class for every leaf value
|
|
1756
|
+
span, plus `string` (quoted runs minus their escapes), `escaped` (one valid
|
|
1757
|
+
string-internal escape sequence) and `brace` (scope delimiters). #264
|
|
1758
|
+
settled which of those merges keep their name and which do not, on a stated
|
|
1759
|
+
rule rather than taste — see
|
|
1760
|
+
[The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5).
|
|
1761
|
+
`ExplainSpanClass` and `data.spans[]` stay proof-only; `src/explain/operators.ts`
|
|
1762
|
+
remains data plus accessors, and the operator table above is its source. The
|
|
1763
|
+
`centrs → highlight` projection is B4 and reads both.
|
|
1764
|
+
|
|
1765
|
+
**Where the operator fill abstains.** Fill order is the resolution mechanism, so
|
|
1766
|
+
a byte that is structurally part of a path or an argument is left
|
|
1767
|
+
`unclassified` for the path/arg fills rather than claimed. Three abstentions,
|
|
1768
|
+
each grounded on the corpus device oracle (`parseil_results.il_text`):
|
|
1769
|
+
|
|
1770
|
+
| Abstention | Grounding |
|
|
1771
|
+
| ---------- | --------- |
|
|
1772
|
+
| `, / = -` outside a `( )` group. `(` opens an expression; `[` opens a **command substitution** and `{` a block or array literal. | The IL for `[ /system/identity/get value-name=name ]` is `(evl /system/identity/get value-name=name)` — path separators and an argument separator, no division and no comparison node. |
|
|
1773
|
+
| A spelling glued immediately after an argument `=`. | `in-interface-list=!LAN`, `.id=*2`, `oid=.1.3.6.1.2.1` — the byte after an argument `=` starts the value. |
|
|
1774
|
+
| A spelling glued into an argument **name**. | `:foreach x in=$list` is `/foreach counter=$x` with no `(in …)` node anywhere; the IL keeps `security.authentication-types=wpa2-psk` as one name and renders `.id` as the single symbol `$.id`, never `(. …)`. |
|
|
1775
|
+
|
|
1776
|
+
The first abstention has a measured cost, and it is now generated rather than
|
|
1777
|
+
asserted. The figures below were prose with no script behind them and no gate
|
|
1778
|
+
on them for three releases, quoting a `224` / `1,259` / `1,035` split that no
|
|
1779
|
+
universe reproduces (#341). The abstention itself is as wide as it ever was —
|
|
1780
|
+
what was wrong was the accounting of it. They are re-derived by
|
|
1781
|
+
`bun run explain:bracket-equals` and gated by
|
|
1782
|
+
`bun run explain:bracket-equals:readme:check`.
|
|
1783
|
+
|
|
1784
|
+
<!-- BEGIN GENERATED bracket-equals — regenerate with `bun run explain:bracket-equals:readme` -->
|
|
1785
|
+
Measured by `bun run explain:bracket-equals` over the pinned corpus:
|
|
1786
|
+
**1,079** `=` bytes sit inside `[ … ]`, across
|
|
1787
|
+
263 of 948 scripts, and **1,067** of them come back
|
|
1788
|
+
`unclassified` — the operator fill abstains on essentially all of it. No
|
|
1789
|
+
later fill rescues them: the `arg` fill claims **0**, because it offers only
|
|
1790
|
+
the `=` its own located argument token names and a `[ … ]` substitution is
|
|
1791
|
+
not where it locates arguments.
|
|
1792
|
+
|
|
1793
|
+
Of those 1,067, **448** are governed by a `find`/`where` query and are
|
|
1794
|
+
genuine comparisons — the device lowers `[find address=$IP]` to
|
|
1795
|
+
`(= $address $IP)` — against **619** plain `arg=value`. So abstaining is
|
|
1796
|
+
42.0% wrong and claiming the lot would be 58.0% wrong; a `where`-aware
|
|
1797
|
+
fill can take the query half later. For contrast the fill does claim
|
|
1798
|
+
**1,402** `=` bytes whose innermost opener is `(`.
|
|
1799
|
+
|
|
1800
|
+
The query reading is the device's, not a guess: on 7.24.2, 93 of the
|
|
1801
|
+
137 scripts carrying one have IL with at least as many `(= …)` nodes.
|
|
1802
|
+
Of the rest, 28 are scripts the console refused outright — their IL is prose,
|
|
1803
|
+
with no nodes of any kind — and 16 use a menu the capture build could not
|
|
1804
|
+
resolve (a forum snippet on `/interface wireless` answers `bad command name
|
|
1805
|
+
wireless`), so the device never reached the query to lower it. Neither is a
|
|
1806
|
+
counter-example to the reading. Whole-corpus `(= …)` head nodes, which also
|
|
1807
|
+
count expression comparisons the fill already claims:
|
|
1808
|
+
7.23.5 1,233, 7.24.2 1,240, 7.25beta3 1,240.
|
|
1809
|
+
<!-- END GENERATED bracket-equals -->
|
|
1810
|
+
|
|
1811
|
+
### Device agreement for the token partition (#264 B4, #263)
|
|
1812
|
+
|
|
1813
|
+
The census above counts what offline analysis *claimed*. This counts what the
|
|
1814
|
+
device *agreed with*. The oracle is the committed stratified slice of the Q13
|
|
1815
|
+
`/console/inspect request=highlight` captures
|
|
1816
|
+
([`test/fixtures/explain/highlight-streams.slice.json`](../../test/fixtures/explain/highlight-streams.slice.json)),
|
|
1817
|
+
so the scorer needs no router, no corpus and no network, and gates in the unit
|
|
1818
|
+
suite rather than in the corpus job. Both oracles must see the same bytes, so
|
|
1819
|
+
the source text is reconstructed from the device's own run-length pairs — never
|
|
1820
|
+
from the corpus row, which still holds the non-ASCII the capture replaced
|
|
1821
|
+
(the #269 rule, one layer up).
|
|
1822
|
+
|
|
1823
|
+
**`highlight` is not one oracle; it is four kinds of answer.** Scoring a byte
|
|
1824
|
+
means first knowing what kind of answer came back, because three of the four
|
|
1825
|
+
are not judgments about syntax at all:
|
|
1826
|
+
|
|
1827
|
+
| Kind | Classes | Why it cannot be scored as agreement |
|
|
1828
|
+
| ---- | ------- | ------------------------------------ |
|
|
1829
|
+
| syntax | `cmd` `dir` `arg` `arg-dot` `arg-scope` `comment` `escaped` `syntax-meta` `variable-local` `variable-global` `variable-auto` `variable-parameter` | — this is the region that can be scored |
|
|
1830
|
+
| non-syntax | `obj-inactive` `obj-dynamic` `obj-disabled` `variable-undefined` `syntax-obsolete` | The source text alone does not decide these. centrs has no class for most of them at all — `ExplainSymbolClass` deliberately excludes `undefined` — and where it *does* claim the byte (an `arg` the device calls `obj-inactive`) it is the **scorer** that abstains, because the device was not making a syntax judgement there. *Why* each one is out of reach is a second question, answered per byte rather than per class; see applicability below. |
|
|
1831
|
+
| silence | `none` | The device's abstention, not a class named "none". Agreeing with silence is not agreement. |
|
|
1832
|
+
| parser-stop | `error` | The device gives up here — see below. |
|
|
1833
|
+
|
|
1834
|
+
**Applicability is a second axis, and it is not the class name.** #263's four
|
|
1835
|
+
categories — offline-decidable, schema-dependent, state-dependent,
|
|
1836
|
+
version-dependent — are assigned from the *pair* (what centrs read the byte as,
|
|
1837
|
+
what the device called it) plus whether the two captures even agreed, because
|
|
1838
|
+
one highlight class carries more than one category. Version disagreement is
|
|
1839
|
+
checked first, ahead even of the device's own silence: a byte one build
|
|
1840
|
+
classifies and the next leaves `none` is a version difference, not an absence.
|
|
1841
|
+
`obj-inactive` on an argument name means the menu's schema has no such field
|
|
1842
|
+
(**schema** — `default-vlan-id`, under an `/interface ethernet switch port` a
|
|
1843
|
+
CHR does not have); the same class on a menu or command name means the package
|
|
1844
|
+
is not installed on this box (**state**: `/zerotier`, `/user-manager`,
|
|
1845
|
+
`/container`). `obj-dynamic` on the routing table `main` is a live object
|
|
1846
|
+
(**state**).
|
|
1847
|
+
`syntax-obsolete` is this version's deprecation table talking, landing on the
|
|
1848
|
+
whitespace where modern syntax wants `=` (`} else {`, `:foreach conn in
|
|
1849
|
+
$conns`) — **version**. And `variable-undefined` is deliberately left
|
|
1850
|
+
**uncategorized**: an undefined reference may be an external global, a
|
|
1851
|
+
query/filter field name or a lexical spelling, nothing committed here separates
|
|
1852
|
+
the three, and an absent binding in *this* document is not evidence of a
|
|
1853
|
+
runtime undefined variable either. Leaving it explicit is the answer, not a gap
|
|
1854
|
+
to be filled by assumption.
|
|
1855
|
+
|
|
1856
|
+
**The device stops classifying at its first `error`, and mostly does not
|
|
1857
|
+
recover.** Measured over the slice, `error` appears **at most once per script
|
|
1858
|
+
and is always exactly one byte wide**, and on long-term 7.23.5 every byte after
|
|
1859
|
+
it comes back `none`. Most of the slice sits in that tail. A percentage computed
|
|
1860
|
+
over the whole stream therefore measures the device giving up, not the parser,
|
|
1861
|
+
which is why the tail is its own outcome and is never folded into agreement. The
|
|
1862
|
+
rule is not absolute across versions, so the report does not assert it: stable
|
|
1863
|
+
7.24.2 and development 7.25beta3 classify again after the `error` in one run and
|
|
1864
|
+
stop on fewer scripts, and
|
|
1865
|
+
`parser-recovered` in the table below is the figure that says by how much —
|
|
1866
|
+
zero would have been the claim, and it is not zero. Both counts are generated,
|
|
1867
|
+
not quoted here, so neither can go stale against the fixture.
|
|
1868
|
+
|
|
1869
|
+
**The projection is declared, never fitted.** `src/explain/highlight-projection.ts`
|
|
1870
|
+
maps each centrs token class to the device classes it predicts, justified by
|
|
1871
|
+
what the class was *defined* to mean rather than by where it happened to land.
|
|
1872
|
+
A projection tuned to the measurement would make the measurement circular. So a
|
|
1873
|
+
class whose intent has no single device counterpart — `string`, which spans a
|
|
1874
|
+
delimiter the device calls `syntax-meta`, escapes it calls `escaped` and an
|
|
1875
|
+
interior it declines to classify, and `value`, for which the device has no class
|
|
1876
|
+
at all — projects nowhere, and its bytes are counted as `unprojected` with the
|
|
1877
|
+
device's own answers tabulated beside them. That table is the evidence #264 B5
|
|
1878
|
+
needs in order to decide which provisional merge deserves to be split; it is not
|
|
1879
|
+
a score.
|
|
1880
|
+
|
|
1881
|
+
<!-- cspell:disable — the example table quotes corpus script paths verbatim, forum-thread typos included -->
|
|
1882
|
+
<!-- BEGIN GENERATED highlight-agreement — regenerate with `bun run explain:highlight-agreement:readme` -->
|
|
1883
|
+
The agreement report is re-derivable with
|
|
1884
|
+
`bun run explain:highlight-agreement` and needs no router and no corpus: it
|
|
1885
|
+
scores the token partition against
|
|
1886
|
+
`test/fixtures/explain/highlight-streams.slice.json`, 70 stratified scripts of
|
|
1887
|
+
committed `/console/inspect request=highlight` output. The figures below are
|
|
1888
|
+
generated from `test/fixtures/explain/highlight-agreement.json` by
|
|
1889
|
+
`bun run explain:highlight-agreement:readme`, gated against it by
|
|
1890
|
+
`bun run explain:highlight-agreement:readme:check`, and the fixture itself is
|
|
1891
|
+
gated against a fresh measurement by
|
|
1892
|
+
`test/unit/explain-highlight-agreement.test.ts`. Of 85,529 bytes at 7.24.2,
|
|
1893
|
+
27,974 are bytes **both** sides decided a syntax class for, and 99.99% of
|
|
1894
|
+
those agree (dev 99.99%, holdout 100.00%). Set `comment` aside — it is 65.63%
|
|
1895
|
+
of that decided region and mostly the corpus's harness-injected `# Source:`
|
|
1896
|
+
banner (#203) — and the remaining 9,615 bytes agree 99.98%. The device stops
|
|
1897
|
+
classifying at its one-byte `error`: 36 of 70 scripts carry one, and the
|
|
1898
|
+
41,968 bytes from there on are not a judgment about anything. The oracle
|
|
1899
|
+
itself moves between captures: 36 stop at 7.24.2 and 43 stop at 7.23.5 and 36
|
|
1900
|
+
stop at 7.25beta3, and 27 of the 70 streams differ between them. **This is a
|
|
1901
|
+
trend line, not a pass gate** — the slice is a per-(split, class) quota over a
|
|
1902
|
+
corpus that is 96.8% two authors, so the percentage describes this slice.
|
|
1903
|
+
|
|
1904
|
+
| centrs class | projects to | agree | disagree | unprojected | of decided |
|
|
1905
|
+
| ------------ | ----------- | ----: | -------: | ----------: | ---------: |
|
|
1906
|
+
| `comment` | `comment` | 18,359 | 0 | 0 | 100.00% |
|
|
1907
|
+
| `variable-local` | `variable-local` | 1,123 | 0 | 0 | 100.00% |
|
|
1908
|
+
| `variable-global` | `variable-global` | 582 | 0 | 0 | 100.00% |
|
|
1909
|
+
| `variable-auto` | `variable-auto` | 92 | 0 | 0 | 100.00% |
|
|
1910
|
+
| `variable-parameter` | `variable-parameter` | 459 | 0 | 0 | 100.00% |
|
|
1911
|
+
| `dir` | `dir` | 1,677 | 0 | 0 | 100.00% |
|
|
1912
|
+
| `cmd` | `cmd` | 2,244 | 0 | 0 | 100.00% |
|
|
1913
|
+
| `arg` | `arg`, `arg-dot`, `arg-scope` | 2,188 | 0 | 0 | 100.00% |
|
|
1914
|
+
| `arg-sep` | `syntax-meta` | 382 | 0 | 0 | 100.00% |
|
|
1915
|
+
| `operator` | `syntax-meta` | 237 | 2 | 0 | 99.16% |
|
|
1916
|
+
| `brace` | `syntax-meta` | 211 | 0 | 0 | 100.00% |
|
|
1917
|
+
| `string` | *abstains* | 0 | 0 | 499 | — |
|
|
1918
|
+
| `escaped` | `escaped` | 418 | 0 | 0 | 100.00% |
|
|
1919
|
+
| `value` | *abstains* | 0 | 0 | 144 | — |
|
|
1920
|
+
|
|
1921
|
+
| outcome | bytes @7.24.2 | bytes @7.23.5 | bytes @7.25beta3 |
|
|
1922
|
+
| ------- | ----: | ----: | ----: |
|
|
1923
|
+
| agree — both decided, projection accepts | 27,972 | 27,218 | 27,972 |
|
|
1924
|
+
| disagree — both decided, projection rejects | 2 | 2 | 2 |
|
|
1925
|
+
| unprojected — no declared projection covers the pair | 643 | 633 | 643 |
|
|
1926
|
+
| offline-silent — device decided, centrs abstained | 4,300 | 4,150 | 4,300 |
|
|
1927
|
+
| non-syntax — the device answered something syntax cannot decide | 847 | 733 | 847 |
|
|
1928
|
+
| device-silent — centrs decided, device said `none` | 5,815 | 5,674 | 5,815 |
|
|
1929
|
+
| both-silent | 3,886 | 3,702 | 3,886 |
|
|
1930
|
+
| parser-stopped — at/after the device's `error` byte, and silent from there | 41,968 | 43,417 | 41,968 |
|
|
1931
|
+
| parser-recovered — past that `error`, and the device classified anyway | 96 | 0 | 96 |
|
|
1932
|
+
|
|
1933
|
+
Where the device decided and offline analysis did not (4,300 bytes at 7.24.2,
|
|
1934
|
+
the next fill's target list): `syntax-meta` 3,122, `comment` 355,
|
|
1935
|
+
`variable-local` 326, `arg` 197, `escaped` 145, `dir` 66, 89 across the rest.
|
|
1936
|
+
Most of the `syntax-meta` share is whitespace the device merged into an
|
|
1937
|
+
adjacent structure run rather than a token centrs missed.
|
|
1938
|
+
|
|
1939
|
+
**Applicability** is the second axis (#263): of the bytes the device did
|
|
1940
|
+
answer, what kind of fact was it answering? Assigned from the pair — what
|
|
1941
|
+
centrs read the byte as and what the device called it — plus whether the
|
|
1942
|
+
captures agreed, never by assigning a whole class to one category. It covers
|
|
1943
|
+
every measured byte, the post-`error` tail included: where one capture stops
|
|
1944
|
+
parsing and the next does not, that tail IS the version disagreement, which is
|
|
1945
|
+
why it dominates the category.
|
|
1946
|
+
|
|
1947
|
+
| applicability @7.24.2 | bytes | leading cells |
|
|
1948
|
+
| ---------------------------- | ----: | ------------- |
|
|
1949
|
+
| offline-decidable | 32,002 | `comment` → `comment` 18,006, `unclassified` → `syntax-meta` 2,999, `cmd` → `cmd` 2,132 |
|
|
1950
|
+
| schema-dependent | 326 | `arg` → `obj-inactive` 326 |
|
|
1951
|
+
| state-dependent | 168 | `dir` → `obj-inactive` 120, `cmd` → `obj-inactive` 21, `unclassified` → `obj-dynamic` 20 |
|
|
1952
|
+
| version-dependent | 1,158 | `comment` → `comment` 353, `unclassified` → `obj-inactive` 159, `unclassified` → `syntax-meta` 123 |
|
|
1953
|
+
| uncategorized | 213 | `unclassified` → `obj-inactive` 90, `unclassified` → `variable-undefined` 75, `value` → `variable-undefined` 21 |
|
|
1954
|
+
| no-device-answer | 51,662 | `none` where the captures agree, wherever it falls |
|
|
1955
|
+
|
|
1956
|
+
One representative run per cell, so B5 reads a fragment rather than a count.
|
|
1957
|
+
`runs` is how often that exact fragment occurs; the location is a slice script
|
|
1958
|
+
and a byte offset into the stream the device saw.
|
|
1959
|
+
|
|
1960
|
+
| outcome | cell | fragment | runs | first at |
|
|
1961
|
+
| ------- | ---- | -------- | ---: | -------- |
|
|
1962
|
+
| disagree | `operator` → `arg` | `"in"` | 1 | `forum/amm0/topic-169456-having-the-where-filter-in-scripting-signifantly-increases-the-execution-time-an/post-0006-snippet-01.rsc` @615 |
|
|
1963
|
+
| unprojected | `string` → `syntax-meta` | `"\""` | 311 | `eworm/ppp-on-up.rsc` @435 |
|
|
1964
|
+
| unprojected | `value` → `syntax-meta` | `"\""` | 88 | `forum/amm0/topic-153357-using-wifiwave2-to-bridge-two-audience-wirelessly-thoughts-4-address-mode/post-0001-snippet-01.rsc` @1436 |
|
|
1965
|
+
| unprojected | `string` → `arg` | `"as-value"` | 1 | `forum/amm0/topic-266499-how-to-condense-variables-passed-through-to-function/post-0020-snippet-01.rsc` @279 |
|
|
1966
|
+
| unprojected | `string` → `cmd` | `"len"` | 1 | `forum/amm0/topic-155978-zerotier-on-mikrotik-a-rosetta-stone-v7-1-1/post-0001-snippet-01.rsc` @1794 |
|
|
1967
|
+
| unprojected | `string` → `dir` | `":"` | 1 | `forum/amm0/topic-155978-zerotier-on-mikrotik-a-rosetta-stone-v7-1-1/post-0001-snippet-01.rsc` @1793 |
|
|
1968
|
+
<!-- END GENERATED highlight-agreement -->
|
|
1969
|
+
<!-- cspell:enable -->
|
|
1970
|
+
|
|
1971
|
+
**What the disagreements are.** Two cells carry all of them, and neither is a
|
|
1972
|
+
lexing accident:
|
|
1973
|
+
|
|
1974
|
+
- `arg` → `syntax-meta`: the `=` byte the `arg` class merges in. The device
|
|
1975
|
+
reads a separator as structure. This is the measured cost of the merge the
|
|
1976
|
+
token-census block already flags, and it is the single largest reason `arg`
|
|
1977
|
+
scores below the other name-coincident classes.
|
|
1978
|
+
- `operator` → `arg`: the space-separated `in` of `:foreach conn in $conns`,
|
|
1979
|
+
which the device reads as the argument **name** `in`. The two oracles
|
|
1980
|
+
genuinely differ here — `:parse` IL carries `in` in the operator set, and the
|
|
1981
|
+
operator-abstention table above already records that `:foreach x in=$list`
|
|
1982
|
+
lowers with no `(in …)` node.
|
|
1983
|
+
|
|
1984
|
+
The third cell is gone. `cmd` → `dir` was the whole of #324 — a known menu path
|
|
1985
|
+
carrying arguments and no verb, where `verbsplit` promoted the last segment;
|
|
1986
|
+
those 12 bytes now abstain instead of disagreeing, and the 14 `dir` bytes the
|
|
1987
|
+
same statements contributed to `agree` abstain with them. That is the price,
|
|
1988
|
+
and [the fix states it](#the-menu-path-with-arguments-324): withdrawing a
|
|
1989
|
+
fabricated verb also withdraws the menu reading it was built on.
|
|
1990
|
+
|
|
1991
|
+
The `unprojected` residue is what #264 B5 read to decide which merges to
|
|
1992
|
+
split — [The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5) records
|
|
1993
|
+
the verdict on each. It was also where #325 lived:
|
|
1994
|
+
a quoted string claimed any `$[…]` substitution inside it, so bytes the device
|
|
1995
|
+
resolves to `cmd` or `dir` came back as one opaque `string` run even though
|
|
1996
|
+
`structure.subcommands` had already resolved that same span. Those cells have
|
|
1997
|
+
largely emptied (`string` → `cmd` 42 → 3, `string` → `dir` 19 → 1, with the
|
|
1998
|
+
same fix recovering padded bare brackets `[ … ]` from `offline-silent`), and
|
|
1999
|
+
the two residues left are stated rather than fixed:
|
|
2000
|
+
|
|
2001
|
+
- `$(…)` — a **parenthesized** interpolation carrying a bracket, as in
|
|
2002
|
+
`apikey-len=$([:len $apikey])`. Only `$[` is scanned inside a string, so the
|
|
2003
|
+
inner command is not resolved there in the first place; this is a parser
|
|
2004
|
+
reach, not a fill-order question (4 bytes on the slice).
|
|
2005
|
+
- a bare-word **argument** inside a substitution — `as-value` in
|
|
2006
|
+
`:put "$[/ip address print as-value]"`. The path fill claims the path and
|
|
2007
|
+
verb; the argument fill reads `statements[].arguments`, and a substitution is
|
|
2008
|
+
not a statement, so it has no candidate to claim (8 bytes).
|
|
2009
|
+
|
|
2010
|
+
The `$` sigil and the `[` / `]` delimiters stay `string`, and `value` bytes
|
|
2011
|
+
are, to the device, simply unclassified — both settled by
|
|
2012
|
+
[The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5) below.
|
|
2013
|
+
|
|
2014
|
+
### The vocabulary, and what earns a class (#264 B5)
|
|
2015
|
+
|
|
2016
|
+
`data.tokens[]` shipped with its `class` marked provisional from B1 onward —
|
|
2017
|
+
emit first, name later (#293 design decision 2). This is the "name later", and
|
|
2018
|
+
the vocabulary below is what `@tikoci/centrs/explain` offers a consumer. It is
|
|
2019
|
+
no longer provisional.
|
|
2020
|
+
|
|
2021
|
+
**The rule a class has to pass.** A merged class is split when both hold, and
|
|
2022
|
+
stays merged otherwise:
|
|
2023
|
+
|
|
2024
|
+
1. **A consumer needs the distinction** — it changes what that consumer renders
|
|
2025
|
+
or decides, not just how a token dump reads.
|
|
2026
|
+
2. **The envelope cannot already supply it** — the distinction is not
|
|
2027
|
+
recoverable by joining `tokens[]` to another published surface on byte
|
|
2028
|
+
offsets.
|
|
2029
|
+
|
|
2030
|
+
The second test is the one that does the work. A class that restates a fact
|
|
2031
|
+
`structure`, `symbols` or `values` already publishes is a second place for that
|
|
2032
|
+
fact to go stale, and the two copies can then disagree about the same bytes. So
|
|
2033
|
+
the question is never "would a name here be nice"; it is "what would a consumer
|
|
2034
|
+
have to do without it". Applied to the five merges the fills carried, exactly
|
|
2035
|
+
one passes:
|
|
2036
|
+
|
|
2037
|
+
| merge | what a consumer would do without it | verdict |
|
|
2038
|
+
| ----- | ----------------------------------- | ------- |
|
|
2039
|
+
| `arg` — the name bytes **and** the `=` | subtract one from `arguments[].valueSpan.start`, which exists only where the STRICT reading read. **5,088 of the 16,526 `=` runs the fill paints (30.8%, in 436 of 948 corpus scripts) sit in a statement whose `arguments` refused** — led by `do=` (2,393), `else=` (328) and `in=` (212), the block binder of every `:if`, `:foreach` and `:while`. For those there is nothing to subtract from. | **split** → `arg-sep` |
|
|
2040
|
+
| `operator` — 26 spellings + 2 aliases | slice the input at the token's span: the spelling *is* the bytes, and `src/explain/operators.ts` maps it to its IL node | keep |
|
|
2041
|
+
| `value` — every leaf, whatever its shape | join `data.values.occurrences[]` on the offset; #225's three axes (shape, observed, schema) live there, and a class per shape would be a fourth copy free to contradict them | keep |
|
|
2042
|
+
| `brace` — scope and array delimiters | `structure.blocks[]` locates every scope block; an array container is the `values.occurrences[]` entry that is some other occurrence's `parent` | keep |
|
|
2043
|
+
| `string` — delimiters, escapes, interior | the opening delimiter is the run's first byte and, where the run is closed, the last is its partner (an unterminated one is already a located defect). The **escapes were a different answer** — see below | **partly split** → `escaped`; delimiters and interior stay merged |
|
|
2044
|
+
|
|
2045
|
+
Re-derive the `arg-sep` figures with `bun run explain:arg-reach`, which prices
|
|
2046
|
+
the same gap it prices the reach fork with.
|
|
2047
|
+
|
|
2048
|
+
**What the split cost.** Nothing, which was the claim and is now the
|
|
2049
|
+
measurement. Classified bytes are identical — 1,023,405 of 1,426,731 (71.73%),
|
|
2050
|
+
equal to the last recorded decimal before and after — and `arg` gave up exactly
|
|
2051
|
+
the 16,526 bytes `arg-sep` claims, one per attribute. Token count rises 214,503
|
|
2052
|
+
→ 231,026 because one run became two. **This is a retag.**
|
|
2053
|
+
|
|
2054
|
+
(Those totals are the corpus as it stood for THIS measurement. The classified
|
|
2055
|
+
total is 1,023,107 today: #347/#348 later classified 8 further bytes, and #355's
|
|
2056
|
+
fail-closed symbol stop then unclaimed 306 of them — variable bytes sitting
|
|
2057
|
+
after a bare apostrophe, which the device never reaches. The identity being
|
|
2058
|
+
claimed here is between the two sides of the split, not with any later total —
|
|
2059
|
+
restating it against today's figure would credit this retag with a movement a
|
|
2060
|
+
different change made. The generated block above carries the current numbers.)
|
|
2061
|
+
|
|
2062
|
+
One count is not a clean split: `arg`'s own run count falls 16,526 → 16,523.
|
|
2063
|
+
Three attributes lost their name run entirely, because an earlier `variable-*`
|
|
2064
|
+
span had already claimed the name bytes — `set $ifcId ssid=$ssid`, and two
|
|
2065
|
+
`on-error=` in one snippet — leaving only the `=` on the residual. Before the
|
|
2066
|
+
split those three emitted a token *called* `arg` that contained nothing but an
|
|
2067
|
+
`=`. That is the merge in miniature, and it is why the number goes down.
|
|
2068
|
+
|
|
2069
|
+
**The agreement number moved a long way, and it is still not a score.** Against
|
|
2070
|
+
the committed device slice, projection disagreements fall **360 → 2** and the
|
|
2071
|
+
decided region goes 98.66% → 99.99% (`arg` alone 85.50% → 100.00%). Read that
|
|
2072
|
+
as the withdrawal of a wrong assertion, not as a parser that got better: centrs
|
|
2073
|
+
stopped calling a separator a name over the 358 runs the device had been
|
|
2074
|
+
reading as `syntax-meta` all along, and not one byte changed which analyzer
|
|
2075
|
+
found it. The two survivors are the `operator` → `arg` `in` of
|
|
2076
|
+
`:foreach conn in $conns`, where the two oracles genuinely differ.
|
|
2077
|
+
|
|
2078
|
+
The projection entry was **declared before the measurement, never fitted to
|
|
2079
|
+
it**: `arg-sep` predicts `syntax-meta` because a separator is structure
|
|
2080
|
+
punctuation, the same reason `brace` and `operator` predict it. The slice then
|
|
2081
|
+
put 358 of 358 there.
|
|
2082
|
+
|
|
2083
|
+
**Where the tolerant reading reaches the envelope (#316's open decision).** It
|
|
2084
|
+
reaches it *as the token partition*, and that is the only place it should.
|
|
2085
|
+
`structure.statements[].arguments` stays the strict all-or-nothing reading,
|
|
2086
|
+
because its consumer renders a runnable request and a half-read argument list
|
|
2087
|
+
changes what the rendered command does. But with the `=` named, `tokens[]`
|
|
2088
|
+
locates the argument on its own:
|
|
2089
|
+
|
|
2090
|
+
```text
|
|
2091
|
+
/ip route add dst-address=1.1.1.1 gateway=$gw
|
|
2092
|
+
arguments: { read: false, why: "a variable value" }
|
|
2093
|
+
tokens: … arg("gateway") arg-sep("=") unclassified("$") variable-parameter("gw")
|
|
2094
|
+
|
|
2095
|
+
:if ($x = 1) do={ :put "hi" }
|
|
2096
|
+
arguments: { read: false, why: "a substitution or expression value" }
|
|
2097
|
+
tokens: … operator("=") … arg("do") arg-sep("=") brace("{") …
|
|
2098
|
+
```
|
|
2099
|
+
|
|
2100
|
+
Neither statement publishes a single argument, and in both a browser consumer
|
|
2101
|
+
can still see a located `gateway=` / `do=`. The second line is the
|
|
2102
|
+
discrimination working as well: the same `=` byte is `operator` inside
|
|
2103
|
+
`($x = 1)` and `arg-sep` in `do=`, decided by fill order and meaning rather
|
|
2104
|
+
than by a byte scan.
|
|
2105
|
+
|
|
2106
|
+
What `tokens[]` still does **not** carry is the *reason* a value was not
|
|
2107
|
+
decoded. `undecided` stays off `ExplainArgumentToken` and out of the
|
|
2108
|
+
envelope: the envelope's value-fact surface is `data.values.occurrences[]`,
|
|
2109
|
+
where absence at an offset already *is* the abstention, and a second spelling
|
|
2110
|
+
of the same abstention would be free to disagree with it. A consumer that needs
|
|
2111
|
+
the reason imports `lexExplainArgumentTokens` from the package root — which is
|
|
2112
|
+
why that export exists.
|
|
2113
|
+
|
|
2114
|
+
**The second split the rule admits, now made: `escaped`.** A *valid* string
|
|
2115
|
+
escape is recoverable from nothing. Only INVALID ones surface elsewhere, as
|
|
2116
|
+
`bad-string-escape` diagnostics, so a consumer holding `tokens[]` has no
|
|
2117
|
+
published surface to join against to find the valid ones — and the device does
|
|
2118
|
+
class them, as one of the 19 highlight classes. Both halves of the rule hold,
|
|
2119
|
+
so the class is earned.
|
|
2120
|
+
|
|
2121
|
+
Unlike `arg-sep` this is a **fill change** rather than a pure relabel of one
|
|
2122
|
+
analyzer's own output: the escape grammar (uppercase-hex `\XX`, the whitespace
|
|
2123
|
+
continuation, the twelve single-character forms) has to be walked, and the
|
|
2124
|
+
bytes come out of two fills at once rather than one.
|
|
2125
|
+
`src/explain/escape-tokens.ts` runs between `arg` and `value`.
|
|
2126
|
+
|
|
2127
|
+
**The walk is not new, and deliberately not a second one.**
|
|
2128
|
+
[String escape validation](#string-escape-validation-247-252) already owned this
|
|
2129
|
+
grammar for `bad-string-escape`. `walkStringEscapes` now returns *both* halves
|
|
2130
|
+
of that one reading — the valid escapes and the first invalid one — so an
|
|
2131
|
+
`escaped` token and a `bad-string-escape` diagnostic cannot disagree about which
|
|
2132
|
+
bytes are an escape. Deriving them from two walks would have made that
|
|
2133
|
+
disagreement possible, and the document is walked once rather than twice. The
|
|
2134
|
+
walk stops at the first invalid escape, so `escaped` is the prefix the validator
|
|
2135
|
+
vouched for: past a malformed escape the string boundaries are themselves in
|
|
2136
|
+
doubt, and painting there would be a claim the walk cannot support.
|
|
2137
|
+
|
|
2138
|
+
**What the split cost: nothing, and the census proves it.** On the corpus
|
|
2139
|
+
`escaped` claims **19,960 bytes over 7,408 tokens**, and every one of those
|
|
2140
|
+
bytes came from a fill that already held it — `string` gives up 17,672 and
|
|
2141
|
+
`value` 2,288, which is exactly 19,960. Classified bytes are **identical to the
|
|
2142
|
+
last decimal**, 1,023,405 of 1,426,731 (71.73076073906013%), so in byte terms
|
|
2143
|
+
this is a retag even though it took a new walk to make — again, the identity is
|
|
2144
|
+
between this change's own before and after, and the corpus total has since moved
|
|
2145
|
+
to 1,023,107 / 241,539 tokens under #347/#348 and #355. Only the token count
|
|
2146
|
+
moves, 231,026 → 241,663: the 7,408 new `escaped` tokens plus the 3,229
|
|
2147
|
+
fragments they leave behind when they split a run they sit inside (`string`
|
|
2148
|
+
+2,643, `value` +586). That second number is well under one per escape because
|
|
2149
|
+
escapes cluster — adjacent ones like `\C3\BD`, and ones flush against a
|
|
2150
|
+
delimiter, add no fragment at all.
|
|
2151
|
+
|
|
2152
|
+
**What it bought.** Against the committed device slice `escaped` claims **418
|
|
2153
|
+
bytes on 100.00% agreement, 0 disagreements** — the 53 `string` → `escaped`
|
|
2154
|
+
runs and the 1 `value` → `escaped` that were `unprojected` before, scored for
|
|
2155
|
+
the first time. Total `unprojected` falls **1,051 → 633 bytes** (−39.8%) and the
|
|
2156
|
+
both-decided region grows 26,794 → 27,212. No byte changed which analyzer found
|
|
2157
|
+
it and no disagreement was introduced.
|
|
2158
|
+
|
|
2159
|
+
The projection entry was again **declared before it was measured**: `escaped`
|
|
2160
|
+
predicts `escaped` because the class means one valid string-internal escape and
|
|
2161
|
+
that is the run the device gives the same name. `escaped` was already listed in
|
|
2162
|
+
`DEVICE_CLASS_KIND` as a syntax class; the agreement report's `unprojected`
|
|
2163
|
+
table showed the bytes were being merged away, it did not pick the target.
|
|
2164
|
+
|
|
2165
|
+
**The residue this does not claim, sized.** 145 bytes across 24 runs in 4 slice
|
|
2166
|
+
scripts stay `unclassified` where the device says `escaped`, and every one of
|
|
2167
|
+
them is the same shape: a `\`-newline **line continuation in code**, outside any
|
|
2168
|
+
string (`… != true) \<newline> do={`). The device's run there swallows the
|
|
2169
|
+
next line's indentation too, so the 145 bytes are 48 of `\`+terminator and 97 of
|
|
2170
|
+
trailing whitespace. That is a different claim from a string escape — centrs has
|
|
2171
|
+
no continuation token, and matching the device would mean deciding how much
|
|
2172
|
+
following whitespace a continuation owns — so it is **not** folded in here. It
|
|
2173
|
+
belongs with the continuation question #225 already owns (what a continuation
|
|
2174
|
+
does to a value's interior); this paragraph is its size, not a plan.
|
|
2175
|
+
|
|
2176
|
+
Delimiters are the near miss that shows the rule discriminates rather than
|
|
2177
|
+
rubber-stamps: `"` bytes are 311 `string` → `syntax-meta` runs and 88
|
|
2178
|
+
`value` → `syntax-meta`, the device classes them too, and they stay merged
|
|
2179
|
+
anyway, because a consumer can find them by looking at the run's own first and
|
|
2180
|
+
last byte. The `$` sigil and the `[` / `]` of a substitution stay `string` on
|
|
2181
|
+
the same ground — `structure.subcommands` already publishes the resolution, and
|
|
2182
|
+
its `innerSpan` locates the brackets around it.
|
|
2183
|
+
|
|
2184
|
+
### Designed, not implemented (the CLI surface, #202b)
|
|
2185
|
+
|
|
2186
|
+
`src/cli/explain.ts` ships the offline command; implemented flags are generated
|
|
2187
|
+
into [`docs/CLI.md`](../../docs/CLI.md) and are not restated here. What the
|
|
2188
|
+
[Surface](#surface-option-a--decided) table above describes and the command does
|
|
2189
|
+
**not** yet accept:
|
|
2190
|
+
|
|
2191
|
+
| Flag / form | Phase | Why it is not accepted yet |
|
|
2192
|
+
| ----------- | ----- | -------------------------- |
|
|
2193
|
+
| `explain <router> '<input>'` | 2 | The grammar is parsed and **refused** with `usage/not-implemented`, naming the router it did not contact. Running the offline analysis under an invocation that asked for probes would report `mode: "offline"` about a request for more. |
|
|
2194
|
+
| `--cursor <byte>` | 2 | It positions `--complete`, which enumerates nothing offline. A cursor with no candidate surface to position is a flag with no effect. |
|
|
2195
|
+
| `--full` | 3 | It lifts smart-sizing limits, and the offline result is never truncated (`input.truncated` is always `false`). Same rule: no accepted flag is a no-op. |
|
|
2196
|
+
|
|
2197
|
+
`--complete` and `--schema` **are** accepted, because they have an offline
|
|
2198
|
+
behavior the spec names: they enumerate nothing and emit the live-target tip.
|
|
2199
|
+
That is the difference — an accepted flag must do something observable.
|
|
2200
|
+
|
|
2201
|
+
## MCP and library surfaces
|
|
2202
|
+
|
|
2203
|
+
- **Consumers use the layer that answers their question.** Byte ranges and
|
|
2204
|
+
their classes (`spans` and the shipped `tokens` facet) serve editor/token
|
|
2205
|
+
consumers; the resolved reading (`structure`, `symbols`, `values`) carries
|
|
2206
|
+
path, verb, arguments, and value facts; doc/version enrichment belongs to
|
|
2207
|
+
rosetta (#175). A consumer that wants enrichment takes
|
|
2208
|
+
the **middle** layer: given `{path, verb, args}` plus a version it does its
|
|
2209
|
+
own lookups, whereas handing it tokens would make it re-derive from bytes a
|
|
2210
|
+
path centrs already resolved. So #223 is a decision about the structure
|
|
2211
|
+
layer, and the token vocabulary (#264) is deliberately outside its scope —
|
|
2212
|
+
the two do not need to be sequenced against each other. The same split says
|
|
2213
|
+
where a schema-shaped gap belongs: the per-path argument list `catalog.ts`
|
|
2214
|
+
refuses to be is enrichment, not lexing.
|
|
2215
|
+
- `centrs_explain` today wraps `canonicalizeExecuteCommand` (offline, no CDB —
|
|
2216
|
+
the only tool that serves without one). It grows toward this spec
|
|
2217
|
+
**following the CLI scheme** (decision): same facets, same envelope-shaped
|
|
2218
|
+
data as the CLI's `--json`, optionally a CDB-resolved target for live
|
|
2219
|
+
evidence. The no-CDB offline path must survive, and the MCP adapter must
|
|
2220
|
+
not widen the structured gate. The current flat `centrs_explain` shape
|
|
2221
|
+
(`{ input, mode, path, verb, attributes, queries, writeShaped }`) is
|
|
2222
|
+
**superseded** when this lands — a deliberate pre-1.0 breaking change, with
|
|
2223
|
+
`commands/mcp/` examples and integration tests updated in the same change;
|
|
2224
|
+
no dual-shape compatibility layer is planned. **That supersession is held out
|
|
2225
|
+
of #202 and tracked in #223** (maintainer decision): `centrs_explain` and
|
|
2226
|
+
rosetta's tool surface must stay aligned for agent usage, which is a
|
|
2227
|
+
cross-project decision. Until #223 lands the MCP keeps its flat shape, so the
|
|
2228
|
+
CLI's `--json` and the MCP's `data` are **deliberately divergent**.
|
|
2229
|
+
- Library: `explainCommand(input, options?: ExplainCommandOptions)` is the
|
|
2230
|
+
offline analysis (`src/explain.ts`, #202a/#202c) and
|
|
2231
|
+
`explainEnvelope(input, options?: ExplainEnvelopeOptions)` wraps it in the
|
|
2232
|
+
standard envelope. Both option types are exported. Phase 2 extends that same
|
|
2233
|
+
analysis bag with `target` and `facets`; `curl` stays an orthogonal rendering
|
|
2234
|
+
request, so offline and live calls do not need competing second-argument
|
|
2235
|
+
shapes. `lsp-routeros-ts` and
|
|
2236
|
+
tikbook are the intended external consumers (hover/diagnostics/completion/
|
|
2237
|
+
semantic tokens over these calls); an LSP *protocol* surface on centrs
|
|
2238
|
+
stays out of scope (#90). A bounded offline browser/editor consumption check
|
|
2239
|
+
now precedes vocabulary stabilization (#264 B5); full live LSP integration
|
|
2240
|
+
remains later work. The entry is `@tikoci/centrs/explain` and it is browser-verified —
|
|
2241
|
+
see [The offline package entry](#the-offline-package-entry-312).
|
|
2242
|
+
|
|
2243
|
+
### The offline package entry (#312)
|
|
2244
|
+
|
|
2245
|
+
**`@tikoci/centrs/explain` -> `src/explain.ts`** is the documented public entry for
|
|
2246
|
+
offline analysis. It is a real boundary, not a naming convention: the module
|
|
2247
|
+
graph reachable from it opens no connection, reads no CDB, and touches no
|
|
2248
|
+
filesystem, so it bundles and runs in a browser, a Worker, or an editor host.
|
|
2249
|
+
|
|
2250
|
+
```sh
|
|
2251
|
+
bun run explain:browser-consumer # boundary + executed consumer report
|
|
2252
|
+
bun run explain:browser-consumer --json # machine-readable
|
|
2253
|
+
```
|
|
2254
|
+
|
|
2255
|
+
Both halves are gated in the unit suite by
|
|
2256
|
+
`test/unit/explain-browser-entry.test.ts`, so neither needs a router or the
|
|
2257
|
+
corpus.
|
|
2258
|
+
|
|
2259
|
+
**Why a gate and not a note.** The entry reached the transport stack until this
|
|
2260
|
+
landed, for two *pure* helpers: `toYaml` (from `retrieve.ts`) and
|
|
2261
|
+
`canonicalizeExecuteCommand`/`isWriteShaped` (from `execute.ts`). Importing
|
|
2262
|
+
either dragged in `mac-telnet` (`node:dgram`, `node:os`), `native-api`
|
|
2263
|
+
(`node:crypto`) and `ssh`. Both now live in `src/core/` — `src/core/yaml.ts` and
|
|
2264
|
+
`src/core/execute-command.ts` — moved character for character, with `execute.ts`
|
|
2265
|
+
re-exporting the gate because it remains that gate's documented public home.
|
|
2266
|
+
**Nothing about the script-vs-structured gate moved with them**; the locked
|
|
2267
|
+
contract table in `test/unit/execute-canonicalize-contract.test.ts` still imports
|
|
2268
|
+
from `execute` and still passes.
|
|
2269
|
+
|
|
2270
|
+
That coupling was re-introduced once already by a one-line import, which is why
|
|
2271
|
+
the static half of the check names the offending **edge** (`explain.ts ->
|
|
2272
|
+
execute.ts -> mac-telnet-console.ts -> mac-telnet.ts`) rather than leaving a
|
|
2273
|
+
reader with the bundler's polyfill complaint four modules downstream.
|
|
2274
|
+
|
|
2275
|
+
**What "runs in a browser" is proven by.** Compilation is not the proof. A
|
|
2276
|
+
bundler silently substitutes browser polyfills, so the check bundles a worker
|
|
2277
|
+
entry around the module with `--target browser`, wraps the output in a function
|
|
2278
|
+
that shadows every host global (`Bun`, `process`, `require`, `module`,
|
|
2279
|
+
`exports`, `__dirname`, `__filename`), and **spawns that bundle as the Worker**.
|
|
2280
|
+
`Bun` is a non-configurable global — `delete globalThis.Bun` throws — so
|
|
2281
|
+
shadowing, not deletion, is what makes the absence real, and doing it in the
|
|
2282
|
+
bundler's banner keeps the harness free of `eval`/`new Function` while making
|
|
2283
|
+
the bundle text on disk the artifact that carries the claim.
|
|
2284
|
+
|
|
2285
|
+
It then compares the whole `ExplainData` against the Bun-native run, field for
|
|
2286
|
+
field: structure, diagnostics, tokens, spans, `canonical`, and
|
|
2287
|
+
`runtimeAcceptance`. Fourteen cases, covering normal and write-shaped commands,
|
|
2288
|
+
multi-statement and nested input, three malformed classes, Unicode, astral, a
|
|
2289
|
+
leading BOM, IPv6, empty input, and the facets switched off. All fourteen are
|
|
2290
|
+
byte-identical today.
|
|
2291
|
+
|
|
2292
|
+
Two things the bundle deliberately exercises rather than bypasses. It imports
|
|
2293
|
+
through the **published subpath** `@tikoci/centrs/explain`, not
|
|
2294
|
+
`../src/explain.ts`, so a typo in `package.json`'s `exports` map fails the gate
|
|
2295
|
+
instead of leaving it green while every real consumer breaks (the graph walk
|
|
2296
|
+
additionally asserts that the `exports` entry and the measured file are the same
|
|
2297
|
+
path). And it calls `resolveExplainFormat(undefined)`, whose `env` default is
|
|
2298
|
+
the `typeof`-guarded `hostEnv()` — nothing else reaches that parameter, so
|
|
2299
|
+
without this a regression to a bare `Bun.env` would pass every case here while a
|
|
2300
|
+
browser caller of that exported function got a `TypeError`.
|
|
2301
|
+
|
|
2302
|
+
The shadowing is **measured from inside the bundle**, not assumed: the worker
|
|
2303
|
+
reports which host globals it could still see, and an empty list is part of the
|
|
2304
|
+
gate. Without it, a wrapper that silently stopped being emitted would leave
|
|
2305
|
+
every "identical" above produced with host APIs in reach, proving nothing. Only
|
|
2306
|
+
`Bun` and `process` are probed, because they are the only two of the seven a
|
|
2307
|
+
bundle can honestly observe — Bun constant-folds `typeof require` to
|
|
2308
|
+
`"function"` at build time, mentioning `__dirname`/`__filename` makes it inject
|
|
2309
|
+
the build machine's absolute paths as local constants (the probe would cause
|
|
2310
|
+
what it reports), and mentioning `module`/`exports` reclassifies the entry as
|
|
2311
|
+
CommonJS, whose top-level `export default` is a syntax error inside the wrapper.
|
|
2312
|
+
All seven stay shadowed regardless.
|
|
2313
|
+
|
|
2314
|
+
The one tolerated host builtin is **`node:net`**, for `isIP` in
|
|
2315
|
+
`src/explain/values.ts` — a pure predicate with no socket in it. A bundler
|
|
2316
|
+
replaces it with a regex pair, so the browser build genuinely runs a *different*
|
|
2317
|
+
`isIP` than Bun does; the two IPv6 cases exist to price that substitution, which
|
|
2318
|
+
is the only place its verdict is observable. A builtin that polyfills to a stub
|
|
2319
|
+
(`node:fs`) or throws (`node:dgram`) could not be checked this way, so the
|
|
2320
|
+
allowlist is a boundary, not a list to extend.
|
|
2321
|
+
|
|
2322
|
+
**What it does not claim.** Not that the `centrs` package works in a browser —
|
|
2323
|
+
it does not, and is not meant to. Not DOM integration, no real browser engine.
|
|
2324
|
+
And these cases are a boundary corpus, not a parser-coverage corpus; measured
|
|
2325
|
+
parser evidence stays in the censuses and
|
|
2326
|
+
[Device agreement](#device-agreement-for-the-token-partition-264-b4-263).
|
|
2327
|
+
|
|
2328
|
+
**What this hands #264 B5.** The comparison serializes the entire public result,
|
|
2329
|
+
so it pins the vocabulary a consumer actually depends on: every `tokens[]` class
|
|
2330
|
+
name, every span pair, and the `ExplainData` key set are now load-bearing across
|
|
2331
|
+
a process boundary. A B5 rename is therefore a visible break here rather than a
|
|
2332
|
+
silent one — which is the point of settling the vocabulary before an external
|
|
2333
|
+
consumer commits to it.
|
|
2334
|
+
|
|
2335
|
+
## Non-goals
|
|
2336
|
+
|
|
2337
|
+
- No LSP server in centrs — the LSP is a consumer the library must support,
|
|
2338
|
+
nothing more.
|
|
2339
|
+
- No execution probes; no state mutation of any kind.
|
|
2340
|
+
- No fan-out.
|
|
2341
|
+
- No rosetta/restraml integration at runtime (decision, amended by #228): no
|
|
2342
|
+
shell-outs, no calls, no vendored *schema* artifact, no version-history facts.
|
|
2343
|
+
The two generated structure tables are built from pinned public sources by
|
|
2344
|
+
scripts in `scripts/`, and centrs vendors its own CLI-Reference ETL rather
|
|
2345
|
+
than depending on rosetta for it. Output may tip toward rosetta for
|
|
2346
|
+
docs/version questions; deeper integration waits on centrs's shape settling
|
|
2347
|
+
and rosetta publishing stable consumable surfaces.
|
|
2348
|
+
- No promise of enum exhaustiveness or of the upstream `completion-tricks`
|
|
2349
|
+
synthetic-probe recipes (still open research in lsp-routeros-ts
|
|
2350
|
+
`BACKLOG.md`) — candidates are labeled as observed, not closed sets.
|
|
2351
|
+
|
|
2352
|
+
## Phase-0 ratification (#185)
|
|
2353
|
+
|
|
2354
|
+
The phase-0 canonicalization grounding lab (#185, rev 2) is complete. It scored
|
|
2355
|
+
a schema-free offline parser against device truth — `:parse` IL and
|
|
2356
|
+
`/console/inspect` highlight, cross-checked on CHR 7.23.2 and 7.24rc2 — over a
|
|
2357
|
+
frozen train/holdout split of the 913-script lsp-routeros-ts corpus, plus
|
|
2358
|
+
constructed corner and mutation suites. Each question below names the spec claim
|
|
2359
|
+
it gated and links its exit artifact (the artifact carries the fixture/split
|
|
2360
|
+
hashes, RouterOS versions, oracle source, per-class precision/recall/abstention,
|
|
2361
|
+
and disagreement samples). Probes are throwaway (`.scratch/`, non-authoritative);
|
|
2362
|
+
the findings here are the deliverable, promoted per
|
|
2363
|
+
`.github/instructions/routeros-grounding.instructions.md`.
|
|
2364
|
+
|
|
2365
|
+
**Ratified as designed:** Q1 statement segmentation · Q2 block topology · Q3
|
|
2366
|
+
`[]` sub-command path re-constitution · Q4 stateful-per-document path context ·
|
|
2367
|
+
Q6 no-schema verb/menu boundary (**decision 3 unchanged**) · Q8 tested REST
|
|
2368
|
+
mapping rules + fail-closed boundary · Q10 `.proplist` recipe · Q13 symbol scope
|
|
2369
|
+
classes.
|
|
2370
|
+
|
|
2371
|
+
**Amended the surface (folded in above):**
|
|
2372
|
+
|
|
2373
|
+
| Finding | Amendment | Exit artifact |
|
|
2374
|
+
| ------- | --------- | ------------- |
|
|
2375
|
+
| **Q16** write-shape | `structure.containsWrite` is three-valued `true \| false \| unknown` (boolean rejected as unsafe); zero false negatives on obvious writes; `unknown` separate from `false` | [#5027578882](https://github.com/tikoci/centrs/issues/185#issuecomment-5027578882) |
|
|
2376
|
+
| **Q6/Q3/Q13/Q14** schema-free floor | a distinct **`ambiguous`** canonicalizer verdict; fail-closed rule promoted to the constitution | [#5025480566](https://github.com/tikoci/centrs/issues/185#issuecomment-5025480566) |
|
|
2377
|
+
| **Q14** malformed/partial | diagnostics carry a **defect region**; fail-closed recovery (no invented commands after a defect; bare-word heads → `unknown`); `--complete` bounded offline | [#5048400737](https://github.com/tikoci/centrs/issues/185#issuecomment-5048400737) |
|
|
2378
|
+
| **Q15** coordinates | ASCII-normalization is **byte-count-preserving** (non-ASCII byte → placeholder, one-for-one) so analyzed offsets ≡ highlight byte offsets; invariants validated 100% | [#5048304595](https://github.com/tikoci/centrs/issues/185#issuecomment-5048304595) |
|
|
2379
|
+
| **Q11** native-api readout | the cell is **not** degraded — `:put [:parse …]` over native `/execute` with `as-string` returns line/column; the old console-only caveat is retired | [#5046924529](https://github.com/tikoci/centrs/issues/185#issuecomment-5046924529) |
|
|
2380
|
+
| **Q17** robustness | product parser bounds nesting depth (over-depth diagnostic) and scans single-pass; determinism + well-formed span trees are property tests | [#5040370326](https://github.com/tikoci/centrs/issues/185#issuecomment-5040370326) |
|
|
2381
|
+
|
|
2382
|
+
**Pre-registered thresholds — outcomes:** `api-candidate`/write-shape precision
|
|
2383
|
+
and coordinate invariants met; the **one hard-0 threshold that was *not* met is
|
|
2384
|
+
Q14's "no fabrication after the first defect"** — the throwaway lab walker
|
|
2385
|
+
invents a command in 3 of 76 mutants, which is exactly why the fail-closed
|
|
2386
|
+
bare-word rule (b) above is a **hard product requirement**, not a nicety. The lab
|
|
2387
|
+
probes were left unpatched by design; the guard belongs in the product parser.
|
|
2388
|
+
|
|
2389
|
+
**Corpus coverage caveat (#203).** Every figure above was measured on the
|
|
2390
|
+
913-script corpus, which is **96.8% two forum authors** (`amm0` 630,
|
|
2391
|
+
`rextended` 254) and **53.8% scripting-directive-bearing**. Pure `add`/`set`
|
|
2392
|
+
configuration is **2.3%** (21 files), genuine device-emitted `/export` output
|
|
2393
|
+
is **1.0%** (9 files), and the three `/export` serializations — `compact`,
|
|
2394
|
+
`terse`, `verbose` — were never captured at phase 0. Nearly every file also
|
|
2395
|
+
opens with a harness-injected `# Source: …` line the device never wrote (884
|
|
2396
|
+
of 913). The ratified answers hold for what was measured; they are **not**
|
|
2397
|
+
evidence of coverage over device configuration, which is the genre most likely
|
|
2398
|
+
to be handed to `explain` and the one that concentrates Q4 path context and
|
|
2399
|
+
Q6's bare-path ambiguity. Quote phase-0 numbers per genre, never as one blended
|
|
2400
|
+
figure. #203 records the capture decision and caveat; #336 owns the current
|
|
2401
|
+
corpus repin and remeasurement.
|
|
2402
|
+
|
|
2403
|
+
Those figures are re-derivable, not eyeballed: `bun run explain:corpus-census`
|
|
2404
|
+
(`scripts/explain-corpus-census.ts`) reproduces the whole genre table from
|
|
2405
|
+
`corpus.sqlite`, with a per-collection stratum breakdown, and is the source of
|
|
2406
|
+
truth over #203's prose where the two disagree — it corrects the issue's
|
|
2407
|
+
export-banner count (9, not 8: the issue's own 22 − 13 shebangs) and its
|
|
2408
|
+
pure-config count (21, not 23: the first pass counted MAC addresses, IPv6
|
|
2409
|
+
literals, and DHCP client-ids as `:` directives, which under-counts exactly the
|
|
2410
|
+
config genre in question). Since phase 0 the corpus has grown to 948 with a
|
|
2411
|
+
`tangentsoft` stratum of 35 genuine `/export … terse` device captures, taking
|
|
2412
|
+
export-banner share to 4.6%; **that stratum is not part of any phase-0 figure
|
|
2413
|
+
above**. Current whole-corpus offline censuses include its source text.
|
|
2414
|
+
|
|
2415
|
+
The pinned snapshot carries **three complete builds** — long-term `7.23.5`,
|
|
2416
|
+
stable `7.24.2` and development `7.25beta3` — each with *both* oracles over all
|
|
2417
|
+
948 scripts, alongside six partial captures (parseIL at 7.20.8, 7.22.1,
|
|
2418
|
+
7.23rc1; highlight at 7.9.2, 7.23.2, 7.24rc2) that cover only 913. Schema v4
|
|
2419
|
+
names that split rather than leaving it to be discovered by hand: the
|
|
2420
|
+
`v_version_coverage` view classifies each build, and
|
|
2421
|
+
`coverage_class = 'complete'` is the filter every version-comparable query uses.
|
|
2422
|
+
`source_scripts` is byte-identical across both repins that reached this
|
|
2423
|
+
snapshot, so every text-derived report — the token and value censuses,
|
|
2424
|
+
`explain:arg-reach`, the browser-consumer proof — is unchanged by construction,
|
|
2425
|
+
and only the IL-derived operator census moved.
|
|
2426
|
+
|
|
2427
|
+
**The three complete builds do not agree with each other, and that is the
|
|
2428
|
+
finding.** Scored over all 948 scripts:
|
|
2429
|
+
|
|
2430
|
+
| oracle | long-term 7.23.5 ↔ stable 7.24.2 | stable 7.24.2 ↔ development 7.25beta3 |
|
|
2431
|
+
| ------ | -------------------------------: | ------------------------------------: |
|
|
2432
|
+
| `:parse` IL (`il_sha256`) | 87 scripts differ | 11 |
|
|
2433
|
+
| `highlight` (count + class set) | 33 | 0 |
|
|
2434
|
+
|
|
2435
|
+
Stable and development are the same build for these purposes; long-term is the
|
|
2436
|
+
one that moves. So an operator-axis or token-partition figure is a claim *about
|
|
2437
|
+
a channel*, not about RouterOS — which is why the census reports its builds as
|
|
2438
|
+
separate columns and states its version set, rather than collapsing them to one
|
|
2439
|
+
number. Long-term is also the channel a real device most likely runs.
|
|
2440
|
+
|
|
2441
|
+
The corpus itself is not in this repo and is not moving here — `lsp-routeros-ts`
|
|
2442
|
+
owns producing snapshots, centrs owns which snapshot it measures against.
|
|
2443
|
+
`scripts/corpus-pin.json` is that second half: the commit and the blob sha256 a
|
|
2444
|
+
census run measures. It identifies the **current** 948-script snapshot only —
|
|
2445
|
+
the phase-0 figures above were measured on the frozen 913-script corpus and no
|
|
2446
|
+
pin describes them. `bun run corpus:fetch` downloads and verifies it, so the
|
|
2447
|
+
censuses run from a bare clone and in CI rather than on one machine (#186).
|
|
2448
|
+
|
|
2449
|
+
**All three export serializations parse clean offline (#203 deliverable 2,
|
|
2450
|
+
re-scored after #207).** Every one of the 120 CHR captures — `compact`, `terse`
|
|
2451
|
+
and `verbose` × `hide-sensitive` / `show-sensitive` × both pinned versions —
|
|
2452
|
+
runs through the promoted modules with **zero segmenter structural notes, 100%
|
|
2453
|
+
Q4 path resolution, and 0.0% Q16 abstention with no blockers anywhere in the
|
|
2454
|
+
stratum**. The last figure is new: before #207's baked menu table, the
|
|
2455
|
+
`isConfirmedNav` hyphen rule abstained on 25 distinct genuine menus. This is
|
|
2456
|
+
the coverage claim that matters for `explain`, and it holds.
|
|
2457
|
+
|
|
2458
|
+
**Q6's bare-path abstention is closed on the same table (#210).** `verbsplit.ts`
|
|
2459
|
+
originally did not consume `menus.ts`, so Q6 kept returning `ambiguous` on
|
|
2460
|
+
`compact`/`verbose` for exactly the bare-path lines the container table can
|
|
2461
|
+
decide. It now reads that table at V4 — the one case the ratified `proposed`
|
|
2462
|
+
rule refuses — and emits a `navigation` resolution when the path is a known
|
|
2463
|
+
menu. Abstention on device-emitted export goes to **0.0% on every stratum**
|
|
2464
|
+
(from 37.6–38.4% on `compact`/`verbose`; `terse` was already 0% by
|
|
2465
|
+
construction, since it never emits a bare menu line), and on the frozen corpus
|
|
2466
|
+
from 4.0% → **1.4%** dev and 1.2% → **0.5%** holdout.
|
|
2467
|
+
|
|
2468
|
+
The table is consulted in `resolveVerb`, never in `splitRun`, so the ratified
|
|
2469
|
+
schema-free rule still prices decision 3 on its own terms and Q16's abstention
|
|
2470
|
+
is unchanged to the decimal (44.8% dev / 46.0% holdout). Precision was checked
|
|
2471
|
+
against the device parser rather than asserted: over 1,689 IL (`:parse`)
|
|
2472
|
+
projections — which distinguish `menuctx` from `cmd` natively — the table never
|
|
2473
|
+
calls a menu what the device read as a command, neither across the whole table
|
|
2474
|
+
nor across the 104 paths this rule flips. The residual is the right residual:
|
|
2475
|
+
`/system/reboot`, `/quit`, `/tool/speed-test` and abbreviated forms like
|
|
2476
|
+
`/ip fire conn` stay `ambiguous`, because the table is a floor and absence from
|
|
2477
|
+
it is not evidence of a command. *(#228 has since decided all but the
|
|
2478
|
+
abbreviated forms — see below.)*
|
|
2479
|
+
|
|
2480
|
+
**The contradicting half of that seam is closed too (#211 B1).**
|
|
2481
|
+
`pathresolve.ts` claimed **every** `/`-led bare path was navigation, with no
|
|
2482
|
+
check that the path is a menu, so `/ip address print` navigated to the
|
|
2483
|
+
non-existent `/ip/address/print` while `verbsplit` read the identical text as
|
|
2484
|
+
verb `print` at `/ip/address` — two promoted modules, contradictory answers,
|
|
2485
|
+
34 dev / 13 holdout statements, `verbsplit` right in every one. `menuNavPath`
|
|
2486
|
+
now consults the same frozen `VERBS` set (R9), which takes those statements to
|
|
2487
|
+
**0 dev / 0 holdout** at no abstention cost: they were already decided, just
|
|
2488
|
+
decided wrongly. The vocabulary moved to the leaf `verbs.ts` so both modules
|
|
2489
|
+
share one object and cannot drift apart, and the invariant — a
|
|
2490
|
+
`verbsplit`-`resolved` statement is never `isNav` — is pinned in
|
|
2491
|
+
`test/unit/explain-pathresolve.test.ts` alongside R9's premise: no path in the
|
|
2492
|
+
generated `MENU_PATHS` table carries a verb segment. That premise is scoped to
|
|
2493
|
+
the table, not to RouterOS — `menus.ts` is a floor, so it cannot rule out an
|
|
2494
|
+
unlisted menu named `.../print`, which R9 would read as a command.
|
|
2495
|
+
|
|
2496
|
+
R9 left the harder half open (**#211 B2**): a bare path with *no* verb
|
|
2497
|
+
(`/system/reboot`, `/quit`, `/terminal/cuu`) was still read as navigation and
|
|
2498
|
+
still cascaded, because the menu table is a floor and absence from it is not
|
|
2499
|
+
evidence of a command. B2 was parked rather than guessed at for exactly that
|
|
2500
|
+
reason — the only rule available would have had to invert the floor contract.
|
|
2501
|
+
|
|
2502
|
+
**#228 closes B2 for the published subset, without touching the floor.** The
|
|
2503
|
+
command axis is the missing *positive* evidence, so R12 refuses navigation into
|
|
2504
|
+
a path a table SAYS is a command rather than into one another table happens not
|
|
2505
|
+
to mention. `/system/reboot` no longer moves the context, so a statement after
|
|
2506
|
+
it resolves where it actually is; `verbsplit` and `pathresolve` agree, which
|
|
2507
|
+
they would otherwise now do *confidently* in opposite directions. What is left
|
|
2508
|
+
is the honest residue of both floors — `/terminal/cuu` is a real no-argument
|
|
2509
|
+
command MikroTik publishes nowhere — and it stays pinned as a KNOWN LIMIT
|
|
2510
|
+
fixture rather than assumed closed. The contradiction that #202 would have
|
|
2511
|
+
rendered is gone.
|
|
2512
|
+
|
|
2513
|
+
**The command axis also outranks the punctuation guess.** `splitRun`'s last
|
|
2514
|
+
rule reads the first space-separated token as the verb, which lands on the
|
|
2515
|
+
operand whenever the command itself was written slash-joined:
|
|
2516
|
+
`/system/gps/monitor once` put the verb on `once`, and
|
|
2517
|
+
`/interface ethernet reset-counters ether1` on `ethernet`. A published command
|
|
2518
|
+
names its own boundary, so the operand goes back to being an argument and
|
|
2519
|
+
`argsAt` moves with it. Over the frozen corpus plus the export stratum, 171 of
|
|
2520
|
+
19,002 statement readings change and every one names a better token; Q16
|
|
2521
|
+
abstention moves 44.8% → **44.6%** dev, 46.0% holdout and 0.0% export unchanged,
|
|
2522
|
+
and exactly one document changes verdict (`unknown` → `false`, a pasted
|
|
2523
|
+
`/system/gps/monitor once` transcript — an abstention #207 recorded as correct
|
|
2524
|
+
*given no evidence*, which there now is). The command axis says which token is
|
|
2525
|
+
the verb; it says nothing about whether that verb mutates, so `write.ts`'s
|
|
2526
|
+
curated vocabularies are unchanged and `/system/reboot` still abstains.
|
|
2527
|
+
|
|
2528
|
+
`splitRun` itself is untouched again — both tables are consumed by the callers —
|
|
2529
|
+
so the ratified schema-free rule keeps pricing decision 3 on its own terms. Two
|
|
2530
|
+
places deliberately do *not* consult the catalog: scripting directives, where
|
|
2531
|
+
R10 was measured 2,088-to-18 and the seven root-level catalog commands all sit
|
|
2532
|
+
where R10 already puts them; and a bare-word head at the ROOT context, because
|
|
2533
|
+
those seven are ordinary English words and the corpus's pasted `import serial` /
|
|
2534
|
+
`import io` / `import os` would otherwise read as RouterOS `/import`. An
|
|
2535
|
+
*inherited* menu context is itself the evidence that the surrounding text is
|
|
2536
|
+
RouterOS, so a relative `stop [find …]` under `/container` does resolve.
|
|
2537
|
+
|
|
2538
|
+
**Importability was also measured, and is a weaker, narrower result than it
|
|
2539
|
+
first appears (#203 deliverable 4).** On CHR, none of the six root `/export`
|
|
2540
|
+
documents imports into a blank same-version CHR; per-menu exports do, and
|
|
2541
|
+
round-trip byte-exactly including continuation-wrapped comments. RouterOS
|
|
2542
|
+
`/import` is fail-fast, so it aborts at the first error and every later
|
|
2543
|
+
statement silently never runs. But **all four failure causes are semantic or
|
|
2544
|
+
device-state rejections, not syntax**: a duplicate object, a built-in object,
|
|
2545
|
+
a missing mandatory value, and a name collision. That was verified rather than
|
|
2546
|
+
assumed — each blocker statement was replayed through `:put [:parse …]` and
|
|
2547
|
+
`/console/inspect request=highlight` on both versions and **accepted every
|
|
2548
|
+
time**, with controls (`no-such-attr=`, an unterminated string, a nonexistent
|
|
2549
|
+
menu) correctly rejected. So these results carry **no** implication about
|
|
2550
|
+
whether `explain` can parse the forms, and they do not license claims about the
|
|
2551
|
+
export styles in general: the captures come from one synthetic configuration on
|
|
2552
|
+
CHR, which has a single NIC and no switch or wireless hardware, so its defconf
|
|
2553
|
+
and its available menus differ from any real device's. Treat the provenance
|
|
2554
|
+
label as **L1-emitted, L2-refuted for these captures on CHR**. Harness and
|
|
2555
|
+
per-document results: `.scratch/explain-lab-q203-import-verify.ts`;
|
|
2556
|
+
validity check: `.scratch/q203-blocker-validity.ts`.
|
|
2557
|
+
|
|
2558
|
+
**Non-blocking / deferred (not ratification-gating):** Q5 expression depth, Q7
|
|
2559
|
+
tokenizer corners, Q9 potential-command taxonomy, Q12 span-vocabulary draft —
|
|
2560
|
+
these refine the surface during implementation but did not gate ratification.
|
|
2561
|
+
Two of the four have since closed. **Q7** became #201 (lexical boundaries —
|
|
2562
|
+
sigil spellings, escape validity, statement-start eligibility) and #199 (the
|
|
2563
|
+
shared scanner's `$[…]`-in-string blind spot), both raised by the Q13 promotion
|
|
2564
|
+
and both now closed. **#199**'s fix is the one shared scanner,
|
|
2565
|
+
`src/explain/quoted-string.ts`, which is what keeps the segmenter, the brace
|
|
2566
|
+
classifier and the block scan from drifting on where a string ends; **#201** is
|
|
2567
|
+
the wider one, spanning statement context, sigil/escape reading and defect
|
|
2568
|
+
detection across several already-promoted modules.
|
|
2569
|
+
**Q12** closed with #264: the vocabulary is settled on a stated rule and is no
|
|
2570
|
+
longer provisional — see
|
|
2571
|
+
[The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5). **Q5** and
|
|
2572
|
+
**Q9** remain phase-1 measurements. The
|
|
2573
|
+
probe framework's disposition (reuse of the mutation suite, coordinate
|
|
2574
|
+
fixtures, and adversarial generators as product test fixtures) is tracked in
|
|
2575
|
+
issue #186 rather than carried into mainline.
|
|
2576
|
+
|
|
2577
|
+
## Definition of done and staging
|
|
2578
|
+
|
|
2579
|
+
Verification follows the [constitution](../../docs/CONSTITUTION.md#done-definition).
|
|
2580
|
+
The phase-0 grounding and promoted phase-1 CLI/library core (#185/#202) are
|
|
2581
|
+
already in the product. The remaining task graph is maintained in #90; issue
|
|
2582
|
+
closure for an earlier slice does not establish the baseline below.
|
|
2583
|
+
|
|
2584
|
+
### Offline baseline acceptance
|
|
2585
|
+
|
|
2586
|
+
The offline capability can advance from `coded` to `verified` when:
|
|
2587
|
+
|
|
2588
|
+
- **Examples and diagnostics:** the offline examples and focused analysis/CLI
|
|
2589
|
+
contract tests pass. Known false confident readings in the baseline task
|
|
2590
|
+
list, starting with #311's missing separator, are fixed with source-located
|
|
2591
|
+
diagnostics and accepted/malformed controls. Unknown or ambiguous input
|
|
2592
|
+
remains explicit; no fabricated statements, paths, values, or runtime acceptance.
|
|
2593
|
+
- **Bounded execution:** the existing public `explainCommand` performance,
|
|
2594
|
+
coordinate, depth, no-throw, and determinism contracts pass. A reproduced
|
|
2595
|
+
performance failure is investigated before changing its threshold (#313). Cost
|
|
2596
|
+
must grow with input size and no faster: a wall-clock budget on a thermally
|
|
2597
|
+
throttled machine measures the machine, so the growth **shape** is pinned by a
|
|
2598
|
+
ratio between two sizes timed back to back, and an absolute threshold is never
|
|
2599
|
+
raised in place of finding the growing term. One input shape is not enough —
|
|
2600
|
+
#313's separator-free input carries no symbols, so it never reaches the
|
|
2601
|
+
symbol/value machinery where the next growth term was hiding (#317), and
|
|
2602
|
+
neither of those carries a `do={…}`, which is where the third one was (#320).
|
|
2603
|
+
A guarded shape is only guarded for the analyzers it actually runs, so a new
|
|
2604
|
+
growth guard states which analyzers its input reaches and is mutation-tested
|
|
2605
|
+
against the term it removed. **Met:** #322's fixed-size depth guard grows
|
|
2606
|
+
nesting 16× (depth 4 → 64) under the public token/curl path and stays below
|
|
2607
|
+
the 6× bound; deterministic companion guards cover colon-led, bare-directive,
|
|
2608
|
+
retry, array/substitution, bracket, and string-interpolation shapes and fail
|
|
2609
|
+
when child-scope traversal is restored.
|
|
2610
|
+
- **Measured token surface:** the total, gapless `--tokens` partition and
|
|
2611
|
+
generated censuses remain reproducible; #264 B4 / #263's projection and
|
|
2612
|
+
device-agreement report are in
|
|
2613
|
+
[Device agreement](#device-agreement-for-the-token-partition-264-b4-263),
|
|
2614
|
+
re-derivable without a router or the corpus. Report decided errors,
|
|
2615
|
+
abstentions, version/state differences, and corpus bias separately — the
|
|
2616
|
+
report's outcome buckets and its four applicability categories are that
|
|
2617
|
+
separation. Classified-byte percentage is not syntax coverage, and agreement
|
|
2618
|
+
is a trend, not a new numerical pass gate. A token-level rule and a token fill
|
|
2619
|
+
read the skip-tolerant argument stream, never the strict REST reading (#316);
|
|
2620
|
+
the two share one boundary walk, the strict reading's reach is unchanged, and
|
|
2621
|
+
the skip-vs-prefix fork is priced rather than assumed —
|
|
2622
|
+
[The token stream a rule reads](#the-token-stream-a-rule-reads-316). That
|
|
2623
|
+
stream reaches the envelope as the token partition and nowhere else, which is
|
|
2624
|
+
what makes the `=` boundary visible to a consumer whose statement published
|
|
2625
|
+
no arguments at all (#264 B5).
|
|
2626
|
+
- **Usable library contract:** one bounded browser/editor consumer imports a
|
|
2627
|
+
documented public offline entry point, runs analysis without transport/CDB
|
|
2628
|
+
dependencies, and verifies structure, diagnostics, tokens, and source-position
|
|
2629
|
+
mapping against the CLI/library result (#312). **Met** by `@tikoci/centrs/explain` and
|
|
2630
|
+
`bun run explain:browser-consumer` — see
|
|
2631
|
+
[The offline package entry](#the-offline-package-entry-312); the boundary is a
|
|
2632
|
+
module-graph gate and the consumer proof executes the browser bundle rather
|
|
2633
|
+
than only compiling it. The token vocabulary that consumer reads is settled by
|
|
2634
|
+
#264 — [The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5) —
|
|
2635
|
+
on a stated rule, with the two splits it makes (`arg-sep`, then `escaped`)
|
|
2636
|
+
and the merges it keeps each recorded there with the surface that already
|
|
2637
|
+
carries the distinction. A complete editor, LSP server, palette
|
|
2638
|
+
reproduction, or SCIP implementation is not required to prove this contract.
|
|
2639
|
+
- **Reachable evidence and explicit residue:** changed RouterOS semantic claims
|
|
2640
|
+
have committed CHR-grounded controls and a reproducible capture/replay path.
|
|
2641
|
+
Every remaining issue has a stated scope and dependency; #225 distinguishes
|
|
2642
|
+
offline value work from live types, #272 owns evidence reachability, and #211's
|
|
2643
|
+
unlisted-path decision retains the current abstention until separately decided.
|
|
2644
|
+
|
|
2645
|
+
Retain the existing checks: `bun run lint`, `bun run test`, `bun run build`, and
|
|
2646
|
+
`bun run lint:ci`, plus the affected CHR grounding/integration procedure from
|
|
2647
|
+
the constitution. Add the consumer and agreement checks with their implementation.
|
|
2648
|
+
No new rules engine, schema snapshot, final-state interpreter, or network-behavior
|
|
2649
|
+
test framework is part of this baseline.
|
|
2650
|
+
|
|
2651
|
+
### Current and following capabilities
|
|
2652
|
+
|
|
2653
|
+
- **Current — live probes (#236, phase 2):** highlight + `:parse` and
|
|
2654
|
+
completion/child/syntax over rest-api/native-api. B4's projection/agreement
|
|
2655
|
+
informs this work; each live cell requires its own CHR-passing examples.
|
|
2656
|
+
- **MCP alignment (#223):** the deliberate flat-to-rich response migration,
|
|
2657
|
+
coordinated with rosetta. Independent of the offline token vocabulary.
|
|
2658
|
+
- **Facet polish (phase 3):** live sizing, `--full`, completion/schema ergonomics.
|
|
2659
|
+
- **Full LSP integration (phase 4):** consume live and offline results in
|
|
2660
|
+
`lsp-routeros-ts`, building on the earlier offline consumer check.
|
|
2661
|
+
|
|
2662
|
+
## Decisions (2026-07-19 round) and remaining opens
|
|
2663
|
+
|
|
2664
|
+
Decided this round (details inline above; recorded in #90):
|
|
2665
|
+
|
|
2666
|
+
- **Option A** — one verb + facet flags; broad-query ladder
|
|
2667
|
+
(path → verbs, path/verb → args, `print` → `.proplist`); complexity of the
|
|
2668
|
+
refined scheme is an accepted risk.
|
|
2669
|
+
- **Smart sizing** over `--depth`; counts + truncation warning for large
|
|
2670
|
+
subtrees; `--full` (name TBD) as the only override. `--format` stays a
|
|
2671
|
+
rendering switch; colorized command breakdown is a rendering feature.
|
|
2672
|
+
- **No offline schema snapshot** — offline is the canonicalizer, full stop;
|
|
2673
|
+
schema/version truth comes from a live router (steered by tip) and
|
|
2674
|
+
docs/version history stays rosetta's domain.
|
|
2675
|
+
*Amended 2026-08-07 (#228).* The rejected candidate was the
|
|
2676
|
+
restraml/deep-inspect command tree: indirect, version-shaped schema data with
|
|
2677
|
+
substantial churn, and that reasoning still holds for it. The carve-out is
|
|
2678
|
+
narrower: offline ships **no schema** — no argument names, no types, no enums,
|
|
2679
|
+
no per-menu verb lists, no `.proplist` — but it does ship a generated
|
|
2680
|
+
**structure** table (path, kind, provenance, published applicability gate)
|
|
2681
|
+
unioned from pinned RouterOS inspect trees and MikroTik's published CLI
|
|
2682
|
+
Reference. Such a table may say whether a path is navigation or a command; it
|
|
2683
|
+
never describes what a command accepts, and absence from it abstains and never
|
|
2684
|
+
rejects. `src/explain/catalog.ts` is that table. Its **command axis is read**
|
|
2685
|
+
by `pathresolve.ts`, `verbsplit.ts` and `write.ts`. *Amended again by #235:*
|
|
2686
|
+
its **menu/settings kind is read too**, unioned with `menus.ts` — decisive for
|
|
2687
|
+
a relative bare path once its inherited context is applied, and for the
|
|
2688
|
+
absolute bare path `verbsplit.ts` used to call `ambiguous`. Presence still
|
|
2689
|
+
decides in one direction only; absence still abstains. It is still a static
|
|
2690
|
+
snapshot of vendor-published structure, which is why the decision is amended
|
|
2691
|
+
rather than argued around.
|
|
2692
|
+
- **No rosetta coupling for now** — at most steering tips; no calls, no
|
|
2693
|
+
artifacts, no maintained bindings.
|
|
2694
|
+
- **centrs-owned span vocabulary** with RouterOS-fidelity color mapping for
|
|
2695
|
+
LSP consumers; raw highlight classes are provenance/debug, not the surface.
|
|
2696
|
+
- **Grow `canonicalizeExecuteCommand`**; rosetta's `canonicalize.ts` is
|
|
2697
|
+
reference-only, divergence expected.
|
|
2698
|
+
- **MCP mirrors the CLI scheme** and result shape.
|
|
2699
|
+
- **Script-scale explain is in scope** — block/scope analysis and
|
|
2700
|
+
sub-command path re-constitution are target canonicalizer capabilities, and
|
|
2701
|
+
per-statement transport classification (execute vs api, `curl` rendering)
|
|
2702
|
+
is part of the base output.
|
|
2703
|
+
|
|
2704
|
+
Amendments from the PR review pass (Codex/CodeRabbit/Copilot on #184, same
|
|
2705
|
+
round):
|
|
2706
|
+
|
|
2707
|
+
- **Positional grammar is target-first** when live (`explain <router>
|
|
2708
|
+
'<input>'`); one positional = offline input. Matches every other
|
|
2709
|
+
router-taking command and the shared selection helpers.
|
|
2710
|
+
- **Transport classification is fail-closed** (`api-candidate` / `execute` /
|
|
2711
|
+
`unknown`); `curl` renders only from explicit, tested mapping rules.
|
|
2712
|
+
- **The gate verdict is reproduced verbatim**; explain-only mutation
|
|
2713
|
+
inference lives in `structure.containsWrite`, never in `writeShaped`.
|
|
2714
|
+
- **Per-fact provenance is an `evidence[]` table** with stable ids; target
|
|
2715
|
+
identity stays in `meta.target`.
|
|
2716
|
+
- **Envelope mirrors `check`** (`ok: true` + `data.verdict` for analyzed
|
|
2717
|
+
input); coordinate contract (byte offsets, exclusive `end`,
|
|
2718
|
+
original-document mapping for LSP) is explicit.
|
|
2719
|
+
- **`:parse` readout is transport-specific** (constitution); native API uses
|
|
2720
|
+
`/execute as-string` and returns the same parser text as REST/console, while
|
|
2721
|
+
omitting `as-string` returns only the job handle (Q11).
|
|
2722
|
+
- **MCP shape change is a deliberate pre-1.0 break**, migrated in one change.
|
|
2723
|
+
|
|
2724
|
+
Still open (implementation opens — none block ratification):
|
|
2725
|
+
|
|
2726
|
+
1. Final flag names (`--full`, `--curl`, facet names) and the smart-sizing
|
|
2727
|
+
thresholds (what counts as "cheap", what triggers truncation).
|
|
2728
|
+
2. The mapping table from RouterOS highlight classes to **colors**, and how
|
|
2729
|
+
unknown/new upstream classes degrade. Phase 0 pinned the raw inputs (Q13's
|
|
2730
|
+
per-occurrence highlight-class corpus; the observed 19 classes on 7.23.2 plus
|
|
2731
|
+
drift on 7.24rc2). The centrs-owned **class list is no longer open**: #264 B5
|
|
2732
|
+
settled it on a stated rule and `data.tokens[].class` is no longer provisional
|
|
2733
|
+
([The vocabulary](#the-vocabulary-and-what-earns-a-class-264-b5)), and the
|
|
2734
|
+
projection onto the device's own classes is reported by
|
|
2735
|
+
`bun run explain:highlight-agreement`
|
|
2736
|
+
([Device agreement](#device-agreement-for-the-token-partition-264-b4-263)).
|
|
2737
|
+
What stays open here is the color map and the drift policy. Live mappings
|
|
2738
|
+
and full LSP integration follow separately.
|
|
2739
|
+
3. Whether the live describe ladder needs result caching per target+version
|
|
2740
|
+
(probe cost vs freshness) — deferred to implementation evidence.
|
|
2741
|
+
|
|
2742
|
+
Resolved during phase 0:
|
|
2743
|
+
|
|
2744
|
+
- **First CLI→REST families (Q8).** Nine mapping rules are
|
|
2745
|
+
runtime-exercised (2xx on 7.23.2 and 7.24rc2): `add`→`PUT`, `print`→`GET`,
|
|
2746
|
+
`get`→`GET/<id>`, `set`→`PATCH/<id>`, `remove`→`DELETE/<id>`,
|
|
2747
|
+
`run(action)`→`POST/<path>/<command>` (demonstrated by `/ip/dns/cache flush`
|
|
2748
|
+
with an empty body, and read as the family it names, not as the one verb),
|
|
2749
|
+
and `proplist`/`query`→`POST/<path>/print`. Selector
|
|
2750
|
+
writes (`set`/`remove [find …]`), singleton-menu `set`, and non-CRUD actions
|
|
2751
|
+
fail closed to `execute`/`unknown` (no `curl`). Placeholder-host and
|
|
2752
|
+
credential-elision rendering details stay an implementation open; `curl` is
|
|
2753
|
+
REST-only (native-api has no curl analogue).
|
|
2754
|
+
- **Ratification question list (#185).** The phase-0 question list and the spec
|
|
2755
|
+
claims each gated were the deliverable of #185; every ratification-gating
|
|
2756
|
+
question is answered.
|