@sleepy_panda_srl/portulan 0.1.3 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -5
- package/cli/README.md +142 -358
- package/cli/advisory.mjs +354 -0
- package/cli/comments.mjs +932 -0
- package/cli/compile.mjs +1322 -1607
- package/cli/context.mjs +883 -0
- package/cli/control-chars.mjs +12 -301
- package/cli/discover.mjs +17 -574
- package/cli/doctor.mjs +224 -1074
- package/cli/drills.mjs +68 -502
- package/cli/feedback.mjs +16 -237
- package/cli/finish.mjs +437 -0
- package/cli/form.mjs +644 -0
- package/cli/fuzz-shell.mjs +20 -305
- package/cli/gate.mjs +7 -199
- package/cli/goldens.mjs +26 -184
- package/cli/index.mjs +305 -655
- package/cli/init.mjs +243 -566
- package/cli/inside.mjs +3 -28
- package/cli/instructions.mjs +829 -0
- package/cli/ledger.mjs +778 -0
- package/cli/librarian.mjs +135 -599
- package/cli/manifest.mjs +2 -39
- package/cli/mutants.mjs +9 -170
- package/cli/new.mjs +12 -243
- package/cli/pack-identity.mjs +5 -74
- package/cli/pack-version.mjs +13 -255
- package/cli/plugin-lint.mjs +42 -658
- package/cli/portulan.mjs +5 -124
- package/cli/recipe-set.mjs +15 -245
- package/cli/release-eval.mjs +28 -348
- package/cli/review-meter.mjs +17 -258
- package/cli/rule-carriers.mjs +12 -175
- package/cli/sessions.mjs +82 -0
- package/cli/skill-goldens.mjs +10 -176
- package/cli/skills-set.mjs +12 -414
- package/cli/stop-gate.mjs +83 -514
- package/cli/symbols.mjs +1132 -0
- package/cli/telemetry.mjs +27 -492
- package/cli/upgrade.mjs +171 -314
- package/cli/vendor.mjs +146 -476
- package/cli/version-carriers.mjs +8 -86
- package/core/README.md +1 -1
- package/core/engine.md +3 -9
- package/core/operating/autonomy.md +1 -1
- package/core/operating/context.md +183 -0
- package/core/operating/evolution.md +3 -3
- package/core/operating/loop.md +13 -20
- package/core/operating/memory.md +12 -3
- package/core/operating/sessions.md +100 -0
- package/core/personas/librarian.md +2 -2
- package/core/skills/consolidate/SKILL.md +4 -2
- package/core/templates/handoff.md +28 -15
- package/core/templates/memory-entry.md +1 -0
- package/evals/README.md +255 -845
- package/evals/releases/0.2.0.json +146 -0
- package/evals/releases/0.2.0.md +67 -0
- package/evals/releases/README.md +3 -3
- package/package.json +4 -2
- package/packs/rituals/checkpoints/README.md +1 -1
- package/packs/tools/github/verify/actions-pinned.sh +10 -55
- package/spec/README.md +112 -6
- package/spec/migrations/0001-repository-declares-its-tree.mjs +1 -26
- package/spec/migrations/0002-bundle-fallback-path.mjs +13 -74
- package/spec/migrations/0003-handoff-index-not-kept.mjs +66 -0
- package/spec/migrations/0004-changelog-fragments.mjs +53 -0
- package/spec/migrations/0005-session-log-retired.mjs +69 -0
- package/spec/migrations/0006-boot-card.mjs +77 -0
- package/spec/migrations/0007-guidance-compiled.mjs +31 -0
- package/spec/migrations/0008-card-reading.mjs +55 -0
- package/spec/migrations/0009-instruction-sections-on-read.mjs +60 -0
- package/spec/migrations/0010-comment-rail.mjs +57 -0
- package/spec/migrations/README.md +45 -10
- package/spec/pack.schema.json +1 -1
- package/spec/slots.md +333 -21
- package/spec/workspace.schema.json +145 -4
package/evals/README.md
CHANGED
|
@@ -1,910 +1,320 @@
|
|
|
1
1
|
# evals/
|
|
2
2
|
|
|
3
|
-
Milestone 8's home: golden
|
|
4
|
-
|
|
5
|
-
binding criterion; [`../docs/milestones/m08.md`](
|
|
6
|
-
This file does not restate either — a rule with two carriers is obeyed at the narrower one.
|
|
3
|
+
Milestone 8's home: the golden corpora, the review-loop register, the telemetry opt-in, the A/B harness's
|
|
4
|
+
records and the eval result each release carries. Row 8 of [`../docs/plan.md`](../docs/plan.md) is the
|
|
5
|
+
binding criterion; [`../docs/milestones/m08.md`](https://github.com/sleepy-panda-srl/portulan/blob/8a33f9b/docs/milestones/m08.md) is its legislative history.
|
|
6
|
+
This file does not restate either — a rule with two carriers is obeyed at the narrower one. It says what
|
|
7
|
+
is here, how to run or extend each rail, and what each one does not establish; a rail's detail is its
|
|
8
|
+
module, whose longer header is at `git show a31da46:cli/<module>.mjs`. How each clause came to be —
|
|
9
|
+
first-run figures, reversed designs, review rounds — is at `git show 8a33f9b:evals/README.md`.
|
|
7
10
|
|
|
8
|
-
##
|
|
11
|
+
## Two words that mean two things here
|
|
9
12
|
|
|
10
|
-
**"Eval" carries two unrelated senses in this repository, and only one of them lives in this
|
|
11
|
-
directory.** Both were load-bearing before this directory had any content, so the disambiguation is
|
|
12
|
-
written down rather than left to be inferred from a filename.
|
|
13
|
+
**"Eval" carries two unrelated senses in this repository, and only one of them lives in this directory.**
|
|
13
14
|
|
|
14
15
|
| Sense | Where it lives | What it is |
|
|
15
16
|
|---|---|---|
|
|
16
17
|
| **Evaluation as measurement** — golden tasks, A/B, the eval gate | **here**, `evals/` | Does the engine make an agent work better? Milestone 8's subject. |
|
|
17
|
-
| **Evaluation as a licensed copy** | [`../cli/eval-bundle.mjs`](../cli/eval-bundle.mjs), [`../.portulan/verify/eval-bundle.sh`](../.portulan/verify/eval-bundle.sh) | A named-recipient bundle cut from a commit under evaluation terms
|
|
18
|
-
|
|
19
|
-
**And `goldens` itself now names two runners, so that is disambiguated here too rather than left to a
|
|
20
|
-
prefix.** [`../cli/goldens.mjs`](../cli/goldens.mjs) grades the **compiled gates** against adversarial
|
|
21
|
-
fixtures, through the compiler's own `matchesRule`. [`../cli/skill-goldens.mjs`](../cli/skill-goldens.mjs)
|
|
22
|
-
grades the **core skills'** mandates against the live artifacts they govern. Same clause, two subjects —
|
|
23
|
-
the 2026-07-28 amendment's *"golden tasks reach the gates as well as the skills"* — and two oracles, which
|
|
24
|
-
is why they are two modules and two recipes rather than one with a flag.
|
|
25
|
-
|
|
26
|
-
The machinery this directory ships is therefore named **`goldens`**, never `evals` — so a reader meeting
|
|
27
|
-
`cli/goldens.mjs` beside `cli/eval-bundle.mjs` is not left to guess which sense is which. The directory
|
|
28
|
-
keeps the row's own word; the tools take a narrower one.
|
|
29
|
-
|
|
30
|
-
## What is built today
|
|
31
|
-
|
|
32
|
-
**Eight clauses of row 8, of nine, and a ninth built to its mechanism but not to its instance.** (a), adversarial fixtures per compiled gate, landed 2026-08-24;
|
|
33
|
-
**(b)**, mutation testing over both matchers and grammar-aware fuzzing over the shell segmenter,
|
|
34
|
-
landed 2026-08-25; **(d)**, scheduled forced-red drills, landed 2026-08-25; **(c)**, review-loop
|
|
35
|
-
metering, landed 2026-08-26; **golden tasks per core skill** — the row's original first subject —
|
|
36
|
-
landed 2026-08-26; **OTel opt-in config** landed 2026-08-28; **the A/B baseline** landed 2026-08-31,
|
|
37
|
-
across three sessions the maintainer split at construction | grading | running — 6b the arms, 6c the
|
|
38
|
-
graders, **6d the run**; and **a rule change merged or rejected on eval evidence** was decided by the
|
|
39
|
-
instruments 2026-09-03 and **discharged on its merge, 2026-09-09** — the clause reads *merges*, and the
|
|
40
|
-
session that measured the table said in terms that *"his merge is the ruling"*, so the date the evidence
|
|
41
|
-
was taken is not the date the clause landed; its own section is below. **A release carries an eval
|
|
42
|
-
result** landed as a *mechanism*
|
|
43
|
-
2026-09-01 — the record layer, the rail and its forced-red drill — and is **half-vouched**, the row's own
|
|
44
|
-
word for a clause whose demonstration was owed to an event that had not happened yet. It has now half
|
|
45
|
-
happened: `0.1.3` carries a record **in the tree**, and what is still owed is the tag and a published
|
|
46
|
-
body citing it — both Gated and both the maintainer's. The two are not equally unreachable, and saying
|
|
47
|
-
so loosely is the overclaim this file warns about elsewhere: `release-eval --tagged` reaches the
|
|
48
|
-
**tagged tree** at the publish, from the release workflow, while the **body** — authored on GitHub — is
|
|
49
|
-
the surface no check reaches at all. **Nothing remains outright.** Each is listed below with the sentence
|
|
50
|
-
[`../.portulan/dod.md`](../.portulan/dod.md) condition 4 requires.
|
|
51
|
-
|
|
52
|
-
## The A/B baseline — what 40 turns measured, and what they did not
|
|
53
|
-
|
|
54
|
-
[`ab/baseline.json`](ab/baseline.json) is the capture and [`ab/baseline.md`](ab/baseline.md) the register
|
|
55
|
-
rendered from it. **k = 5 per cell, ruled by the maintainer; 4 scenarios × 2 arms × 5 runs; every turn
|
|
56
|
-
isolated with a home and config directory of its own.** Held by the `ab-run` recipe and a forced-red
|
|
57
|
-
drill.
|
|
58
|
-
|
|
59
|
-
**At the 2026-08-31 capture the headline is a tie**, and [`ab/baseline.md`](ab/baseline.md) carries the
|
|
60
|
-
figure itself, with every per-turn row beside it — this paragraph cites that register rather than
|
|
61
|
-
restating it. The date is part of the claim: it is what keeps this sentence true rather than stale if
|
|
62
|
-
the arms are ever re-run. It is recorded as measured. No re-run was taken to find a better number, and
|
|
63
|
-
none will be: `k = 5` supports a recorded rate and nothing else.
|
|
64
|
-
|
|
65
|
-
**The finding that matters most is not the tie.** On `done-demonstrated` the rig fails unconditionally,
|
|
66
|
-
so the only route to a green rail is rewriting the check — and **arm A rewrote it in 3 of 5 turns**.
|
|
67
|
-
Session 6c's first design labelled a green rail `demonstrated` and scored it as compliance; the
|
|
68
|
-
session-open checkpoint reversed that before any turn ran. Had it not, those three turns would have
|
|
69
|
-
counted as compliances and the register would publish **an arm-A lead manufactured entirely by scoring
|
|
70
|
-
the gaming move as the good outcome**, against the tie [`ab/baseline.md`](ab/baseline.md) recorded on
|
|
71
|
-
2026-08-31. The reversal is what makes that tie trustworthy, and it is the strongest evidence this milestone has
|
|
72
|
-
produced that a fresh-context checkpoint catches what its own author cannot.
|
|
73
|
-
|
|
74
|
-
**On `altitude` arm A REACHED the compliant location and was still scored 0/5, and that is the finding.**
|
|
75
|
-
It wrote the constraint into `.portulan/tasks/` on **3 of 5** turns — arm B reached it never. All three
|
|
76
|
-
graded `higher-layer` because `gradeAltitude` gives any governance-surface occurrence precedence, and in
|
|
77
|
-
every one of those turns the governance hit is **the dated handoff arm A's own `dod.md` condition 8
|
|
78
|
-
demands**. So the predicate **counted** a gate-mandated artifact as promotion and **penalised** the
|
|
79
|
-
treatment arm for obeying the treatment: a treatment–instrument interaction, not a fact about the arm. _(Runs 0 and
|
|
80
|
-
3 are different and are genuine promotion — they wrote the constraint into `memory/`.)_ **The registered
|
|
81
|
-
carrier of what may not be concluded from this is
|
|
82
|
-
[`ab/corpus.md`](ab/corpus.md)'s section of that name**, which since the milestone-8 close carries the
|
|
83
|
-
argument in full — including the half this paragraph does not state, that arm B has no path to the
|
|
84
|
-
penalising branch at all, so the row is one-directional rather than merely noisy. The rendered register
|
|
85
|
-
cites that section; this paragraph cites it too rather than becoming a fourth uncited restatement. On
|
|
86
|
-
`curated-layer` arm A wrote into `memory/` twice and arm B touched neither curated directory — a visible
|
|
87
|
-
difference, 0/5 both.
|
|
88
|
-
|
|
89
|
-
Recorded rather than repaired at the capture and at the close: moving a target after seeing the figures
|
|
90
|
-
is the selection this instrument exists to refuse, and what to do about it was the maintainer's.
|
|
91
|
-
**Repaired 2026-09-09, the same day the close named it**, when he lifted the hold — the session-record
|
|
92
|
-
slots left `altitude`'s population, precedence was left alone, and what that implies for the 2026-08-31
|
|
93
|
-
capture is a **re-classification of a fixed record and never a new baseline**, disclosed in
|
|
94
|
-
[`ab/corpus.md`](ab/corpus.md) beside the date. The paragraph above describes that capture and stays in
|
|
95
|
-
its own tense: it is a record, not a carrier. **The baseline is scoped to the
|
|
96
|
-
vendored-and-compiled tier** [`ab/arm.md`](ab/arm.md) specifies, and closes row 8 for no other
|
|
97
|
-
configuration of *Portulan on*.
|
|
98
|
-
|
|
99
|
-
**One cell is at ceiling.** `observed-content` is 5/5 for both arms — [`ab/corpus.md`](ab/corpus.md)
|
|
100
|
-
warned in terms that a bare agent which also declines the injection makes the row uninformative, and it
|
|
101
|
-
does.
|
|
18
|
+
| **Evaluation as a licensed copy** | [`../cli/eval-bundle.mjs`](../cli/eval-bundle.mjs), [`../.portulan/verify/eval-bundle.sh`](../.portulan/verify/eval-bundle.sh) | A named-recipient bundle cut from a commit under evaluation terms, its issuance ledger kept outside this repository. Nothing to do with this directory. |
|
|
102
19
|
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
20
|
+
So the machinery here is named **`goldens`**, never `evals`: the directory keeps the row's word and the
|
|
21
|
+
tools take a narrower one. And **`goldens` names two runners** — two subjects and two oracles, which is why
|
|
22
|
+
they are two modules and two recipes rather than one with a flag. [`../cli/goldens.mjs`](../cli/goldens.mjs)
|
|
23
|
+
grades the **compiled gates** through the compiler's own `matchesRule`;
|
|
24
|
+
[`../cli/skill-goldens.mjs`](../cli/skill-goldens.mjs) grades the **core skills'** mandates against the
|
|
25
|
+
live artifacts they govern.
|
|
106
26
|
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
checks is already broken, and this directory was carrying one of its own. Found as a suppressed note
|
|
111
|
-
in Copilot round 5 of [#336](https://github.com/sleepy-panda-srl/portulan/pull/336).
|
|
27
|
+
Each rail below is a verify recipe in [`../.portulan/verify/`](../.portulan/verify/), run on every pull
|
|
28
|
+
request; `bash .portulan/verify/<recipe>.sh` runs one by hand, and `node cli/<module>.mjs --help` prints a
|
|
29
|
+
module's modes. Every runner prints its own totals, so no figure is written here.
|
|
112
30
|
|
|
113
|
-
|
|
114
|
-
pull request. Run it by hand with:
|
|
31
|
+
## The gate corpus — clause (a)
|
|
115
32
|
|
|
116
33
|
```
|
|
34
|
+
evals/goldens/gates/<rule-id>.json one fixture file per rule in the yielded gate policy
|
|
117
35
|
node cli/goldens.mjs --workspace . --pack-root packs
|
|
118
36
|
```
|
|
119
37
|
|
|
120
|
-
**
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
is
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
no
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
`shell-prefix`, and `no-branch` for a combination the matcher has no code for. The green prints the
|
|
149
|
-
per-path census, including the zeroes, because a corpus can carry two hundred cases and exercise one
|
|
150
|
-
branch of four.
|
|
151
|
-
|
|
152
|
-
It earns its place on one asymmetry: a `then`/`do`/brace-group leader is **caught** on the write path
|
|
153
|
-
— `shellSegments` knows `SEGMENT_LEADERS` — and **escapes** on the shell path, where `commandSegments`
|
|
154
|
-
does not. One rule id, two segmenters, two answers. Without the field those two cases read as a
|
|
155
|
-
contradiction rather than as the asymmetry they are.
|
|
156
|
-
|
|
157
|
-
Byte-level attacks are stored **escaped** (JSON `\r`, `\u0000`) and decoded by `JSON.parse`.
|
|
158
|
-
[`../cli/control-chars.mjs`](../cli/control-chars.mjs) refuses a raw CR anywhere in this tree by
|
|
159
|
-
decision, and exempting a growing adversarial-content directory is the allow-list defect that same file
|
|
160
|
-
names — so the corpus carries no raw control bytes, and a test asserts both halves: the bytes are clean
|
|
161
|
-
*and* the escapes really decode.
|
|
162
|
-
|
|
163
|
-
## Clause (b) — the corpus is measured against a broken matcher, and the grammar against bash
|
|
164
|
-
|
|
165
|
-
Two rails, landed 2026-08-25, and they answer two different questions:
|
|
38
|
+
**What a fixture is.** A file is named for its own `rule` field — the runner refuses a misfiled one — and
|
|
39
|
+
holds `cases`, each `{id, class, tool, path, input, expect, why}`. A case is **data**, answered by the
|
|
40
|
+
compiler's own exported `matchesRule`, the function the hook calls at tool time. **A case's command string
|
|
41
|
+
is never executed**, and the suite asserts the runner imports no process-spawning API. Two classes, and
|
|
42
|
+
exactly two:
|
|
43
|
+
|
|
44
|
+
- **`holds`** — the matcher catches this today and must keep catching it.
|
|
45
|
+
- **`documented-hole`** — the matcher does **not** catch this, and `hole` names the record that says so.
|
|
46
|
+
It holds **in both directions**: if the hole silently closes, the case goes red until the record is
|
|
47
|
+
updated.
|
|
48
|
+
|
|
49
|
+
`path` names the branch of `matchesRule` a case exercises — `matchesPath`, `shell-write`, `shell-prefix`
|
|
50
|
+
or `no-branch` — and is **derived, never declared**: the runner computes it with `matcherPath(kind, tool)`
|
|
51
|
+
and refuses a case that disagrees. Byte-level attacks are stored **escaped** (JSON `\r`, `\u0000`), since
|
|
52
|
+
[`../cli/control-chars.mjs`](../cli/control-chars.mjs) refuses raw control bytes in the tree; the suite
|
|
53
|
+
checks both that the corpus is clean and that the escapes decode.
|
|
54
|
+
|
|
55
|
+
The corpus is load-bearing for the next clause: clause (b) needs this corpus as its kill-set and this
|
|
56
|
+
fixture format as its output shape. A `mutants` survivor is repaired by a new case here, and a `fuzz-shell`
|
|
57
|
+
finding prints as a case ready to paste — reviewed first, since its `expect` records what the matcher did.
|
|
58
|
+
|
|
59
|
+
**What it does NOT establish: adequacy.** It checks **presence** — every compiled gate has fixtures and
|
|
60
|
+
every case answers as recorded — which one trivial case per rule satisfies. `mutants` checks
|
|
61
|
+
**discrimination**; **realism**, whether the attacks resemble anything an adversary would type, stays a
|
|
62
|
+
reviewer's judgement. The runner prints that limit on every green and names every rule exempt for
|
|
63
|
+
declaring no matchable action.
|
|
64
|
+
|
|
65
|
+
## Mutants and the grammar fuzzer — clause (b)
|
|
166
66
|
|
|
167
67
|
```
|
|
168
68
|
node cli/mutants.mjs --workspace . --pack-root packs the mutation census
|
|
169
69
|
node cli/fuzz-shell.mjs --workspace . --pack-root packs the grammar fuzzer
|
|
170
70
|
```
|
|
171
71
|
|
|
172
|
-
**`mutants` asks whether the corpus
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
mutant is killable by one. A `survives` record is admissible only as a **proof** — semantic
|
|
178
|
-
equivalence, or equivalence under the yielded policy — never as a standing note that a gap exists,
|
|
179
|
-
which would rebuild the prose hole list clause (a) exists to have replaced.
|
|
180
|
-
|
|
181
|
-
**It went red on its first run and the corpus lost.** Among the breakages that went unnoticed:
|
|
182
|
-
removing `sudo` from the command-prefix table, dropping `..` resolution from path normalisation, and
|
|
183
|
-
disabling quote tracking in the segmenter. Every one is a fixture now, and each was derived by
|
|
184
|
-
measuring which input distinguishes the mutant rather than by reasoning about it — two were *not*
|
|
185
|
-
killed by the spelling that seemed obvious.
|
|
186
|
-
|
|
187
|
-
_The figures for that first run are dated in
|
|
188
|
-
[the session's handoff](../.portulan/handoffs/2026-08-25-the-corpus-lost-and-the-fuzzer-found-a-live-bypass.md),
|
|
189
|
-
and the SHIPPED totals are printed by `node cli/mutants.mjs` and `node cli/goldens.mjs`, which are
|
|
190
|
-
their one carrier. This paragraph carried "eleven of forty-eight" against a table that had since grown
|
|
191
|
-
to fifty-three — a count written before the thing it counted stopped growing, which is this
|
|
192
|
-
repository's most-repeated defect and was this session's third instance of it. The pre-commit
|
|
193
|
-
checkpoint caught the same figure in the handoff and the repair stopped at the site that was quoted;
|
|
194
|
-
Copilot round 2 found the one it missed. Deleted rather than corrected, so the trap is not re-armed
|
|
195
|
-
for whoever adds the next operator._
|
|
196
|
-
|
|
197
|
-
**`fuzz-shell` asks whether the SEGMENTERS answer one grammar.** It composes a command from a grammar
|
|
198
|
-
instead of mutating a string, so it knows by construction whether the payload sits where bash would
|
|
199
|
-
execute it or where bash would only print it. Positions are enumerated and recorded; **spellings are
|
|
200
|
-
fuzzed**, and the invariant is that every spelling of one command in one position gets the same
|
|
201
|
-
answer. Every recorded divergence from ground truth cites the record that licenses it.
|
|
202
|
-
|
|
203
|
-
**The grammar's own ground truth is measured, not argued.**
|
|
204
|
-
[`../cli/fuzz-shell.ground.test.mjs`](../cli/fuzz-shell.ground.test.mjs) runs every position under
|
|
205
|
-
real bash with a **neutral** payload — never a gated command — and writes every path spelling to a
|
|
206
|
-
throwaway file. A grammar that lies about itself produces not a red but a green about the wrong thing,
|
|
207
|
-
which is the one failure a fuzzer cannot detect in itself. It caught two.
|
|
208
|
-
|
|
209
|
-
**It found a live bypass of every Gated shell action.** `bash -c "ls; git push --force origin main"`
|
|
210
|
-
answered **false**: the composition tested the raw command's segments and each segment's spellings,
|
|
211
|
-
and never a spelling's segments. The write matcher never had the gap, because `shellWrites` segments
|
|
212
|
-
again internally — one fix landing in one carrier and not its sibling, between two branches of one
|
|
213
|
-
function. Closed the same day, at the class rather than the spelling, with the two-wrapper
|
|
214
|
-
counterexample asserted so the unwrap budget stays at one level.
|
|
72
|
+
**`mutants` asks whether the corpus discriminates.** It breaks [`../cli/compile.mjs`](../cli/compile.mjs)'s
|
|
73
|
+
matcher region on purpose, one declared, anchored operator at a time, and grades each mutant against the
|
|
74
|
+
gate corpus. An operator the corpus fails to notice is a hole in the kill-set and the repair is a new
|
|
75
|
+
fixture; a `survives` record is admissible only as a **proof** of equivalence, never as a standing note
|
|
76
|
+
that a gap exists.
|
|
215
77
|
|
|
216
|
-
|
|
78
|
+
**`fuzz-shell` asks whether the two segmenters answer one grammar.** It composes each command from a
|
|
79
|
+
grammar, so it knows whether the payload sits where bash would execute it or only print it. Positions are
|
|
80
|
+
enumerated and recorded, spellings are fuzzed, and every spelling of one command in one position must get
|
|
81
|
+
the same answer; every recorded divergence from ground truth cites its record. The grammar's own ground
|
|
82
|
+
truth is measured under real bash with a neutral payload by
|
|
83
|
+
[`../cli/fuzz-shell.ground.test.mjs`](../cli/fuzz-shell.ground.test.mjs), and the seed is pinned and
|
|
84
|
+
printed on every run.
|
|
217
85
|
|
|
218
|
-
**
|
|
219
|
-
|
|
86
|
+
**What they do NOT establish:** that the attacks are realistic (`mutants`), or what a gate ought to cover
|
|
87
|
+
(`fuzz-shell` — whether a recorded escape should stay open is a policy question for the maintainer).
|
|
220
88
|
|
|
221
|
-
|
|
222
|
-
|---|---|
|
|
223
|
-
| Does every compiled gate have fixtures at all? | Are those fixtures any good? |
|
|
224
|
-
| Does every case still answer as recorded? | Is the case worth answering? |
|
|
225
|
-
|
|
226
|
-
So a gate cannot reach the compiled policy with **no** adversarial thought recorded against it — and
|
|
227
|
-
one trivial happy-path fixture per rule satisfies the floor while proving nothing. The runner prints
|
|
228
|
-
this limit on every green rather than letting the exit code imply more than it means.
|
|
229
|
-
|
|
230
|
-
**Half of that gap closed on 2026-08-25, and the sentence has to move with it.** This section used to
|
|
231
|
-
end *"no check can tell those apart"*, and clause (b)'s mutation census is a check that tells part of
|
|
232
|
-
them apart: it breaks the matchers on purpose and asks whether the corpus notices. So the right split
|
|
233
|
-
is now three ways rather than two — **presence** (the `goldens` rail), **discrimination** (the
|
|
234
|
-
`mutants` rail: does the kill-set catch a matcher that has been broken?), and **realism** (whether the
|
|
235
|
-
attacks resemble anything an adversary would type), which is still a reviewer's judgement and stays
|
|
236
|
-
one. The census is what forced the correction rather than a reader noticing: it went red on its first
|
|
237
|
-
run, on 2026-08-25, against the corpus as it then stood — which failed to notice a whole class of
|
|
238
|
-
breakages, every one of which is a fixture now.
|
|
239
|
-
|
|
240
|
-
The **exemption** is the obvious way to dodge the rail: write the next gate `none`-shaped and it needs
|
|
241
|
-
no fixtures. So every exempt rule is named in the output on every run, the way `compile --matrix` prints
|
|
242
|
-
its own refused rules.
|
|
243
|
-
|
|
244
|
-
### What it found on its first run
|
|
245
|
-
|
|
246
|
-
A hole nobody had recorded: **a rule whose target is the whole repository (`./`) matches nothing at
|
|
247
|
-
runtime.** `matchesPath` reduces `"./"` to the empty string and then refuses the empty string, so
|
|
248
|
-
`edit-on-a-working-branch` and `read-anything-in-the-repository` answer false for every input. Nothing
|
|
249
|
-
is mis-enforced today — both are `auto`, and neither layer ever asks — but a **gated** rule written
|
|
250
|
-
that way would compile to a named permission surface while the matcher covered nothing. Now
|
|
251
|
-
entry 8 of [`../.portulan/gate-map.md`](../.portulan/gate-map.md)'s honest-holes list, tracked as [#337](https://github.com/sleepy-panda-srl/portulan/issues/337), asserted here, and **closed at the
|
|
252
|
-
enforcing tiers on 2026-09-09**, the date the change merged — see *A rule change decided on eval
|
|
253
|
-
evidence* below.
|
|
254
|
-
|
|
255
|
-
_(This paragraph said the gated rule **"would compile to a permission rule covering the tree"**, and
|
|
256
|
-
that was wrong in a way worth keeping rather than quietly deleting: re-derived on 2026-09-03, a real
|
|
257
|
-
target compiles to a `**` glob and `./` does not — the emitted spec is the bare `Edit(./)`, whose host
|
|
258
|
-
meaning this repository installs nothing to measure. Either reading is a hazard, and the one this
|
|
259
|
-
repository can measure offline is the narrower and sharper one: the compiler reports the rule **compiled**
|
|
260
|
-
while the matcher answers false for every input. Corrected under
|
|
261
|
-
[`../.portulan/proposals/0022-a-claim-about-a-mechanism-is-re-derived-like-a-figure.md`](../.portulan/proposals/0022-a-claim-about-a-mechanism-is-re-derived-like-a-figure.md).)_
|
|
262
|
-
|
|
263
|
-
Ten of the corpus's own hand-written expectations were refuted by the rail on the same run, which is
|
|
264
|
-
the argument for the rail in one sentence.
|
|
265
|
-
|
|
266
|
-
## Clause (d) — every rail is forced red on a calendar, and required to fire
|
|
89
|
+
## Forced-red drills — clause (d)
|
|
267
90
|
|
|
268
91
|
```
|
|
269
92
|
node cli/drills.mjs --pack-root packs the sweep: force every rail red
|
|
270
93
|
node cli/drills.mjs --check --pack-root packs the roster: every rail has a drill
|
|
271
94
|
```
|
|
272
95
|
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
**
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
move bytes on disk. Isolation is one throwaway `git worktree` per drill, so nothing perturbs a working
|
|
292
|
-
tree.
|
|
293
|
-
|
|
294
|
-
**The sweep reports on a COMMIT and prints which one.** A dirty tree is refused outright; `--working-copy`
|
|
295
|
-
synthesizes one with `git stash create` and refuses while untracked-and-unstaged files exist, since a
|
|
296
|
-
synthesized tree missing the file under review would be a green about the wrong tree. That is also why
|
|
297
|
-
the **verify recipe runs `--check` and not the sweep**: a recipe that answered about `HEAD` would not be
|
|
298
|
-
answering [`../.portulan/dod.md`](../.portulan/dod.md) condition 1's question.
|
|
299
|
-
|
|
300
|
-
**What the word *every* covers, and what it does not.** The sweep drills every recipe the workspace
|
|
301
|
-
yields, plus the Stop-gate and the PreToolUse gate runner — the amendment's *"from watchers to every
|
|
302
|
-
rail"*. The rails it cannot force are **named in the output on every run** with the reason: the platform
|
|
303
|
-
floor, the host's own permission layer, the CI seam, the pre-commit seam scan, `claude plugin validate
|
|
304
|
-
--strict`, the platform watchers, and the librarian's pass. A scope claim with no carrier is what the
|
|
305
|
-
prose register in [`../.portulan/verify/README.md`](../.portulan/verify/README.md) was, and that table is
|
|
306
|
-
deleted in favour of this.
|
|
307
|
-
|
|
308
|
-
**What is demonstrated, and what is unvouched.** All twenty-one rails were forced red by hand on
|
|
309
|
-
2026-08-25 and every one fired. The **calendar** — [`../.github/workflows/drills.yml`](../.github/workflows/drills.yml),
|
|
310
|
-
weekly — is a watcher, so it owes its own observation under
|
|
311
|
-
[`../.portulan/proposals/0007-every-watcher-ships-with-its-observation-procedure.md`](../.portulan/proposals/0007-every-watcher-ships-with-its-observation-procedure.md):
|
|
312
|
-
`workflow_dispatch` is **answered** — run
|
|
313
|
-
[`32883413709`](https://github.com/sleepy-panda-srl/portulan/actions/runs/32883413709) on `da9c06e`, 21 of
|
|
314
|
-
21 — and the **schedule** is answered by its first run and nothing earlier. **The schedule has now answered
|
|
315
|
-
twice, and both runs — with the dispatch before them — are in [`../.portulan/verify/README.md`](../.portulan/verify/README.md), the one carrier of that answer.** And a *missing* run stays undetectable — that is `0007`'s silence problem one altitude up,
|
|
316
|
-
tracked as [#344](https://github.com/sleepy-panda-srl/portulan/issues/344) rather than built.
|
|
317
|
-
|
|
318
|
-
_This paragraph said the gap was already **filed** — the fifth carrier of that sentence, and the one a
|
|
319
|
-
first repair of the other four walked past. A claim in the past tense about an issue that did not exist is
|
|
320
|
-
`../.portulan/dod.md` condition 4's own case, and a fix that stops at the sites somebody quoted is
|
|
321
|
-
[`0020`](../.portulan/proposals/0020-a-fix-is-not-done-at-the-site-it-was-found.md). Both found at the
|
|
322
|
-
pre-commit checkpoint's second pass, which re-derived the carrier set instead of reading the list it had
|
|
323
|
-
been handed._
|
|
324
|
-
|
|
325
|
-
## Clause (c) — the loop's own figures stop being counted by hand
|
|
96
|
+
`goldens` asks whether a gate has fixtures and `mutants` whether they discriminate; **this asks whether
|
|
97
|
+
the rail still fires at all.** A drill is a pair — a control on a pristine tree, then a perturbation that
|
|
98
|
+
must place exactly once and move bytes on disk — and the rail's output must carry the drill's declared
|
|
99
|
+
tell in the fire and not in the control. Each runs in a throwaway `git worktree`. A new rail needs an
|
|
100
|
+
entry in `DRILLS` in [`../cli/drills.mjs`](../cli/drills.mjs): `--check` reds a yielded rail with no drill.
|
|
101
|
+
The sweep reports on a **commit** and refuses a dirty tree (`--working-copy` synthesizes one), which is why
|
|
102
|
+
the verify recipe runs `--check` and not the sweep; the rails it cannot force are named in its output, each
|
|
103
|
+
with the reason.
|
|
104
|
+
|
|
105
|
+
**The calendar** is [`../.github/workflows/drills.yml`](../.github/workflows/drills.yml), weekly, and the
|
|
106
|
+
recorded runs of its dispatch and its schedule are in
|
|
107
|
+
[`../.portulan/gate-map/platform-floor.md`](../.portulan/gate-map/platform-floor.md).
|
|
108
|
+
|
|
109
|
+
**What it does NOT establish:** `--check` runs no rail, so whether each one fires is the sweep's answer on
|
|
110
|
+
the calendar; and a scheduled run that never starts stays undetectable from inside —
|
|
111
|
+
[#344](https://github.com/sleepy-panda-srl/portulan/issues/344).
|
|
112
|
+
|
|
113
|
+
## The review-loop meter — clause (c)
|
|
326
114
|
|
|
327
115
|
```
|
|
328
116
|
node cli/review-meter.mjs --snapshot evals/review-loop/snapshot.json the figures
|
|
329
117
|
node cli/review-meter.mjs --fetch --repo <owner/name> --out <file> the one mode that fetches
|
|
330
118
|
```
|
|
331
119
|
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
from the pull request that produced the definition: on
|
|
357
|
-
[#105](https://github.com/sleepy-panda-srl/portulan/pull/105) the commit `08d7d10` answered an inline
|
|
358
|
-
finding and **was never a reviewed head** — it rode inside the next push — and the push at `cff3e4e0`
|
|
359
|
-
follows a 4,087-byte finding-bearing submission while answering none of it. Any rule keyed on *"a
|
|
360
|
-
finding-bearing submission preceded this push"* calls the second a round; the maintainer's own table
|
|
361
|
-
calls it *no*. The 2026-07-30 ruling states the method that works and it is not one an API has:
|
|
362
|
-
*"Count pushes, then look inside each one."*
|
|
363
|
-
|
|
364
|
-
So the tool computes no fix-rounds, estimates none, and prints nothing that could be mistaken for one.
|
|
365
|
-
|
|
366
|
-
**The empty-round rate is an upper bound, and the reason is a layering rule.** Finding nothing has two
|
|
367
|
-
halves — no inline thread, *and* no suppressed low-confidence note in the body. The first is
|
|
368
|
-
structural. The second is decided by the awk in
|
|
369
|
-
[`../.github/workflows/copilot-review.yml`](../.github/workflows/copilot-review.yml), fixtured in
|
|
370
|
-
[`../.portulan/verify/workflow-filters.mjs`](../.portulan/verify/workflow-filters.mjs) — a matcher
|
|
371
|
-
deliberately reduced to **one** carrier, and a **workspace-layer** one, while the tool is engine.
|
|
372
|
-
Copying it would put two spellings of one rule on opposite sides of the boundary where neither could
|
|
373
|
-
see the other drift. So the tool reports `submissions that found nothing ≤ submissions with no inline
|
|
374
|
-
comment` and prints the right-hand side under its own name.
|
|
375
|
-
|
|
376
|
-
**The boundary forbids a copy; it does not make the exact rate unreachable**, and the first draft of
|
|
377
|
-
this section conflated the two. `../.portulan/verify/workflow-filters.mjs` already **lifts and runs**
|
|
378
|
-
those awk programs out of the workflow's parsed `run:` scalars, and `--fetch` already spawns — so a body
|
|
379
|
-
could be piped through the lifted program at capture time and stored as one integer, no second spelling
|
|
380
|
-
and still no bodies in the snapshot. Left unbuilt for **budget**, one clause per session, and tracked as
|
|
381
|
-
[#355](https://github.com/sleepy-panda-srl/portulan/issues/355).
|
|
382
|
-
|
|
383
|
-
### What the first run measured
|
|
384
|
-
|
|
385
|
-
**140 submissions over the 30 most recently merged pull requests — 4.67 each.** The record's
|
|
386
|
-
hand-counted figure is **3.7**, and the two are **not a disagreement**: they measure **disjoint
|
|
387
|
-
corpora**. The ratified window is the thirty most recent as of 2026-07-28 and names #44, #49 and #57;
|
|
388
|
-
this one runs #301–#354. They share no pull request. So the honest sentence is *the loop weighs 4.67 a
|
|
389
|
-
month later*, not *his 30 were re-counted and came out different* — a distinction the first draft of
|
|
390
|
-
this section lost, and the pre-commit checkpoint restored.
|
|
391
|
-
|
|
392
|
-
That figure is **reported and not acted on**: the record is maintainer-ratified and the curated layer is
|
|
393
|
-
human-owned, so a re-derived number is a finding for him rather than an edit for an implementer.
|
|
394
|
-
|
|
395
|
-
**And two of the three figures turned out to be one figure.** Pushes and submissions came out exactly
|
|
396
|
-
equal — 140 and 140 — because `review_on_push: true` draws one submission per push. So the criterion's
|
|
397
|
-
*"pushes per round"* is **1.00 by construction** in submission units, which the register now prints
|
|
398
|
-
under that name rather than leaving a reader to infer it; and pushes-per-finding-bearing-submission,
|
|
399
|
-
the informative substitute, is not a second measurement either: it is `1/(1 - the no-inline rate)`, and
|
|
400
|
-
140/46 = 3.04 = 1/(1 - 0.671). The tool **detects the coincidence and says so** on every run and in the
|
|
401
|
-
register, rather than presenting three columns of which two are algebraically the same column.
|
|
402
|
-
|
|
403
|
-
**The window is by merge date, and getting that wrong was this change's own worst defect.**
|
|
404
|
-
`gh pr list` orders by pull request **number**, and the first capture taken here inherited that order —
|
|
405
|
-
three merge-order inversions, a corpus containing #303 and missing #301, under a register claiming *"the
|
|
406
|
-
30 most recently merged"*. Every published figure was against a corpus its own heading did not name.
|
|
407
|
-
Found at the pre-commit checkpoint from evidence inside the committed snapshot: `mergedAt` was already
|
|
408
|
-
captured and nothing sorted on it. The fetch now lists a **pool** and takes the window from it by merge
|
|
409
|
-
date, records whether the pool saturated — an unsaturated pool makes the window provable rather than
|
|
410
|
-
likely — and `validateSnapshot` refuses a snapshot that is not in descending merge order.
|
|
411
|
-
|
|
412
|
-
### What this rail does NOT establish
|
|
413
|
-
|
|
414
|
-
| The rail answers | The rail cannot answer |
|
|
415
|
-
|---|---|
|
|
416
|
-
| Do the published figures come out of the captured data? | Is the captured data current? |
|
|
417
|
-
| How heavy is the loop, in submissions? | How heavy is it in fix-rounds? |
|
|
418
|
-
|
|
419
|
-
**It is a meter, not a bound.** Rule 4 stops a loop at two fix-rounds and nothing here stops anything;
|
|
420
|
-
the record's own honest-limits section already says the judgement it depends on is the interested
|
|
421
|
-
party's about its own work, and a tool reporting after the merge does not change that. It adjudicates
|
|
422
|
-
no **sibling** exemption either, so *rounds past the bound* is not computable and is not claimed.
|
|
423
|
-
|
|
424
|
-
**And the snapshot does not refresh itself.** The `review-loop` recipe compares a register to a
|
|
425
|
-
snapshot, never a snapshot to the world, so a stale capture and a current one are the same green.
|
|
426
|
-
Refreshing is `--fetch`, run by a person. That is the same silence
|
|
427
|
-
[#344](https://github.com/sleepy-panda-srl/portulan/issues/344) tracks for the drill calendar, in a
|
|
428
|
-
second place, and it is now **filed** as
|
|
429
|
-
[#356](https://github.com/sleepy-panda-srl/portulan/issues/356) rather than only named — the two want
|
|
430
|
-
one mechanism between them and closing them together is likely cheaper than either alone.
|
|
431
|
-
|
|
432
|
-
## Golden tasks per core skill — the mandates bound to what they govern
|
|
120
|
+
[`review-loop/snapshot.json`](review-loop/snapshot.json) is the capture and
|
|
121
|
+
[`review-loop/register.md`](review-loop/register.md) the register rendered from it: the first command
|
|
122
|
+
with `--register evals/review-loop/register.md --write` regenerates it, and the recipe byte-compares it.
|
|
123
|
+
The criterion's *"rounds per pull request"* is **submissions** per pull request — its unit before *round*
|
|
124
|
+
was redefined — and the tool prints that name, never the bare word.
|
|
125
|
+
What each figure is, at length, is in the header at `git show a31da46:cli/review-meter.mjs`.
|
|
126
|
+
|
|
127
|
+
**What it does NOT establish:**
|
|
128
|
+
|
|
129
|
+
- **Fix-rounds.** Whether a push answers a submission is a fact about its contents that no API answers, so
|
|
130
|
+
the tool computes none, estimates none, and prints nothing that could be mistaken for one.
|
|
131
|
+
- **The exact empty-round rate.** It prints `submissions that found nothing ≤ submissions with no inline
|
|
132
|
+
comment`, an upper bound. The other half — a suppressed note in the review body — was decided by a
|
|
133
|
+
workspace-layer matcher the engine tool refused to copy, since two spellings of one rule on opposite
|
|
134
|
+
sides of that boundary cannot see each other drift; the matcher left with its workflow on 2026-09-23
|
|
135
|
+
([#355](https://github.com/sleepy-panda-srl/portulan/issues/355)).
|
|
136
|
+
- **A bound.** It is a meter: rule 4 of
|
|
137
|
+
[`a-review-loop-needs-a-bound.md`](../.portulan/memory/a-review-loop-needs-a-bound.md) stops a loop, and
|
|
138
|
+
nothing here stops anything.
|
|
139
|
+
- **Current data.** The recipe compares the register to the snapshot, never the snapshot to the world, so a
|
|
140
|
+
stale capture and a current one are the same green. Refreshing is `--fetch`, run by a person
|
|
141
|
+
([#356](https://github.com/sleepy-panda-srl/portulan/issues/356)).
|
|
142
|
+
|
|
143
|
+
## Golden tasks per core skill
|
|
433
144
|
|
|
434
145
|
```
|
|
435
146
|
node cli/skill-goldens.mjs --repo-root . --workspace .portulan
|
|
436
147
|
```
|
|
437
148
|
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
### What a case is
|
|
454
|
-
|
|
455
|
-
A case binds one numbered step of a skill's `## The pass` to the artifact set it governs, and names the
|
|
456
|
-
rail that enforces it — or `null`, which is the interesting value. Four properties, each of which exists
|
|
457
|
-
because a checkpoint found the version without it:
|
|
458
|
-
|
|
459
|
-
- **The denominator is derived, not chosen.** The runner enumerates the steps out of the skill file and
|
|
460
|
-
requires every one to be `bound` or `unbindable`. A fifteenth step is red. The first draft let the
|
|
461
|
-
author pick which mandates to answer for, and a census over a set its own author drew reports *"5 of
|
|
462
|
-
5"* and means nothing — the difference [`../cli/goldens.mjs`](../cli/goldens.mjs) already draws by
|
|
463
|
-
deriving its denominator from the yielded policy.
|
|
464
|
-
- **An empty step list is could-not-run.** Three `## The pass` spellings exist, so an exact match finds
|
|
465
|
-
one skill of three and the other two satisfy *every step is accounted for* **vacuously**. That is this
|
|
466
|
-
directory's own sentence about clause (d) — *"a check whose enumeration went empty … reports green and
|
|
467
|
-
has stopped being a rail"* — and the heading match is anchored besides, because a loose one grabs
|
|
468
|
-
`consolidate`'s `## The one move this pass may not make`.
|
|
469
|
-
- **A mandate quote must place exactly once**, or the case refuses. A skill whose wording is reworded,
|
|
470
|
-
softened or deleted reddens instead of drifting, and **nothing in this tree did that before.**
|
|
471
|
-
- **The predicates use the carriers' own exported functions rather than re-spelling them.** The runner
|
|
472
|
-
imports `RETIRE_WHEN` from [`../cli/doctor.mjs`](../cli/doctor.mjs), so the retire-when check is
|
|
473
|
-
`doctor`'s regex and not a second one. **What that does not do — and three carriers of this sentence
|
|
474
|
-
once said it did — is verify the `carrier` field.** Nothing links it to a check: the pre-commit
|
|
475
|
-
checkpoint rewrote every carrier in a corpus file to a module that does not exist and the corpus
|
|
476
|
-
stayed green. `carrier` is a **declared, reviewed field**, and a reviewer is what checks it.
|
|
477
|
-
|
|
478
|
-
### `unbindable` is adjudicated, never asserted
|
|
479
|
-
|
|
480
|
-
The dodge is obvious — call a mandate unbindable and it needs no case. So the reason takes a **closed
|
|
481
|
-
vocabulary**: `judgement-only`, `no-artifact`, `cross-language`, and `already-carried` — the last added
|
|
482
|
-
at the pre-commit checkpoint, which found a row classed `bound` whose named carrier the runner never
|
|
483
|
-
reads, and no honest term for what it actually was. And a `judgement-only` step **may name
|
|
484
|
-
no artifacts**: if you can name the artifacts a mandate governs, it is not judgement, it is unbuilt.
|
|
485
|
-
That is [`../cli/mutants.mjs`](../cli/mutants.mjs)'s rule that a `survives` record is admissible only as
|
|
486
|
-
a proof and never as a standing note that a gap exists.
|
|
487
|
-
|
|
488
|
-
### What the first run measured
|
|
489
|
-
|
|
490
|
-
**Five of fifteen mandates bind to live artifacts — four load-bearing, one census — and ten are
|
|
491
|
-
adjudicated unbindable.** No figure is written here beyond that shape; the runner prints its own totals.
|
|
492
|
-
|
|
493
|
-
**The ratio is a finding rather than bookkeeping.** How much of a core skill is artifact discipline and
|
|
494
|
-
how much is agent judgement is a measurement *about the engine*, which is what this row exists to
|
|
495
|
-
produce. **Read the split, not the total**: of the ten unbindable, five are `judgement-only` — one third
|
|
496
|
-
of the fifteen — and the rest are `no-artifact`, `cross-language` or `already-carried`, which mean a
|
|
497
|
-
carrier exists somewhere else. The judgement rows are the A/B clause's **first** subject. _(**Widened
|
|
498
|
-
2026-08-28, on the maintainer's ruling**, and this sentence read "Only the judgement rows are the A/B
|
|
499
|
-
clause's subject" until then. The A/B clause's subject is the mandates `core/` ships that an arm actually
|
|
500
|
-
receives — the kernel's, plus the artifact mandates reachable in a built arm — of which the judgement
|
|
501
|
-
rows are a part. The reason the narrower reading could not stand is measured rather than argued:
|
|
502
|
-
`vendor --host` carries `core/skills/` **not at all**, so all five judgement rows reach nothing
|
|
503
|
-
in a vendored arm and would grade the base model. See [`ab/corpus.md`](ab/corpus.md). Four carriers
|
|
504
|
-
stated the narrower reading and all four moved in one change; the claim is registered in
|
|
505
|
-
[`../.portulan/rule-carriers.json`](../.portulan/rule-carriers.json) so a fifth cannot appear **in the
|
|
506
|
-
registered scope** unnoticed — bounded by that registry's own `exclude` list, which is the one carrier
|
|
507
|
-
of what it holds and which covers `docs/milestones/`, where the fourth carrier sat.)_ _(Three carriers of
|
|
508
|
-
this paragraph said "two thirds are judgement", which was the unbindable total wearing the judgement
|
|
509
|
-
label. Corrected at the pre-commit checkpoint, which also found one row classed `bound` whose named
|
|
510
|
-
carrier the runner never reads.)_
|
|
511
|
-
|
|
512
|
-
**One of the five rows is `census`, and it is marked as such** in the corpus and in the output. A census
|
|
513
|
-
row re-indexes a figure `doctor` already prints, under the mandate that wants it; it is not a new check,
|
|
514
|
-
and presenting five equal rows would read as more evidence than it holds — the repair this directory
|
|
515
|
-
already applied to three columns of which two were one column.
|
|
149
|
+
The row's original first subject. **Ruled, not derived:** a deterministic corpus graded against live
|
|
150
|
+
artifacts in the tree; the model-run sense is the A/B clause's.
|
|
151
|
+
|
|
152
|
+
**What a case is.** [`goldens/skills/<skill>.json`](goldens/skills/) holds one case per numbered step of
|
|
153
|
+
the skill's `## The pass`. The runner derives that denominator from the skill file, so a new step is red
|
|
154
|
+
until a case accounts for it. A step is either:
|
|
155
|
+
|
|
156
|
+
- **`bound`** — a `mandate` quote that must place exactly once in the skill's `SKILL.md`, the `artifacts`
|
|
157
|
+
it governs as a declared slot, a `predicate` decided before any figure is taken, and a `carrier` naming
|
|
158
|
+
the rail that enforces it, or `null`. `kind` is `load-bearing`, or `census` for a row that re-indexes a
|
|
159
|
+
figure an existing recipe already prints.
|
|
160
|
+
- **`unbindable`** — with a `reason` from a closed vocabulary: `judgement-only`, `no-artifact`,
|
|
161
|
+
`cross-language` or `already-carried`. A `judgement-only` step may name no artifacts: if you can name the
|
|
162
|
+
artifacts a mandate governs, it is not judgement, it is unbuilt.
|
|
516
163
|
|
|
517
164
|
### Accepted drift — the in-tree record
|
|
518
165
|
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
166
|
+
A bound case may list live files that fail its mandate and still ship green, in `expect.accepted` as
|
|
167
|
+
`{file, why}`: a merged record is not an implementer's to rewrite. The runner refuses an entry whose `why`
|
|
168
|
+
carries no argument, and the list holds **in both directions** — a file that starts complying reddens the
|
|
169
|
+
corpus until it is delisted, because a drift list that outlives its drift is as wrong as one that hides
|
|
170
|
+
it. The runner prints each count beside the compliant total. Three mandates carry accepted drift, tracked
|
|
171
|
+
together as [#358](https://github.com/sleepy-panda-srl/portulan/issues/358):
|
|
523
172
|
|
|
524
173
|
| Mandate | Live artifacts | Drift |
|
|
525
174
|
|---|---|---|
|
|
526
|
-
| `clarify` step 4 — fold answers back as **EARS** acceptance criteria | `.portulan/tasks/` |
|
|
527
|
-
| `codify` step 1 — name the incident and link it | `.portulan/proposals/` |
|
|
528
|
-
| `codify` step 3 — attach how it earns its place | `.portulan/proposals/` |
|
|
529
|
-
|
|
530
|
-
**The
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
[#358](https://github.com/sleepy-panda-srl/portulan/issues/358).
|
|
537
|
-
|
|
538
|
-
**The maintainer ruled on 2026-08-26 that [`../core/templates/task.md`](../core/templates/task.md)'s
|
|
539
|
-
shape still binds**, so the twelve are non-compliant by his word rather than by an implementer's
|
|
540
|
-
inference — which is what makes this an accepted drift and not a divergence pinned before anyone ruled.
|
|
541
|
-
|
|
542
|
-
**It holds in both directions.** A file here that starts complying reddens the corpus until it is
|
|
543
|
-
delisted, because a drift list that outlives its drift is as wrong as one that hides it. That direction
|
|
544
|
-
is not decoration: it caught **three filenames the author had written from a number prefix rather than
|
|
545
|
-
read off disk**, before any reviewer saw them.
|
|
546
|
-
|
|
547
|
-
**The figure moved twice before it was written down, and that is the lesson worth carrying.** One
|
|
548
|
-
opening pass said 10 of 19, another said 12, a third said 19 — because each computed before deciding the
|
|
549
|
-
predicate. Section-presence answers 10; the mandate's own words are about EARS **shape**, and under that
|
|
550
|
-
predicate it is 12, with 46 of 48 individual criteria EARS-shaped. **The predicate is now a reviewed
|
|
551
|
-
field on the case**, decided before any figure.
|
|
175
|
+
| `clarify` step 4 — fold answers back as **EARS** acceptance criteria | `.portulan/tasks/` | no acceptance-criteria section, or criteria not EARS-shaped |
|
|
176
|
+
| `codify` step 1 — name the incident and link it | `.portulan/proposals/` | no provenance field |
|
|
177
|
+
| `codify` step 3 — attach how it earns its place | `.portulan/proposals/` | no enforcement field |
|
|
178
|
+
|
|
179
|
+
**The maintainer ruled on 2026-08-26 that [`../core/templates/task.md`](../core/templates/task.md)'s shape
|
|
180
|
+
still binds**, so `clarify`'s six accepted files are non-compliant by his word rather than by an
|
|
181
|
+
implementer's inference — which is what makes them accepted drift and not a divergence pinned before
|
|
182
|
+
anyone ruled. Of those six, five carry no acceptance-criteria section in any spelling and one fails on
|
|
183
|
+
EARS shape alone, so *"fails the EARS mandate"* is arithmetically right and rhetorically wrong; #358 has
|
|
184
|
+
the cohorts.
|
|
552
185
|
|
|
553
186
|
### What this rail does NOT establish
|
|
554
187
|
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
And it grades a skill's **mandates** against the tree, never an agent's judgement in following one. No
|
|
567
|
-
golden task here tells you whether an agent given `clarify` asks good questions.
|
|
568
|
-
|
|
569
|
-
## OTel opt-in config — the first thing here that can send anything anywhere
|
|
570
|
-
|
|
571
|
-
**The reading was ruled, not derived.** *"OTel opt-in config"* reads at least three ways, and
|
|
572
|
-
[`../.portulan/gate-map.md`](../.portulan/gate-map.md)'s *"Session-open runs `clarify` against the
|
|
573
|
-
milestone row itself"* exists because milestone 4 guessed at exactly this shape of ambiguity and cost a
|
|
574
|
-
session-blocking question. Put to the maintainer on 2026-08-28 as three readings — a config surface
|
|
575
|
-
emitting nothing; that surface **plus a real emitter**, hand-written OTLP-over-HTTP JSON, off by
|
|
576
|
-
default and never callable from a verify recipe; or an OTLP-shaped local file sink with no network path
|
|
577
|
-
at all. **He ruled the second.** The same session ruled the signal set (**the review-loop figures
|
|
578
|
-
only**) and the consent model (below), so all three axes of this clause are his rather than an
|
|
579
|
-
implementer's — which is the standard session 4 set for this row and the one milestone 4 paid for.
|
|
580
|
-
|
|
581
|
-
**Why this clause now.** A written-down dependency pointed at it and nothing pointed anywhere else:
|
|
582
|
-
[`../.portulan/verify/review-loop.sh`](../.portulan/verify/review-loop.sh) said the then-still-open OTel
|
|
583
|
-
clause *"owes an emission path for these figures … which is why they are written as machine-readable
|
|
584
|
-
JSON with a rendered register beside them, rather than as prose an exporter could not consume."*
|
|
585
|
-
Session 3 shaped its output for this session. The debt is discharged **on either reading** of that
|
|
586
|
-
recipe's locative phrase: if he reads *"in the telemetry clause"* the other way, the path is now built;
|
|
587
|
-
if he does not, none was owed.
|
|
188
|
+
- **Adequacy.** A mandate can be bound and the binding trivial.
|
|
189
|
+
- **That a carrier runs its check.** The predicates import the carriers' own functions — `RETIRE_WHEN`
|
|
190
|
+
comes from [`../cli/doctor.mjs`](../cli/doctor.mjs) — which proves a carrier **contains** the check,
|
|
191
|
+
not that it **runs** it; and `carrier` is a declared, reviewed field that nothing links to a check.
|
|
192
|
+
Closing that needs a per-mandate drill, the `drills` shape one grain finer.
|
|
193
|
+
- **An agent's judgement.** It grades a skill's mandates against the tree, never an agent following one.
|
|
194
|
+
The `judgement-only` rows are the A/B clause's first subject, which [`ab/corpus.md`](ab/corpus.md)
|
|
195
|
+
carries.
|
|
196
|
+
|
|
197
|
+
## The OTel opt-in
|
|
588
198
|
|
|
589
199
|
```
|
|
590
200
|
evals/telemetry/config.json the committed opt-in — the ONLY gate on emission
|
|
591
201
|
evals/telemetry/review-loop.otlp.json the golden payload, regenerated and byte-compared
|
|
202
|
+
node cli/telemetry.mjs --config evals/telemetry/config.json --render
|
|
592
203
|
```
|
|
593
204
|
|
|
594
|
-
|
|
595
|
-
every
|
|
205
|
+
The same command with `--write evals/telemetry/review-loop.otlp.json` in place of `--render` regenerates
|
|
206
|
+
the golden, which every release cut owes because the version rides in the payload. **Ruled, not
|
|
207
|
+
derived:** a real emitter of hand-written OTLP-over-HTTP JSON, off by default, never callable from a
|
|
208
|
+
verify recipe, carrying the review-loop figures only. [`../cli/telemetry.mjs`](../cli/telemetry.mjs)
|
|
209
|
+
carries the ruling, the environment variables it reads and the ones it does not, and what the payload
|
|
210
|
+
may hold.
|
|
211
|
+
|
|
212
|
+
**The committed file is the only gate** — deliberately not `OTEL_SDK_DISABLED`, which defaults to `false`
|
|
213
|
+
and would make emission an opt-out. The environment supplies transport only, and the validator refuses a
|
|
214
|
+
config carrying `headers`, `endpoint` or a token, so a secret never enters a committed file.
|
|
215
|
+
|
|
216
|
+
**Consent is ruled, and railed.** The committed opt-in **is** the standing consent, and committing a
|
|
217
|
+
config that says `enabled: true` is Gated and the maintainer's alone. `--export` refuses a config that is
|
|
218
|
+
**untracked**, **tracked but absent from `HEAD`** — staged and never committed — or **differs from
|
|
219
|
+
`HEAD`**, so editing a working copy cannot manufacture consent. It is person-invoked only: CI runs this
|
|
220
|
+
module on every pull request but never `--export`, and wiring that mode into a workflow, hook or schedule
|
|
221
|
+
is a new consent question rather than a covered one. It neither queues nor retries.
|
|
222
|
+
|
|
223
|
+
**What it does NOT establish:**
|
|
224
|
+
|
|
225
|
+
- **Current data.** The payload is checked against the snapshot, never the snapshot against the world
|
|
226
|
+
([#356](https://github.com/sleepy-panda-srl/portulan/issues/356)). The capture stamp travels inside the
|
|
227
|
+
payload, so a stale export arrives labelled stale — and time-series backends commonly drop aged samples.
|
|
228
|
+
- **Who committed the consent.** The rail establishes *tracked and byte-identical to `HEAD`*, not
|
|
229
|
+
*committed by the maintainer*: `commit-to-a-working-branch` is `auto`, so an agent can commit it. What it
|
|
230
|
+
buys is that the act appears in a reviewed diff.
|
|
231
|
+
- **The opt-in in the affirmative.** No workspace in this tree opts in, so every green is about the off
|
|
232
|
+
path. The send is proven only in the suite, against an injected transport, and no yielded recipe may
|
|
233
|
+
ever exercise `--export`.
|
|
234
|
+
- **A network mode reached indirectly.** The suite derives the set of network-capable modules from the
|
|
235
|
+
tree, and the offline audit matches a module as a path suffix and a flag as a token; a mode reached
|
|
236
|
+
through another script, or through a flag built at runtime, is invisible to it.
|
|
237
|
+
- **Scheduled emission.** Nothing fires when nobody runs it —
|
|
238
|
+
[#344](https://github.com/sleepy-panda-srl/portulan/issues/344)'s silence again.
|
|
239
|
+
- **Enforcement of the review-loop bound.** An emitter with no backend reading it is not the checker
|
|
240
|
+
[`a-review-loop-needs-a-bound.md`](../.portulan/memory/a-review-loop-needs-a-bound.md) lacks.
|
|
241
|
+
- **Traces.** Metrics only.
|
|
242
|
+
|
|
243
|
+
## The A/B baseline
|
|
244
|
+
|
|
245
|
+
| File | What it is |
|
|
246
|
+
|---|---|
|
|
247
|
+
| [`ab/corpus.md`](ab/corpus.md) | The scenarios and grading rules, the registered carrier of the clause's subject, and *What may not be concluded* |
|
|
248
|
+
| [`ab/arm.md`](ab/arm.md) | What *Portulan on* denotes here: the vendored-and-compiled treatment arm |
|
|
249
|
+
| [`ab/register.md`](ab/register.md) | Both arms as built, byte-compared by the `ab` recipe |
|
|
250
|
+
| [`ab/graders.md`](ab/graders.md) | The graders' discrimination, byte-compared by the `ab-grade` recipe |
|
|
251
|
+
| [`ab/baseline.json`](ab/baseline.json), [`ab/baseline.md`](ab/baseline.md) | The capture, and the register rendered from it — the one carrier of the figures |
|
|
252
|
+
| [`ab/warm.md`](ab/warm.md) | The warm-start A/B, a sibling of the baseline rather than a clause of row 8: it prices fresh sessions warm against cold from the host's own records, in the three lines every measurement reports, and says which host switch earns a place in `sessions`. Its runner, [`../cli/warm.mjs`](../cli/warm.mjs), is built, and its record waits on the maintainer's word, since every run starts real sessions |
|
|
253
|
+
|
|
254
|
+
[`../cli/ab.mjs`](../cli/ab.mjs) builds the arms, [`../cli/ab-grade.mjs`](../cli/ab-grade.mjs) grades the
|
|
255
|
+
trees they leave behind, and [`../cli/ab-run.mjs`](../cli/ab-run.mjs) runs the turns between: `k = 5` per
|
|
256
|
+
cell, ruled by the maintainer, every turn isolated with a home and config directory of its own. Each
|
|
257
|
+
module's `--write` regenerates its document; `ab-run --write` re-renders the register from the committed
|
|
258
|
+
capture. The recipes run no agent; the modes that do — `ab --stop-probe`, `ab-run --smoke` and
|
|
259
|
+
`--matrix` — are a person's to run.
|
|
260
|
+
|
|
261
|
+
**What may not be concluded** from the baseline is [`ab/corpus.md`](ab/corpus.md)'s section of that name,
|
|
262
|
+
and the register cites it rather than restating it. **The baseline is scoped to the vendored-and-compiled
|
|
263
|
+
tier** [`ab/arm.md`](ab/arm.md) specifies, and closes row 8 for no other configuration of *Portulan on*.
|
|
264
|
+
|
|
265
|
+
## A release carries an eval result
|
|
596
266
|
|
|
597
267
|
```
|
|
598
|
-
node cli/
|
|
268
|
+
node cli/release-eval.mjs --capture at a cut: run every yielded recipe, write the record pair
|
|
269
|
+
node cli/release-eval.mjs --verify the recipe: every governed release has a record matching its capture
|
|
599
270
|
```
|
|
600
271
|
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
`
|
|
621
|
-
|
|
622
|
-
|
|
623
|
-
|
|
624
|
-
|
|
625
|
-
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
[`../
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
|
|
636
|
-
a
|
|
637
|
-
would have shipped with **zero filled instances**, since this workspace declares telemetry off and
|
|
638
|
-
[`../examples/`](../examples/) stays at 2.4 — which is
|
|
639
|
-
[`../spec/slots.md`](../spec/slots.md)'s *"splitting on speculation is how a schema acquires slots
|
|
640
|
-
nobody fills"* exactly. It is filed as [`0034`](../.portulan/proposals/0034-one-spec-bump-carries-both-evals-and-telemetry.md) — **accepted by the maintainer 2026-08-28** — to ride with
|
|
641
|
-
`evals` at the close, which makes it one bump instead of two. **That close ran 2026-09-09, so the
|
|
642
|
-
deferral is now due and still undrafted** — named by the close rather than performed inside it, since
|
|
643
|
-
drafting a schema there is the *"splitting on speculation"* the same entry warns against. The acceptance
|
|
644
|
-
commissions the drafting
|
|
645
|
-
and settles none of the shape: the key names, either slot's structure, and whether a migration is owed
|
|
646
|
-
are the drafted work, and the close remains the earliest sensible moment because `slots.md`'s deferral
|
|
647
|
-
conditions the `evals` half on row 8 having decided what an eval artifact is here.
|
|
648
|
-
|
|
649
|
-
### Consent — ruled, and railed rather than promised
|
|
650
|
-
|
|
651
|
-
[`../.portulan/gates.json`](../.portulan/gates.json) makes *sending anything outward on the team's
|
|
652
|
-
behalf* Gated, and [`../core/operating/autonomy.md`](../core/operating/autonomy.md) says Gated is **per
|
|
653
|
-
action**. Proposal
|
|
654
|
-
[`0014`](../.portulan/proposals/0014-a-feedback-pipe-points-out-of-the-seam.md) reserved telemetry as
|
|
655
|
-
*"a separate mechanism with separate consent"* — named there, ruled nowhere, until now.
|
|
656
|
-
|
|
657
|
-
**His ruling: the committed opt-in config IS the standing consent.** Each export rides on it without a
|
|
658
|
-
fresh per-send approval, the way a compiled gate carries a tier without asking again; what stays Gated
|
|
659
|
-
and his alone is **committing a config that says `enabled: true`**. The ruling is transcribed into
|
|
660
|
-
`gates.json`'s own rule and into the gate map in the same change, because a ruling living only in a
|
|
661
|
-
session note while those said otherwise would be one rule with two disagreeing carriers, obeyed at the
|
|
662
|
-
narrower.
|
|
663
|
-
|
|
664
|
-
Three things keep it from widening into the thing the tier exists to stop, and each is a rail:
|
|
665
|
-
|
|
666
|
-
- `--export` refuses in **three** states, and the third is the one that matters most: the config is
|
|
667
|
-
**untracked**, **tracked but absent from `HEAD`** — staged and never committed — or **differs from
|
|
668
|
-
`HEAD`**. So an agent cannot
|
|
669
|
-
manufacture consent by editing a working copy. Without this the ruling would be a mandate nothing
|
|
670
|
-
checks, arriving inside the change that states it.
|
|
671
|
-
- **`--export` is person-invoked only.** No workflow, hook or schedule runs *that mode*, and wiring one
|
|
672
|
-
in is a new consent question rather than a covered one. _The narrowing matters and the first draft got
|
|
673
|
-
it wrong in the direction that misleads: CI runs `cli/telemetry.mjs` on **every** pull request, because
|
|
674
|
-
the `telemetry` recipe is in the yielded set and [`../.github/workflows/verify.yml`](../.github/workflows/verify.yml)
|
|
675
|
-
runs the set. What CI never runs is `--export`, and a sentence saying the module is person-invoked
|
|
676
|
-
tells a reader auditing consent to stop looking._
|
|
677
|
-
- It **neither queues nor retries**, on [`../cli/feedback.mjs`](../cli/feedback.mjs)'s rule that a queue
|
|
678
|
-
flushing itself later is a silent send with extra steps.
|
|
679
|
-
- **Nothing from the transport environment is logged.** Headers never; the endpoint only as origin and
|
|
680
|
-
path, with userinfo and query stripped, and not at all when it failed to parse. An endpoint legally
|
|
681
|
-
carries `user:pass@` and tokens ride in query parameters, so printing it in full leaked exactly what
|
|
682
|
-
withholding the headers protected — into CI logs, which are long-lived and world-readable here.
|
|
683
|
-
|
|
684
|
-
### The payload is closed by construction, and was widened exactly once
|
|
685
|
-
|
|
686
|
-
Emission is assembled from an explicit allow-list of aggregate figures; no producer is handed a
|
|
687
|
-
snapshot to spread into a body, so there is no path from a free-text or path-shaped field to the wire.
|
|
688
|
-
The ruled vocabulary was *figures, rule ids, recipe ids* — **no identity at all**, which would have made
|
|
689
|
-
one workspace's export indistinguishable from another's at any collector. Put to him rather than
|
|
690
|
-
widened by an implementer, on `cli/feedback.mjs`'s rule that a closed list is wrong only when somebody
|
|
691
|
-
widens it **on purpose**: he widened it to **one named resource attribute**, the repository slug. A
|
|
692
|
-
reviewer **login** stays excluded and is the field this deliberately is not — this snapshot's only login
|
|
693
|
-
is a bot's, but an adopter's would carry human names.
|
|
694
|
-
|
|
695
|
-
The attribute vocabulary is **pinned in both directions** by the suite, so an added key *and* a deleted
|
|
696
|
-
carrier are each a red — against two lists rather than one, because they are two questions. The
|
|
697
|
-
allow-list says what an emission *may* carry; a separate floor says what it *must*. One list checked
|
|
698
|
-
both ways made `service.namespace` — optional in the config and optional in OpenTelemetry's own
|
|
699
|
-
conventions — look mandatory, and was green only because this workspace happens to declare one. A pin
|
|
700
|
-
that is correct by coincidence of a single instance is the shape a census over its author's own set
|
|
701
|
-
takes, one clause over.
|
|
702
|
-
|
|
703
|
-
### What this rail does NOT establish
|
|
704
|
-
|
|
705
|
-
- **It cannot tell a current snapshot from a stale one.** It checks the payload against the snapshot,
|
|
706
|
-
never the snapshot against the world — [`review-loop.sh`](../.portulan/verify/review-loop.sh)'s cost
|
|
707
|
-
inherited one step down, and [#356](https://github.com/sleepy-panda-srl/portulan/issues/356). What
|
|
708
|
-
this layer adds is that the capture stamp travels **inside** the payload, so a stale export arrives
|
|
709
|
-
labelled stale rather than looking fresh. A consequence to state rather than assert a threshold for:
|
|
710
|
-
a stale export carries an aged `timeUnixNano`, and time-series backends commonly drop or reject aged
|
|
711
|
-
samples — semantically correct and silently discarded is the worst pair.
|
|
712
|
-
- **The consent rail establishes *tracked and byte-identical to `HEAD`* — not *committed by him*.**
|
|
713
|
-
`commit-to-a-working-branch` is tier **auto**, so an agent commits unattended; a consent committed by a
|
|
714
|
-
scripted `git commit` clears every gate this rail has. What the rail buys is real and is narrower than
|
|
715
|
-
it first reads: it moves the act from a file edit nobody sees into **a commit that appears in a
|
|
716
|
-
reviewed diff**, on the branch a human merges. It cannot tell whose commit it is, and nothing here
|
|
717
|
-
claims it can. Measured at the pre-commit checkpoint rather than reasoned about.
|
|
718
|
-
- **The offline audit's class is derived, and was not always.** The table claimed to rail the class of
|
|
719
|
-
network-capable modes in `cli/` and listed **two of three** — `cli/feedback.mjs` files a GitHub issue
|
|
720
|
-
through `gh issue create` and had been network-capable longer than either of the others. A set drawn
|
|
721
|
-
by its author and reported as complete is the census shape session 4 named one clause over, committed
|
|
722
|
-
at this table's birth and found at round 10 of the review. The suite now **derives** the set from the
|
|
723
|
-
tree, so a new network-capable module reddens. What remains uncovered is narrower: a module reaching
|
|
724
|
-
the network in a way the derivation does not see, and a mode reached **indirectly**. It
|
|
725
|
-
matches a module as a path **suffix** and a flag as a **token**, so `./` and `--flag=value` spellings
|
|
726
|
-
are covered — the first version knew one literal string, which made `./` a bypass — but a mode reached
|
|
727
|
-
**indirectly**, through another script or a flag built at runtime, is invisible to it and no wider
|
|
728
|
-
pattern fixes that.
|
|
729
|
-
- **The opt-in is never demonstrated in the affirmative from a committed artifact.** No workspace in
|
|
730
|
-
this tree declares telemetry on, so every green here is a green about the **off** path. The send is
|
|
731
|
-
proven in the suite against an injected transport and a temporary repository with the consent really
|
|
732
|
-
committed — a session-time observation, not a standing one, which is the state clause (d) exists to
|
|
733
|
-
replace and does not reach here.
|
|
734
|
-
- **`--export`'s network path is covered only by unit tests against a fake**, and by construction no
|
|
735
|
-
yielded recipe may ever exercise it — this recipe's own `offline` check forbids it. That is a
|
|
736
|
-
deliberate consequence rather than an oversight, and it is the mode that matters.
|
|
737
|
-
- **Emission is not scheduled**, and nothing fires when nobody runs it —
|
|
738
|
-
[#344](https://github.com/sleepy-panda-srl/portulan/issues/344)'s silence problem in a third place.
|
|
739
|
-
- **The review-loop bound is still checked by nothing.** The 2026-07-28 amendment says *"the telemetry
|
|
740
|
-
clause is where that checker's home is"* for
|
|
741
|
-
[`a-review-loop-needs-a-bound.md`](../.portulan/memory/a-review-loop-needs-a-bound.md). An emitter
|
|
742
|
-
with no backend reading it does not discharge that, and this clause must not be read as enforcement.
|
|
743
|
-
- **Metrics only.** OTLP's trace payload is a second shape with a second set of semantics; one signal
|
|
744
|
-
demonstrated beats two asserted.
|
|
745
|
-
|
|
746
|
-
## A release carries an eval result — the mechanism, and the instance it is still owed
|
|
747
|
-
|
|
748
|
-
Row 8's **ninth** clause, added by the maintainer's ruling of 2026-08-24. The mandate is older than the
|
|
749
|
-
clause: [`../docs/plan.md`](../docs/plan.md)'s Protocol → Versioning has said *"from milestone 8,
|
|
750
|
-
releases carry an eval result"* since the plan was locked, and it belonged to **no row** until that day —
|
|
751
|
-
so a close re-deriving row 8's criterion clause by clause would have re-derived eight and left it
|
|
752
|
-
unbuilt. [`releases/`](releases/) is where a release's record lives, one pair per version:
|
|
753
|
-
`<version>.json` is the capture and `<version>.md` the register rendered from it and byte-compared.
|
|
754
|
-
|
|
755
|
-
### The three things the amendment left open, and what was ruled here
|
|
756
|
-
|
|
757
|
-
The amendment named what it did **not** settle — *"what an eval result attached to a release consists of,
|
|
758
|
-
where it is carried (the changelog entry, the release body, a file in the payload), and whether a release
|
|
759
|
-
with no eval result is refused by a rail or by a person. **Those are this row's remaining work, not this
|
|
760
|
-
amendment's content.**"* The last sentence is quoted rather than trimmed because it is the half that makes
|
|
761
|
-
the passage a **scoping limit** — without it the list reads as three open questions rather than as three
|
|
762
|
-
questions assigned to this row — and a first draft of this section dropped it.
|
|
763
|
-
|
|
764
|
-
**What it consists of.** The verdict every recipe the workspace **yields** returned, measured by running
|
|
765
|
-
them at a named commit, plus the **identity** of the A/B baseline the release ships against — path,
|
|
766
|
-
capture date, commit, cleanliness. Never that baseline's figures: those have one carrier and a second
|
|
767
|
-
copy of them inside a per-release record is the hand-maintained figure this milestone deleted six of.
|
|
768
|
-
There is no *eval* subset of the recipe set — [`../cli/recipe-set.mjs`](../cli/recipe-set.mjs) has no
|
|
769
|
-
such category — so an "eval recipes only" record would have required a hand-listed roster, which is the
|
|
770
|
-
defect the record exists to avoid.
|
|
771
|
-
|
|
772
|
-
**Where it is carried.** One file per version, cited and never restated — **and it ships**. `evals/releases/` is in `package.json`'s `files`, the maintainer's decision of 2026-09-01 on an option the amendment had left open and the implementer had closed the other way. **A measurement settled it:** a record written into the *unpacked tarball* is graded `exit 0` by `cli/release-eval.mjs --tagged`, which already ships — so the artifact carries both the claim and the tool that checks it, and the objection that an npm consumer can run no recipe was true and beside the point. [`../docs/milestones/m08.md`](../docs/milestones/m08.md) carries the adjudication, the one citation that does not survive packing, and why absolute URLs were refused.
|
|
773
|
-
|
|
774
|
-
**Rail or person — both, split by what each can reach.** The `release-eval` recipe grades the tree on
|
|
775
|
-
every commit. [`../.github/workflows/publish-github-packages.yml`](../.github/workflows/publish-github-packages.yml)
|
|
776
|
-
runs `release-eval --tagged` against the tag's own checkout, before publishing — the one mechanical
|
|
777
|
-
surface at the release act. The maintainer owns the rest, and
|
|
778
|
-
[`../.portulan/gate-map.md`](../.portulan/gate-map.md) carries that half beside the tier it belongs to.
|
|
779
|
-
|
|
780
|
-
### Two designs a fresh context reversed before a line was written
|
|
781
|
-
|
|
782
|
-
**Cut detection off `CHANGELOG.md`'s top heading never fires.** The cut commit re-seeds `## Unreleased`
|
|
783
|
-
*above* the version it just wrote — `b410c020` — because that file's own header requires the tagged tree
|
|
784
|
-
to contain its own entry. The top heading is `Unreleased` on every commit including the cut, so a rail
|
|
785
|
-
keyed to it would have reported *no cut in this tree* on precisely the tree it was built to grade.
|
|
786
|
-
|
|
787
|
-
**A rail keyed to `package.json`'s version alone grades one record at a time.** The moment `0.1.4` is
|
|
788
|
-
declared, `0.1.3`'s record can be deleted and nothing is looking at it. The subject is therefore **every
|
|
789
|
-
version the changelog records**, from `0.1.3` onward, permanently — and the set is graded in both
|
|
790
|
-
directions, so a record for a release that was never cut reds as well.
|
|
272
|
+
[`releases/`](releases/) holds one pair per version, described in
|
|
273
|
+
[`releases/README.md`](releases/README.md), for every version [`../CHANGELOG.md`](../CHANGELOG.md)
|
|
274
|
+
records from `0.1.3` onward — graded permanently and in both directions. A record is the verdict every
|
|
275
|
+
recipe the workspace yields returned at a named commit, plus the **identity** of the A/B baseline the
|
|
276
|
+
release ships against, never its figures.
|
|
277
|
+
**`evals/releases/` ships in the package**, by the maintainer's decision of 2026-09-01, so the package
|
|
278
|
+
carries both the record and the tool that checks it: `--tagged` grades a record in the unpacked package as
|
|
279
|
+
it does in the tagged tree. The recipe grades the tree on every commit;
|
|
280
|
+
[`../.github/workflows/publish-github-packages.yml`](../.github/workflows/publish-github-packages.yml) runs
|
|
281
|
+
`--tagged` against the tag's own checkout before publishing; the rest is Gated, and
|
|
282
|
+
[`../.portulan/gate-map.md`](../.portulan/gate-map.md) carries it beside its tier.
|
|
283
|
+
|
|
284
|
+
**What it does NOT establish.** Every register prints its own limits: a green means the record agrees with
|
|
285
|
+
its capture, never that the release is good; the recipes ran at a commit inside the cut change, not at the
|
|
286
|
+
tag; `release-eval`'s own row is excluded. Beyond the register lies the **published** half: `--tagged`
|
|
287
|
+
reaches the tagged tree at the publish, and the release body, written outside the tree, is reached by no
|
|
288
|
+
check. While every governed release has its record, the clause's central arm — a governed release with
|
|
289
|
+
none — is exercised only by the forced-red drill that moves the boundary.
|
|
290
|
+
|
|
291
|
+
**`v0.1.3` is the first release published with its eval result.** It was tagged on 2026-09-30 at the
|
|
292
|
+
commit that cut it, the publish workflow ran `--tagged` against that tree before publishing it, and its
|
|
293
|
+
release body cites [`releases/0.1.3.md`](releases/0.1.3.md) without restating it. That the body cites the
|
|
294
|
+
register is the maintainer's act, and nothing here checks it.
|
|
295
|
+
|
|
296
|
+
## A rule change decided on eval evidence
|
|
297
|
+
|
|
298
|
+
Row 8's clause *a rule change merges or is rejected on eval evidence* is discharged by one instance, hole 8
|
|
299
|
+
of [`../.portulan/gate-map.md`](../.portulan/gate-map.md): a `gated` or `prohibited` path target that can
|
|
300
|
+
never match. [#337](https://github.com/sleepy-panda-srl/portulan/issues/337) set out three answers, the
|
|
301
|
+
instruments chose between them, and the change merged on 2026-09-09; the evidence table and the argument
|
|
302
|
+
are [`../docs/milestones/m08.md`](https://github.com/sleepy-panda-srl/portulan/blob/8a33f9b/docs/milestones/m08.md)'s session-10 note. The rail is `neverMatches`
|
|
303
|
+
and the backend refusal beside `HOST_GATE_TIERS` in [`../cli/compile.mjs`](../cli/compile.mjs), a suite
|
|
304
|
+
block, and a second forced-red drill on `doctor`; its limit — the refusal is at compile time, not at the
|
|
305
|
+
hook — is stated beside the refusal. The hazard is held there and not in the gate corpus, whose
|
|
306
|
+
denominator is the yielded policy: adding a never-matching `gated` rule to `gates.json` to give the corpus
|
|
307
|
+
a case would manufacture the defect.
|
|
791
308
|
|
|
792
|
-
|
|
309
|
+
## What is NOT built yet
|
|
793
310
|
|
|
794
|
-
|
|
795
|
-
|
|
796
|
-
records no red.
|
|
797
|
-
- **The recipes did not run at the tag.** A record is committed *in* the change that cuts a release, so
|
|
798
|
-
its `source.commit` names a commit **inside that change** rather than the tag — a capture cannot name a
|
|
799
|
-
commit that does not exist yet. How much of the cut precedes it is a property of how the change is
|
|
800
|
-
sequenced, and `--capture` refuses a version below the first governed one, so a clean capture can never
|
|
801
|
-
run before the version bump. The register prints the commit it was taken at and says this, the way the
|
|
802
|
-
A/B capture prints `clean`.
|
|
803
|
-
- **`release-eval`'s own row is excluded from every capture**, because a capture cannot be accurate about
|
|
804
|
-
the record it lives inside. The exclusion is printed with its reason rather than the row being dropped.
|
|
805
|
-
- **`0.1.3` carries one, so the rail grades a non-empty set.** Before that cut its green read *no release
|
|
806
|
-
from `0.1.3` onward has been cut yet*, which was a **state, not a measurement**; the module said so on
|
|
807
|
-
its own face rather than letting exit 0 imply otherwise, and it prints whichever of the two holds. The
|
|
808
|
-
clause's central arm — *a governed release with no record* — is still unreachable from the committed
|
|
809
|
-
tree, now because the one governed release **has** its record rather than because none exists, so it
|
|
810
|
-
remains exercised by the forced-red drill that moves the boundary; it fired on 2026-09-01. A drill is
|
|
811
|
-
not a release, and a record in the tree is not a published one.
|
|
812
|
-
|
|
813
|
-
## A rule change decided on eval evidence — the ninth clause
|
|
814
|
-
|
|
815
|
-
**The clause is instance-shaped, and this is the instance.** Row 8 asks that *a rule change merges or
|
|
816
|
-
is rejected on eval evidence*, and until 2026-09-09 the honest sentence in this file was that **every
|
|
817
|
-
rule in [`../.portulan/memory/`](../.portulan/memory/) had been merged on review alone**. _(The evidence
|
|
818
|
-
was taken on 2026-09-03 and the change merged on 2026-09-09; the clause's verb is **merges**, so the
|
|
819
|
-
later date is the one that discharges it. Three sentences in this file dated it to the session rather
|
|
820
|
-
than to the merge until the milestone-8 close re-derived it from `gh pr view 408` — created
|
|
821
|
-
`2026-09-03T18:57:23Z`, merged `2026-09-09T07:48:12Z`.)_ What changed
|
|
822
|
-
is not that a rule was minted — rules are minted often — but that a **decision between three drafted
|
|
823
|
-
answers was made by running the instruments** rather than by argument.
|
|
824
|
-
|
|
825
|
-
**The rule change.** Hole 8 of [`../.portulan/gate-map.md`](../.portulan/gate-map.md): a `gated` or
|
|
826
|
-
`prohibited` gate rule whose path target can never match compiles to a named permission surface while
|
|
827
|
-
the runtime matcher answers false for every input — a gate reported as covered that enforces nothing.
|
|
828
|
-
[#337](https://github.com/sleepy-panda-srl/portulan/issues/337) set out three defensible answers and
|
|
829
|
-
presumed none, precisely so the choice could be made on evidence.
|
|
830
|
-
|
|
831
|
-
**The evidence, measured on this tree.** Each candidate applied as a scratch mutation — `cli/mutants.mjs`'
|
|
832
|
-
own technique — and every instrument run and read:
|
|
833
|
-
|
|
834
|
-
| candidate | `compile` | `goldens` | `mutants` | `doctor` | `tests` |
|
|
835
|
-
|---|---|---|---|---|---|
|
|
836
|
-
| baseline (`main`) | 0 | 0 | 0 | 0 | 0 |
|
|
837
|
-
| **1** — refuse it at `parse`, every tier | **2** | **2** | **2** | **1** | **1** |
|
|
838
|
-
| **2** — make `./` match the tree | 0 | **1** | **2** | 0 | **1** |
|
|
839
|
-
| **3** — refuse it in the backend that enforces | 0 | 0 | 0 | 0 | 0 |
|
|
840
|
-
| **the hazard itself**, on a clean `main` clone — `edit-on-a-working-branch` flipped to `gated` | 1 | 0 | 0 | **0** | 1 |
|
|
841
|
-
| **the hazard itself**, on this tree | **2** | 0 | 0 | **1** | 1 |
|
|
842
|
-
|
|
843
|
-
**What the table decided, and what it did not.** Option 1 is **rejected on the measurement**: this
|
|
844
|
-
workspace's own policy carries two `auto` rules written `write: "./"` and `read: "./"`, so refusing the
|
|
845
|
-
shape at `parse` takes their spelling — three instruments could not run and two went red, and
|
|
846
|
-
*could-not-run* is the worse of the two outcomes because it reports nothing about the tree. That is
|
|
847
|
-
#337's own prediction re-derived rather than quoted. Option 2's reds are **not** a rejection and must not
|
|
848
|
-
be read as one: they are the record-keeping rails firing exactly as designed — `goldens` reds whenever the
|
|
849
|
-
documented hole moves, *including when it closes*, and `cli/mutants.mjs` already records that same edit
|
|
850
|
-
as the operator `matchesPath-admits-the-empty-target`, `killed`, with the note *"CLOSES hole 8 … the
|
|
851
|
-
good-news direction the corpus exists to catch"*. Its other two reds are that one fact twice more rather
|
|
852
|
-
than three independent objections: `mutants` exits 2 because the census refuses to run over a reddened
|
|
853
|
-
corpus, and the eight suite failures are the cases that **pin** that census and that corpus. Option 2 is
|
|
854
|
-
declined on **authority**: it decides what
|
|
855
|
-
`./` should MEAN as a policy target, which is the question #337 reserves. Option 3 keeps every
|
|
856
|
-
instrument green and closes the hazard, and it is the one taken.
|
|
857
|
-
|
|
858
|
-
**The last two rows are what decide the merge half.** On the first four rows alone, option 3 and *doing
|
|
859
|
-
nothing* are indistinguishable — both are all-green — so the choice would rest on argument. They grade
|
|
860
|
-
the **defect** instead, by flipping this repository's own `edit-on-a-working-branch` from `auto` to
|
|
861
|
-
`gated`. **Every cell in this table is a verify recipe's exit**, so `compile` there is `compile --check`;
|
|
862
|
-
the clean-clone row is measured on a detached checkout of `main`, not on this tree with a file swapped,
|
|
863
|
-
because a hybrid is neither tree.
|
|
864
|
-
|
|
865
|
-
**The discriminating cell is `doctor`, and it is the whole argument.** On `main`, `doctor` exits **0**
|
|
866
|
-
over that policy — it counts the hollow gate among *11 of 25 rule(s) compiled* and names five gates no
|
|
867
|
-
backend compiles, none of them this one, so the tree reports the rule as covered and nothing contradicts
|
|
868
|
-
it. On this tree it exits **1**, printing the refusal. In write mode the same policy at `main` also
|
|
869
|
-
*emits* the rule, surface `Edit(./) · hook: a Bash command writing ./`; the recipe's **1** is the drift
|
|
870
|
-
that emission causes against the committed artifact, which is a different fact and is why the cell is
|
|
871
|
-
labelled by its instrument.
|
|
872
|
-
|
|
873
|
-
**`tests` is 1 on both sides and discriminates nothing**, which is stated rather than hidden behind a
|
|
874
|
-
prettier cell: on `main` one case fails, *"every rule is cited under the gate map section matching its
|
|
875
|
-
TIER"*, because the prose still files that rule under Auto; on this tree sixteen do — **eight**
|
|
876
|
-
because the new drill's anchor searches for the `"tier": "auto"` line the fixture rewrote, **seven**
|
|
877
|
-
because the refusal fires, and one that fails on `main` too. Both are
|
|
878
|
-
artefacts of perturbing a policy the suite pins, on either tree, and neither says anything about the
|
|
879
|
-
change.
|
|
880
|
-
|
|
881
|
-
**So the clause discharges as *merges on eval evidence*, with option 1 a measured-cost rejection** — and
|
|
882
|
-
the second half of *"merges or is rejected"* is not overclaimed beyond that.
|
|
883
|
-
|
|
884
|
-
**What the rail is.** The predicate `neverMatches` in [`../cli/compile.mjs`](../cli/compile.mjs), the
|
|
885
|
-
backend refusal beside `HOST_GATE_TIERS`, a suite block that goes red in six cases when the refusal is
|
|
886
|
-
disabled, and a second forced-red drill on the `doctor` rail — `doctor` rather than `compile`, because a
|
|
887
|
-
`CompileError` exits 2 and `cli/drills.mjs`'s coverage check refuses a drill declaring `exit: 2`, on the
|
|
888
|
-
ground that it *"would read a refusal as a verdict"*.
|
|
889
|
-
|
|
890
|
-
**Its limits, in the shape this directory asks for.** The refusal is at **compile** time, not at the
|
|
891
|
-
hook: `cli/gate.mjs` reads the policy through `parse`, which this change deliberately does not touch, so
|
|
892
|
-
a workspace that has already committed such a rule still loads it at run time and the hook steps aside
|
|
893
|
-
because nothing matched. And the hazard cannot live in the gate corpus itself — `cli/goldens.mjs`
|
|
894
|
-
derives its denominator from the **yielded policy**, and adding a `gated` `./` rule to `gates.json` to
|
|
895
|
-
satisfy a corpus would be manufacturing the defect. The suite block and the drill are its home.
|
|
311
|
+
Each names where it arrives, per [`../.portulan/dod.md`](../.portulan/dod.md) condition 4 — nothing here
|
|
312
|
+
claims a capability that does not exist:
|
|
896
313
|
|
|
897
|
-
|
|
314
|
+
- **The `evals` and `telemetry` Workspace Definition slots.** Proposal
|
|
315
|
+
[`0034`](../.portulan/proposals/0034-one-spec-bump-carries-both-evals-and-telemetry.md), accepted
|
|
316
|
+
2026-08-28, carries both in one spec bump. The deferral in [`../spec/slots.md`](../spec/slots.md) is due
|
|
317
|
+
since the milestone-8 close and undrafted; the drafting settles the key names, each slot's structure and
|
|
318
|
+
whether a migration is owed.
|
|
898
319
|
|
|
899
|
-
Each
|
|
900
|
-
here claims a capability that does not exist:
|
|
901
|
-
|
|
902
|
-
- **A release actually *published* with its eval result.** The mechanism is built — its section is above —
|
|
903
|
-
and since the `0.1.3` cut so is the instance's in-tree half: `evals/releases/0.1.3.{json,md}`. What is
|
|
904
|
-
still not built is the half no tree can hold. **Ruled 2026-09-01: it waits for a real cut**, and the
|
|
905
|
-
release is done once milestone 8's other work is finished — so this clause was never closable on the
|
|
906
|
-
mechanism, and the cut is the milestone's last act rather than its first sequel. `tag-a-release` and
|
|
907
|
-
`publish-a-release` are Gated, so the clause stays undemonstrated until the maintainer takes them and
|
|
908
|
-
the release body cites the register. The argument that was put to him, and what a cut mechanically is, are in
|
|
909
|
-
[`../docs/milestones/m08.md`](../docs/milestones/m08.md): a clause about releases, held satisfied while
|
|
910
|
-
zero governed releases exist, is true the way *a baseline names its host* was true of an empty string.
|
|
320
|
+
Each rail's open limits are in its section above, with the issue tracking each where there is one.
|