@sleepy_panda_srl/portulan 0.1.2 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (81) hide show
  1. package/README.md +7 -6
  2. package/cli/README.md +142 -291
  3. package/cli/advisory.mjs +354 -0
  4. package/cli/comments.mjs +932 -0
  5. package/cli/compile.mjs +1443 -1210
  6. package/cli/context.mjs +883 -0
  7. package/cli/control-chars.mjs +12 -301
  8. package/cli/discover.mjs +17 -574
  9. package/cli/doctor.mjs +225 -1071
  10. package/cli/drills.mjs +1107 -0
  11. package/cli/feedback.mjs +16 -237
  12. package/cli/finish.mjs +437 -0
  13. package/cli/form.mjs +644 -0
  14. package/cli/fuzz-shell.mjs +595 -0
  15. package/cli/gate.mjs +7 -198
  16. package/cli/goldens.mjs +311 -0
  17. package/cli/index.mjs +305 -655
  18. package/cli/init.mjs +244 -564
  19. package/cli/inside.mjs +3 -28
  20. package/cli/instructions.mjs +829 -0
  21. package/cli/ledger.mjs +778 -0
  22. package/cli/librarian.mjs +135 -599
  23. package/cli/manifest.mjs +2 -39
  24. package/cli/mutants.mjs +764 -0
  25. package/cli/new.mjs +12 -243
  26. package/cli/pack-identity.mjs +6 -68
  27. package/cli/pack-version.mjs +13 -248
  28. package/cli/plugin-lint.mjs +42 -658
  29. package/cli/portulan.mjs +5 -124
  30. package/cli/recipe-set.mjs +16 -240
  31. package/cli/release-eval.mjs +748 -0
  32. package/cli/review-meter.mjs +500 -0
  33. package/cli/rule-carriers.mjs +12 -175
  34. package/cli/sessions.mjs +82 -0
  35. package/cli/skill-goldens.mjs +424 -0
  36. package/cli/skills-set.mjs +12 -414
  37. package/cli/stop-gate.mjs +126 -454
  38. package/cli/symbols.mjs +1132 -0
  39. package/cli/telemetry.mjs +582 -0
  40. package/cli/upgrade.mjs +171 -314
  41. package/cli/vendor.mjs +146 -476
  42. package/cli/version-carriers.mjs +9 -74
  43. package/core/README.md +1 -1
  44. package/core/engine.md +3 -9
  45. package/core/operating/autonomy.md +1 -1
  46. package/core/operating/context.md +183 -0
  47. package/core/operating/evolution.md +3 -3
  48. package/core/operating/loop.md +13 -20
  49. package/core/operating/memory.md +12 -3
  50. package/core/operating/sessions.md +100 -0
  51. package/core/personas/librarian.md +2 -2
  52. package/core/skills/consolidate/SKILL.md +4 -2
  53. package/core/templates/gate-policy.md +27 -8
  54. package/core/templates/handoff.md +28 -15
  55. package/core/templates/memory-entry.md +1 -0
  56. package/evals/README.md +320 -0
  57. package/evals/releases/0.1.3.json +134 -0
  58. package/evals/releases/0.1.3.md +64 -0
  59. package/evals/releases/0.2.0.json +146 -0
  60. package/evals/releases/0.2.0.md +67 -0
  61. package/evals/releases/README.md +32 -0
  62. package/package.json +9 -2
  63. package/packs/rituals/checkpoints/README.md +9 -2
  64. package/packs/rituals/checkpoints/pack.json +1 -1
  65. package/packs/rituals/checkpoints/skills/pre-commit/SKILL.md +32 -2
  66. package/packs/tools/github/verify/actions-pinned.sh +10 -55
  67. package/spec/README.md +112 -6
  68. package/spec/migrations/0001-repository-declares-its-tree.mjs +1 -26
  69. package/spec/migrations/0002-bundle-fallback-path.mjs +13 -74
  70. package/spec/migrations/0003-handoff-index-not-kept.mjs +66 -0
  71. package/spec/migrations/0004-changelog-fragments.mjs +53 -0
  72. package/spec/migrations/0005-session-log-retired.mjs +69 -0
  73. package/spec/migrations/0006-boot-card.mjs +77 -0
  74. package/spec/migrations/0007-guidance-compiled.mjs +31 -0
  75. package/spec/migrations/0008-card-reading.mjs +55 -0
  76. package/spec/migrations/0009-instruction-sections-on-read.mjs +60 -0
  77. package/spec/migrations/0010-comment-rail.mjs +57 -0
  78. package/spec/migrations/README.md +45 -10
  79. package/spec/pack.schema.json +1 -1
  80. package/spec/slots.md +359 -23
  81. package/spec/workspace.schema.json +145 -4
@@ -26,15 +26,15 @@
26
26
 
27
27
  ```json
28
28
  {
29
- "portulan": { "gates": "1.0" },
29
+ "portulan": { "spec": "2.2" },
30
30
 
31
31
  "why": "{Why this team gates what it gates — the sentence a new agent reads first. Name the blast radius you are protecting, not the tools you are restricting.}",
32
32
 
33
33
  "floor": {
34
34
  "branch": "{default branch}",
35
- "require_pull_request": true,
36
- "block_force_push": true,
37
- "checks": []
35
+ "checks": [],
36
+ "reviews": 0,
37
+ "resolve_conversations": true
38
38
  },
39
39
 
40
40
  "rules": [
@@ -45,11 +45,17 @@
45
45
  "reason": "{Why, and what to do instead. Prohibited means no yes makes this acceptable — if a human could approve it, it is Gated, not Prohibited.}"
46
46
  },
47
47
  {
48
- "id": "{rule-id}",
48
+ "id": "force-push-without-a-lease",
49
49
  "tier": "gated",
50
- "action": { "shell": "{command prefix}" },
50
+ "action": { "shell": "git push --force" },
51
51
  "reason": "{Why a human must approve each time. Prefer the undoability test: what does it cost if this was wrong and nobody noticed for a week?}"
52
52
  },
53
+ {
54
+ "id": "delete-a-ref",
55
+ "tier": "gated",
56
+ "action": { "shell": "git push --delete" },
57
+ "reason": "{Why a deletion waits for a person. These two ids and actions are filled in already — they are the two destructive ref spellings the floor backend can express, and a `floor` block with no rule reaching it refuses to compile.}"
58
+ },
53
59
  {
54
60
  "id": "{rule-id}",
55
61
  "tier": "propose",
@@ -60,6 +66,16 @@
60
66
  }
61
67
  ```
62
68
 
69
+ > **`floor` is exactly four keys, and they are the four `compile` reads** — `branch`, `checks`,
70
+ > `reviews`, `resolve_conversations`. It is not a free-form block: `reviews` is a non-negative integer
71
+ > and `resolve_conversations` a boolean, both required, because a floor that omitted them would export
72
+ > one weaker than the one already in force. _(Until 2026-08-22 this skeleton emitted
73
+ > `require_pull_request` and `block_force_push` instead — two keys no version of the compiler has ever
74
+ > read — beside a `portulan.gates` version key it does not read either. A scaffold the next documented
75
+ > step refuses is the defect this template's own closing paragraph warns about, one layer up;
76
+ > [#329](https://github.com/sleepy-panda-srl/portulan/issues/329) reported it, and `new`'s test now
77
+ > scaffolds this file and compiles it rather than only checking that it was written.)_
78
+ >
63
79
  > **`checks: []` is deliberate and you should probably leave it that way at first.** A floor requiring a
64
80
  > status check that no workflow job reports is a red nobody caused — a fresh repository reports no checks,
65
81
  > and a real one reports something else, so any name guessed here fails on the first run. Add the check
@@ -72,5 +88,8 @@
72
88
  > print every rule against both rather than reasoning about it — the refusals are printed, never silent.
73
89
  >
74
90
  > **Start with the rules that destroy rather than create.** A ref deletion and a force-push are the two
75
- > the floor backend can express, they are the two whose damage is hardest to undo, and they are a policy's
76
- > honest first pair.
91
+ > **destructive ref spellings** the floor backend can express — it also emits `pull_request` and
92
+ > `required_status_checks` for a `propose` rule once `floor.checks` is non-empty, which is the paragraph
93
+ > above — and they are the two whose damage is hardest to undo, so they are a policy's
94
+ > honest first pair — so the skeleton above ships them filled in rather than as placeholders, which is
95
+ > also what keeps the `floor` block reachable. Their `reason` is still yours to write.
@@ -1,25 +1,23 @@
1
1
  # Template — Handoff
2
2
 
3
- > A **handoff** records a decision and its *why*, so the next agent — or the next context window —
4
- > inherits the reasoning and not just the result. Write one **at the close of every session** — that part
5
- > is not conditional, and short is fine — and also when compacting, when hitting an iteration cap, or when
6
- > passing work across a persona boundary. Decisions without their why get silently re-litigated or
7
- > reversed. **Drop any section that has nothing to say** — the date and the decisions-with-why are the
8
- > only irreducible parts, and a handoff of five lines is a valid handoff. The date lives in the
9
- > **filename**, leading, in ISO form — `YYYY-MM-DD-{slug}.md` — not in a field inside the document: that
10
- > is what a gate checks and what makes the series sort chronologically. That is what keeps the
11
- > every-session rule from becoming ceremony: it scales down here, inside the artifact, rather than by
12
- > being skipped. _(Provenance: Cognition — handoffs record decisions and rationale. The every-session rule
13
- > and why it is binary rather than discretionary are in `../operating/loop.md`.)_
3
+ > A **handoff** carries the work still open, so the next agent — or the next context window — starts
4
+ > from where things stand rather than from the transcript. Write one when a session ends with work not
5
+ > committed and pushed, when compacting, when hitting an iteration cap, or when passing work across a
6
+ > persona boundary. Work that is committed carries its own why, in its commit message, so a handoff
7
+ > says nothing the diff or a commit already says. **Drop any section with nothing to say**; five lines
8
+ > is a valid handoff. The date leads the **filename**, in ISO form — `YYYY-MM-DD-{slug}.md` — because
9
+ > that is what a gate checks and what sorts the series. _(Provenance: Cognition — handoffs record
10
+ > decisions and their rationale. When one is owed, and why the commit carries the rest:
11
+ > `../operating/loop.md`.)_
12
+ >
13
+ > **The commit a change lands in:** a subject that says what changed, in the present tense and under 72
14
+ > characters; a blank line; a body of a few lines that says why, and what was rejected when that matters.
14
15
 
15
16
  ---
16
17
 
17
18
  # Handoff — {task / thread}
18
19
 
19
- **State.** {where things actually are right now — done, in-progress, blocked — in a few lines.}
20
-
21
- **Decisions + why.** {each decision taken and the reason it was taken; the reasons are the payload.}
22
- - {decision} — because {why}; alternatives considered: {…}.
20
+ **State.** {where things stand: what is done and committed, what is in progress and uncommitted.}
23
21
 
24
22
  **Open questions.** {what is undecided, and who decides it — flag the human-owned ones.}
25
23
 
@@ -27,3 +25,18 @@
27
25
 
28
26
  **Recoverability.** {anything left in a partial state, and how to make it safe; see
29
27
  `../operating/safety.md`.}
28
+
29
+ ---
30
+
31
+ _A filled one, for a session that stopped mid-change:_
32
+
33
+ # Handoff — retry the flaky upload test
34
+
35
+ **State.** The retry wrapper is written and its unit test passes; it is stashed, not committed, because
36
+ the integration suite still fails once in about ten runs.
37
+
38
+ **Open questions.** Whether three attempts is the right cap — the owner of the upload service decides.
39
+
40
+ **Next action.** `git stash pop`, run the integration suite twenty times, and read the one failure's log.
41
+
42
+ **Recoverability.** Nothing was pushed; dropping the stash loses only the wrapper.
@@ -8,6 +8,7 @@
8
8
  ---
9
9
 
10
10
  **type:** rule | decision | reference | glossary
11
+ **dated:** {YYYY-MM-DD — the day this record's text last changed; an edit re-dates it. The per-record cap binds by it where the workspace declares a cutoff}
11
12
  **scope:** {the persona / repo / workspace this belongs to — memory is per-agent, not global}
12
13
  **provenance:** {one of the two forms below — a rule carrying neither is taste, and `doctor` rejects it}
13
14
 
@@ -0,0 +1,320 @@
1
+ # evals/
2
+
3
+ Milestone 8's home: the golden corpora, the review-loop register, the telemetry opt-in, the A/B harness's
4
+ records and the eval result each release carries. Row 8 of [`../docs/plan.md`](../docs/plan.md) is the
5
+ binding criterion; [`../docs/milestones/m08.md`](https://github.com/sleepy-panda-srl/portulan/blob/8a33f9b/docs/milestones/m08.md) is its legislative history.
6
+ This file does not restate either — a rule with two carriers is obeyed at the narrower one. It says what
7
+ is here, how to run or extend each rail, and what each one does not establish; a rail's detail is its
8
+ module, whose longer header is at `git show a31da46:cli/<module>.mjs`. How each clause came to be —
9
+ first-run figures, reversed designs, review rounds — is at `git show 8a33f9b:evals/README.md`.
10
+
11
+ ## Two words that mean two things here
12
+
13
+ **"Eval" carries two unrelated senses in this repository, and only one of them lives in this directory.**
14
+
15
+ | Sense | Where it lives | What it is |
16
+ |---|---|---|
17
+ | **Evaluation as measurement** — golden tasks, A/B, the eval gate | **here**, `evals/` | Does the engine make an agent work better? Milestone 8's subject. |
18
+ | **Evaluation as a licensed copy** | [`../cli/eval-bundle.mjs`](../cli/eval-bundle.mjs), [`../.portulan/verify/eval-bundle.sh`](../.portulan/verify/eval-bundle.sh) | A named-recipient bundle cut from a commit under evaluation terms, its issuance ledger kept outside this repository. Nothing to do with this directory. |
19
+
20
+ So the machinery here is named **`goldens`**, never `evals`: the directory keeps the row's word and the
21
+ tools take a narrower one. And **`goldens` names two runners** — two subjects and two oracles, which is why
22
+ they are two modules and two recipes rather than one with a flag. [`../cli/goldens.mjs`](../cli/goldens.mjs)
23
+ grades the **compiled gates** through the compiler's own `matchesRule`;
24
+ [`../cli/skill-goldens.mjs`](../cli/skill-goldens.mjs) grades the **core skills'** mandates against the
25
+ live artifacts they govern.
26
+
27
+ Each rail below is a verify recipe in [`../.portulan/verify/`](../.portulan/verify/), run on every pull
28
+ request; `bash .portulan/verify/<recipe>.sh` runs one by hand, and `node cli/<module>.mjs --help` prints a
29
+ module's modes. Every runner prints its own totals, so no figure is written here.
30
+
31
+ ## The gate corpus — clause (a)
32
+
33
+ ```
34
+ evals/goldens/gates/<rule-id>.json one fixture file per rule in the yielded gate policy
35
+ node cli/goldens.mjs --workspace . --pack-root packs
36
+ ```
37
+
38
+ **What a fixture is.** A file is named for its own `rule` field — the runner refuses a misfiled one — and
39
+ holds `cases`, each `{id, class, tool, path, input, expect, why}`. A case is **data**, answered by the
40
+ compiler's own exported `matchesRule`, the function the hook calls at tool time. **A case's command string
41
+ is never executed**, and the suite asserts the runner imports no process-spawning API. Two classes, and
42
+ exactly two:
43
+
44
+ - **`holds`** — the matcher catches this today and must keep catching it.
45
+ - **`documented-hole`** — the matcher does **not** catch this, and `hole` names the record that says so.
46
+ It holds **in both directions**: if the hole silently closes, the case goes red until the record is
47
+ updated.
48
+
49
+ `path` names the branch of `matchesRule` a case exercises — `matchesPath`, `shell-write`, `shell-prefix`
50
+ or `no-branch` — and is **derived, never declared**: the runner computes it with `matcherPath(kind, tool)`
51
+ and refuses a case that disagrees. Byte-level attacks are stored **escaped** (JSON `\r`, `\u0000`), since
52
+ [`../cli/control-chars.mjs`](../cli/control-chars.mjs) refuses raw control bytes in the tree; the suite
53
+ checks both that the corpus is clean and that the escapes decode.
54
+
55
+ The corpus is load-bearing for the next clause: clause (b) needs this corpus as its kill-set and this
56
+ fixture format as its output shape. A `mutants` survivor is repaired by a new case here, and a `fuzz-shell`
57
+ finding prints as a case ready to paste — reviewed first, since its `expect` records what the matcher did.
58
+
59
+ **What it does NOT establish: adequacy.** It checks **presence** — every compiled gate has fixtures and
60
+ every case answers as recorded — which one trivial case per rule satisfies. `mutants` checks
61
+ **discrimination**; **realism**, whether the attacks resemble anything an adversary would type, stays a
62
+ reviewer's judgement. The runner prints that limit on every green and names every rule exempt for
63
+ declaring no matchable action.
64
+
65
+ ## Mutants and the grammar fuzzer — clause (b)
66
+
67
+ ```
68
+ node cli/mutants.mjs --workspace . --pack-root packs the mutation census
69
+ node cli/fuzz-shell.mjs --workspace . --pack-root packs the grammar fuzzer
70
+ ```
71
+
72
+ **`mutants` asks whether the corpus discriminates.** It breaks [`../cli/compile.mjs`](../cli/compile.mjs)'s
73
+ matcher region on purpose, one declared, anchored operator at a time, and grades each mutant against the
74
+ gate corpus. An operator the corpus fails to notice is a hole in the kill-set and the repair is a new
75
+ fixture; a `survives` record is admissible only as a **proof** of equivalence, never as a standing note
76
+ that a gap exists.
77
+
78
+ **`fuzz-shell` asks whether the two segmenters answer one grammar.** It composes each command from a
79
+ grammar, so it knows whether the payload sits where bash would execute it or only print it. Positions are
80
+ enumerated and recorded, spellings are fuzzed, and every spelling of one command in one position must get
81
+ the same answer; every recorded divergence from ground truth cites its record. The grammar's own ground
82
+ truth is measured under real bash with a neutral payload by
83
+ [`../cli/fuzz-shell.ground.test.mjs`](../cli/fuzz-shell.ground.test.mjs), and the seed is pinned and
84
+ printed on every run.
85
+
86
+ **What they do NOT establish:** that the attacks are realistic (`mutants`), or what a gate ought to cover
87
+ (`fuzz-shell` — whether a recorded escape should stay open is a policy question for the maintainer).
88
+
89
+ ## Forced-red drills — clause (d)
90
+
91
+ ```
92
+ node cli/drills.mjs --pack-root packs the sweep: force every rail red
93
+ node cli/drills.mjs --check --pack-root packs the roster: every rail has a drill
94
+ ```
95
+
96
+ `goldens` asks whether a gate has fixtures and `mutants` whether they discriminate; **this asks whether
97
+ the rail still fires at all.** A drill is a pair — a control on a pristine tree, then a perturbation that
98
+ must place exactly once and move bytes on disk — and the rail's output must carry the drill's declared
99
+ tell in the fire and not in the control. Each runs in a throwaway `git worktree`. A new rail needs an
100
+ entry in `DRILLS` in [`../cli/drills.mjs`](../cli/drills.mjs): `--check` reds a yielded rail with no drill.
101
+ The sweep reports on a **commit** and refuses a dirty tree (`--working-copy` synthesizes one), which is why
102
+ the verify recipe runs `--check` and not the sweep; the rails it cannot force are named in its output, each
103
+ with the reason.
104
+
105
+ **The calendar** is [`../.github/workflows/drills.yml`](../.github/workflows/drills.yml), weekly, and the
106
+ recorded runs of its dispatch and its schedule are in
107
+ [`../.portulan/gate-map/platform-floor.md`](../.portulan/gate-map/platform-floor.md).
108
+
109
+ **What it does NOT establish:** `--check` runs no rail, so whether each one fires is the sweep's answer on
110
+ the calendar; and a scheduled run that never starts stays undetectable from inside —
111
+ [#344](https://github.com/sleepy-panda-srl/portulan/issues/344).
112
+
113
+ ## The review-loop meter — clause (c)
114
+
115
+ ```
116
+ node cli/review-meter.mjs --snapshot evals/review-loop/snapshot.json the figures
117
+ node cli/review-meter.mjs --fetch --repo <owner/name> --out <file> the one mode that fetches
118
+ ```
119
+
120
+ [`review-loop/snapshot.json`](review-loop/snapshot.json) is the capture and
121
+ [`review-loop/register.md`](review-loop/register.md) the register rendered from it: the first command
122
+ with `--register evals/review-loop/register.md --write` regenerates it, and the recipe byte-compares it.
123
+ The criterion's *"rounds per pull request"* is **submissions** per pull request — its unit before *round*
124
+ was redefined — and the tool prints that name, never the bare word.
125
+ What each figure is, at length, is in the header at `git show a31da46:cli/review-meter.mjs`.
126
+
127
+ **What it does NOT establish:**
128
+
129
+ - **Fix-rounds.** Whether a push answers a submission is a fact about its contents that no API answers, so
130
+ the tool computes none, estimates none, and prints nothing that could be mistaken for one.
131
+ - **The exact empty-round rate.** It prints `submissions that found nothing ≤ submissions with no inline
132
+ comment`, an upper bound. The other half — a suppressed note in the review body — was decided by a
133
+ workspace-layer matcher the engine tool refused to copy, since two spellings of one rule on opposite
134
+ sides of that boundary cannot see each other drift; the matcher left with its workflow on 2026-09-23
135
+ ([#355](https://github.com/sleepy-panda-srl/portulan/issues/355)).
136
+ - **A bound.** It is a meter: rule 4 of
137
+ [`a-review-loop-needs-a-bound.md`](../.portulan/memory/a-review-loop-needs-a-bound.md) stops a loop, and
138
+ nothing here stops anything.
139
+ - **Current data.** The recipe compares the register to the snapshot, never the snapshot to the world, so a
140
+ stale capture and a current one are the same green. Refreshing is `--fetch`, run by a person
141
+ ([#356](https://github.com/sleepy-panda-srl/portulan/issues/356)).
142
+
143
+ ## Golden tasks per core skill
144
+
145
+ ```
146
+ node cli/skill-goldens.mjs --repo-root . --workspace .portulan
147
+ ```
148
+
149
+ The row's original first subject. **Ruled, not derived:** a deterministic corpus graded against live
150
+ artifacts in the tree; the model-run sense is the A/B clause's.
151
+
152
+ **What a case is.** [`goldens/skills/<skill>.json`](goldens/skills/) holds one case per numbered step of
153
+ the skill's `## The pass`. The runner derives that denominator from the skill file, so a new step is red
154
+ until a case accounts for it. A step is either:
155
+
156
+ - **`bound`** — a `mandate` quote that must place exactly once in the skill's `SKILL.md`, the `artifacts`
157
+ it governs as a declared slot, a `predicate` decided before any figure is taken, and a `carrier` naming
158
+ the rail that enforces it, or `null`. `kind` is `load-bearing`, or `census` for a row that re-indexes a
159
+ figure an existing recipe already prints.
160
+ - **`unbindable`** — with a `reason` from a closed vocabulary: `judgement-only`, `no-artifact`,
161
+ `cross-language` or `already-carried`. A `judgement-only` step may name no artifacts: if you can name the
162
+ artifacts a mandate governs, it is not judgement, it is unbuilt.
163
+
164
+ ### Accepted drift — the in-tree record
165
+
166
+ A bound case may list live files that fail its mandate and still ship green, in `expect.accepted` as
167
+ `{file, why}`: a merged record is not an implementer's to rewrite. The runner refuses an entry whose `why`
168
+ carries no argument, and the list holds **in both directions** — a file that starts complying reddens the
169
+ corpus until it is delisted, because a drift list that outlives its drift is as wrong as one that hides
170
+ it. The runner prints each count beside the compliant total. Three mandates carry accepted drift, tracked
171
+ together as [#358](https://github.com/sleepy-panda-srl/portulan/issues/358):
172
+
173
+ | Mandate | Live artifacts | Drift |
174
+ |---|---|---|
175
+ | `clarify` step 4 — fold answers back as **EARS** acceptance criteria | `.portulan/tasks/` | no acceptance-criteria section, or criteria not EARS-shaped |
176
+ | `codify` step 1 — name the incident and link it | `.portulan/proposals/` | no provenance field |
177
+ | `codify` step 3 — attach how it earns its place | `.portulan/proposals/` | no enforcement field |
178
+
179
+ **The maintainer ruled on 2026-08-26 that [`../core/templates/task.md`](../core/templates/task.md)'s shape
180
+ still binds**, so `clarify`'s six accepted files are non-compliant by his word rather than by an
181
+ implementer's inference — which is what makes them accepted drift and not a divergence pinned before
182
+ anyone ruled. Of those six, five carry no acceptance-criteria section in any spelling and one fails on
183
+ EARS shape alone, so *"fails the EARS mandate"* is arithmetically right and rhetorically wrong; #358 has
184
+ the cohorts.
185
+
186
+ ### What this rail does NOT establish
187
+
188
+ - **Adequacy.** A mandate can be bound and the binding trivial.
189
+ - **That a carrier runs its check.** The predicates import the carriers' own functions — `RETIRE_WHEN`
190
+ comes from [`../cli/doctor.mjs`](../cli/doctor.mjs) — which proves a carrier **contains** the check,
191
+ not that it **runs** it; and `carrier` is a declared, reviewed field that nothing links to a check.
192
+ Closing that needs a per-mandate drill, the `drills` shape one grain finer.
193
+ - **An agent's judgement.** It grades a skill's mandates against the tree, never an agent following one.
194
+ The `judgement-only` rows are the A/B clause's first subject, which [`ab/corpus.md`](ab/corpus.md)
195
+ carries.
196
+
197
+ ## The OTel opt-in
198
+
199
+ ```
200
+ evals/telemetry/config.json the committed opt-in — the ONLY gate on emission
201
+ evals/telemetry/review-loop.otlp.json the golden payload, regenerated and byte-compared
202
+ node cli/telemetry.mjs --config evals/telemetry/config.json --render
203
+ ```
204
+
205
+ The same command with `--write evals/telemetry/review-loop.otlp.json` in place of `--render` regenerates
206
+ the golden, which every release cut owes because the version rides in the payload. **Ruled, not
207
+ derived:** a real emitter of hand-written OTLP-over-HTTP JSON, off by default, never callable from a
208
+ verify recipe, carrying the review-loop figures only. [`../cli/telemetry.mjs`](../cli/telemetry.mjs)
209
+ carries the ruling, the environment variables it reads and the ones it does not, and what the payload
210
+ may hold.
211
+
212
+ **The committed file is the only gate** — deliberately not `OTEL_SDK_DISABLED`, which defaults to `false`
213
+ and would make emission an opt-out. The environment supplies transport only, and the validator refuses a
214
+ config carrying `headers`, `endpoint` or a token, so a secret never enters a committed file.
215
+
216
+ **Consent is ruled, and railed.** The committed opt-in **is** the standing consent, and committing a
217
+ config that says `enabled: true` is Gated and the maintainer's alone. `--export` refuses a config that is
218
+ **untracked**, **tracked but absent from `HEAD`** — staged and never committed — or **differs from
219
+ `HEAD`**, so editing a working copy cannot manufacture consent. It is person-invoked only: CI runs this
220
+ module on every pull request but never `--export`, and wiring that mode into a workflow, hook or schedule
221
+ is a new consent question rather than a covered one. It neither queues nor retries.
222
+
223
+ **What it does NOT establish:**
224
+
225
+ - **Current data.** The payload is checked against the snapshot, never the snapshot against the world
226
+ ([#356](https://github.com/sleepy-panda-srl/portulan/issues/356)). The capture stamp travels inside the
227
+ payload, so a stale export arrives labelled stale — and time-series backends commonly drop aged samples.
228
+ - **Who committed the consent.** The rail establishes *tracked and byte-identical to `HEAD`*, not
229
+ *committed by the maintainer*: `commit-to-a-working-branch` is `auto`, so an agent can commit it. What it
230
+ buys is that the act appears in a reviewed diff.
231
+ - **The opt-in in the affirmative.** No workspace in this tree opts in, so every green is about the off
232
+ path. The send is proven only in the suite, against an injected transport, and no yielded recipe may
233
+ ever exercise `--export`.
234
+ - **A network mode reached indirectly.** The suite derives the set of network-capable modules from the
235
+ tree, and the offline audit matches a module as a path suffix and a flag as a token; a mode reached
236
+ through another script, or through a flag built at runtime, is invisible to it.
237
+ - **Scheduled emission.** Nothing fires when nobody runs it —
238
+ [#344](https://github.com/sleepy-panda-srl/portulan/issues/344)'s silence again.
239
+ - **Enforcement of the review-loop bound.** An emitter with no backend reading it is not the checker
240
+ [`a-review-loop-needs-a-bound.md`](../.portulan/memory/a-review-loop-needs-a-bound.md) lacks.
241
+ - **Traces.** Metrics only.
242
+
243
+ ## The A/B baseline
244
+
245
+ | File | What it is |
246
+ |---|---|
247
+ | [`ab/corpus.md`](ab/corpus.md) | The scenarios and grading rules, the registered carrier of the clause's subject, and *What may not be concluded* |
248
+ | [`ab/arm.md`](ab/arm.md) | What *Portulan on* denotes here: the vendored-and-compiled treatment arm |
249
+ | [`ab/register.md`](ab/register.md) | Both arms as built, byte-compared by the `ab` recipe |
250
+ | [`ab/graders.md`](ab/graders.md) | The graders' discrimination, byte-compared by the `ab-grade` recipe |
251
+ | [`ab/baseline.json`](ab/baseline.json), [`ab/baseline.md`](ab/baseline.md) | The capture, and the register rendered from it — the one carrier of the figures |
252
+ | [`ab/warm.md`](ab/warm.md) | The warm-start A/B, a sibling of the baseline rather than a clause of row 8: it prices fresh sessions warm against cold from the host's own records, in the three lines every measurement reports, and says which host switch earns a place in `sessions`. Its runner, [`../cli/warm.mjs`](../cli/warm.mjs), is built, and its record waits on the maintainer's word, since every run starts real sessions |
253
+
254
+ [`../cli/ab.mjs`](../cli/ab.mjs) builds the arms, [`../cli/ab-grade.mjs`](../cli/ab-grade.mjs) grades the
255
+ trees they leave behind, and [`../cli/ab-run.mjs`](../cli/ab-run.mjs) runs the turns between: `k = 5` per
256
+ cell, ruled by the maintainer, every turn isolated with a home and config directory of its own. Each
257
+ module's `--write` regenerates its document; `ab-run --write` re-renders the register from the committed
258
+ capture. The recipes run no agent; the modes that do — `ab --stop-probe`, `ab-run --smoke` and
259
+ `--matrix` — are a person's to run.
260
+
261
+ **What may not be concluded** from the baseline is [`ab/corpus.md`](ab/corpus.md)'s section of that name,
262
+ and the register cites it rather than restating it. **The baseline is scoped to the vendored-and-compiled
263
+ tier** [`ab/arm.md`](ab/arm.md) specifies, and closes row 8 for no other configuration of *Portulan on*.
264
+
265
+ ## A release carries an eval result
266
+
267
+ ```
268
+ node cli/release-eval.mjs --capture at a cut: run every yielded recipe, write the record pair
269
+ node cli/release-eval.mjs --verify the recipe: every governed release has a record matching its capture
270
+ ```
271
+
272
+ [`releases/`](releases/) holds one pair per version, described in
273
+ [`releases/README.md`](releases/README.md), for every version [`../CHANGELOG.md`](../CHANGELOG.md)
274
+ records from `0.1.3` onward — graded permanently and in both directions. A record is the verdict every
275
+ recipe the workspace yields returned at a named commit, plus the **identity** of the A/B baseline the
276
+ release ships against, never its figures.
277
+ **`evals/releases/` ships in the package**, by the maintainer's decision of 2026-09-01, so the package
278
+ carries both the record and the tool that checks it: `--tagged` grades a record in the unpacked package as
279
+ it does in the tagged tree. The recipe grades the tree on every commit;
280
+ [`../.github/workflows/publish-github-packages.yml`](../.github/workflows/publish-github-packages.yml) runs
281
+ `--tagged` against the tag's own checkout before publishing; the rest is Gated, and
282
+ [`../.portulan/gate-map.md`](../.portulan/gate-map.md) carries it beside its tier.
283
+
284
+ **What it does NOT establish.** Every register prints its own limits: a green means the record agrees with
285
+ its capture, never that the release is good; the recipes ran at a commit inside the cut change, not at the
286
+ tag; `release-eval`'s own row is excluded. Beyond the register lies the **published** half: `--tagged`
287
+ reaches the tagged tree at the publish, and the release body, written outside the tree, is reached by no
288
+ check. While every governed release has its record, the clause's central arm — a governed release with
289
+ none — is exercised only by the forced-red drill that moves the boundary.
290
+
291
+ **`v0.1.3` is the first release published with its eval result.** It was tagged on 2026-09-30 at the
292
+ commit that cut it, the publish workflow ran `--tagged` against that tree before publishing it, and its
293
+ release body cites [`releases/0.1.3.md`](releases/0.1.3.md) without restating it. That the body cites the
294
+ register is the maintainer's act, and nothing here checks it.
295
+
296
+ ## A rule change decided on eval evidence
297
+
298
+ Row 8's clause *a rule change merges or is rejected on eval evidence* is discharged by one instance, hole 8
299
+ of [`../.portulan/gate-map.md`](../.portulan/gate-map.md): a `gated` or `prohibited` path target that can
300
+ never match. [#337](https://github.com/sleepy-panda-srl/portulan/issues/337) set out three answers, the
301
+ instruments chose between them, and the change merged on 2026-09-09; the evidence table and the argument
302
+ are [`../docs/milestones/m08.md`](https://github.com/sleepy-panda-srl/portulan/blob/8a33f9b/docs/milestones/m08.md)'s session-10 note. The rail is `neverMatches`
303
+ and the backend refusal beside `HOST_GATE_TIERS` in [`../cli/compile.mjs`](../cli/compile.mjs), a suite
304
+ block, and a second forced-red drill on `doctor`; its limit — the refusal is at compile time, not at the
305
+ hook — is stated beside the refusal. The hazard is held there and not in the gate corpus, whose
306
+ denominator is the yielded policy: adding a never-matching `gated` rule to `gates.json` to give the corpus
307
+ a case would manufacture the defect.
308
+
309
+ ## What is NOT built yet
310
+
311
+ Each names where it arrives, per [`../.portulan/dod.md`](../.portulan/dod.md) condition 4 — nothing here
312
+ claims a capability that does not exist:
313
+
314
+ - **The `evals` and `telemetry` Workspace Definition slots.** Proposal
315
+ [`0034`](../.portulan/proposals/0034-one-spec-bump-carries-both-evals-and-telemetry.md), accepted
316
+ 2026-08-28, carries both in one spec bump. The deferral in [`../spec/slots.md`](../spec/slots.md) is due
317
+ since the milestone-8 close and undrafted; the drafting settles the key names, each slot's structure and
318
+ whether a migration is owed.
319
+
320
+ Each rail's open limits are in its section above, with the issue tracking each where there is one.
@@ -0,0 +1,134 @@
1
+ {
2
+ "portulan": {
3
+ "releaseEval": "1"
4
+ },
5
+ "version": "0.1.3",
6
+ "captured": "2026-09-16",
7
+ "source": {
8
+ "commit": "fac7b6d36ce3aa53ce93381ef8cce07e503608e4",
9
+ "clean": true
10
+ },
11
+ "host": {
12
+ "node": "v26.8.1",
13
+ "platform": "darwin"
14
+ },
15
+ "recipes": [
16
+ {
17
+ "id": "docs",
18
+ "exit": 0
19
+ },
20
+ {
21
+ "id": "json",
22
+ "exit": 0
23
+ },
24
+ {
25
+ "id": "doctor",
26
+ "exit": 0
27
+ },
28
+ {
29
+ "id": "tests",
30
+ "exit": 0
31
+ },
32
+ {
33
+ "id": "plugin",
34
+ "exit": 0
35
+ },
36
+ {
37
+ "id": "compile",
38
+ "exit": 0
39
+ },
40
+ {
41
+ "id": "workflow-filters",
42
+ "exit": 0
43
+ },
44
+ {
45
+ "id": "index",
46
+ "exit": 0
47
+ },
48
+ {
49
+ "id": "control-chars",
50
+ "exit": 0
51
+ },
52
+ {
53
+ "id": "rule-carriers",
54
+ "exit": 0
55
+ },
56
+ {
57
+ "id": "pack-version",
58
+ "exit": 0
59
+ },
60
+ {
61
+ "id": "pack-identity",
62
+ "exit": 0
63
+ },
64
+ {
65
+ "id": "payload",
66
+ "exit": 0
67
+ },
68
+ {
69
+ "id": "eval-bundle",
70
+ "exit": 0
71
+ },
72
+ {
73
+ "id": "goldens",
74
+ "exit": 0
75
+ },
76
+ {
77
+ "id": "mutants",
78
+ "exit": 0
79
+ },
80
+ {
81
+ "id": "fuzz-shell",
82
+ "exit": 0
83
+ },
84
+ {
85
+ "id": "version-carriers",
86
+ "exit": 0
87
+ },
88
+ {
89
+ "id": "skill-goldens",
90
+ "exit": 0
91
+ },
92
+ {
93
+ "id": "review-loop",
94
+ "exit": 0
95
+ },
96
+ {
97
+ "id": "telemetry",
98
+ "exit": 0
99
+ },
100
+ {
101
+ "id": "drills",
102
+ "exit": 0
103
+ },
104
+ {
105
+ "id": "ab",
106
+ "exit": 0
107
+ },
108
+ {
109
+ "id": "ab-grade",
110
+ "exit": 0
111
+ },
112
+ {
113
+ "id": "ab-run",
114
+ "exit": 0
115
+ },
116
+ {
117
+ "id": "tools/github:actions-pinned",
118
+ "exit": 0
119
+ }
120
+ ],
121
+ "excluded": [
122
+ {
123
+ "id": "release-eval",
124
+ "why": "a capture cannot be accurate about the record it is inside — captured before the record is written it is red for the record's absence, and captured after it is stale the moment the register is rendered. Its verdict for this release is the rail's own run on the pull request carrying this file."
125
+ }
126
+ ],
127
+ "abBaseline": {
128
+ "snapshot": "evals/ab/baseline.json",
129
+ "register": "evals/ab/baseline.md",
130
+ "captured": "2026-08-31T09:05:32.699Z",
131
+ "commit": "a642d551eec6c146c67ef0acfe3f8ba535c5ce6e",
132
+ "clean": false
133
+ }
134
+ }
@@ -0,0 +1,64 @@
1
+ # Eval result — Portulan 0.1.3
2
+
3
+ > **Generated. Do not edit.** Rendered from [`evals/releases/0.1.3.json`](0.1.3.json) by [`cli/release-eval.mjs`](../../cli/release-eval.mjs) and byte-compared by the `release-eval` verify recipe. This is the eval result milestone 8 requires a release to carry — `docs/plan.md`, Protocol → Versioning.
4
+
5
+ ## What this release was measured at
6
+
7
+ | | |
8
+ |---|---|
9
+ | Version | `0.1.3` |
10
+ | Captured | 2026-09-16 |
11
+ | Commit | `fac7b6d36ce3aa53ce93381ef8cce07e503608e4` |
12
+ | Tree at capture | clean |
13
+ | Node | `v26.8.1` |
14
+ | Platform | `darwin` |
15
+
16
+ ## The rails, and what each returned
17
+
18
+ 26 of the 27 recipes this workspace yielded at capture. The set is derived from [`cli/recipe-set.mjs`](../../cli/recipe-set.mjs) and never listed by hand.
19
+
20
+ | Recipe | Verdict |
21
+ |---|---|
22
+ | `docs` | green |
23
+ | `json` | green |
24
+ | `doctor` | green |
25
+ | `tests` | green |
26
+ | `plugin` | green |
27
+ | `compile` | green |
28
+ | `workflow-filters` | green |
29
+ | `index` | green |
30
+ | `control-chars` | green |
31
+ | `rule-carriers` | green |
32
+ | `pack-version` | green |
33
+ | `pack-identity` | green |
34
+ | `payload` | green |
35
+ | `eval-bundle` | green |
36
+ | `goldens` | green |
37
+ | `mutants` | green |
38
+ | `fuzz-shell` | green |
39
+ | `version-carriers` | green |
40
+ | `skill-goldens` | green |
41
+ | `review-loop` | green |
42
+ | `telemetry` | green |
43
+ | `drills` | green |
44
+ | `ab` | green |
45
+ | `ab-grade` | green |
46
+ | `ab-run` | green |
47
+ | `tools/github:actions-pinned` | green |
48
+
49
+ **`release-eval` is excluded**, and the reason is printed rather than the row silently dropped: a capture cannot be accurate about the record it is inside — captured before the record is written it is red for the record's absence, and captured after it is stale the moment the register is rendered. Its verdict for this release is the rail's own run on the pull request carrying this file.
50
+
51
+ ## The A/B baseline this release ships against
52
+
53
+ [`evals/ab/baseline.md`](../../evals/ab/baseline.md), rendered from `evals/ab/baseline.json` — captured 2026-08-31T09:05:32.699Z at `a642d551eec6c146c67ef0acfe3f8ba535c5ce6e`, over a tree that was **not clean**.
54
+
55
+ **Its figures are not repeated here.** They have one carrier and this is not it.
56
+
57
+ ## What this record does not establish
58
+
59
+ - **A green says this record agrees with its own capture — never that the release is good.** Every line below is what the recipes returned at one commit, and a recipe's own green establishes only what that recipe's documentation says it establishes.
60
+ - **The recipes were not run at the tag.** They ran at `fac7b6d36ce3aa53ce93381ef8cce07e503608e4`, and this record is committed *in* the cut change — so the tagged tree is this commit plus whatever that change still adds after the capture, the record itself at minimum. A record cannot be captured at a commit that does not exist yet, and printing the one it was captured at is the only honest form.
61
+ - **`release-eval` is excluded from the rows above**, because a capture cannot be accurate about the record it is inside. Its verdict for this release is the rail's own run on the pull request that carries this file.
62
+ - **This record measures the Portulan repository's own build**, at the commit named above — never the workspace, project or package this release is installed into. Nothing here has looked at your tree.
63
+ - **The tree was clean at capture**, so the commit named above is what was measured.
64
+ - **The A/B baseline's figures are NOT restated here.** This record names which baseline the release ships against; the figures and everything that may not be concluded from them live in that register alone.