@sleepy_panda_srl/portulan 0.1.1 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -19
- package/cli/README.md +155 -125
- package/cli/compile.mjs +797 -40
- package/cli/doctor.mjs +59 -2
- package/cli/drills.mjs +1541 -0
- package/cli/fuzz-shell.mjs +880 -0
- package/cli/gate.mjs +3 -2
- package/cli/goldens.mjs +469 -0
- package/cli/index.mjs +92 -42
- package/cli/init.mjs +4 -1
- package/cli/inside.mjs +38 -0
- package/cli/mutants.mjs +925 -0
- package/cli/pack-identity.mjs +9 -2
- package/cli/pack-version.mjs +7 -0
- package/cli/recipe-set.mjs +48 -1
- package/cli/release-eval.mjs +1068 -0
- package/cli/review-meter.mjs +741 -0
- package/cli/skill-goldens.mjs +590 -0
- package/cli/skills-set.mjs +166 -4
- package/cli/stop-gate.mjs +301 -28
- package/cli/telemetry.mjs +1047 -0
- package/cli/version-carriers.mjs +224 -0
- package/core/operating/memory.md +7 -8
- package/core/operating/verification.md +4 -12
- package/core/templates/gate-policy.md +27 -8
- package/evals/README.md +910 -0
- package/evals/releases/0.1.3.json +134 -0
- package/evals/releases/0.1.3.md +64 -0
- package/evals/releases/README.md +32 -0
- package/package.json +7 -2
- package/packs/README.md +11 -21
- package/packs/rituals/checkpoints/README.md +8 -1
- package/packs/rituals/checkpoints/pack.json +1 -1
- package/packs/rituals/checkpoints/skills/pre-commit/SKILL.md +32 -2
- package/packs/tools/README.md +1 -4
- package/spec/README.md +19 -22
- package/spec/migrations/README.md +2 -3
- package/spec/slots.md +30 -11
package/cli/gate.mjs
CHANGED
|
@@ -49,8 +49,9 @@
|
|
|
49
49
|
// `bash -c "git push …"` is invisible to it — measured, and the reason
|
|
50
50
|
// `../core/operating/autonomy.md` calls the platform floor the gate that holds when this
|
|
51
51
|
// layer fails. This runner unwraps one level of `sh -c` / `bash -c` / `zsh -c` before matching.
|
|
52
|
-
// 2. A SHELL WRITE to a path a `write:` rule protects. `Edit(./docs/vision.md)` denies three
|
|
53
|
-
// tools
|
|
52
|
+
// 2. A SHELL WRITE to a path a `write:` rule protects. `Edit(./docs/vision.md)` denies all three
|
|
53
|
+
// file-editing tools — ONE emitted pattern, which the host matches for every one of them — and
|
|
54
|
+
// `echo x >> docs/vision.md` is a fourth way to the same bytes. `matchesRule` now
|
|
54
55
|
// answers for `Bash` on a write rule, by a table of redirections and file-writing commands
|
|
55
56
|
// that `./compile.mjs` states in full.
|
|
56
57
|
//
|
package/cli/goldens.mjs
ADDED
|
@@ -0,0 +1,469 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Grade this workspace's compiled gates against a corpus of adversarial fixtures.
|
|
3
|
+
//
|
|
4
|
+
// Milestone 8, clause (a): *golden tasks reach the gates as well as the skills, so a matcher ships
|
|
5
|
+
// with the attack cases that prove its coverage instead of prose describing it.*
|
|
6
|
+
//
|
|
7
|
+
// ## What this is FOR, at its real size
|
|
8
|
+
//
|
|
9
|
+
// `../.portulan/gate-map.md` carries an honest-holes list. That list was wrong when first published —
|
|
10
|
+
// four items, five missing, the plainest of them a newline — and it was corrected by a supervisor who
|
|
11
|
+
// tried to DEFEAT the matcher rather than read it. Fourteen review rounds then found three more live
|
|
12
|
+
// bypasses the attack pass itself had not tried. Eight in all, every one found after the gate was
|
|
13
|
+
// called done.
|
|
14
|
+
//
|
|
15
|
+
// A hole list is a claim like any other, and the only thing that checks it is somebody attacking it.
|
|
16
|
+
// This runner is where those attacks live so they are re-run on every commit instead of on the day
|
|
17
|
+
// somebody happens to look.
|
|
18
|
+
//
|
|
19
|
+
// ## The two things it rails, and the one it does NOT
|
|
20
|
+
//
|
|
21
|
+
// 1. **Coverage is measured, not named.** A rule that compiles to a matcher and carries no fixture
|
|
22
|
+
// is RED. So a new gate cannot ship with prose standing in for attack cases — the failure
|
|
23
|
+
// `../.portulan/memory/a-checkers-coverage-is-measured-not-named.md` names, turned into a check.
|
|
24
|
+
// 2. **A documented hole stays documented in BOTH directions.** A `documented-hole` case asserts the
|
|
25
|
+
// matcher's CURRENT behaviour and names the hole record it belongs to. If the hole silently
|
|
26
|
+
// closes, that case goes red too — because a hole list that still lists a closed hole is as wrong
|
|
27
|
+
// as one that hides an open one, which is a sentence the gate map has had to write about itself.
|
|
28
|
+
//
|
|
29
|
+
// **What it does not rail: adequacy.** This is a PRESENCE floor. One trivial happy-path fixture per
|
|
30
|
+
// rule satisfies clause 1 while proving nothing adversarial, and no check here can tell the
|
|
31
|
+
// difference — judging whether a corpus is a real attack is exactly the judgement a reviewer is for.
|
|
32
|
+
// Written down rather than left to be discovered, because a rail whose limit is unstated gets read as
|
|
33
|
+
// the guarantee it is not. What the floor buys is narrower and still worth having: a gate cannot reach
|
|
34
|
+
// the compiled policy with NO adversarial thought recorded against it at all.
|
|
35
|
+
//
|
|
36
|
+
// ## How a case is graded — and what is deliberately never done to it
|
|
37
|
+
//
|
|
38
|
+
// Every case is answered by `./compile.mjs`'s own exported `matchesRule`, the same function
|
|
39
|
+
// `./gate.mjs` calls at hook time. Never a re-implementation: "two implementations of one matcher is
|
|
40
|
+
// the drift this repository keeps finding, and a *matcher* that drifts does not look wrong — it looks
|
|
41
|
+
// like a matcher" (`./compile.mjs`).
|
|
42
|
+
//
|
|
43
|
+
// **A fixture's command string is DATA and is never executed.** The corpus contains
|
|
44
|
+
// `git push --force`, `rm -rf docs` and constitution-write spellings by design — running one would be
|
|
45
|
+
// somewhere between tripping the gate under test and destroying the tree it runs in. There is no code
|
|
46
|
+
// path from a fixture to a subprocess, and `./goldens.test.mjs` asserts that this module spawns
|
|
47
|
+
// nothing at all.
|
|
48
|
+
//
|
|
49
|
+
// ## Which policy it grades
|
|
50
|
+
//
|
|
51
|
+
// The one the workspace YIELDS — its own `gates.json` plus the fragments its composed packs
|
|
52
|
+
// contribute — never the declared file alone. A rail scoped to the declared list would let a composed
|
|
53
|
+
// gate ship with no fixtures while this check stayed green, which is the class `../.portulan/dod.md`
|
|
54
|
+
// condition 1 names in so many words and `./gate.mjs` already had repaired.
|
|
55
|
+
//
|
|
56
|
+
// Resolution is HERMETIC: no host plugin-cache discovery, ever. `compile --matrix` without a pinned
|
|
57
|
+
// root refuses on this maintainer's machine as SHADOWED — an installed cache copy at one version
|
|
58
|
+
// against the tree's at another — so a runner that consulted the host would grade the machine rather
|
|
59
|
+
// than the tree, and its verdict would move with what happens to be installed. The recipe names its
|
|
60
|
+
// root as well, on the pin discipline `../.portulan/verify/compile.sh` and `./pinned-roots.live.test.mjs`
|
|
61
|
+
// already hold the compiler to.
|
|
62
|
+
//
|
|
63
|
+
// Exit 0 green · 1 red · 2 could not run.
|
|
64
|
+
|
|
65
|
+
import fs from "node:fs";
|
|
66
|
+
import path from "node:path";
|
|
67
|
+
import { pathToFileURL } from "node:url";
|
|
68
|
+
|
|
69
|
+
import {
|
|
70
|
+
CompileError,
|
|
71
|
+
READ_TOOLS,
|
|
72
|
+
WRITE_TOOLS,
|
|
73
|
+
composeFragments,
|
|
74
|
+
matchesRule,
|
|
75
|
+
packContributions,
|
|
76
|
+
parse,
|
|
77
|
+
policyDeclaration,
|
|
78
|
+
resolveWorkspace,
|
|
79
|
+
} from "./compile.mjs";
|
|
80
|
+
|
|
81
|
+
/** Where the corpus lives, relative to the repository root. */
|
|
82
|
+
export const CORPUS_DIR = "evals/goldens/gates";
|
|
83
|
+
|
|
84
|
+
/** The action kinds `matchesRule` can answer for. A rule shaped otherwise has no matcher to attack. */
|
|
85
|
+
export const MATCHABLE = ["shell", "write", "read"];
|
|
86
|
+
|
|
87
|
+
/**
|
|
88
|
+
* The two case classes, and why there are exactly two.
|
|
89
|
+
*
|
|
90
|
+
* `holds` is a regression pin: the matcher catches this today and must keep catching it. Every one of
|
|
91
|
+
* the eight bypasses found on #60 is one of these — they are CLOSED holes, and the corpus exists so
|
|
92
|
+
* they cannot quietly re-open.
|
|
93
|
+
*
|
|
94
|
+
* `documented-hole` is the other direction: the matcher does NOT catch this, the gate map says so, and
|
|
95
|
+
* the case exists to keep that admission true. A third class for "we hope this gets fixed" was
|
|
96
|
+
* considered and refused — it would be a wish, and a wish that goes green when someone fixes it is
|
|
97
|
+
* indistinguishable from a hole that closed without its record being updated.
|
|
98
|
+
*/
|
|
99
|
+
export const CLASSES = ["holds", "documented-hole"];
|
|
100
|
+
|
|
101
|
+
/**
|
|
102
|
+
* Which branch of `matchesRule` a case exercises — the session-open checkpoint's adjustment 3.
|
|
103
|
+
*
|
|
104
|
+
* **Why a case must say this.** One rule id can be answered by two different segmenters with two
|
|
105
|
+
* different answers, and the corpus contains a live example: a `then`/`do`/brace-group leader is
|
|
106
|
+
* CAUGHT on the write path (`shellSegments` knows `SEGMENT_LEADERS`) and ESCAPES on the shell path
|
|
107
|
+
* (`commandSegments` does not). A reader meeting those two cases without this field would take the
|
|
108
|
+
* pair for a contradiction rather than for the asymmetry it is.
|
|
109
|
+
*
|
|
110
|
+
* **And why it is DERIVED rather than declared.** `matcherPath` computes it from the rule's action
|
|
111
|
+
* kind and the case's tool, and `grade` refuses a case whose declared path disagrees. A field written
|
|
112
|
+
* from memory on 212 cases is a second carrier of what `WRITE_TOOLS`, `READ_TOOLS` and the action kind
|
|
113
|
+
* already decide — and this repository's standing finding is that the second carrier is the one that
|
|
114
|
+
* goes wrong. Writing it down keeps it readable; deriving it keeps it true.
|
|
115
|
+
*/
|
|
116
|
+
export const PATHS = ["matchesPath", "shell-write", "shell-prefix", "no-branch"];
|
|
117
|
+
|
|
118
|
+
/**
|
|
119
|
+
* The branch `matchesRule` will take for this rule kind and tool.
|
|
120
|
+
*
|
|
121
|
+
* `no-branch` is a real answer and not a gap: it names a combination the matcher has no code for, and
|
|
122
|
+
* a case declaring it is asserting exactly that — `Bash` against a `read:` rule, or a write tool
|
|
123
|
+
* against a `shell:` one. Those cases matter because the asymmetry between action kinds is the sort of
|
|
124
|
+
* thing a reader assumes away.
|
|
125
|
+
*/
|
|
126
|
+
export function matcherPath(kind, tool) {
|
|
127
|
+
if (kind === "shell") return tool === "Bash" ? "shell-prefix" : "no-branch";
|
|
128
|
+
if (kind === "write") {
|
|
129
|
+
if (WRITE_TOOLS.includes(tool)) return "matchesPath";
|
|
130
|
+
// Bash reaches a `write:` rule through `commandSegments` × `spellings` and then `shellWrites`,
|
|
131
|
+
// which segments AGAIN with `shellSegments`. Named for the entry, since that is what a reader
|
|
132
|
+
// needs to find the code; the docblock above carries the two-segmenter consequence.
|
|
133
|
+
return tool === "Bash" ? "shell-write" : "no-branch";
|
|
134
|
+
}
|
|
135
|
+
if (kind === "read") return READ_TOOLS.includes(tool) ? "matchesPath" : "no-branch";
|
|
136
|
+
return "no-branch";
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
export class CouldNotRun extends Error {}
|
|
140
|
+
|
|
141
|
+
/** Read one JSON file, or say which file and why not. */
|
|
142
|
+
function readJson(file, what) {
|
|
143
|
+
let text;
|
|
144
|
+
try {
|
|
145
|
+
text = fs.readFileSync(file, "utf8");
|
|
146
|
+
} catch (cause) {
|
|
147
|
+
throw new CouldNotRun(`${what} at ${file} cannot be read — ${cause.code ?? cause.message}`);
|
|
148
|
+
}
|
|
149
|
+
try {
|
|
150
|
+
return JSON.parse(text);
|
|
151
|
+
} catch (cause) {
|
|
152
|
+
throw new CouldNotRun(`${what} at ${file} is not valid JSON — ${cause.message}`);
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
/**
|
|
157
|
+
* The rules this workspace yields, parsed.
|
|
158
|
+
*
|
|
159
|
+
* Composition runs BEFORE `parse`, exactly as `compile` does it, so a pack's fragment is graded by the
|
|
160
|
+
* same code that grades a hand-written rule. Following the compiler's order rather than inventing one
|
|
161
|
+
* is the whole reason this function is four lines of plumbing and no judgement.
|
|
162
|
+
*/
|
|
163
|
+
export function yieldedRules(named, { packRoots = null } = {}) {
|
|
164
|
+
const { workspaceRoot, workspaceDir } = resolveWorkspace(named);
|
|
165
|
+
const { file: policyFile, declared, reason } = policyDeclaration(workspaceRoot, workspaceDir);
|
|
166
|
+
if (!declared && !fs.existsSync(policyFile)) {
|
|
167
|
+
// **The fallback is THREE states and the message names which one.** This said
|
|
168
|
+
// `${workspaceDir}/workspace.json declares none` for all of them, which is a sentence that is
|
|
169
|
+
// false twice over: when the manifest does not exist there is nothing to have declared
|
|
170
|
+
// anything, and when it named a policy this compiler refused, it declared one and was
|
|
171
|
+
// overruled. Both send a reader to audit the one file that is not at fault — the failure
|
|
172
|
+
// `../.portulan/memory/verify-preconditions-fail-closed.md` names, and the failure
|
|
173
|
+
// `compile.mjs` had already learned at `undeclaredPolicyMessage`, whose own comment says
|
|
174
|
+
// "only the caller knows to name what the absence costs". I copied that call site's SHAPE
|
|
175
|
+
// and not its discipline. Reported as a suppressed note by Copilot, round 8 on #336.
|
|
176
|
+
const why =
|
|
177
|
+
reason === "no-manifest"
|
|
178
|
+
? `there is no ${workspaceDir}/workspace.json, so nothing here has been authored as a workspace yet`
|
|
179
|
+
: reason === "refused"
|
|
180
|
+
? `${workspaceDir}/workspace.json DOES name a gate policy, and it was refused — the manifest is not at fault, the path it names is`
|
|
181
|
+
: `${workspaceDir}/workspace.json declares no \`gates\` key, which is a legitimate shape and means the policy is expected at the default path`;
|
|
182
|
+
throw new CouldNotRun(`no gate policy at ${policyFile}: ${why}. There is nothing to grade fixtures against`);
|
|
183
|
+
}
|
|
184
|
+
const policy = readJson(policyFile, "the gate policy");
|
|
185
|
+
// `discovery: null` and `forced: false`, always. See the header: a rail that consults the host
|
|
186
|
+
// grades the host.
|
|
187
|
+
const { contributions, unresolved } = packContributions(workspaceRoot, workspaceDir, {
|
|
188
|
+
...(packRoots === null ? { named: [] } : { packRoots }),
|
|
189
|
+
discovery: null,
|
|
190
|
+
forced: false,
|
|
191
|
+
});
|
|
192
|
+
const composed = composeFragments(policy, contributions);
|
|
193
|
+
return { workspaceRoot, rules: parse(composed.policy).rules, unresolved };
|
|
194
|
+
}
|
|
195
|
+
|
|
196
|
+
/**
|
|
197
|
+
* Split the yielded rules into the ones a fixture can attack and the ones it cannot.
|
|
198
|
+
*
|
|
199
|
+
* A rule whose action is `none` has no tool-level surface at all — `spend-money-or-register-a-domain`
|
|
200
|
+
* and `send-something-outside-this-repository` are the standing examples, and both say so in their own
|
|
201
|
+
* `action.none` prose. There is no input to hand `matchesRule` for them, so requiring fixtures would be
|
|
202
|
+
* requiring a test of nothing.
|
|
203
|
+
*
|
|
204
|
+
* **They are named in the output on every run regardless**, and that is the load-bearing half. The
|
|
205
|
+
* exemption is the obvious way to dodge this rail — write the next gate `none`-shaped and it needs no
|
|
206
|
+
* attack cases — so the census prints them the way `compile --matrix` prints its own refused rules,
|
|
207
|
+
* rather than letting the exemption be silent.
|
|
208
|
+
*/
|
|
209
|
+
export function partition(rules) {
|
|
210
|
+
const matchable = [];
|
|
211
|
+
const exempt = [];
|
|
212
|
+
for (const rule of rules) {
|
|
213
|
+
const kind = MATCHABLE.find((k) => typeof rule.action?.[k] === "string");
|
|
214
|
+
if (kind) matchable.push({ ...rule, kind });
|
|
215
|
+
else exempt.push({ ...rule, why: Object.keys(rule.action ?? {}).join(", ") || "no action" });
|
|
216
|
+
}
|
|
217
|
+
return { matchable, exempt };
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
/** Every fixture file in the corpus, read and shape-checked. */
|
|
221
|
+
export function readCorpus(repoRoot, dir = CORPUS_DIR) {
|
|
222
|
+
const base = path.join(repoRoot, dir);
|
|
223
|
+
let names;
|
|
224
|
+
try {
|
|
225
|
+
names = fs.readdirSync(base).filter((n) => n.endsWith(".json")).sort();
|
|
226
|
+
} catch (cause) {
|
|
227
|
+
throw new CouldNotRun(`the fixture corpus at ${base} cannot be read — ${cause.code ?? cause.message}`);
|
|
228
|
+
}
|
|
229
|
+
const files = [];
|
|
230
|
+
for (const name of names) {
|
|
231
|
+
const file = path.join(base, name);
|
|
232
|
+
const doc = readJson(file, "a fixture file");
|
|
233
|
+
const where = `${dir}/${name}`;
|
|
234
|
+
if (typeof doc.rule !== "string" || doc.rule === "") {
|
|
235
|
+
throw new CouldNotRun(`${where} names no \`rule\` — a fixture file attacks exactly one rule and must say which`);
|
|
236
|
+
}
|
|
237
|
+
// **One fixture file per rule, named for it — checked, because this file already SAYS it.**
|
|
238
|
+
// The red for a missing fixture prints `add evals/goldens/gates/<rule-id>.json`, `evals/README.md`
|
|
239
|
+
// documents the layout, and nothing enforced either: a renamed or misfiled file validated
|
|
240
|
+
// cleanly and still graded, so the convention held by everyone remembering it. That is the
|
|
241
|
+
// shape `../.portulan/memory/a-mandate-nothing-checks-is-already-broken.md` names, and this
|
|
242
|
+
// module was carrying one of its own while its whole subject is mandates that need checkers.
|
|
243
|
+
// Reported as a suppressed note by Copilot, round 5 on #336.
|
|
244
|
+
if (name !== `${doc.rule}.json`) {
|
|
245
|
+
throw new CouldNotRun(
|
|
246
|
+
`${where} declares rule \`${doc.rule}\` — one fixture file per rule, named for it, so rename it to ` +
|
|
247
|
+
`${dir}/${doc.rule}.json or correct the \`rule\` field. A misfiled corpus grades correctly and ` +
|
|
248
|
+
`reviews badly, which is the worse of the two`,
|
|
249
|
+
);
|
|
250
|
+
}
|
|
251
|
+
if (!Array.isArray(doc.cases) || doc.cases.length === 0) {
|
|
252
|
+
throw new CouldNotRun(`${where} carries no \`cases\` — an empty fixture file is coverage that is not there`);
|
|
253
|
+
}
|
|
254
|
+
for (const [i, c] of doc.cases.entries()) {
|
|
255
|
+
const at = `${where} case ${i} (${c?.id ?? "unnamed"})`;
|
|
256
|
+
if (typeof c?.id !== "string" || c.id === "") throw new CouldNotRun(`${at} has no \`id\``);
|
|
257
|
+
if (!CLASSES.includes(c?.class)) {
|
|
258
|
+
throw new CouldNotRun(`${at} declares class ${JSON.stringify(c?.class)} — one of ${CLASSES.join(" / ")}`);
|
|
259
|
+
}
|
|
260
|
+
if (typeof c?.tool !== "string" || c.tool === "") throw new CouldNotRun(`${at} names no \`tool\``);
|
|
261
|
+
if (!PATHS.includes(c?.path)) {
|
|
262
|
+
throw new CouldNotRun(`${at} declares path ${JSON.stringify(c?.path)} — one of ${PATHS.join(" / ")}`);
|
|
263
|
+
}
|
|
264
|
+
if (typeof c?.expect !== "boolean") throw new CouldNotRun(`${at} declares no boolean \`expect\``);
|
|
265
|
+
if (typeof c?.why !== "string" || c.why.trim() === "") {
|
|
266
|
+
throw new CouldNotRun(`${at} carries no \`why\` — an attack case nobody can read is not reviewable`);
|
|
267
|
+
}
|
|
268
|
+
if (c.class === "documented-hole" && (typeof c.hole !== "string" || c.hole.trim() === "")) {
|
|
269
|
+
throw new CouldNotRun(`${at} is a documented-hole and names no \`hole\` — the record it keeps true is the point of the class`);
|
|
270
|
+
}
|
|
271
|
+
if (c.input === null || typeof c.input !== "object" || Array.isArray(c.input)) {
|
|
272
|
+
throw new CouldNotRun(`${at} declares no \`input\` object`);
|
|
273
|
+
}
|
|
274
|
+
}
|
|
275
|
+
files.push({ where, doc });
|
|
276
|
+
}
|
|
277
|
+
return files;
|
|
278
|
+
}
|
|
279
|
+
|
|
280
|
+
/**
|
|
281
|
+
* Grade the corpus against the rules. Returns findings; the caller decides what they cost.
|
|
282
|
+
*
|
|
283
|
+
* Every finding names the file, the case and the repair. A red that does not say what to do about it
|
|
284
|
+
* costs a reader the same time twice.
|
|
285
|
+
*/
|
|
286
|
+
export function grade(rules, corpus) {
|
|
287
|
+
const { matchable, exempt } = partition(rules);
|
|
288
|
+
const byId = new Map(matchable.map((r) => [r.id, r]));
|
|
289
|
+
const covered = new Set();
|
|
290
|
+
const findings = [];
|
|
291
|
+
const byPath = new Map();
|
|
292
|
+
let cases = 0;
|
|
293
|
+
|
|
294
|
+
for (const { where, doc } of corpus) {
|
|
295
|
+
const rule = byId.get(doc.rule);
|
|
296
|
+
if (!rule) {
|
|
297
|
+
const isExempt = exempt.some((e) => e.id === doc.rule);
|
|
298
|
+
findings.push({
|
|
299
|
+
where,
|
|
300
|
+
what: isExempt
|
|
301
|
+
? `attacks \`${doc.rule}\`, which declares no matchable action — it has no tool-level surface, so there is nothing to attack. Delete the file, or give the rule an action a matcher can read`
|
|
302
|
+
: `attacks \`${doc.rule}\`, which the yielded policy does not declare. Either the rule was renamed and this file was not, or the fixture outlived its gate`,
|
|
303
|
+
});
|
|
304
|
+
continue;
|
|
305
|
+
}
|
|
306
|
+
covered.add(rule.id);
|
|
307
|
+
for (const c of doc.cases) {
|
|
308
|
+
cases += 1;
|
|
309
|
+
// The declared path is checked against the DERIVED one before the case is graded. A case
|
|
310
|
+
// that mislabels its own branch is a case a reader will draw the wrong conclusion from,
|
|
311
|
+
// and the label is the half no green would otherwise test.
|
|
312
|
+
const derived = matcherPath(rule.kind, c.tool);
|
|
313
|
+
byPath.set(derived, (byPath.get(derived) ?? 0) + 1);
|
|
314
|
+
if (c.path !== derived) {
|
|
315
|
+
findings.push({
|
|
316
|
+
where: `${where} → ${c.id}`,
|
|
317
|
+
what:
|
|
318
|
+
`declares path \`${c.path}\` and \`matchesRule\` takes \`${derived}\` for a ` +
|
|
319
|
+
`${rule.kind}: rule reached through ${c.tool}. The path is derived from the action kind and the ` +
|
|
320
|
+
`tool, so this is a mislabel rather than a disagreement — correct the case`,
|
|
321
|
+
});
|
|
322
|
+
continue;
|
|
323
|
+
}
|
|
324
|
+
// The one call that matters, and it is the compiler's own. Never re-implemented here.
|
|
325
|
+
const actual = matchesRule(rule, c.tool, c.input);
|
|
326
|
+
if (actual === c.expect) continue;
|
|
327
|
+
findings.push({
|
|
328
|
+
where: `${where} → ${c.id}`,
|
|
329
|
+
what:
|
|
330
|
+
c.class === "holds"
|
|
331
|
+
? `REGRESSION: \`${rule.id}\` used to answer ${c.expect} for this and now answers ${actual}. ${c.why}`
|
|
332
|
+
: `the documented hole \`${c.hole}\` has MOVED: this case expects ${c.expect} and the matcher answers ${actual}. ` +
|
|
333
|
+
`If the hole closed, that is good news and the record must say so — update ${c.hole}, then change this case to \`holds\`. ` +
|
|
334
|
+
`A hole list that still lists a closed hole is as wrong as one that hides an open one`,
|
|
335
|
+
});
|
|
336
|
+
}
|
|
337
|
+
}
|
|
338
|
+
|
|
339
|
+
for (const rule of matchable) {
|
|
340
|
+
if (covered.has(rule.id)) continue;
|
|
341
|
+
findings.push({
|
|
342
|
+
where: `${CORPUS_DIR}/`,
|
|
343
|
+
what:
|
|
344
|
+
`\`${rule.id}\` (${rule.tier}, a \`${rule.kind}:\` rule) compiles to a matcher and no fixture attacks it. ` +
|
|
345
|
+
`Coverage is measured, not named — add ${CORPUS_DIR}/${rule.id}.json with at least one case`,
|
|
346
|
+
});
|
|
347
|
+
}
|
|
348
|
+
|
|
349
|
+
return { findings, cases, matchable, exempt, covered, byPath };
|
|
350
|
+
}
|
|
351
|
+
|
|
352
|
+
function usage() {
|
|
353
|
+
return [
|
|
354
|
+
"usage: node cli/goldens.mjs [--check] [--workspace <dir>] [--pack-root <dir>]",
|
|
355
|
+
"",
|
|
356
|
+
" Grades evals/goldens/gates/ against the gate policy the workspace YIELDS.",
|
|
357
|
+
" --check is the default and the only mode; the flag is accepted so the recipe reads",
|
|
358
|
+
" like its siblings.",
|
|
359
|
+
"",
|
|
360
|
+
" Exit 0 green · 1 red · 2 could not run.",
|
|
361
|
+
].join("\n");
|
|
362
|
+
}
|
|
363
|
+
|
|
364
|
+
export function run(argv = [], { stdout = process.stdout, stderr = process.stderr, cwd = process.cwd() } = {}) {
|
|
365
|
+
const say = (line = "") => stdout.write(`${line}\n`);
|
|
366
|
+
// Before every other argument decision, so asking for help cannot be outranked by a complaint
|
|
367
|
+
// about the rest of the command line — the contract `./portulan.mjs` states.
|
|
368
|
+
if (argv.includes("--help") || argv.includes("-h")) {
|
|
369
|
+
say(usage());
|
|
370
|
+
return 0;
|
|
371
|
+
}
|
|
372
|
+
let named = cwd;
|
|
373
|
+
let packRoots = null;
|
|
374
|
+
try {
|
|
375
|
+
for (let i = 0; i < argv.length; i += 1) {
|
|
376
|
+
if (argv[i] === "--check") continue;
|
|
377
|
+
if (argv[i] === "--workspace") {
|
|
378
|
+
named = argv[i + 1];
|
|
379
|
+
i += 1;
|
|
380
|
+
if (named === undefined) throw new CouldNotRun("--workspace needs a directory");
|
|
381
|
+
} else if (argv[i] === "--pack-root") {
|
|
382
|
+
const root = argv[i + 1];
|
|
383
|
+
i += 1;
|
|
384
|
+
// A single leading `-` is refused too, the way `compile` and `doctor` refuse it:
|
|
385
|
+
// `--pack-root -h` would otherwise be consumed as a path and fail later as an
|
|
386
|
+
// unreadable one. A directory genuinely named `-name` is `./-name`.
|
|
387
|
+
if (root === undefined || root.startsWith("-")) throw new CouldNotRun("--pack-root needs a directory");
|
|
388
|
+
let stat = null;
|
|
389
|
+
try {
|
|
390
|
+
stat = fs.statSync(root);
|
|
391
|
+
} catch (cause) {
|
|
392
|
+
throw new CouldNotRun(`--pack-root ${root} cannot be read — ${cause.code ?? cause.message}`);
|
|
393
|
+
}
|
|
394
|
+
if (!stat.isDirectory()) throw new CouldNotRun(`--pack-root ${root} is not a directory`);
|
|
395
|
+
(packRoots ??= []).push(path.resolve(root));
|
|
396
|
+
} else throw new CouldNotRun(`unknown argument ${JSON.stringify(argv[i])}`);
|
|
397
|
+
}
|
|
398
|
+
|
|
399
|
+
const { workspaceRoot, rules, unresolved } = yieldedRules(named, { packRoots });
|
|
400
|
+
const corpus = readCorpus(workspaceRoot);
|
|
401
|
+
const { findings, cases, matchable, exempt, byPath } = grade(rules, corpus);
|
|
402
|
+
|
|
403
|
+
// An unresolved pack is a declaration the workspace believes it composed, so its fragments are
|
|
404
|
+
// absent from the denominator and this run's coverage claim is narrower than it looks. Printed
|
|
405
|
+
// before the verdict, for the reason `compile` prints it there.
|
|
406
|
+
for (const u of unresolved) say(`pack ${u.name} UNRESOLVED — ${u.why}; its gate fragments are not in this census`);
|
|
407
|
+
|
|
408
|
+
say(`goldens: ${cases} case(s) over ${matchable.length} matchable rule(s) in ${corpus.length} fixture file(s)`);
|
|
409
|
+
// Per matcher branch, and printed even at zero. A corpus can carry 200 cases and exercise one
|
|
410
|
+
// branch of three; the total says nothing about that and this line says it plainly.
|
|
411
|
+
say(`goldens: by matcher path — ${PATHS.map((p) => `${p} ${byPath.get(p) ?? 0}`).join(" · ")}`);
|
|
412
|
+
// The exemption, named on every run — see `partition`.
|
|
413
|
+
if (exempt.length) {
|
|
414
|
+
say(`goldens: ${exempt.length} rule(s) declare no matchable action and are exempt from fixtures:`);
|
|
415
|
+
for (const e of exempt) say(` ${e.id} (${e.tier}, ${e.why})`);
|
|
416
|
+
}
|
|
417
|
+
|
|
418
|
+
if (findings.length) {
|
|
419
|
+
for (const f of findings) stderr.write(`goldens: ${f.where}\n ${f.what}\n`);
|
|
420
|
+
stderr.write(`RED — ${findings.length} finding(s) in the gate corpus\n`);
|
|
421
|
+
return 1;
|
|
422
|
+
}
|
|
423
|
+
say("GREEN — every matchable gate carries fixtures, and every case answers as recorded");
|
|
424
|
+
// Said out loud rather than implied by the green, because the limit is the half a reader
|
|
425
|
+
// otherwise supplies wrongly. See the header.
|
|
426
|
+
say("goldens: this is a PRESENCE floor — whether a corpus is a real attack is a reviewer's judgement, not this rail's");
|
|
427
|
+
return 0;
|
|
428
|
+
} catch (error) {
|
|
429
|
+
if (error instanceof CouldNotRun || error instanceof CompileError) {
|
|
430
|
+
stderr.write(`goldens: ${error.message}\n`);
|
|
431
|
+
return 2;
|
|
432
|
+
}
|
|
433
|
+
// **An unexpected throw is COULD-NOT-RUN, never a red**, and this arm exists because the first
|
|
434
|
+
// draft rethrew: a `ReferenceError` from a typo crashed the process, node exited 1, and
|
|
435
|
+
// `../.portulan/verify/goldens.sh` faithfully translated that into "RED — verify recipe failed"
|
|
436
|
+
// about a corpus nothing had finished grading. A crash that reads as a verdict is the shape
|
|
437
|
+
// this repository has now met in three tools — it is why `pack-version.sh` and
|
|
438
|
+
// `eval-bundle.sh` grew `[ -f ]` preconditions. The stack rides along, because a
|
|
439
|
+
// could-not-run nobody can debug is only half an answer.
|
|
440
|
+
stderr.write(`goldens: could not finish grading — ${error?.stack ?? error}\n`);
|
|
441
|
+
return 2;
|
|
442
|
+
}
|
|
443
|
+
}
|
|
444
|
+
|
|
445
|
+
// The entry guard, in the ONE form `./rule-carriers.mjs` designates after this repository got it wrong
|
|
446
|
+
// twice. `file://${argv[1]}` is NOT that form: `import.meta.url` percent-encodes, this working copy
|
|
447
|
+
// lives under a path with spaces, and the comparison fails — so the tool exits 0 having run nothing.
|
|
448
|
+
// **A green that is the tool never starting.** This file shipped that spelling for exactly one
|
|
449
|
+
// measurement: the first run of `node cli/goldens.mjs` printed nothing and exited 0, on a corpus that
|
|
450
|
+
// did not exist yet and against a policy nothing had read. The third time this repository has met it,
|
|
451
|
+
// and the reason the designated form is copied rather than re-derived.
|
|
452
|
+
//
|
|
453
|
+
// The realpath fallback covers the symlink case an npm `bin` produces, in a `try` because a missing
|
|
454
|
+
// path must answer no rather than throw.
|
|
455
|
+
function isMain() {
|
|
456
|
+
const invoked = process.argv[1];
|
|
457
|
+
if (!invoked) return false;
|
|
458
|
+
if (import.meta.url === pathToFileURL(invoked).href) return true;
|
|
459
|
+
try {
|
|
460
|
+
return import.meta.url === pathToFileURL(fs.realpathSync(invoked)).href;
|
|
461
|
+
} catch {
|
|
462
|
+
return false;
|
|
463
|
+
}
|
|
464
|
+
}
|
|
465
|
+
|
|
466
|
+
// `process.exitCode` rather than `process.exit`, which `./control-chars.mjs` settled for this
|
|
467
|
+
// repository: exiting outright can truncate a pipe that has not drained — and a truncated line IS exit
|
|
468
|
+
// 0 with no output, the precise shape of the false green the guard above was fixed for.
|
|
469
|
+
if (isMain()) process.exitCode = run(process.argv.slice(2));
|