done-done-axi 0.5.2 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -3
- package/dist/cli.js +62 -15
- package/dist/proof.js +58 -0
- package/docs/reference.md +28 -5
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -125,11 +125,11 @@ done-done init
|
|
|
125
125
|
```yaml
|
|
126
126
|
journeys:
|
|
127
127
|
smoke:
|
|
128
|
-
command: npm start > $DONE_DONE_EVIDENCE_ROOT/${RUN_ID}.log
|
|
128
|
+
command: npm start > $DONE_DONE_EVIDENCE_ROOT/${RUN_ID}.log && node record-steps.mjs
|
|
129
129
|
core: true
|
|
130
130
|
```
|
|
131
131
|
|
|
132
|
-
The command runs from the repo root with `DONE_DONE_EVIDENCE_ROOT` (a new folder for this run in the evidence root) and `DONE_DONE_RUN_ID` set, and `${RUN_ID}` in it is replaced. A run that writes nothing into its folder fails. A journey that only runs code checks (`npm test`, `yarn run check`, `tsc`, `pytest`, a linter or a build) is code-only: its log says the code checks pass, not that the product works, so `init --journey` refuses it and `check`, `receipt` and `gate` never count it as proof ([the rule](docs/reference.md#real-product-evidence)).
|
|
132
|
+
The command runs from the repo root with `DONE_DONE_EVIDENCE_ROOT` (a new folder for this run in the evidence root) and `DONE_DONE_RUN_ID` set, and `${RUN_ID}` in it is replaced. A run that writes nothing into its folder fails. The command also writes `steps.json` beside its captures: the target version, the state it started from, the steps a user takes and the result a user sees. `check`, `receipt` and `gate` read the claim against that record, so a test log wrapped in a script cannot prove a claim about the product ([the record](docs/reference.md#step-record)). A journey that only runs code checks (`npm test`, `yarn run check`, `tsc`, `pytest`, a linter or a build) is code-only: its log says the code checks pass, not that the product works, so `init --journey` refuses it and `check`, `receipt` and `gate` never count it as proof ([the rule](docs/reference.md#real-product-evidence)).
|
|
133
133
|
|
|
134
134
|
Name your **core journey** with `core: true`: the one thing a user cannot live without, driven end to end on the real target (for a remote terminal app: pair, open a pane, type, see the output). If your app ships something users install and upgrade (an npm CLI, an Android app, a host service), also record an **upgrade journey** with `upgrade: true` that installs your previous release, upgrades over it in place, then drives the core journey: a fresh install alone is not how users get your release. `doctor` fails until both are named ([the rule](docs/reference.md#core-journey-and-release-path)).
|
|
135
135
|
|
|
@@ -138,7 +138,7 @@ Optional: `done-done init --hook <claude|codex|opencode|gemini|pi>` adds a sessi
|
|
|
138
138
|
Record the recipe in the repo so anyone can re-run the proof, then commit it (and `.claude/settings.json`, `.codex/hooks.json`, `.gemini/settings.json`, `.opencode/plugins/done-done.js` or `.pi/extensions/done-done.js` if you added a hook):
|
|
139
139
|
|
|
140
140
|
```sh
|
|
141
|
-
done-done init --journey smoke --core -- 'npm start > $DONE_DONE_EVIDENCE_ROOT/${RUN_ID}.log'
|
|
141
|
+
done-done init --journey smoke --core -- 'npm start > $DONE_DONE_EVIDENCE_ROOT/${RUN_ID}.log && node record-steps.mjs'
|
|
142
142
|
git add .done-done.yaml
|
|
143
143
|
git commit -m "Add done-done"
|
|
144
144
|
```
|
package/dist/cli.js
CHANGED
|
@@ -22,8 +22,9 @@ import { lint } from './lint.js';
|
|
|
22
22
|
import { look } from './look.js';
|
|
23
23
|
import { scanSecrets } from './secrets.js';
|
|
24
24
|
import { mapChange } from './affected.js';
|
|
25
|
-
import { codeOnly, codeOnlyJourneys, configFault, MARK, namedJourney, releaseArtifacts, RULE, runsFault } from './proof.js';
|
|
25
|
+
import { codeOnly, codeOnlyJourneys, configFault, MARK, namedJourney, releaseArtifacts, RULE, runsFault, STEPS_FILE, STEPS_SHAPE, stepRecordFault, stepsFault } from './proof.js';
|
|
26
26
|
const PASS = 0, FAIL = 1, USAGE = 2, INCONCLUSIVE = 3, WAITING = 4;
|
|
27
|
+
const VERSION = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
|
|
27
28
|
// The shape of a journey, so a new user or agent can write one without opening the reference.
|
|
28
29
|
// done-done sets DONE_DONE_EVIDENCE_ROOT (the run's own folder) and replaces ${RUN_ID}, so the example works for any evidence root.
|
|
29
30
|
const journeyExample = 'journeys: { smoke: { command: "<drive the core user journey on the real app> > $DONE_DONE_EVIDENCE_ROOT/${RUN_ID}.log", core: true } }';
|
|
@@ -33,6 +34,8 @@ const JOURNEYS_TEXT = `journeys (.done-done.yaml): the command runs with DONE_DO
|
|
|
33
34
|
real product: a journey whose command only runs tests, typecheck, lint or a build (npm test, yarn run check, tsc, pytest, or a test or guard
|
|
34
35
|
script such as bash tests/x.test.sh or node scripts/floor-guard.mjs) is code-only;
|
|
35
36
|
init --journey refuses it, and check, receipt and gate refuse proof that comes only from code-only runs or a config of only code-only journeys
|
|
37
|
+
step record: the journey also writes steps.json into $DONE_DONE_EVIDENCE_ROOT, shaped ${STEPS_SHAPE};
|
|
38
|
+
check, receipt and gate refuse a claim that no cited run's step record covers (a run recorded before step records only warns)
|
|
36
39
|
core (journeys.<name>.core: true): the core user journey, the one thing a user cannot live without, driven end to end on the real target
|
|
37
40
|
(for a remote terminal: pair, open a pane, type, see the output); doctor requires one, here or as a "(core journey)" entry in the verify skill's feature map
|
|
38
41
|
upgrade (journeys.<name>.upgrade: true): the release path users take: install the previous shipped version, upgrade over it in place, then drive the
|
|
@@ -430,7 +433,26 @@ function openRun(repo, cfg, journey, command, id) {
|
|
|
430
433
|
const runId = id ?? newRunId(h.commit);
|
|
431
434
|
mkdirSync(cfg.evidence.root, { recursive: true });
|
|
432
435
|
mkdirSync(runDir(cfg, runId));
|
|
433
|
-
return { schema: 1, run_id: runId, journey, command, ...h, started_at: new Date().toISOString(), ended_at: null, exit: null, files: [] };
|
|
436
|
+
return { schema: 1, run_id: runId, journey, command, ...h, started_at: new Date().toISOString(), ended_at: null, exit: null, files: [], tool: VERSION };
|
|
437
|
+
}
|
|
438
|
+
// The step record a run's journey (or driving agent) wrote beside its captures; it is a record, not a capture, so it is not a run file.
|
|
439
|
+
function readSteps(cfg, rec) {
|
|
440
|
+
delete rec.steps;
|
|
441
|
+
delete rec.steps_error;
|
|
442
|
+
const file = join(runDir(cfg, rec.run_id), STEPS_FILE);
|
|
443
|
+
if (!existsSync(file))
|
|
444
|
+
return;
|
|
445
|
+
try {
|
|
446
|
+
const steps = JSON.parse(readFileSync(file, 'utf8'));
|
|
447
|
+
const bad = stepRecordFault(steps);
|
|
448
|
+
if (bad)
|
|
449
|
+
rec.steps_error = bad;
|
|
450
|
+
else
|
|
451
|
+
rec.steps = steps;
|
|
452
|
+
}
|
|
453
|
+
catch (e) {
|
|
454
|
+
rec.steps_error = `it is not JSON (${e.message})`;
|
|
455
|
+
}
|
|
434
456
|
}
|
|
435
457
|
// The first line of a run's log that names a failure, trimmed: what a non-zero exit is about. Empty when none does.
|
|
436
458
|
function firstFailure(log) {
|
|
@@ -441,7 +463,9 @@ function firstFailure(log) {
|
|
|
441
463
|
}
|
|
442
464
|
function closeRun(repo, cfg, rec, log) {
|
|
443
465
|
const now = head(repo);
|
|
466
|
+
readSteps(cfg, rec);
|
|
444
467
|
rec.files = captured(cfg, rec.run_id)
|
|
468
|
+
.filter((p) => basename(p) !== STEPS_FILE || dirname(p) !== rec.run_id)
|
|
445
469
|
.sort()
|
|
446
470
|
.map((p) => {
|
|
447
471
|
const b = readFileSync(join(cfg.evidence.root, p));
|
|
@@ -802,7 +826,7 @@ async function runCheck(argv, run) {
|
|
|
802
826
|
continue;
|
|
803
827
|
}
|
|
804
828
|
const rec = (atNow.length ? atNow : claims).reduce((a, b) => (b.started_at > a.started_at ? b : a));
|
|
805
|
-
runs.set(rec.run_id, { run_id: rec.run_id, journey: rec.journey, command: rec.command, ...(rec.feature && { feature: rec.feature }) });
|
|
829
|
+
runs.set(rec.run_id, { run_id: rec.run_id, journey: rec.journey, command: rec.command, ...(rec.feature && { feature: rec.feature }), ...(rec.tool && { tool: rec.tool }), ...(rec.steps && { steps: rec.steps }), ...(rec.steps_error && { steps_error: rec.steps_error }) });
|
|
806
830
|
// Only a capture tied to a run record can reach the judge, so a stray cited file is never sent.
|
|
807
831
|
const c = { name, bytes, sha256: hash, media: v };
|
|
808
832
|
if (v.status === 'ok' && !record && pair)
|
|
@@ -835,6 +859,15 @@ async function runCheck(argv, run) {
|
|
|
835
859
|
const codeRuns = runs.size ? runsFault([...runs.values()]) : null;
|
|
836
860
|
if (codeRuns)
|
|
837
861
|
findings.push({ level: 'fail', why: codeRuns });
|
|
862
|
+
// The user steps behind the captures: a cited real run needs a step record covering the claim. Runs from before step records warn.
|
|
863
|
+
const warnings = [];
|
|
864
|
+
if (claim && runs.size && !codeRuns) {
|
|
865
|
+
const s = stepsFault([...runs.values()], claim.outcome);
|
|
866
|
+
if (s.fault)
|
|
867
|
+
findings.push({ level: 'fail', why: s.fault });
|
|
868
|
+
if (s.warning)
|
|
869
|
+
warnings.push(s.warning);
|
|
870
|
+
}
|
|
838
871
|
// Reuse: fail only when every cited file was already used for a different claim id. The ledger lives in the
|
|
839
872
|
// evidence root next to the run records it guards: one per app (repo), shared by every checkout of it, never in git.
|
|
840
873
|
const ledger = join(cfg.evidence.root, '.done-done', 'ledger');
|
|
@@ -890,12 +923,12 @@ async function runCheck(argv, run) {
|
|
|
890
923
|
if (j.notJudged.length)
|
|
891
924
|
judged.not_judged = j.notJudged;
|
|
892
925
|
judged.note = 'PNG and JPEG screens and text captures (logs, JSON, CLI output) are judged; video and other files are not';
|
|
893
|
-
const rep = report(findings, files, { cfg, repo, claim: claim.outcome, source: values.from, citationBase: values['citation-base'], judge: judged, judgeLevel });
|
|
926
|
+
const rep = report(findings, files, { cfg, repo, claim: claim.outcome, source: values.from, citationBase: values['citation-base'], judge: judged, judgeLevel, warnings });
|
|
894
927
|
if (rep.code === PASS) {
|
|
895
928
|
mkdirSync(dirname(ledger), { recursive: true });
|
|
896
929
|
appendFileSync(ledger, hashes.filter((h) => !used.get(h)?.has(claimId)).map((h) => `${h} ${claimId}\n`).join(''));
|
|
897
930
|
}
|
|
898
|
-
return { code: rep.code, result: rep.result, judge: judged, judgeBlock, claim: claim?.outcome, skill: claim?.skill, files, cited, runs: [...runs.values()], head: now, findings };
|
|
931
|
+
return { code: rep.code, result: rep.result, judge: judged, judgeBlock, claim: claim?.outcome, skill: claim?.skill, files, cited, runs: [...runs.values()], head: now, findings, warnings };
|
|
899
932
|
}
|
|
900
933
|
async function cmdCheck(argv) {
|
|
901
934
|
const r = await runCheck(argv);
|
|
@@ -933,8 +966,13 @@ function receiptFault(rec, short) {
|
|
|
933
966
|
if (badJudge)
|
|
934
967
|
return `receipt for ${short} has a malformed judge block`;
|
|
935
968
|
const code = runsFault(rec.runs);
|
|
936
|
-
|
|
969
|
+
if (code)
|
|
970
|
+
return `receipt for ${short} is no real-product proof: ${code}`;
|
|
971
|
+
const steps = stepsFault(rec.runs, rec.claim);
|
|
972
|
+
return steps.fault ? `receipt for ${short} has no step record covering its claim: ${steps.fault}` : null;
|
|
937
973
|
}
|
|
974
|
+
// The warning a well-formed receipt still carries: its runs were recorded before step records.
|
|
975
|
+
const receiptWarning = (rec) => stepsFault(rec.runs, rec.claim).warning;
|
|
938
976
|
// Verify a recorded receipt for `headArg`, which may be a full sha or a ref (HEAD, a branch or a tag).
|
|
939
977
|
// Read-only, so a missing or changed record is a failure, not a usage error.
|
|
940
978
|
function verifyReceipt(repo, cfg, headArg) {
|
|
@@ -986,7 +1024,7 @@ function verifyReceipt(repo, cfg, headArg) {
|
|
|
986
1024
|
}
|
|
987
1025
|
if (bad.length)
|
|
988
1026
|
return fail(`cited evidence no longer matches the receipt: ${bad.join('; ')}`);
|
|
989
|
-
return { code: PASS, result: { command: 'receipt', result: 'pass', head: commit, tree: rec.tree, claim: rec.claim, ...(rec.skill && { skill: rec.skill }), files: rec.files.length, at: rec.at, schema: rec.schema, judge: j, help: ['This receipt is valid for this head.'] } };
|
|
1027
|
+
return { code: PASS, result: { command: 'receipt', result: 'pass', head: commit, tree: rec.tree, claim: rec.claim, ...(rec.skill && { skill: rec.skill }), files: rec.files.length, at: rec.at, schema: rec.schema, judge: j, ...(receiptWarning(rec) && { warnings: [receiptWarning(rec)] }), help: ['This receipt is valid for this head.'] } };
|
|
990
1028
|
}
|
|
991
1029
|
async function cmdReceipt(argv) {
|
|
992
1030
|
const vi = argv.indexOf('--verify');
|
|
@@ -1024,7 +1062,7 @@ function recordReceipt(repo, cfg, r) {
|
|
|
1024
1062
|
throw new UsageError('receipt: the check cited no evidence files, so no receipt was written; a receipt must bind at least one cited evidence file', 'Cite real proof: done-done receipt --outcome "<claim>" <files...>');
|
|
1025
1063
|
const rec = { schema: 2, repo: cfg.app, head: r.head.commit, tree: r.head.tree, claim: r.claim ?? '', ...(r.skill && { skill: r.skill }), result: 'pass', at: new Date().toISOString(), files: r.cited, runs: r.runs, judge: r.judgeBlock };
|
|
1026
1064
|
const path = writeReceipt(cfg, rec);
|
|
1027
|
-
const result = { command: 'receipt', result: 'pass', head: r.head.commit, tree: r.head.tree, claim: rec.claim, ...(rec.skill && { skill: rec.skill }), files: rec.files.length, receipt: shown(repo, path) };
|
|
1065
|
+
const result = { command: 'receipt', result: 'pass', head: r.head.commit, tree: r.head.tree, claim: rec.claim, ...(rec.skill && { skill: rec.skill }), files: rec.files.length, receipt: shown(repo, path), ...(r.warnings.length && { warnings: r.warnings }) };
|
|
1028
1066
|
return { code: PASS, result: { ...result, ...(r.judge && { judge: r.judge }), help: [`Verify this receipt (merge gate): done-done receipt --verify ${r.head.commit}`] } };
|
|
1029
1067
|
}
|
|
1030
1068
|
// Why evidence from `from` no longer proves `to`, or '' if it still does.
|
|
@@ -1070,6 +1108,8 @@ function checkHelp(code, findings, files, { cfg, repo, claim, source, citationBa
|
|
|
1070
1108
|
? `${absent} named file${absent > 1 ? 's are' : ' is'} not in this checkout. If the evidence is in another folder, check against it: ${form}`
|
|
1071
1109
|
: `${absent} named file${absent > 1 ? 's are' : ' is'} not under --citation-base ${arg(citationBase)} either: check the paths in the Checks block`);
|
|
1072
1110
|
}
|
|
1111
|
+
if (findings.some((f) => f.why.includes('step record')))
|
|
1112
|
+
help.push(`Have the journey write ${STEPS_FILE} into $DONE_DONE_EVIDENCE_ROOT, shaped ${STEPS_SHAPE}, naming the steps and the result the claim is about; then re-run it (${captureHint(cfg)}) and cite the new files`);
|
|
1073
1113
|
// Proof refused as code-only needs a journey that drives the product, not a re-capture of the same run.
|
|
1074
1114
|
if (findings.some((f) => f.why.endsWith(RULE)))
|
|
1075
1115
|
help.push(`Capture the real product: ${captureHint(cfg).startsWith('done-done init') ? captureHint(cfg) : `${captureHint(cfg)}, or ${RECORD_REAL}`}, then cite its files`);
|
|
@@ -1101,6 +1141,8 @@ function report(findings, files, ctx) {
|
|
|
1101
1141
|
result.findings = rows.items;
|
|
1102
1142
|
if (rows.omitted)
|
|
1103
1143
|
result.findings_omitted = rows.omitted;
|
|
1144
|
+
if (ctx.warnings?.length)
|
|
1145
|
+
result.warnings = ctx.warnings;
|
|
1104
1146
|
result.help = checkHelp(code, findings, files, ctx);
|
|
1105
1147
|
return { code, result };
|
|
1106
1148
|
}
|
|
@@ -1330,11 +1372,12 @@ ${inst}${cells}${dom}${before}
|
|
|
1330
1372
|
2. ${cfg.instance ? 'Drive that running instance' : 'Launch the app as the skill says, then drive'} that feature through the real user path, step by step as its recipe says.
|
|
1331
1373
|
Never add steps or data to make the outcome look true: capture what that path shows, even when it contradicts the outcome.
|
|
1332
1374
|
3. Save PNG screenshots of what a user sees once the path is done (the resulting state the outcome is about) into ${evidence} (also in $DONE_DONE_EVIDENCE_ROOT); when the outcome shows in text (a CLI's output, a log or JSON), save that text there as a .txt, .log or .json file instead.
|
|
1333
|
-
The judge checks every capture there against the outcome, so a capture of an earlier step (such as an empty or half-filled form, or text from before the outcome) fails the drive: keep those in ${scratch}. Save nothing else there, and never edit a capture.
|
|
1334
|
-
4.
|
|
1375
|
+
The judge checks every capture there against the outcome, so a capture of an earlier step (such as an empty or half-filled form, or text from before the outcome) fails the drive: keep those in ${scratch}. Save nothing else there but the step record of step 4, and never edit a capture.
|
|
1376
|
+
4. Write ${STEPS_FILE} into ${evidence} (also in $DONE_DONE_EVIDENCE_ROOT), the record of what you did, as JSON shaped ${STEPS_SHAPE}: the app version you drove (its package or build version, else the commit), the state it started from, each step a user takes in your own words, and what a user can see at the end. Write what you did, never what the outcome needs.
|
|
1377
|
+
5. Stop everything you started.
|
|
1335
1378
|
|
|
1336
1379
|
Do not change any file in the repository. Do not run done-done: it refuses inside a drive, and done-done answers the run after you stop. Put logs and other scratch files in ${scratch} (also in $TMPDIR), never in /tmp.
|
|
1337
|
-
Keep working until all
|
|
1380
|
+
Keep working until all five steps are done: your next message without a tool call ends the drive and must be the answer.
|
|
1338
1381
|
If you cannot launch the app or reach the feature, say why in blocker and save no capture.
|
|
1339
1382
|
Answer with feature (the feature file you drove, such as features/<name>.md), captures (the file names you saved) and blocker ("" when you drove it),
|
|
1340
1383
|
for example {"feature":"features/<name>.md","captures":["<name>.png"],"blocker":""}.`;
|
|
@@ -3657,7 +3700,7 @@ async function cmdLook(argv) {
|
|
|
3657
3700
|
// The folder of a branch where a receipt copied from the evidence root is committed, for `gate`.
|
|
3658
3701
|
const GATE_DIR = '.done-done-receipts';
|
|
3659
3702
|
// The reason no committed receipt vouches for `tip`, or null when one does. Reads git objects only: no config, no judge, no evidence files.
|
|
3660
|
-
function gateFault(repo, tip, dir, pushed) {
|
|
3703
|
+
function gateFault(repo, tip, dir, pushed, warnings) {
|
|
3661
3704
|
const short = tip.slice(0, 12);
|
|
3662
3705
|
const names = git(repo, '-c', 'core.quotepath=off', 'ls-tree', '-r', '--name-only', tip, '--', dir).split('\n').filter((n) => n.endsWith('.json'));
|
|
3663
3706
|
if (!names.length)
|
|
@@ -3689,6 +3732,9 @@ function gateFault(repo, tip, dir, pushed) {
|
|
|
3689
3732
|
const at = Date.parse(rec.at);
|
|
3690
3733
|
if (!(Math.floor(at / 1000) <= Math.floor(pushed / 1000)))
|
|
3691
3734
|
return `receipt time ${rec.at} is not before the push (${new Date(pushed).toISOString()})`;
|
|
3735
|
+
const warn = receiptWarning(rec);
|
|
3736
|
+
if (warn)
|
|
3737
|
+
warnings.push(warn);
|
|
3692
3738
|
return null;
|
|
3693
3739
|
})();
|
|
3694
3740
|
if (!why)
|
|
@@ -3713,8 +3759,9 @@ function cmdGate(argv) {
|
|
|
3713
3759
|
const pushed = values.pushed === undefined ? Number(git(repo, 'log', '-1', '--format=%ct', tip)) * 1000 : Date.parse(values.pushed);
|
|
3714
3760
|
if (Number.isNaN(pushed))
|
|
3715
3761
|
throw new UsageError(`gate: --pushed ${values.pushed} is not a time`, 'Pass an ISO time such as 2026-10-20T09:05:56Z');
|
|
3716
|
-
const
|
|
3717
|
-
|
|
3762
|
+
const warnings = [];
|
|
3763
|
+
const why = gateFault(repo, tip, (values.dir ?? GATE_DIR).replace(/\/+$/, ''), pushed, warnings);
|
|
3764
|
+
console.log(why === null ? ['PASS', ...warnings.map((w) => `WARN: ${w}`)].join('\n') : `FAIL: ${why}`);
|
|
3718
3765
|
return why === null ? PASS : FAIL;
|
|
3719
3766
|
}
|
|
3720
3767
|
function cmdLint(argv) {
|
|
@@ -4009,7 +4056,7 @@ async function main(argv) {
|
|
|
4009
4056
|
return PASS;
|
|
4010
4057
|
}
|
|
4011
4058
|
if (cmd === '--version') {
|
|
4012
|
-
console.log(
|
|
4059
|
+
console.log(VERSION);
|
|
4013
4060
|
return PASS;
|
|
4014
4061
|
}
|
|
4015
4062
|
if (cmd === '--help' || cmd === '-h' || cmd === 'help') {
|
package/dist/proof.js
CHANGED
|
@@ -102,6 +102,24 @@ export function configFault(journeys) {
|
|
|
102
102
|
const code = codeOnlyJourneys(journeys);
|
|
103
103
|
return names.length && code.length === names.length ? `every journey in .done-done.yaml is code-only (${code.join(', ')}): ${RULE}` : null;
|
|
104
104
|
}
|
|
105
|
+
export const STEPS_FILE = 'steps.json';
|
|
106
|
+
export const STEPS_SHAPE = '{"target": {"version": "<version of the app driven>"}, "start": "<the state it started from>", "steps": ["<a step a user takes>", ...], "result": "<what a user can see at the end>"}';
|
|
107
|
+
// Why parsed JSON is not a step record, or null.
|
|
108
|
+
export function stepRecordFault(s) {
|
|
109
|
+
const r = s;
|
|
110
|
+
const text = (v) => typeof v === 'string' && v.trim() !== '';
|
|
111
|
+
if (!r || typeof r !== 'object' || Array.isArray(r))
|
|
112
|
+
return `it is not a JSON object shaped ${STEPS_SHAPE}`;
|
|
113
|
+
if (!text(r.target?.version))
|
|
114
|
+
return 'target.version is missing';
|
|
115
|
+
if (!text(r.start))
|
|
116
|
+
return 'start is missing';
|
|
117
|
+
if (!Array.isArray(r.steps) || !r.steps.length || !r.steps.every(text))
|
|
118
|
+
return 'steps must be a non-empty list of strings';
|
|
119
|
+
if (!text(r.result))
|
|
120
|
+
return 'result is missing';
|
|
121
|
+
return null;
|
|
122
|
+
}
|
|
105
123
|
// Why these capture runs are no real-product evidence (none recorded, or every one code-only), or null.
|
|
106
124
|
export function runsFault(runs) {
|
|
107
125
|
if (!Array.isArray(runs) || !runs.length)
|
|
@@ -112,6 +130,46 @@ export function runsFault(runs) {
|
|
|
112
130
|
return null;
|
|
113
131
|
return `every cited capture comes from a code-only run (${[...new Set(runs.map((r) => r.journey ?? r.command))].join(', ')}): ${RULE}`;
|
|
114
132
|
}
|
|
133
|
+
// Words that carry no claim: the claim's own scaffolding, not what a user does or sees.
|
|
134
|
+
const GENERIC = new Set('that this with from into when then than have will does your their they them user users able now works work working shows show shown once after before also each every what which while where there here only just more most some such same been being were should would could done proof proves prove outcome correct correctly properly real product the and for are can its'.split(' '));
|
|
135
|
+
const words = (t) => t.toLowerCase().match(/[a-z0-9]{3,}/g) ?? [];
|
|
136
|
+
// A word of the claim is in the record when the record has a word that starts like it (attach, attaches, attached).
|
|
137
|
+
const same = (a, b) => a.startsWith(b.slice(0, Math.min(5, b.length))) || b.startsWith(a.slice(0, Math.min(5, a.length)));
|
|
138
|
+
// The words of the claim that the step record never mentions, or [] when it covers the claim: at least half of the claim's
|
|
139
|
+
// words are in the record, and one of them in its result. A pure word match, no model: a test log's record ("ran the unit
|
|
140
|
+
// suite") cannot cover a claim about a pane, and a CLI transcript whose record names the command and its output covers a CLI claim.
|
|
141
|
+
export function uncovered(outcome, s) {
|
|
142
|
+
const terms = [...new Set(words(outcome).filter((w) => w.length >= 4 && !GENERIC.has(w)))];
|
|
143
|
+
if (!terms.length)
|
|
144
|
+
return [];
|
|
145
|
+
const all = words([s.start, ...s.steps, s.result].join(' '));
|
|
146
|
+
const result = words(s.result);
|
|
147
|
+
const hit = terms.filter((t) => all.some((w) => same(t, w)));
|
|
148
|
+
return hit.length * 2 >= terms.length && hit.some((t) => result.some((w) => same(t, w))) ? [] : terms.filter((t) => !hit.includes(t));
|
|
149
|
+
}
|
|
150
|
+
// Whether the runs behind a claim carry a step record covering it. A fault: no real run (one that is not code-only) has one.
|
|
151
|
+
// A warning: a real run was recorded before step records existed, so it keeps working but nothing ties its captures to user steps.
|
|
152
|
+
export function stepsFault(runs, outcome) {
|
|
153
|
+
const real = runs.filter((r) => !codeOnly(r.command));
|
|
154
|
+
const why = (r) => {
|
|
155
|
+
if (r.steps_error)
|
|
156
|
+
return `run ${r.run_id}'s step record ${STEPS_FILE} is invalid: ${r.steps_error}`;
|
|
157
|
+
if (!r.steps)
|
|
158
|
+
return `run ${r.run_id} has no step record (${STEPS_FILE} beside its captures)`;
|
|
159
|
+
const bad = stepRecordFault(r.steps);
|
|
160
|
+
if (bad)
|
|
161
|
+
return `run ${r.run_id}'s step record ${STEPS_FILE} is invalid: ${bad}`;
|
|
162
|
+
const miss = uncovered(outcome, r.steps);
|
|
163
|
+
return miss.length ? `run ${r.run_id}'s step record (${r.steps.steps.length} step${r.steps.steps.length > 1 ? 's' : ''}, target ${r.steps.target.version}) never mentions ${miss.join(', ')}` : '';
|
|
164
|
+
};
|
|
165
|
+
const reasons = real.map(why);
|
|
166
|
+
if (!real.length || reasons.some((w) => !w))
|
|
167
|
+
return { fault: null, warning: null };
|
|
168
|
+
const old = real.filter((r) => !r.tool);
|
|
169
|
+
if (old.length)
|
|
170
|
+
return { fault: null, warning: `run ${old.map((r) => r.run_id).join(', ')} was recorded before step records, so nothing shows the user steps behind its captures; re-run the journey so it writes ${STEPS_FILE}` };
|
|
171
|
+
return { fault: `no cited capture comes from a run whose step record covers the claim: ${reasons.join('; ')}`, warning: null };
|
|
172
|
+
}
|
|
115
173
|
export const MARK = { core: '(core journey)', upgrade: '(upgrade journey)' };
|
|
116
174
|
// Where the role is named: the journey that carries it and drives the product, else the feature-map entry with its mark, else null.
|
|
117
175
|
export function namedJourney(role, journeys, featureMap = '') {
|
package/docs/reference.md
CHANGED
|
@@ -97,6 +97,8 @@ journeys:
|
|
|
97
97
|
command: node scripts/capture-review.mjs --out $DONE_DONE_EVIDENCE_ROOT
|
|
98
98
|
```
|
|
99
99
|
|
|
100
|
+
The script saves its captures into that folder and writes `steps.json` beside them: the target version, start state, user steps and end result ([Step record](#step-record)).
|
|
101
|
+
|
|
100
102
|
Then run it:
|
|
101
103
|
|
|
102
104
|
```sh
|
|
@@ -255,9 +257,30 @@ A command is **code-only** when every step in it (split on `&&`, `||`, `;`, `|`
|
|
|
255
257
|
|
|
256
258
|
- `init --journey` refuses a code-only command (exit 2, nothing written), and `init`'s verify skill refuses a code-only launch or feature drive.
|
|
257
259
|
- `check` and `receipt` stop before the judge (exit 2) when every journey in `.done-done.yaml` is code-only, naming them, or when every cited capture comes from a code-only run, naming its journey or command. A test log cited beside real captures is fine: they carry the proof.
|
|
258
|
-
- A receipt records the run behind each cited capture (`runs`), so `receipt --verify` and `gate` apply the same rule with no config: a receipt whose runs are all code-only, or one with no `runs` (written before this rule), fails.
|
|
260
|
+
- A receipt records the run behind each cited capture (`runs`), so `receipt --verify` and `gate` apply the same rule with no config (and the [step record](#step-record) rule after it): a receipt whose runs are all code-only, or one with no `runs` (written before this rule), fails.
|
|
259
261
|
- `doctor`'s `real_product` row is `missing` (and doctor exits 2) when every journey is code-only, and a `warning` naming the code-only journeys beside real ones.
|
|
260
262
|
|
|
263
|
+
### Step record
|
|
264
|
+
|
|
265
|
+
A capture shows a screen; the **step record** says what a user did to reach it. Every journey (and every agent drive) writes `steps.json` into its run folder, `$DONE_DONE_EVIDENCE_ROOT`, beside the captures:
|
|
266
|
+
|
|
267
|
+
```json
|
|
268
|
+
{
|
|
269
|
+
"target": { "version": "0.6.0" },
|
|
270
|
+
"start": "a fresh host with no paired phone",
|
|
271
|
+
"steps": ["pair the phone", "open a pane", "type `echo hi`"],
|
|
272
|
+
"result": "the pane shows hi on its own line"
|
|
273
|
+
}
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
`target.version` is the version of the app driven, `start` the state it began from (for an upgrade journey: the previous version installed), `steps` the steps a user takes, in order, and `result` what a user can see at the end. All four are required; a file that is not JSON or lacks one is recorded as invalid. `run` and `verify` read the file when the run closes and keep it in the run record (`steps`, with `tool`, the done-done version that recorded the run, and `steps_error` when the file is invalid); it is a record, not a capture, so it is not listed in the run's files and the judge never reads it. A journey writes it as it drives, from what it really did, never from the claim.
|
|
277
|
+
|
|
278
|
+
The check reads the claim against the record, with no model: at least half of the claim's words (four letters or more, without scaffolding such as `user`, `can`, `now`, `works`) must appear in the record's `start`, `steps` and `result`, matching word beginnings (`attach`, `attaches`, `attached`), and at least one of them in `result`. This runs after the code-only rule, which stays the first filter, and with it:
|
|
279
|
+
|
|
280
|
+
- `check` and `receipt` stop before the judge (exit 2) when no cited capture comes from a run (not code-only) whose step record covers the claim, naming each run and the words its record never mentions. A log from a wrapper script whose record says `ran the unit suite` cannot satisfy a claim about attaching to a pane; a real CLI transcript whose record names the command and its output still satisfies a claim about that command. A cited run with no step record, or an invalid one, is refused the same way, with the shape to write.
|
|
281
|
+
- A receipt keeps each run's step record (`runs[].steps`), so `receipt --verify` and `gate` apply the same rule to the receipt's claim with no config or evidence: a receipt whose runs have no covering record fails.
|
|
282
|
+
- A run recorded before step records existed (its run record has no `tool`, or a receipt run has none) keeps working: `check`, `receipt`, `receipt --verify` and `gate` still pass, but say so in `warnings` (`gate` prints `WARN: ...` after `PASS`): re-run the journey so it writes `steps.json`.
|
|
283
|
+
|
|
261
284
|
### Core journey and release path
|
|
262
285
|
|
|
263
286
|
A proof that drives some feature of the app does not show the app works when the one flow users need is broken, and a release proven by a fresh install does not show the upgrade users take works. So each app names two journeys, and `init`, `doctor` and the verify skill share one model-free rule for them (`src/proof.ts`).
|
|
@@ -333,7 +356,7 @@ A plain check is never a verdict on its own. A cited file that fails one of thes
|
|
|
333
356
|
|
|
334
357
|
Files that are in no run record give an inconclusive result when another cited capture is one the judge reads: nothing ties them to a commit. So does a file that more than one run record claims when none of those runs is at the current commit (possible only with records from before each run had its own folder). Byte-identical captures from several runs are fine when one of those runs is at the current commit: that run ties the file to the commit. A file whose path is named in a run record but whose bytes no longer match fails as `changed after run <run-id>` (exit 2, before the judge), not inconclusive: the path ties it to that run, so editing it after the run is tampering, not untied evidence.
|
|
335
358
|
|
|
336
|
-
A config whose journeys are all code-only, or cited captures that all come from code-only runs, stop the check the same way, before the judge, naming the journeys ([Real-product evidence](#real-product-evidence)).
|
|
359
|
+
A config whose journeys are all code-only, or cited captures that all come from code-only runs, stop the check the same way, before the judge, naming the journeys ([Real-product evidence](#real-product-evidence)). So does a claim that no cited run's step record covers ([Step record](#step-record)).
|
|
337
360
|
|
|
338
361
|
`evidence: not proven: <cause>` stops the check before its judge is asked, always: exit 2, no result, no receipt. `checks.not_proven` is gone: a config that still sets it is a config error (exit 2) that says to delete the line.
|
|
339
362
|
|
|
@@ -429,7 +452,7 @@ done-done receipt --verify <head> # read-only: check a recorded receipt
|
|
|
429
452
|
|
|
430
453
|
`receipt` takes the same claim inputs as `check` and runs that check. It writes `<root>/.done-done/receipts/<head>.json` only when the check passes on the current HEAD, the tracked tree is clean, and the check cited at least one evidence file; a dirty tracked tree, a failed or inconclusive check, a pass that cited no evidence, or a moved HEAD writes nothing and exits non-zero. The record holds the repo, the head and tree sha, the claim outcome, every cited file with its sha256, the run behind each cited capture (`runs`: run id, journey and command, the command `null` for a run opened by hand or driven by the agent) and a timestamp. Every receipt is schema 2 and holds a `judge` block: the route, model, independence and author, and each judged file's path, sha256 and verdict. A cited file is stored relative to the repo, or as `<evidence root>/<path>` when it is under an evidence root outside the repo, so the receipt still finds it from another checkout or a restored evidence root.
|
|
431
454
|
|
|
432
|
-
`receipt --verify <head>` changes nothing. The argument may be a full commit sha or a ref (`HEAD`, a branch or a tag), resolved to the commit with `git rev-parse --verify <arg>^{commit}` before the receipt lookup; an unresolvable argument fails with the same missing-receipt shape. It succeeds only when the receipt exists, is PASS, the current HEAD is still that head, and every cited file still hashes to the recorded sha256. A receipt is written only after a judge pass, so it reads schema 2 only: any other receipt (one recorded without the judge, or by an older build that let a run past its gate) fails as malformed, and a schema 2 receipt needs a well-formed `judge` block whose every verdict is `pass` for a cited file with the same sha256, which the result shows, and `runs` that are not all code-only ([Real-product evidence](#real-product-evidence)); a receipt with no `runs`, written before that rule, fails: re-run `receipt` for the head. A receipt from `--from` (or a `verify` run that drove the skill) also records the verify skill that ran as `skill` (for example `done-done-verify-<app>`), which `--verify` and so the merge gate show; an older receipt without it still verifies. A missing or malformed receipt and a hash mismatch are failures (exit 1), not usage errors. When the mismatched path is named in a run record, the failure names it as `changed after run <run-id>`.
|
|
455
|
+
`receipt --verify <head>` changes nothing. The argument may be a full commit sha or a ref (`HEAD`, a branch or a tag), resolved to the commit with `git rev-parse --verify <arg>^{commit}` before the receipt lookup; an unresolvable argument fails with the same missing-receipt shape. It succeeds only when the receipt exists, is PASS, the current HEAD is still that head, and every cited file still hashes to the recorded sha256. A receipt is written only after a judge pass, so it reads schema 2 only: any other receipt (one recorded without the judge, or by an older build that let a run past its gate) fails as malformed, and a schema 2 receipt needs a well-formed `judge` block whose every verdict is `pass` for a cited file with the same sha256, which the result shows, and `runs` that are not all code-only ([Real-product evidence](#real-product-evidence)) and carry a step record covering the claim ([Step record](#step-record); a run recorded before step records passes with a `warnings` entry); a receipt with no `runs`, written before that rule, fails: re-run `receipt` for the head. A receipt from `--from` (or a `verify` run that drove the skill) also records the verify skill that ran as `skill` (for example `done-done-verify-<app>`), which `--verify` and so the merge gate show; an older receipt without it still verifies. A missing or malformed receipt and a hash mismatch are failures (exit 1), not usage errors. When the mismatched path is named in a run record, the failure names it as `changed after run <run-id>`.
|
|
433
456
|
|
|
434
457
|
### Verify a change in one run
|
|
435
458
|
|
|
@@ -562,11 +585,11 @@ npx -y done-done-axi gate "$PR_HEAD_SHA"
|
|
|
562
585
|
|
|
563
586
|
The receipt is committed in a follow-up commit, because a receipt cannot sit inside the commit it proves (its hash would change). `gate [<head>] [--dir <folder>] [--pushed <time>]` prints `PASS` or `FAIL: <reason>` and exits 0, 1, or 2 on a usage error. It passes when a receipt in `.done-done-receipts/` (or `--dir`):
|
|
564
587
|
|
|
565
|
-
- is a well-formed pass from a judged run (the same shape `receipt --verify` requires, so its `runs` are not all code-only, see [Real-product evidence](#real-product-evidence)) and is filed as `<its head>.json`;
|
|
588
|
+
- is a well-formed pass from a judged run (the same shape `receipt --verify` requires, so its `runs` are not all code-only, see [Real-product evidence](#real-product-evidence), and carry a step record covering its claim, see [Step record](#step-record)) and is filed as `<its head>.json`;
|
|
566
589
|
- has a head that is `<head>` (default `HEAD`) or an ancestor of it, with every file changed since that head inside the receipts folder, so the proof still covers the code;
|
|
567
590
|
- was made no later than the push (`--pushed`, an ISO time; default the commit time of `<head>`), compared to the second.
|
|
568
591
|
|
|
569
|
-
`gate` trusts the committed JSON: it does not re-hash the evidence, which only `receipt --verify` can do, on the machine that holds it. It says so by name and does not claim more.
|
|
592
|
+
A receipt whose runs were recorded before step records passes with `WARN: ...` lines after `PASS`. `gate` trusts the committed JSON: it does not re-hash the evidence, which only `receipt --verify` can do, on the machine that holds it. It says so by name and does not claim more.
|
|
570
593
|
|
|
571
594
|
### Where proof lives
|
|
572
595
|
|
package/package.json
CHANGED