done-done-axi 0.6.4 → 0.6.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +11 -13
- package/dist/cli.js +147 -47
- package/dist/config.js +36 -2
- package/dist/dom.js +83 -9
- package/dist/judge.js +62 -3
- package/dist/look.js +23 -5
- package/docs/examples/statechart-pilot/pane-attach-race.md +1 -1
- package/docs/index.html +8 -6
- package/docs/reference.md +58 -16
- package/package.json +1 -1
- package/skills/done-done/SKILL.md +5 -1
package/README.md
CHANGED
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
|
|
12
12
|
A command-line check of real screenshots, recordings and output.
|
|
13
13
|
|
|
14
|
-
For developers using coding agents. Tests can pass while the app is still broken.
|
|
14
|
+
For developers using coding agents, alone or in a [Firstmate](#use-with-firstmate-or-another-agent-fleet) fleet. Tests can pass while the app is still broken.
|
|
15
15
|
|
|
16
16
|
<img src="docs/media/demo.png" width="320" alt="Real sample app: Release checklist shows Capture the real app checked and Done. The actual claude-haiku-5-5 judge passes that captured screenshot at sample commit 08d635e46a98.">
|
|
17
17
|
|
|
@@ -20,9 +20,13 @@ For developers using coding agents. Tests can pass while the app is still broken
|
|
|
20
20
|
```sh
|
|
21
21
|
npm install -g done-done-axi
|
|
22
22
|
cd your-app && done-done init
|
|
23
|
-
done-done
|
|
23
|
+
done-done verify --outcome "<what works now>"
|
|
24
24
|
```
|
|
25
25
|
|
|
26
|
+
The last line is your first result: your agent drives the real app, a judge checks each capture, and you get a pass with a receipt, or the judge's reason it did not pass.
|
|
27
|
+
|
|
28
|
+
**With or without Firstmate.** On its own, you or your agent run these three lines. In a [Firstmate](https://github.com/kunchenguid/firstmate) fleet, two files make every worker prove its task, plus an optional merge gate that checks that proof ([how](#use-with-firstmate-or-another-agent-fleet)).
|
|
29
|
+
|
|
26
30
|
[Website](https://umeranjum17.github.io/done-done-axi/) · [Setup](#install-in-3-steps) · [Agent skill](#use-it-from-your-agent) · [Reference](docs/reference.md) · [npm](https://www.npmjs.com/package/done-done-axi)
|
|
27
31
|
|
|
28
32
|
## Install in 3 steps
|
|
@@ -30,16 +34,10 @@ done-done doctor
|
|
|
30
34
|
You need **[Node 22.18+](https://nodejs.org) and git**, plus your own signed-in coding agent or API key for the judge. Early preview; see [what's next](#whats-next).
|
|
31
35
|
|
|
32
36
|
1. **Install.** The first line above puts `done-done` (and the long name `done-done-axi`) on your PATH. No `sudo`? Add `--prefix ~/.local` and put `~/.local/bin` on your `PATH`.
|
|
33
|
-
2. **Set up your repo.** `init`
|
|
34
|
-
3. **
|
|
35
|
-
|
|
36
|
-
Review and commit the capture recipe and verify skill, then ask for proof:
|
|
37
|
-
|
|
38
|
-
```sh
|
|
39
|
-
done-done verify --outcome "<what a user can now do>"
|
|
40
|
-
```
|
|
37
|
+
2. **Set up your repo.** `init` asks you, in a terminal, which coding agent drives your app and which judge checks it. done-done never picks either for you. It then writes a minimal `.done-done.yaml` and a verify skill: how to launch and drive your app. Outside a terminal, pass `--agent <agent> --model <model> --judge <agent> --judge-model <model>` instead.
|
|
38
|
+
3. **Get your first result.** Say what works now, in plain words. `verify` has your agent drive the real product the way a user does and save screenshots or output. A fresh judge checks whether each capture shows your claim. A pass names the capture and writes a receipt for this commit. Anything else stops at a gate with the judge's reason, and `done-done respond` answers it. No judge pass, no receipt.
|
|
41
39
|
|
|
42
|
-
|
|
40
|
+
No result? `done-done doctor` changes nothing and names each missing piece, such as a verify skill `init` could not write, with the exact command that fixes it. Commit `.done-done.yaml` and the verify skill so later runs and your teammates use the same recipe. You choose the agents and models; captures go to your chosen judge's provider, on your own plan or key. [Manual CLI](#manual-cli) walks through recording a journey, checking it and saving a receipt.
|
|
43
41
|
|
|
44
42
|
## Why
|
|
45
43
|
|
|
@@ -153,7 +151,7 @@ Another fleet works the same way: add the include text to whatever your workers
|
|
|
153
151
|
|
|
154
152
|
## Learn more
|
|
155
153
|
|
|
156
|
-
- [Vision](VISION.md):
|
|
154
|
+
- [Vision](VISION.md): who done-done is for, the problem, where it is going, what it is not, and the next milestones.
|
|
157
155
|
- [Worked example: a CLI project](docs/example-cli.md): set up, one journey, check and receipt on a command-line tool.
|
|
158
156
|
- [Worked example: create and maintain a verify skill](docs/example-create-maintain.md): init writes the skill, a source change makes it drift, maintain fixes it, then a real browser journey is judged.
|
|
159
157
|
- [Full reference](docs/reference.md): every command and option, the Checks block rules, each file check, and both config files.
|
|
@@ -161,7 +159,7 @@ Another fleet works the same way: add the include text to whatever your workers
|
|
|
161
159
|
|
|
162
160
|
## What's next
|
|
163
161
|
|
|
164
|
-
So far: `init`, `doctor`, `run` (including `--open` and `--close`), `check`, `receipt`, `affected` (the journeys a diff needs), `lint` (a no-model gate on a Checks block), `gate` (a no-model CI check of a committed receipt), `verify` (by journey or by your coding agent, with a gate you answer with `respond`), `maintain`, the capture judge (always on), the `design` visual check, the `look` step, the run tools (`status`, `attach`, `logs`, `rerun`, `abort`, `signoff`, `history`) and the opt-in session hooks for Claude Code, Codex, opencode, Gemini CLI and pi. Coming next:
|
|
162
|
+
So far: `init`, `doctor`, `run` (including `--open` and `--close`), `check`, `receipt`, `affected` (the journeys a diff needs), `lint` (a no-model gate on a Checks block), `gate` (a no-model CI check of a committed receipt), `verify` (by journey or by your coding agent, with a gate you answer with `respond`), `maintain`, the capture judge (always on), the `design` visual check, the `look` step, the run tools (`status`, `attach`, `logs`, `rerun`, `abort`, `signoff`, `history`) and the opt-in session hooks for Claude Code, Codex, opencode, Gemini CLI and pi. Coming next: the [milestones in the vision](VISION.md#next-milestones).
|
|
165
163
|
|
|
166
164
|
## Develop
|
|
167
165
|
|
package/dist/cli.js
CHANGED
|
@@ -48,7 +48,9 @@ const CHECKS_TEXT = `Checks block (--from): the newest "## Checks" heading, then
|
|
|
48
48
|
outcome: <what a user can now do>
|
|
49
49
|
repro: <command that failed before and passes now, or n/a; a fix PR needs one>
|
|
50
50
|
proof: <the command or path that drove the real product; name the repo's done-done-verify-<app> skill, or say "no verify skill">
|
|
51
|
-
evidence: <the evidence file paths, or "not proven: <cause>"
|
|
51
|
+
evidence: <the evidence file paths, or "not proven: <cause>">
|
|
52
|
+
approved_design (.done-done.yaml): a non-empty manifest of {screen, size: <width>x<height>, theme, mock: <repo PNG/JPEG>, capture: <unique filename>};
|
|
53
|
+
every claim needs every listed live screen, paired with its mock and a passing design verdict; missing or unsure matches fail by screen/size/theme`;
|
|
52
54
|
const RUNS_TEXT = `runs: each verify or maintain run lives in $DONE_DONE_HOME/runs/<app>/<run-id> (run.json, events.jsonl, logs/<step>.log,
|
|
53
55
|
agents/<n>-<name>.json for each agent call) and goes on when the command that started it ends;
|
|
54
56
|
verify steps: resolve, drive, check, judge (always; skipped only after a failed check), triage and fix (only with auto_fix.verify above 0), gate, receipt, report;
|
|
@@ -128,6 +130,8 @@ const FLAGS = {
|
|
|
128
130
|
'--run <run-id>': 'the run to show (default: the newest run of this checkout)',
|
|
129
131
|
'--probe': 'ask the configured agent one JSON-schema question and require a valid answer (one call at the agent\'s model: its cheap default where it has one, else its own; set agent_config.<agent>.model to choose)',
|
|
130
132
|
'--mock <screen>=<mock>': 'judge this screen next to its approved mock (repeat per screen; the screen is judged even when not listed)',
|
|
133
|
+
'--mock <file>': 'with look: the approved design of this page (PNG or JPEG); the judge the user set compares the capture with it, as design --mock does, and a defect fails look',
|
|
134
|
+
'--act <selector>': 'with look: click the element this CSS selector names, then require the page to change what a user can read (its text, elements or address) within 5s; repeat to click several in order; look measures and captures the page they leave',
|
|
131
135
|
'--step <name>': 'the step log to show (verify: resolve, drive, check, judge, gate, receipt, report; maintain: locate, hygiene, source-wave, reconcile, triage, fix, report)',
|
|
132
136
|
'--feature <name>': 'with no journey, the author agent drives this feature of the verify skill (features/<name>.md); default: the only feature, else the agent picks the one the outcome is about',
|
|
133
137
|
'--action <approve|skip|fix>': 'at a spend gate, approve: spend and run the journey; skip: end the run with nothing spent. At the judge gate only fix or skip: fix ends it fail, then the product or the verify skill is fixed (done-done maintain checks it against the source); skip ends it fail without the fix',
|
|
@@ -220,12 +224,14 @@ const COMMANDS = {
|
|
|
220
224
|
passed, 1 a screen failed, 3 none failed but one is unsure`],
|
|
221
225
|
},
|
|
222
226
|
look: {
|
|
223
|
-
usage: [['done-done look <url> [--out <file.png>]', 'open a page in headless Chrome at a phone viewport, capture it with its page-check record, and fail on any squashed, clipped or cut-off UI']],
|
|
224
|
-
flags: ['--out <file.png>'],
|
|
225
|
-
more: [`look: one
|
|
227
|
+
usage: [['done-done look <url> [--act <selector>]... [--mock <file>] [--out <file.png>]', 'open a page in headless Chrome at a phone viewport, click each action, capture it with its page-check record, and fail on any squashed, clipped or cut-off UI, an action that changes nothing, or a mismatch with the design']],
|
|
228
|
+
flags: ['--act <selector>', '--mock <file>', '--out <file.png>'],
|
|
229
|
+
more: [`look: one step a journey calls as its command or inside its script; with no --mock, no judge and no model. It measures horizontal
|
|
226
230
|
overflow, squashed text, text clipped by its container and elements cut off by the left, right or top edge, writes the
|
|
227
|
-
screenshot and its .dom.json record, and exits 1 naming each element, kind and text, 0 when there is none.
|
|
228
|
-
|
|
231
|
+
screenshot and its .dom.json record, and exits 1 naming each element, kind and text, 0 when there is none. Each --act is
|
|
232
|
+
clicked and must change the page. With a look block in .done-done.yaml (the approved reference: tokens, max_control) it
|
|
233
|
+
also fails on a control drawn as the browser's own and on one taller than max_control. A hit always fails: never wrap it
|
|
234
|
+
in || true. Exit 3 when the only doubt is an unsure judge. Needs Chrome or Chromium (DONE_DONE_CHROME names the binary)`],
|
|
229
235
|
},
|
|
230
236
|
gate: {
|
|
231
237
|
usage: [['done-done gate [<head>] [--dir <folder>] [--pushed <time>]', 'no-model check for a hosted runner: PASS or FAIL: <reason> for a receipt committed under .done-done-receipts/']],
|
|
@@ -764,6 +770,7 @@ async function runCheck(argv, run) {
|
|
|
764
770
|
const captures = [];
|
|
765
771
|
// The run behind each cited capture, for the real-product rule and the receipt.
|
|
766
772
|
const runs = new Map();
|
|
773
|
+
const warnings = [];
|
|
767
774
|
// base_ref: a capture in <run id>/before/ of a run that drove its base ref pairs with <run id>/<the same name>.
|
|
768
775
|
const pairOf = (inRoot) => {
|
|
769
776
|
const [id, dir, ...rest] = inRoot?.split(sep) ?? [];
|
|
@@ -809,7 +816,7 @@ async function runCheck(argv, run) {
|
|
|
809
816
|
const inRoot = under(cfg.evidence.root, path);
|
|
810
817
|
const pair = pairOf(inRoot);
|
|
811
818
|
if (cfg.dom && !pair && ['png', 'jpeg', 'webp', 'gif', 'mp4', 'webm'].includes(v.type))
|
|
812
|
-
for (const why of domIssues(cfg.dom, path))
|
|
819
|
+
for (const why of domIssues(cfg.dom, path, cfg.look))
|
|
813
820
|
findings.push({ level: 'fail', why: `${name}: ${why}` });
|
|
814
821
|
// A base capture may equal its head capture byte for byte: the pair's judge, not this check, says the screen did not change.
|
|
815
822
|
if (seen.has(hash) && !record && !pair)
|
|
@@ -839,7 +846,7 @@ async function runCheck(argv, run) {
|
|
|
839
846
|
continue;
|
|
840
847
|
}
|
|
841
848
|
const rec = (atNow.length ? atNow : claims).reduce((a, b) => (b.started_at > a.started_at ? b : a));
|
|
842
|
-
runs.set(rec.run_id, { run_id: rec.run_id, journey: rec.journey, command: rec.command, ...(rec.feature && { feature: rec.feature }), ...(rec.tool && { tool: rec.tool }), ...(rec.steps && { steps: rec.steps }), ...(rec.steps_error && { steps_error: rec.steps_error }) });
|
|
849
|
+
runs.set(rec.run_id, { run_id: rec.run_id, commit: rec.commit, journey: rec.journey, command: rec.command, ...(rec.feature && { feature: rec.feature }), ...(rec.tool && { tool: rec.tool }), ...(rec.steps && { steps: rec.steps }), ...(rec.steps_error && { steps_error: rec.steps_error }) });
|
|
843
850
|
// Only a capture tied to a run record can reach the judge, so a stray cited file is never sent.
|
|
844
851
|
const c = { name, bytes, sha256: hash, media: v };
|
|
845
852
|
if (v.status === 'ok' && !record && pair)
|
|
@@ -849,7 +856,11 @@ async function runCheck(argv, run) {
|
|
|
849
856
|
if (rec.base)
|
|
850
857
|
heads.push([c, inRoot, rec]);
|
|
851
858
|
}
|
|
852
|
-
const
|
|
859
|
+
const kept = [];
|
|
860
|
+
const stale = staleness(repo, cfg, rec.commit, now.commit, rec, kept);
|
|
861
|
+
for (const w of kept)
|
|
862
|
+
if (!warnings.includes(w))
|
|
863
|
+
warnings.push(w);
|
|
853
864
|
if (stale)
|
|
854
865
|
findings.push({ level: 'fail', why: `${name}: ${stale} (run ${rec.run_id})` });
|
|
855
866
|
if (rec.dirty && !cfg.evidence.allow_dirty)
|
|
@@ -857,6 +868,19 @@ async function runCheck(argv, run) {
|
|
|
857
868
|
if (rec.exit !== null && rec.exit !== 0)
|
|
858
869
|
findings.push({ level: 'fail', why: `${name}: its run command exited ${rec.exit} (run ${rec.run_id})` });
|
|
859
870
|
}
|
|
871
|
+
// The approved inventory is the contract, not the subset the claim happened to cite. Enforce it on every
|
|
872
|
+
// claim when configured, so rewording a design-done/release outcome cannot bypass missing screens.
|
|
873
|
+
const approved = cfg.approved_design.map(entry => ({ entry, live: captures.filter(c => screen(c) && basename(cite(c.name)) === entry.capture) }));
|
|
874
|
+
const coverageFaults = [];
|
|
875
|
+
for (const { entry: e, live } of approved) {
|
|
876
|
+
const label = `${e.screen} / ${e.size} / ${e.theme}`;
|
|
877
|
+
if (live.length !== 1)
|
|
878
|
+
coverageFaults.push(`${label}: ${live.length ? 'ambiguous live captures' : 'no live capture'} (${e.capture})`);
|
|
879
|
+
else if (`${live[0].media.w}x${live[0].media.h}` !== e.size)
|
|
880
|
+
coverageFaults.push(`${label}: live capture is ${live[0].media.w}x${live[0].media.h}, not the approved size`);
|
|
881
|
+
}
|
|
882
|
+
for (const why of coverageFaults)
|
|
883
|
+
findings.push({ level: 'fail', why: `approved design: ${why}` });
|
|
860
884
|
// A head capture of a base_ref run with no base capture of its name is reported, then judged alone.
|
|
861
885
|
for (const [c, inRoot, rec] of heads)
|
|
862
886
|
if (!(c.before = befores.get(inRoot ?? '')))
|
|
@@ -873,7 +897,6 @@ async function runCheck(argv, run) {
|
|
|
873
897
|
if (codeRuns)
|
|
874
898
|
findings.push({ level: 'fail', why: codeRuns });
|
|
875
899
|
// The user steps behind the captures: a cited real run needs a step record covering the claim. Runs from before step records warn.
|
|
876
|
-
const warnings = [];
|
|
877
900
|
if (claim && runs.size && !codeRuns) {
|
|
878
901
|
const s = stepsFault([...runs.values()], claim.outcome);
|
|
879
902
|
if (s.fault)
|
|
@@ -901,6 +924,10 @@ async function runCheck(argv, run) {
|
|
|
901
924
|
findings.push({ level: 'fail', why: `every evidence file was already used as proof of a different claim (claim id ${[...new Set(hashes.flatMap(others))].join(', ')}), not this one (claim id ${claimId}, from ${idFrom}); if it is the same claim, pass --claim-id with its id` });
|
|
902
925
|
// A failed plain check is no verdict: the judge is not asked, so the check stops with every cause and no result.
|
|
903
926
|
const failed = findings.filter((f) => f.level === 'fail');
|
|
927
|
+
if (coverageFaults.length) {
|
|
928
|
+
const rep = report(findings, files, { cfg, repo, claim: claim?.outcome, source: values.from, citationBase: values['citation-base'], warnings });
|
|
929
|
+
return { ...rep, claim: claim?.outcome, files, cited, runs: [...runs.values()], head: now, findings, warnings };
|
|
930
|
+
}
|
|
904
931
|
if (failed.length || !claim) {
|
|
905
932
|
const ctx = { cfg, repo, claim: claim?.outcome, source: values.from, citationBase: values['citation-base'] };
|
|
906
933
|
throw new UsageError(`check: the judge was not asked, so there is no verdict: ${failed.map((f) => f.why).join('; ')}`, checkHelp(FAIL, findings, files, ctx)[0]);
|
|
@@ -928,15 +955,52 @@ async function runCheck(argv, run) {
|
|
|
928
955
|
const { choice } = j;
|
|
929
956
|
const judged = { ...(choice && { route: choice.route, model: choice.model, independence: choice.independence, author: choice.author.agent ? `${choice.author.agent} (model ${choice.author.model || 'none'})` : 'no agent set' }), judged: rows.items.map(({ sha256, ...r }) => r) };
|
|
930
957
|
if (choice) {
|
|
931
|
-
const
|
|
932
|
-
judgeBlock = { route: choice.route, model: choice.model, independence: choice.independence, author: choice.author, verdicts: j.judged.map((r) => ({ path:
|
|
958
|
+
const byHash = new Map(captures.map((c) => [c.sha256, receiptPath(repo, cfg, cite(c.name))]));
|
|
959
|
+
judgeBlock = { route: choice.route, model: choice.model, independence: choice.independence, author: choice.author, verdicts: j.judged.map((r) => ({ path: byHash.get(r.sha256), sha256: r.sha256, verdict: r.verdict, ...(r.seen && { seen: r.seen }) })) };
|
|
933
960
|
}
|
|
934
961
|
if (rows.omitted)
|
|
935
962
|
judged.judged_omitted = rows.omitted;
|
|
936
963
|
if (j.notJudged.length)
|
|
937
964
|
judged.not_judged = j.notJudged;
|
|
938
965
|
judged.note = 'PNG and JPEG screens and text captures (logs, JSON, CLI output) are judged; video and other files are not';
|
|
966
|
+
let matches = [];
|
|
967
|
+
if (approved.length) {
|
|
968
|
+
const pairs = approved.map(({ entry, live }) => {
|
|
969
|
+
const path = resolve(repo, entry.mock);
|
|
970
|
+
let committed;
|
|
971
|
+
try {
|
|
972
|
+
committed = execFileSync('git', ['show', `${values['trusted-ref'] ?? now.commit}:${relative(repo, path)}`], { cwd: repo, stdio: ['ignore', 'pipe', 'pipe'] });
|
|
973
|
+
}
|
|
974
|
+
catch {
|
|
975
|
+
throw new UsageError(`approved design: ${entry.screen} / ${entry.size} / ${entry.theme}: approved mock is not committed: ${entry.mock}`);
|
|
976
|
+
}
|
|
977
|
+
if (under(repo, realpathSync(path)) === null)
|
|
978
|
+
throw new UsageError(`approved design: mock leaves the repo: ${entry.mock}`);
|
|
979
|
+
const mock = loadScreen(path);
|
|
980
|
+
if (mock.sha256 !== sha256(committed))
|
|
981
|
+
throw new UsageError(`approved design: approved mock differs from the committed reference: ${entry.mock}`);
|
|
982
|
+
return { ...live[0], mock };
|
|
983
|
+
});
|
|
984
|
+
for (const [i, p] of pairs.entries())
|
|
985
|
+
if (`${p.mock.media.w}x${p.mock.media.h}` !== approved[i].entry.size)
|
|
986
|
+
throw new UsageError(`approved design: ${approved[i].entry.screen} / ${approved[i].entry.size} / ${approved[i].entry.theme}: mock has the wrong pixel size`);
|
|
987
|
+
const verdicts = await design(cfg, choice, pairs, run ? join(run.dir, 'agents') : join(home(), 'agents'));
|
|
988
|
+
matches = verdicts.map((v, i) => {
|
|
989
|
+
const { screen, size, theme } = approved[i].entry;
|
|
990
|
+
if (v.verdict !== 'pass')
|
|
991
|
+
findings.push({ level: 'fail', why: `approved design: ${screen} / ${size} / ${theme}: ${v.verdict === 'unsure' ? 'match not proven: ' + v.reason : v.defects.map(d => `${d.kind}: ${d.where}`).join('; ')}`, file: v.path });
|
|
992
|
+
const mock = pairs[i].mock;
|
|
993
|
+
const mockPath = receiptPath(repo, cfg, cite(mock.name));
|
|
994
|
+
if (!cited.some(f => f.path === mockPath))
|
|
995
|
+
cited.push({ path: mockPath, sha256: mock.sha256 });
|
|
996
|
+
return { screen, size, theme, path: receiptPath(repo, cfg, cite(v.path)), sha256: pairs[i].sha256, mock: mockPath, mock_sha256: mock.sha256, verdict: v.verdict };
|
|
997
|
+
});
|
|
998
|
+
if (judgeBlock)
|
|
999
|
+
judgeBlock.approved_design = matches;
|
|
1000
|
+
}
|
|
939
1001
|
const rep = report(findings, files, { cfg, repo, claim: claim.outcome, source: values.from, citationBase: values['citation-base'], judge: judged, judgeLevel, warnings });
|
|
1002
|
+
if (matches.length)
|
|
1003
|
+
rep.result.design = matches;
|
|
940
1004
|
if (rep.code === PASS) {
|
|
941
1005
|
mkdirSync(dirname(ledger), { recursive: true });
|
|
942
1006
|
appendFileSync(ledger, hashes.filter((h) => !used.get(h)?.has(claimId)).map((h) => `${h} ${claimId}\n`).join(''));
|
|
@@ -978,6 +1042,8 @@ function receiptFault(rec, short) {
|
|
|
978
1042
|
|| !j.verdicts.every((v) => v && v.verdict === 'pass' && rec.files.some((f) => f.path === v.path && f.sha256 === v.sha256));
|
|
979
1043
|
if (badJudge)
|
|
980
1044
|
return `receipt for ${short} has a malformed judge block`;
|
|
1045
|
+
if (j.approved_design !== undefined && (!Array.isArray(j.approved_design) || !j.approved_design.length || !j.approved_design.every(v => v && v.verdict === 'pass' && [v.screen, v.size, v.theme].every(s => typeof s === 'string' && s.trim()) && rec.files.some(f => f.path === v.path && f.sha256 === v.sha256) && rec.files.some(f => f.path === v.mock && f.sha256 === v.mock_sha256))))
|
|
1046
|
+
return `receipt for ${short} has a malformed approved-design verdict`;
|
|
981
1047
|
const code = runsFault(rec.runs);
|
|
982
1048
|
if (code)
|
|
983
1049
|
return `receipt for ${short} is no real-product proof: ${code}`;
|
|
@@ -1079,7 +1145,7 @@ function recordReceipt(repo, cfg, r) {
|
|
|
1079
1145
|
return { code: PASS, result: { ...result, ...(r.judge && { judge: r.judge }), help: [`Verify this receipt (merge gate): done-done receipt --verify ${r.head.commit}`] } };
|
|
1080
1146
|
}
|
|
1081
1147
|
// Why evidence from `from` no longer proves `to`, or '' if it still does.
|
|
1082
|
-
function staleness(repo, cfg, from, to) {
|
|
1148
|
+
function staleness(repo, cfg, from, to, run, warnings = []) {
|
|
1083
1149
|
if (from === to)
|
|
1084
1150
|
return '';
|
|
1085
1151
|
let changed;
|
|
@@ -1092,6 +1158,19 @@ function staleness(repo, cfg, from, to) {
|
|
|
1092
1158
|
const outside = changed.filter((p) => !cfg.evidence.ignore_paths.some((g) => matchesGlob(p, g)));
|
|
1093
1159
|
if (changed.length && !outside.length)
|
|
1094
1160
|
return '';
|
|
1161
|
+
// A head change that touches no journey or feature the run covered leaves its proof valid; both commits are reported.
|
|
1162
|
+
const needs = outside.filter((p) => !p.startsWith(`${GATE_DIR}/`));
|
|
1163
|
+
if (run && (run.journey || run.feature) && needs.length) {
|
|
1164
|
+
const m = mapDiff(repo, cfg, needs, git(repo, 'ls-files', '-z').split('\0').filter(Boolean));
|
|
1165
|
+
if (!m.journeys.some((j) => j.name === run.journey) && !m.features.some((f) => f.name === run.feature)) {
|
|
1166
|
+
warnings.push(`run ${run.run_id} is for commit ${from.slice(0, 12)} and HEAD is ${to.slice(0, 12)}: its ${run.journey ? `journey ${run.journey}` : `feature ${run.feature}`} is untouched by the ${needs.length} changed file${needs.length === 1 ? '' : 's'} (done-done affected)`);
|
|
1167
|
+
return '';
|
|
1168
|
+
}
|
|
1169
|
+
}
|
|
1170
|
+
if (run && changed.length && !needs.length) {
|
|
1171
|
+
warnings.push(`run ${run.run_id} is for commit ${from.slice(0, 12)} and HEAD is ${to.slice(0, 12)}: only gate receipts changed`);
|
|
1172
|
+
return '';
|
|
1173
|
+
}
|
|
1095
1174
|
return `evidence is for commit ${from.slice(0, 12)}, HEAD is ${to.slice(0, 12)}${outside.length ? ` (changed: ${outside.slice(0, 3).join(', ')}${outside.length > 3 ? ', ...' : ''})` : ''}`;
|
|
1096
1175
|
}
|
|
1097
1176
|
// Quote an argument only when a shell would split it.
|
|
@@ -2722,6 +2801,7 @@ const SCAFFOLD_SCHEMA = {
|
|
|
2722
2801
|
type: 'object',
|
|
2723
2802
|
properties: {
|
|
2724
2803
|
surface: STR,
|
|
2804
|
+
first_release: { type: 'boolean' },
|
|
2725
2805
|
launch: { type: 'object', properties: { steps: STR, source: STR, safe: { type: 'boolean' }, why: STR }, required: ['steps', 'source', 'safe', 'why'], additionalProperties: false },
|
|
2726
2806
|
ready: STR,
|
|
2727
2807
|
harness: STR,
|
|
@@ -2729,7 +2809,7 @@ const SCAFFOLD_SCHEMA = {
|
|
|
2729
2809
|
isolate: STR,
|
|
2730
2810
|
features: { type: 'array', minItems: 1, items: { type: 'object', properties: { title: STR, reach: STR, drive: STR, source: STR, capture: { type: 'string', enum: ['screenshot', 'text'] }, core: { type: 'boolean' }, upgrade: { type: 'boolean' } }, required: ['title', 'reach', 'drive', 'source', 'capture', 'core', 'upgrade'], additionalProperties: false } },
|
|
2731
2811
|
},
|
|
2732
|
-
required: ['surface', 'launch', 'ready', 'harness', 'observe', 'isolate', 'features'],
|
|
2812
|
+
required: ['surface', 'first_release', 'launch', 'ready', 'harness', 'observe', 'isolate', 'features'],
|
|
2733
2813
|
additionalProperties: false,
|
|
2734
2814
|
};
|
|
2735
2815
|
const NO_EDIT = 'The drive changes no repo file: note findings in the run\'s report; after the drive, run done-done maintain to see what drifted, then fix the skill';
|
|
@@ -2802,8 +2882,9 @@ A proof drives the running product the way a user does: a test suite, typecheck,
|
|
|
2802
2882
|
- observe: the proof to capture: screenshots of the resulting state, terminal transcripts and exit codes, response bodies, files written.
|
|
2803
2883
|
- isolate: how each run keeps its own instance and state, and never drives a session or account the user owns.
|
|
2804
2884
|
- features: up to ${MAX_FEATURES} things a user does with the app, the most important first, each {title, reach: how a user gets there, drive: the concrete steps through the repo's real handles (the command and its arguments, the route, the page label, the journey name), source: the repo file that shows the feature, capture: screenshot for a screen, text for output, core, upgrade}.
|
|
2885
|
+
- first_release: true only when the repo explicitly says no earlier release yet; false when an earlier version shipped or you cannot tell. This records upgrade proof as pending, never as passed. Do not invent a previous version or an upgrade drive when none exists.
|
|
2805
2886
|
- core: true on exactly one feature, listed first: the core user journey, the one thing a user cannot live without, driven end to end on the real target the way a user does it (for a remote terminal app: pair, open a pane, type, see the output), never a part of it or a setting. Every other feature has core false.
|
|
2806
|
-
${ships.length ? `- upgrade: this app ships an installable or upgradable artifact (${ships.slice(0, 3).join('; ')}), so users get each release by upgrading over the version they have. Set upgrade true on one feature whose drive installs the previous shipped version, upgrades over it in place the way a user does (install the new APK over the old one, update the host or CLI package in place), then drives the core journey; a fresh install alone is no release proof. It may be the core feature itself when its drive does all of that.` : '- upgrade: false on every feature: no tracked file shows the app ships an installable or upgradable artifact.'}
|
|
2887
|
+
${ships.length ? `- upgrade: this app ships an installable or upgradable artifact (${ships.slice(0, 3).join('; ')}), so users get each release by upgrading over the version they have. Set upgrade true on one feature whose drive installs the previous shipped version, upgrades over it in place the way a user does (install the new APK over the old one, update the host or CLI package in place), then drives the core journey; a fresh install alone is no release proof. It may be the core feature itself when its drive does all of that. Exception: when first_release is true, upgrade may be false on every feature; the skill will record the upgrade journey as pending until there is a published version to upgrade from.` : '- upgrade: false on every feature: no tracked file shows the app ships an installable or upgradable artifact.'}
|
|
2807
2888
|
Each source is one path exactly as the tracked file list spells it, with no other words (scripts/verify/drive.sh, never a list or a note); launch.source may be empty only when safe is false.
|
|
2808
2889
|
|
|
2809
2890
|
${context}`;
|
|
@@ -2847,7 +2928,7 @@ function scaffoldProblem(a, tracked, shown, ships) {
|
|
|
2847
2928
|
const cores = kept.filter((f) => f.core).length;
|
|
2848
2929
|
if (cores !== 1)
|
|
2849
2930
|
return `names ${cores} core journeys among its first ${MAX_FEATURES} features, and the skill needs exactly one: the one thing a user cannot live without, end to end on the real target`;
|
|
2850
|
-
if (ships.length && !kept.some((f) => f.upgrade))
|
|
2931
|
+
if (ships.length && !kept.some((f) => f.upgrade) && s.first_release !== true)
|
|
2851
2932
|
return `names no upgrade journey, and the app ships an installable artifact (${ships.slice(0, 3).join('; ')}): a fresh install alone is no release proof`;
|
|
2852
2933
|
return '';
|
|
2853
2934
|
}
|
|
@@ -2890,7 +2971,7 @@ ${found.join('\n')}
|
|
|
2890
2971
|
## Core journey
|
|
2891
2972
|
|
|
2892
2973
|
${link(core)} is the core user journey: the one thing a user cannot live without. A proof that the app works drives it end to end on the real target; a proof of another feature alone does not show the app works.
|
|
2893
|
-
${ships.length ? `\n## Release proof\n\nThis app ships an installable artifact (${ships.slice(0, 3).join('; ')}), and users get a release by upgrading over the version they have. ${link(upgrade)} is the release proof: install the previous shipped version, upgrade over it in place, then drive the core journey. A fresh install alone is no release proof
|
|
2974
|
+
${ships.length ? `\n## Release proof\n\nThis app ships an installable artifact (${ships.slice(0, 3).join('; ')}), and users get a release by upgrading over the version they have. ${upgrade ? `${link(upgrade)} is the release proof: install the previous shipped version, upgrade over it in place, then drive the core journey. A fresh install alone is no release proof.` : 'Upgrade journey pending: no earlier release yet. Before the next release, install the first published version, upgrade over it in place, then drive the core journey. This is not upgrade proof; doctor still reports the missing upgrade journey until one is named.'}\n` : ''}
|
|
2894
2975
|
## Evidence
|
|
2895
2976
|
|
|
2896
2977
|
What to capture: ${clause(s.observe)}. Exercise the real user path, not internal setters; done-done's judge checks every capture in the run folder against the outcome, so save only captures of the resulting state there and keep any of earlier steps outside it; verify side effects alongside what is visible. Evidence goes to the run folder \`done-done run\` prints.
|
|
@@ -3812,9 +3893,9 @@ async function cmdDoctor(argv) {
|
|
|
3812
3893
|
emit(result);
|
|
3813
3894
|
return missingRequired.length ? USAGE : PASS;
|
|
3814
3895
|
}
|
|
3815
|
-
// The
|
|
3896
|
+
// The look step: one page measured, with its actions clicked, against the project's look reference and the --mock design.
|
|
3816
3897
|
async function cmdLook(argv) {
|
|
3817
|
-
const { values, positionals } = parseArgs({ args: argv, options: { out: { type: 'string' } }, allowPositionals: true });
|
|
3898
|
+
const { values, positionals } = parseArgs({ args: argv, options: { out: { type: 'string' }, mock: { type: 'string' }, act: { type: 'string', multiple: true } }, allowPositionals: true });
|
|
3818
3899
|
if (positionals.length !== 1)
|
|
3819
3900
|
throw new UsageError('look needs one page URL', 'Re-run with the page to check: done-done look <url>');
|
|
3820
3901
|
// A path to a page on disk opens as its file:// URL.
|
|
@@ -3822,23 +3903,36 @@ async function cmdLook(argv) {
|
|
|
3822
3903
|
const capture = resolve(values.out ?? join(process.env.DONE_DONE_EVIDENCE_ROOT || '.', 'look.png'));
|
|
3823
3904
|
if (!capture.endsWith('.png'))
|
|
3824
3905
|
throw new UsageError(`--out ${values.out} must be a .png file`);
|
|
3906
|
+
const cfg = loadConfig(repoOrCwd());
|
|
3907
|
+
// The mock and the judge are checked before Chrome starts, so a bad one costs no capture.
|
|
3908
|
+
const mock = values.mock === undefined ? undefined : loadScreen(values.mock);
|
|
3909
|
+
const choice = mock && (await preflightJudge(cfg, false));
|
|
3825
3910
|
mkdirSync(dirname(capture), { recursive: true });
|
|
3826
|
-
const record = await look(url, capture, 390, 844);
|
|
3827
|
-
const hits = lookIssues(record);
|
|
3911
|
+
const record = await look(url, capture, 390, 844, values.act ?? []);
|
|
3912
|
+
const hits = lookIssues(record, cfg.look);
|
|
3913
|
+
const [judged] = mock && choice ? await design(cfg, choice, [{ ...loadScreen(capture), mock }], join(home(), 'agents')) : [];
|
|
3914
|
+
for (const d of judged?.defects ?? [])
|
|
3915
|
+
hits.push(`${url} differs from the approved design ${values.mock}: ${d.kind}: ${d.where}`);
|
|
3916
|
+
const code = hits.length ? FAIL : judged?.verdict === 'unsure' ? INCONCLUSIVE : PASS;
|
|
3828
3917
|
const shown = cap(hits);
|
|
3918
|
+
const again = `done-done look ${[positionals[0], ...(values.act ?? []).flatMap((a) => ['--act', a]), ...(values.mock ? ['--mock', values.mock] : []), ...(values.out ? ['--out', values.out] : [])].map(quote).join(' ')}`;
|
|
3829
3919
|
emit({
|
|
3830
3920
|
command: 'look',
|
|
3831
3921
|
// look measures the layout; it gives no verdict on the outcome, which only the judged check does.
|
|
3832
|
-
result:
|
|
3922
|
+
result: code === FAIL ? 'fail' : code === INCONCLUSIVE ? 'inconclusive' : 'clean',
|
|
3833
3923
|
url,
|
|
3834
3924
|
viewport: '390x844',
|
|
3835
3925
|
capture,
|
|
3836
3926
|
record: domRecord(capture),
|
|
3927
|
+
...(cfg.look && { reference: { ...cfg.look, ...(values.mock && { mock: values.mock }) } }),
|
|
3928
|
+
...(judged && { judge: { route: choice.route, model: choice.model, verdict: judged.verdict, ...(judged.reason && { reason: judged.reason }), ...(judged.cached && { cached: true }) } }),
|
|
3837
3929
|
hits: shown.items,
|
|
3838
3930
|
...(shown.omitted && { hits_omitted: shown.omitted }),
|
|
3839
|
-
help:
|
|
3931
|
+
help: code === FAIL ? [`Fix each element or action named above so it fits the 390 px wide screen, the design and its reference, then re-run: ${again}`]
|
|
3932
|
+
: code === INCONCLUSIVE ? [`The judge is unsure the page matches ${values.mock}: look at both yourself, then re-run: ${again}`]
|
|
3933
|
+
: ['No squashed, clipped or cut-off UI on this page; look at the capture to check the rest'],
|
|
3840
3934
|
});
|
|
3841
|
-
return
|
|
3935
|
+
return code;
|
|
3842
3936
|
}
|
|
3843
3937
|
// The folder of a branch where a receipt copied from the evidence root is committed, for `gate`.
|
|
3844
3938
|
const GATE_DIR = '.done-done-receipts';
|
|
@@ -3916,6 +4010,25 @@ function cmdLint(argv) {
|
|
|
3916
4010
|
console.log(!v.ok ? `FAIL: ${v.why}` : v.warn ? `WARN: ${v.warn}` : 'PASS');
|
|
3917
4011
|
return v.ok ? PASS : FAIL;
|
|
3918
4012
|
}
|
|
4013
|
+
// A screen or mock file for the design judge.
|
|
4014
|
+
function loadScreen(name) {
|
|
4015
|
+
let bytes;
|
|
4016
|
+
try {
|
|
4017
|
+
bytes = readFileSync(name);
|
|
4018
|
+
}
|
|
4019
|
+
catch (e) {
|
|
4020
|
+
throw new UsageError(`${name}: cannot read (${e.code ?? e.message})`, 'Pass screen and mock files that exist');
|
|
4021
|
+
}
|
|
4022
|
+
return { name, bytes, sha256: sha256(bytes), media: checkMedia(name, bytes) };
|
|
4023
|
+
}
|
|
4024
|
+
function repoOrCwd() {
|
|
4025
|
+
try {
|
|
4026
|
+
return repoRoot();
|
|
4027
|
+
}
|
|
4028
|
+
catch {
|
|
4029
|
+
return process.cwd();
|
|
4030
|
+
}
|
|
4031
|
+
}
|
|
3919
4032
|
async function cmdDesign(argv) {
|
|
3920
4033
|
const { values, positionals } = parseArgs({ args: argv, options: { mock: { type: 'string', multiple: true } }, allowPositionals: true });
|
|
3921
4034
|
const mocks = new Map();
|
|
@@ -3928,25 +4041,8 @@ async function cmdDesign(argv) {
|
|
|
3928
4041
|
const names = [...new Set([...positionals, ...[...mocks.keys()].filter((p) => !positionals.some((n) => resolve(n) === p))])];
|
|
3929
4042
|
if (!names.length)
|
|
3930
4043
|
throw new UsageError('design needs at least one screen file', 'Re-run with the screens to check: done-done design <screens...>');
|
|
3931
|
-
const
|
|
3932
|
-
|
|
3933
|
-
try {
|
|
3934
|
-
bytes = readFileSync(name);
|
|
3935
|
-
}
|
|
3936
|
-
catch (e) {
|
|
3937
|
-
throw new UsageError(`${name}: cannot read (${e.code ?? e.message})`, 'Pass screen and mock files that exist');
|
|
3938
|
-
}
|
|
3939
|
-
return { name, bytes, sha256: sha256(bytes), media: checkMedia(name, bytes) };
|
|
3940
|
-
};
|
|
3941
|
-
const screens = names.map((n) => ({ ...load(n), ...(mocks.has(resolve(n)) && { mock: load(mocks.get(resolve(n))) }) }));
|
|
3942
|
-
let repo;
|
|
3943
|
-
try {
|
|
3944
|
-
repo = repoRoot();
|
|
3945
|
-
}
|
|
3946
|
-
catch {
|
|
3947
|
-
repo = process.cwd();
|
|
3948
|
-
}
|
|
3949
|
-
const cfg = loadConfig(repo);
|
|
4044
|
+
const screens = names.map((n) => ({ ...loadScreen(n), ...(mocks.has(resolve(n)) && { mock: loadScreen(mocks.get(resolve(n))) }) }));
|
|
4045
|
+
const cfg = loadConfig(repoOrCwd());
|
|
3950
4046
|
const choice = await preflightJudge(cfg, false);
|
|
3951
4047
|
const judged = await design(cfg, choice, screens, join(home(), 'agents'));
|
|
3952
4048
|
const count = (v) => judged.filter((j) => j.verdict === v).length;
|
|
@@ -3992,6 +4088,13 @@ function defaultBranch(repo) {
|
|
|
3992
4088
|
}
|
|
3993
4089
|
}
|
|
3994
4090
|
const isAncestor = (repo, a, b) => spawnSync('git', ['merge-base', '--is-ancestor', a, b], { cwd: repo, stdio: 'ignore' }).status === 0;
|
|
4091
|
+
// The journeys and features `changed` needs (src/affected.ts), from the verify skill at the working tree.
|
|
4092
|
+
function mapDiff(repo, cfg, changed, tracked, noBase) {
|
|
4093
|
+
const featDir = join(repo, cfg.skill, 'features');
|
|
4094
|
+
const features = Object.fromEntries(skillFeatures(repo, cfg).map((n) => [n, repoText(join(featDir, `${n}.md`)) ?? '']));
|
|
4095
|
+
const coreLine = featureMap(repo, cfg).split('\n').find((l) => /^\s*[-*] /.test(l) && l.includes(MARK.core));
|
|
4096
|
+
return mapChange({ changed, tracked, journeys: cfg.journeys, skill: cfg.skill, features, coreFeature: coreLine?.match(/\]\(([^)#]+)\.md\)/)?.[1], noBase });
|
|
4097
|
+
}
|
|
3995
4098
|
// Which journeys the change needs (src/affected.ts) and whether HEAD's proof covers them. The change is read from `baseArg`,
|
|
3996
4099
|
// else from the newest receipt before HEAD (the commits since the last proof), else from the merge base with the default
|
|
3997
4100
|
// branch; with none, the mapping is uncertain and the core journey is needed.
|
|
@@ -4025,10 +4128,7 @@ function affectedOf(repo, cfg, baseArg) {
|
|
|
4025
4128
|
// Untracked files are no part of the change until added, as everywhere else in done-done.
|
|
4026
4129
|
const uncommitted = list('diff', '--name-only', '--no-renames', '-z', 'HEAD').filter(counts);
|
|
4027
4130
|
const changed = [...new Set([...(base ? list('diff', '--name-only', '--no-renames', '-z', base.commit, 'HEAD').filter(counts) : []), ...uncommitted])];
|
|
4028
|
-
const
|
|
4029
|
-
const features = Object.fromEntries(skillFeatures(repo, cfg).map((n) => [n, repoText(join(featDir, `${n}.md`)) ?? '']));
|
|
4030
|
-
const coreLine = featureMap(repo, cfg).split('\n').find((l) => /^\s*[-*] /.test(l) && l.includes(MARK.core));
|
|
4031
|
-
const m = mapChange({ changed, tracked, journeys: cfg.journeys, skill: cfg.skill, features, coreFeature: coreLine?.match(/\]\(([^)#]+)\.md\)/)?.[1], noBase: base ? undefined : 'no receipt before HEAD and no default branch to diff against' });
|
|
4131
|
+
const m = mapDiff(repo, cfg, changed, tracked, base ? undefined : 'no receipt before HEAD and no default branch to diff against');
|
|
4032
4132
|
const needed = [...m.journeys.map((j) => j.name), ...m.features.map((f) => `feature ${f.name}`)];
|
|
4033
4133
|
// Proof: a valid receipt for HEAD whose runs include every journey and feature the change needs, with nothing uncommitted.
|
|
4034
4134
|
const verified = verifyReceipt(repo, cfg, 'HEAD');
|
package/dist/config.js
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
import { existsSync, readFileSync } from 'node:fs';
|
|
2
2
|
import { homedir } from 'node:os';
|
|
3
|
-
import { basename, isAbsolute, join, resolve } from 'node:path';
|
|
3
|
+
import { basename, dirname, isAbsolute, join, resolve } from 'node:path';
|
|
4
4
|
import { parse } from 'yaml';
|
|
5
5
|
import { checkMatrix } from './matrix.js';
|
|
6
|
-
import { parseDom } from './dom.js';
|
|
6
|
+
import { parseDom, parseLook } from './dom.js';
|
|
7
7
|
export class UsageError extends Error {
|
|
8
8
|
help;
|
|
9
9
|
// The exact next command, when the error has one.
|
|
@@ -66,6 +66,8 @@ const KEYS = {
|
|
|
66
66
|
instance: { scope: 'repo', type: 'instance' },
|
|
67
67
|
matrix: { scope: 'repo', type: 'matrix' },
|
|
68
68
|
dom: { scope: 'repo', type: 'dom' },
|
|
69
|
+
look: { scope: 'repo', type: 'look' },
|
|
70
|
+
approved_design: { scope: 'repo', type: 'approved_design' },
|
|
69
71
|
base_ref: { scope: 'repo', type: 'string' },
|
|
70
72
|
'evidence.root': { scope: 'repo', type: 'string' },
|
|
71
73
|
'evidence.allow_dirty': { scope: 'repo', type: 'boolean' },
|
|
@@ -211,6 +213,28 @@ function apply(target, text, file, scope) {
|
|
|
211
213
|
target[key] = { command: v.command, ready: has(r.url) ? { url: r.url } : { command: r.command }, ready_timeout: v.ready_timeout ?? '30s' };
|
|
212
214
|
continue;
|
|
213
215
|
}
|
|
216
|
+
if (key === 'approved_design') {
|
|
217
|
+
const fail = () => { throw new UsageError(`${file}: approved_design must be a non-empty list of unique {screen, size: <width>x<height> in pixels, theme, mock: <repo-relative PNG/JPEG>, capture: <unique PNG/JPEG filename>}`); };
|
|
218
|
+
const relativeImage = (p) => typeof p === 'string' && !isAbsolute(p) && !p.split(/[\\/]/).includes('..') && /\.(png|jpe?g)$/i.test(p);
|
|
219
|
+
if (!Array.isArray(value) || !value.length)
|
|
220
|
+
fail();
|
|
221
|
+
const rows = value;
|
|
222
|
+
const ids = new Set(), names = new Set();
|
|
223
|
+
for (const r of rows) {
|
|
224
|
+
if (!isMap(r) || Object.keys(r).length !== 5 || !['screen', 'size', 'theme', 'mock', 'capture'].every(k => typeof r[k] === 'string' && r[k].trim()))
|
|
225
|
+
fail();
|
|
226
|
+
if (!/^[1-9]\d*x[1-9]\d*$/.test(r.size) || !relativeImage(r.mock) || !relativeImage(r.capture) || /[\\/]/.test(r.capture))
|
|
227
|
+
fail();
|
|
228
|
+
const id = JSON.stringify([r.screen, r.size, r.theme]);
|
|
229
|
+
if (ids.has(id) || names.has(r.capture))
|
|
230
|
+
fail();
|
|
231
|
+
ids.add(id);
|
|
232
|
+
names.add(r.capture);
|
|
233
|
+
}
|
|
234
|
+
target.approved_design = rows;
|
|
235
|
+
target.from[key] = scope;
|
|
236
|
+
continue;
|
|
237
|
+
}
|
|
214
238
|
if (key === 'dom') {
|
|
215
239
|
const d = parseDom(value);
|
|
216
240
|
if (typeof d === 'string')
|
|
@@ -219,6 +243,14 @@ function apply(target, text, file, scope) {
|
|
|
219
243
|
target.dom = d;
|
|
220
244
|
continue;
|
|
221
245
|
}
|
|
246
|
+
if (key === 'look') {
|
|
247
|
+
const l = parseLook(value, dirname(file));
|
|
248
|
+
if (typeof l === 'string')
|
|
249
|
+
throw new UsageError(`${file}: look ${l}`);
|
|
250
|
+
target.from[key] = scope;
|
|
251
|
+
target.look = l;
|
|
252
|
+
continue;
|
|
253
|
+
}
|
|
222
254
|
if (!ok)
|
|
223
255
|
throw new UsageError(`${file}: ${key} must be ${want[spec.type] ?? `a ${spec.type}`}${key === 'matrix' ? `: ${checkMatrix(value)}` : ''}`);
|
|
224
256
|
if (key === 'agent_config')
|
|
@@ -279,6 +311,8 @@ function defaultConfig(repo) {
|
|
|
279
311
|
instance: null,
|
|
280
312
|
matrix: null,
|
|
281
313
|
dom: null,
|
|
314
|
+
look: null,
|
|
315
|
+
approved_design: [],
|
|
282
316
|
base_ref: '',
|
|
283
317
|
evidence: { local_root: join(home(), 'evidence'), root: '', allow_dirty: false, ignore_paths: [] },
|
|
284
318
|
secrets: { patterns: {}, allow: [] },
|
package/dist/dom.js
CHANGED
|
@@ -5,10 +5,12 @@
|
|
|
5
5
|
// "console_errors": ["<message>", ...], "failed_requests": [{"url": "<request url>", "status": <HTTP status, 0 when none>}],
|
|
6
6
|
// "squashed_text": [{"element": "<element>", "text": "<its text, ~40 chars>", "chars": <width>, "lines": <n>, "words": <n>}],
|
|
7
7
|
// "clipped_text": [{"element": "<element>", "text": "<its text>", "by": "<the container that cuts it>"}],
|
|
8
|
-
// "cut_off": [{"element": "<element>", "text": "<its text>", "edges": ["left" | "right" | "top", ...]}]
|
|
9
|
-
//
|
|
8
|
+
// "cut_off": [{"element": "<element>", "text": "<its text>", "edges": ["left" | "right" | "top", ...]}],
|
|
9
|
+
// "controls": [{"element": "<element>", "text": "<its label>", "native": <true when drawn as the browser's own control>, "height": <px, absent for a textarea or list box>}],
|
|
10
|
+
// "actions": [{"selector": "<what was clicked>", "text": "<its label>", "failed": "<why it did not complete, absent when it did>"}]}
|
|
11
|
+
// squashed_text, clipped_text, cut_off, controls and actions are optional, so a record written before them stays valid.
|
|
10
12
|
import { readFileSync } from 'node:fs';
|
|
11
|
-
import { basename } from 'node:path';
|
|
13
|
+
import { basename, resolve } from 'node:path';
|
|
12
14
|
// The measurement the recipe passes to its browser tool's eval, so no agent invents one: every visible element with its
|
|
13
15
|
// own non-empty text is squashed when its box is under 8 characters wide (box width / the average glyph width of its
|
|
14
16
|
// font) while its text is wider than the box, or when its text sits on more line boxes than it has words.
|
|
@@ -67,6 +69,39 @@ export const LOOK = `() => {
|
|
|
67
69
|
}
|
|
68
70
|
return { overflow: document.documentElement.scrollWidth > vw, overflowing: [...document.querySelectorAll('body *')].filter((e) => e.getBoundingClientRect().right > vw + 1).slice(0, 10).map(name), ...out };
|
|
69
71
|
}`.replace(/\s*\n\s*/g, ' ');
|
|
72
|
+
// Every visible control with its label and height, and whether it is drawn as the browser's own: its background and
|
|
73
|
+
// border are what the browser gives the same control (same tag, input type and colour scheme) on a page with no styles.
|
|
74
|
+
// A checkbox, radio, range, colour or file input is not measured, and a textarea or list box has no height to hold.
|
|
75
|
+
export const CONTROLS = `() => {
|
|
76
|
+
const name = (e) => e.tagName.toLowerCase() + (e.id ? '#' + e.id : e.classList[0] ? '.' + e.classList[0] : '');
|
|
77
|
+
const f = document.createElement('iframe'); f.style.cssText = 'position:fixed;width:0;height:0;border:0'; document.documentElement.append(f);
|
|
78
|
+
const d = f.contentDocument; d.open(); d.write('<!doctype html><body>'); d.close();
|
|
79
|
+
const keys = ['backgroundColor', 'borderTopStyle', 'borderTopWidth', 'borderTopColor', 'borderTopLeftRadius']; const out = [];
|
|
80
|
+
for (const e of document.querySelectorAll('button, select, textarea, input, [role=button]')) {
|
|
81
|
+
const s = getComputedStyle(e); const r = e.getBoundingClientRect();
|
|
82
|
+
if (!e.getClientRects().length || s.visibility !== 'visible' || r.width < 1 || r.height < 1 || (e.tagName === 'INPUT' && ['checkbox', 'radio', 'range', 'color', 'file'].includes(e.type))) continue;
|
|
83
|
+
let native = false;
|
|
84
|
+
if (['BUTTON', 'SELECT', 'TEXTAREA', 'INPUT'].includes(e.tagName)) { const t = d.createElement(e.tagName); if (e.tagName === 'INPUT') t.type = e.type; t.style.colorScheme = s.colorScheme; d.body.append(t); const u = f.contentWindow.getComputedStyle(t); native = keys.every((k) => s[k] === u[k]); }
|
|
85
|
+
const tall = e.tagName === 'TEXTAREA' || (e.tagName === 'SELECT' && (e.multiple || e.size > 1));
|
|
86
|
+
out.push({ element: name(e), text: (e.getAttribute('aria-label') || e.innerText || e.value || e.placeholder || '').replace(/\\s+/g, ' ').trim().slice(0, 40), native, ...(!tall && { height: Math.round(r.height) }) });
|
|
87
|
+
}
|
|
88
|
+
f.remove(); return out;
|
|
89
|
+
}`.replace(/\s*\n\s*/g, ' ');
|
|
90
|
+
// What a user can read of the page: its address, how many elements it has and its visible text. An action completes when
|
|
91
|
+
// this changes: a card moves, a reply shows as saved, a toast says so.
|
|
92
|
+
const STATE = "(() => location.href + '\\n' + document.querySelectorAll('body *').length + '\\n' + document.body.innerText)()";
|
|
93
|
+
// Where to click the action `sel` names, with its label and the page state before it, or why it cannot be clicked.
|
|
94
|
+
export const AIM = `(sel) => {
|
|
95
|
+
const e = document.querySelector(sel); if (!e) return { text: '', failed: 'is not on the page' };
|
|
96
|
+
e.scrollIntoView({ block: 'center' }); const r = e.getBoundingClientRect(); const x = r.left + r.width / 2; const y = r.top + r.height / 2;
|
|
97
|
+
const text = (e.getAttribute('aria-label') || e.innerText || e.value || '').replace(/\\s+/g, ' ').trim().slice(0, 40);
|
|
98
|
+
if (e.disabled || e.getAttribute('aria-disabled') === 'true') return { text, failed: 'is disabled' };
|
|
99
|
+
const hit = document.elementFromPoint(x, y);
|
|
100
|
+
if (!hit || (hit !== e && !e.contains(hit))) return { text, failed: 'is covered by ' + (hit ? hit.tagName.toLowerCase() + (hit.id ? '#' + hit.id : hit.classList[0] ? '.' + hit.classList[0] : '') : 'nothing that takes a click') };
|
|
101
|
+
return { text, x, y, before: ${STATE} };
|
|
102
|
+
}`.replace(/\s*\n\s*/g, ' ');
|
|
103
|
+
// Whether the page state differs from `before` within `ms`, checked each 50 ms.
|
|
104
|
+
export const CHANGED = `async (before, ms) => { for (let t = 0; t <= ms; t += 50) { if (${STATE} !== before) return true; await new Promise((r) => setTimeout(r, 50)); } return false; }`;
|
|
70
105
|
const isMap = (v) => !!v && typeof v === 'object' && !Array.isArray(v);
|
|
71
106
|
const strings = (a) => Array.isArray(a) && a.every((s) => typeof s === 'string');
|
|
72
107
|
// An optional record list: absent, or every item has an element and text and passes `more`.
|
|
@@ -100,8 +135,41 @@ export function parseDom(v) {
|
|
|
100
135
|
return 'fail_on_overflow must be true or false';
|
|
101
136
|
return { allow_console: v.allow_console ?? [], allow_requests: v.allow_requests ?? [], fail_on_overflow: v.fail_on_overflow ?? true };
|
|
102
137
|
}
|
|
138
|
+
// The look block read from config in `repo`, or the reason it is not one. max_control is a number of px, or the name of
|
|
139
|
+
// a custom property in the tokens file (--ctl-h: 2.75rem), read as px with a rem or em of 16 px.
|
|
140
|
+
export function parseLook(v, repo) {
|
|
141
|
+
if (v == null)
|
|
142
|
+
v = {};
|
|
143
|
+
if (!isMap(v))
|
|
144
|
+
return 'a mapping with optional tokens and max_control';
|
|
145
|
+
for (const k of Object.keys(v))
|
|
146
|
+
if (!['tokens', 'max_control'].includes(k))
|
|
147
|
+
return `unknown key ${k} (tokens, max_control)`;
|
|
148
|
+
const { tokens, max_control: max } = v;
|
|
149
|
+
if (tokens !== undefined && (typeof tokens !== 'string' || !tokens))
|
|
150
|
+
return 'tokens must be the path of the design tokens file';
|
|
151
|
+
if (typeof max === 'number')
|
|
152
|
+
return max > 0 ? { ...(tokens && { tokens }), max_control: max } : 'max_control must be more than 0 px';
|
|
153
|
+
if (max === undefined)
|
|
154
|
+
return tokens ? { tokens } : {};
|
|
155
|
+
if (typeof max !== 'string' || !/^--[\w-]+$/.test(max))
|
|
156
|
+
return 'max_control must be a number of px or the name of a token such as --ctl-h';
|
|
157
|
+
if (!tokens)
|
|
158
|
+
return `max_control ${max} names a token, so tokens must name the file that sets it`;
|
|
159
|
+
let text;
|
|
160
|
+
try {
|
|
161
|
+
text = readFileSync(resolve(repo, tokens), 'utf8');
|
|
162
|
+
}
|
|
163
|
+
catch (e) {
|
|
164
|
+
return `tokens ${tokens} does not read: ${e.code ?? e.message}`;
|
|
165
|
+
}
|
|
166
|
+
const m = new RegExp(`(?<![\\w-])${max}\\s*:\\s*([\\d.]+)(px|rem|em)\\s*[;}]`).exec(text);
|
|
167
|
+
if (!m)
|
|
168
|
+
return `max_control ${max} is not set in ${tokens} as a px, rem or em size`;
|
|
169
|
+
return { tokens, max_control: Number(m[1]) * (m[2] === 'px' ? 1 : 16) };
|
|
170
|
+
}
|
|
103
171
|
// Every non-allowed issue the capture's page-check record shows; a missing or malformed record is an issue too.
|
|
104
|
-
export function domIssues(dom, capture) {
|
|
172
|
+
export function domIssues(dom, capture, ref) {
|
|
105
173
|
const path = domRecord(capture);
|
|
106
174
|
let r;
|
|
107
175
|
try {
|
|
@@ -114,22 +182,28 @@ export function domIssues(dom, capture) {
|
|
|
114
182
|
if (!isMap(r) || typeof r.url !== 'string' || typeof r.overflow !== 'boolean' || !strings(r.overflowing ?? []) || !strings(r.console_errors)
|
|
115
183
|
|| !Array.isArray(r.failed_requests) || !r.failed_requests.every((q) => isMap(q) && typeof q.url === 'string' && typeof q.status === 'number')
|
|
116
184
|
|| !list(r.squashed_text, (q) => [q.chars, q.lines, q.words].every((n) => typeof n === 'number'))
|
|
117
|
-
|| !list(r.clipped_text, (q) => typeof q.by === 'string') || !list(r.cut_off, (q) => strings(q.edges))
|
|
118
|
-
|
|
185
|
+
|| !list(r.clipped_text, (q) => typeof q.by === 'string') || !list(r.cut_off, (q) => strings(q.edges))
|
|
186
|
+
|| !list(r.controls, (q) => typeof q.native === 'boolean' && ['number', 'undefined'].includes(typeof q.height))
|
|
187
|
+
|| !(r.actions === undefined || (Array.isArray(r.actions) && r.actions.every((q) => isMap(q) && typeof q.selector === 'string' && typeof q.text === 'string' && ['string', 'undefined'].includes(typeof q.failed)))))
|
|
188
|
+
return [`page-check record ${basename(path)} is not {url, overflow, overflowing, console_errors, failed_requests: [{url, status}], squashed_text: [{element, text, chars, lines, words}], clipped_text: [{element, text, by}], cut_off: [{element, text, edges}], controls: [{element, text, native, height}], actions: [{selector, text, failed}]}`];
|
|
119
189
|
const allowed = (list, s) => list.some((re) => new RegExp(re).test(s));
|
|
120
190
|
return [
|
|
121
|
-
...lookIssues({ ...r, overflow: dom.fail_on_overflow && r.overflow }),
|
|
191
|
+
...lookIssues({ ...r, overflow: dom.fail_on_overflow && r.overflow }, ref),
|
|
122
192
|
...r.console_errors.filter((m) => !allowed(dom.allow_console, m)).map((m) => `${r.url} console error: ${m}`),
|
|
123
193
|
...r.failed_requests.filter((q) => (q.status === 0 || q.status >= 400) && !allowed(dom.allow_requests, q.url)).map((q) => `${r.url} request failed (${q.status || 'no response'}): ${q.url}`),
|
|
124
194
|
];
|
|
125
195
|
}
|
|
126
|
-
// The look defects a page-check record shows: horizontal overflow, squashed, clipped and cut-off text or elements
|
|
127
|
-
|
|
196
|
+
// The look defects a page-check record shows: horizontal overflow, squashed, clipped and cut-off text or elements, and an
|
|
197
|
+
// action that did not complete; against a reference, also a control drawn as the browser's own or taller than max_control.
|
|
198
|
+
export function lookIssues(r, ref) {
|
|
128
199
|
const quoted = (t) => (t ? ` "${t}"` : '');
|
|
129
200
|
return [
|
|
130
201
|
...(r.overflow ? [`${r.url} overflows the viewport horizontally${r.overflowing?.length ? ` (${r.overflowing.join(', ')})` : ''}`] : []),
|
|
131
202
|
...(r.squashed_text ?? []).map((q) => `${r.url} squashed text: ${q.element} "${q.text}" is ${q.chars} characters wide (lines ${q.lines}, words ${q.words})`),
|
|
132
203
|
...(r.clipped_text ?? []).map((q) => `${r.url} clipped text: ${q.element}${quoted(q.text)} is cut by ${q.by}`),
|
|
133
204
|
...(r.cut_off ?? []).map((q) => `${r.url} cut off: ${q.element}${quoted(q.text)} runs past the ${q.edges.join(' and ')} edge of the viewport`),
|
|
205
|
+
...(ref ? r.controls ?? [] : []).filter((q) => q.native).map((q) => `${r.url} unstyled control: ${q.element}${quoted(q.text)} is drawn as the browser's own control, not with the design's style`),
|
|
206
|
+
...(ref?.max_control ? r.controls ?? [] : []).filter((q) => q.height > ref.max_control + 0.5).map((q) => `${r.url} oversized control: ${q.element}${quoted(q.text)} is ${q.height} px tall, over the ${ref.max_control} px the design allows`),
|
|
207
|
+
...(r.actions ?? []).filter((q) => q.failed).map((q) => `${r.url} action did not complete: ${q.selector}${quoted(q.text)} ${q.failed}`),
|
|
134
208
|
];
|
|
135
209
|
}
|
package/dist/judge.js
CHANGED
|
@@ -276,11 +276,72 @@ export async function judge(cfg, choose, outcome, captures, records, fallback) {
|
|
|
276
276
|
}
|
|
277
277
|
}
|
|
278
278
|
}
|
|
279
|
+
async function judgeBatch(cfg, choice, outcome, captures, records, dir) {
|
|
280
|
+
const stamp = new Date().toISOString().replace(/[:.]/g, '-');
|
|
281
|
+
const eligible = captures.filter(judgeable);
|
|
282
|
+
const notJudged = captures.filter((c) => !judgeable(c)).map((c) => c.name);
|
|
283
|
+
const selected = eligible;
|
|
284
|
+
const suffix = [];
|
|
285
|
+
const rows = selected.map((c) => {
|
|
286
|
+
const pair = c.before && judgeable(c.before) && screen(c.before) === screen(c) ? c.before : undefined;
|
|
287
|
+
return { c, pair, row: { path: c.name, sha256: c.sha256, ...(pair && { before: pair.name }) } };
|
|
288
|
+
});
|
|
289
|
+
const { route, model } = choice;
|
|
290
|
+
const key = createHash('sha256').update(JSON.stringify([VERSION, captures.map((c) => [c.sha256, c.before?.sha256]), outcome, route, model, cfg.judge.floor])).digest('hex');
|
|
291
|
+
const hit = readCache(dir, key);
|
|
292
|
+
if (Array.isArray(hit?.judged) && hit.judged.length === rows.length && hit.judged.every((r) => ['pass', 'fail', 'unsure'].includes(r?.verdict) && typeof r.reason === 'string' && (r.parts === undefined || coverageOf(r))))
|
|
293
|
+
return { judged: [...hit.judged.map((r) => ({ ...r, cached: true })), ...suffix], notJudged, choice, missing: hit.missing ?? [] };
|
|
294
|
+
const questions = {};
|
|
295
|
+
const images = {};
|
|
296
|
+
const state = { outcome };
|
|
297
|
+
for (const [i, { c, pair }] of rows.entries()) {
|
|
298
|
+
const id = `capture_${i}`;
|
|
299
|
+
const input = screen(c) ? prepare(c) : c.bytes.length > MAX_TEXT ? `${c.name} text exceeds ${MAX_TEXT} bytes` : { text: c.bytes.toString('utf8') };
|
|
300
|
+
const before = pair && (screen(pair) ? prepare(pair) : pair.bytes.length > MAX_TEXT ? `${pair.name} text exceeds ${MAX_TEXT} bytes` : { text: pair.bytes.toString('utf8') });
|
|
301
|
+
if (typeof input === 'string' || typeof before === 'string')
|
|
302
|
+
throw new UsageError(`judge: ${c.name} cannot be sent to the judge: ${input}`, 'Capture it again as the error says, then re-run this command');
|
|
303
|
+
const base = 'text' in input ? before ? TEXT_PAIR : TEXT_QUESTION : before ? PAIR : QUESTION;
|
|
304
|
+
const beforeId = `before_${i}`;
|
|
305
|
+
const captureId = `capture_${i}`;
|
|
306
|
+
if ('mime' in input)
|
|
307
|
+
images[captureId] = input;
|
|
308
|
+
if (before && 'mime' in before)
|
|
309
|
+
images[beforeId] = before;
|
|
310
|
+
state[id] = { ...(before && 'text' in before && { before: before.text }), ...(typeof input === 'object' && 'text' in input && { capture: input.text }) };
|
|
311
|
+
questions[id] = { ...base, images: 'text' in input ? [] : before && 'mime' in before ? [beforeId, captureId] : [captureId], instructions: `${base.instructions} ${PARTS}`, floor: cfg.judge.floor, ...(route === 'openai:key' ? {} : { answerFields: FIELDS }) };
|
|
312
|
+
}
|
|
313
|
+
const call = join(records, `judge-${stamp}-1`);
|
|
314
|
+
const be = await backend(cfg, choice, (request) => `${call}-${request}.agent.json`, questions);
|
|
315
|
+
const answers = await askJudge(cfg, be, choice, state, questions, images);
|
|
316
|
+
const judged = rows.map(({ row, c }, i) => {
|
|
317
|
+
const a = answers[`capture_${i}`];
|
|
318
|
+
const shape = questions[`capture_${i}`];
|
|
319
|
+
const reason = a.personReason || a.rationale || shape.options[String(a.answer)] || '';
|
|
320
|
+
const cov = coverageOf(a);
|
|
321
|
+
return { ...row, verdict: a.abstained ? 'unsure' : a.answer === 'correct' ? 'pass' : a.answer === 'wrong' ? 'fail' : 'unsure', reason: a.abstained ? `not confident enough (${a.reason})` : reason, ...(a.rationale && { seen: a.rationale }), ...(route !== 'openai:key' && cov?.parts.length ? cov : {}), cached: false };
|
|
322
|
+
});
|
|
323
|
+
const named = new Map();
|
|
324
|
+
const shown = new Set();
|
|
325
|
+
for (const r of judged) {
|
|
326
|
+
for (const p of r.parts ?? [])
|
|
327
|
+
if (norm(p))
|
|
328
|
+
named.set(norm(p), named.get(norm(p)) ?? p.trim());
|
|
329
|
+
if (r.verdict === 'pass')
|
|
330
|
+
for (const i of r.covers ?? [])
|
|
331
|
+
shown.add(norm(r.parts?.[i] ?? ''));
|
|
332
|
+
}
|
|
333
|
+
const missing = [...named].filter(([n]) => !shown.has(n)).map(([, p]) => p);
|
|
334
|
+
writeCache(dir, key, { judged, missing, route, model });
|
|
335
|
+
return { judged: [...judged, ...suffix], notJudged, choice, missing };
|
|
336
|
+
}
|
|
279
337
|
async function judgeOn(cfg, choose, outcome, captures, records) {
|
|
280
338
|
const dir = join(home(), 'runs', cfg.app, 'judge');
|
|
281
339
|
const stamp = new Date().toISOString().replace(/[:.]/g, '-');
|
|
282
340
|
const judged = [];
|
|
283
341
|
const notJudged = [];
|
|
342
|
+
const batchChoice = captures.filter(judgeable).length > 1 ? choose() : undefined;
|
|
343
|
+
if (batchChoice && batchChoice.route !== 'openai:key')
|
|
344
|
+
return judgeBatch(cfg, batchChoice, outcome, captures, records, dir);
|
|
284
345
|
let choice;
|
|
285
346
|
let be;
|
|
286
347
|
let calls = 0;
|
|
@@ -337,9 +398,7 @@ async function judgeOn(cfg, choose, outcome, captures, records) {
|
|
|
337
398
|
writeCache(dir, key, { verdict: verdict.verdict, reason: verdict.reason, ...seen, ...coverage, route, model });
|
|
338
399
|
judged.push(verdict);
|
|
339
400
|
}
|
|
340
|
-
//
|
|
341
|
-
// parts differently is still talking about the same part. Every distinct part any reply names must be covered by a
|
|
342
|
-
// capture that passes, and one no passing capture shows is missing, under the name the reply wrote.
|
|
401
|
+
// Every distinct part any reply names must be covered by a passing capture.
|
|
343
402
|
const named = new Map(); // normalised label -> the label to show
|
|
344
403
|
const shown = new Set();
|
|
345
404
|
for (const r of judged) {
|
package/dist/look.js
CHANGED
|
@@ -3,14 +3,17 @@ import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
|
|
|
3
3
|
import { tmpdir } from 'node:os';
|
|
4
4
|
import { join } from 'node:path';
|
|
5
5
|
import { UsageError } from './config.js';
|
|
6
|
-
import { domRecord, LOOK, SQUASHED_TEXT } from './dom.js';
|
|
6
|
+
import { AIM, CHANGED, CONTROLS, domRecord, LOOK, SQUASHED_TEXT } from './dom.js';
|
|
7
7
|
import { findBin } from './agents.js';
|
|
8
8
|
// done-done look: opens one page in headless Chrome at a phone viewport over the DevTools protocol (Node's own
|
|
9
|
-
// WebSocket, no browser library),
|
|
9
|
+
// WebSocket, no browser library), clicks each action in `acts` in order (a CSS selector each) and waits for it to complete,
|
|
10
|
+
// then writes the screenshot and its page-check record next to it, and returns the record.
|
|
10
11
|
// Chrome runs without its sandbox so it starts in containers: point look at your own app, not untrusted sites.
|
|
11
12
|
const BROWSERS = ['google-chrome', 'google-chrome-stable', 'chromium', 'chromium-browser', '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'];
|
|
12
13
|
const WAIT = 30_000;
|
|
13
|
-
|
|
14
|
+
// How long an action has to change what a user can read of the page.
|
|
15
|
+
const ACT = 5_000;
|
|
16
|
+
export async function look(url, capture, width, height, acts = []) {
|
|
14
17
|
const bin = process.env.DONE_DONE_CHROME ? findBin(process.env.DONE_DONE_CHROME) : BROWSERS.map(findBin).find(Boolean);
|
|
15
18
|
if (!bin)
|
|
16
19
|
throw new UsageError(`look needs Chrome or Chromium: ${process.env.DONE_DONE_CHROME ? `DONE_DONE_CHROME=${process.env.DONE_DONE_CHROME} is not an executable` : `none of ${BROWSERS.slice(0, 4).join(', ')} is on PATH`}`, 'Install Chrome or Chromium, or set DONE_DONE_CHROME to its binary');
|
|
@@ -93,8 +96,23 @@ export async function look(url, capture, width, height) {
|
|
|
93
96
|
throw new Error(`${url} did not load: ${nav.errorText}`);
|
|
94
97
|
await loaded;
|
|
95
98
|
// Web fonts change text widths, so measure once they are in and two frames have painted.
|
|
96
|
-
|
|
97
|
-
|
|
99
|
+
const settle = () => evaluate('document.fonts.ready.then(() => new Promise((r) => requestAnimationFrame(() => requestAnimationFrame(r))))');
|
|
100
|
+
await settle();
|
|
101
|
+
// Each action is a real click at its centre, so a control under a toast or overlay fails as covered.
|
|
102
|
+
const actions = [];
|
|
103
|
+
for (const selector of acts) {
|
|
104
|
+
const at = await evaluate(`(${AIM})(${JSON.stringify(selector)})`);
|
|
105
|
+
if (!at.failed) {
|
|
106
|
+
for (const type of ['mousePressed', 'mouseReleased'])
|
|
107
|
+
await send('Input.dispatchMouseEvent', { type, x: at.x, y: at.y, button: 'left', clickCount: 1 });
|
|
108
|
+
if (!(await evaluate(`(${CHANGED})(${JSON.stringify(at.before)}, ${ACT})`)))
|
|
109
|
+
at.failed = `changed nothing a user can read on the page (its text, elements or address) within ${ACT / 1000}s`;
|
|
110
|
+
}
|
|
111
|
+
actions.push({ selector, text: at.text, ...(at.failed && { failed: at.failed }) });
|
|
112
|
+
}
|
|
113
|
+
if (acts.length)
|
|
114
|
+
await evaluate('scrollTo(0, 0)').then(settle);
|
|
115
|
+
const record = { url, ...(await evaluate(`(${LOOK})()`)), console_errors: consoleErrors, failed_requests: failed, squashed_text: await evaluate(`(${SQUASHED_TEXT})()`), controls: await evaluate(`(${CONTROLS})()`), ...(acts.length && { actions }) };
|
|
98
116
|
const shot = await send('Page.captureScreenshot', { format: 'png' });
|
|
99
117
|
writeFileSync(capture, Buffer.from(shot.data, 'base64'));
|
|
100
118
|
writeFileSync(domRecord(capture), `${JSON.stringify(record, null, 2)}\n`);
|
|
@@ -15,7 +15,7 @@ A journey entry in the format of muxr's project skill (`.agents/skills/done-done
|
|
|
15
15
|
|
|
16
16
|
## Driving it with the private stack
|
|
17
17
|
|
|
18
|
-
Needs a separately authorized, isolated Herdr lab session. Never drive
|
|
18
|
+
Needs a separately authorized, isolated Herdr lab session. Never drive a user's live Herdr.
|
|
19
19
|
|
|
20
20
|
1. Start a lab host against a named lab Herdr session with two agent panes, A and B, and pair a lab phone client (Umer).
|
|
21
21
|
2. Hold A's `pane.agent_status_changed` subscribe at a socket proxy so the kit's tree read waits for A's watch, then close pane A. This is step 4 of the counterexample.
|
package/docs/index.html
CHANGED
|
@@ -43,6 +43,7 @@
|
|
|
43
43
|
figure { margin:0 auto 16px; max-width:560px; }
|
|
44
44
|
.quickstart .eyebrow { margin-bottom:6px; }
|
|
45
45
|
.quickstart pre { max-width:560px; margin:0 auto 12px; text-align:left; }
|
|
46
|
+
.quickstart .note { max-width:560px; margin:0 auto 8px; }
|
|
46
47
|
.shot { border:1px solid #393b4c; border-radius:12px; overflow:hidden; background:#11131c; }
|
|
47
48
|
figcaption { font-size:12px; color:var(--muted); margin-top:8px; line-height:1.4; }
|
|
48
49
|
.eyebrow { display:block; color:var(--accent); font-size:11px; font-weight:700; letter-spacing:.07em; margin:0 0 10px; }
|
|
@@ -84,7 +85,7 @@
|
|
|
84
85
|
<header class="hero">
|
|
85
86
|
<h1>Prove your app really works.</h1>
|
|
86
87
|
<p class="definition">A command-line check of real screenshots, recordings and output.</p>
|
|
87
|
-
<p class="audience">For developers using coding agents.</p>
|
|
88
|
+
<p class="audience">For developers using coding agents, alone or in a <a href="#firstmate">Firstmate</a> fleet.</p>
|
|
88
89
|
<p class="problem">Tests pass. Your agent says “done”. The app is still broken.</p>
|
|
89
90
|
<div class="actions"><a class="btn" href="#install">Install in 3 steps</a><a class="btn quiet" href="#see">See it work</a><a class="github" href="https://github.com/umeranjum17/done-done-axi">GitHub ↗</a></div>
|
|
90
91
|
<div class="start">
|
|
@@ -96,7 +97,9 @@
|
|
|
96
97
|
<p class="eyebrow">START HERE · NODE 22.18+ + GIT</p>
|
|
97
98
|
<pre><code>npm install -g done-done-axi
|
|
98
99
|
cd your-app && done-done init
|
|
99
|
-
done-done
|
|
100
|
+
done-done verify --outcome "<what works now>"</code></pre>
|
|
101
|
+
<p class="note">The last line is your first result: your agent drives the real app, a judge checks each capture, and you get a pass with a receipt, or the judge’s reason it did not pass.</p>
|
|
102
|
+
<p class="note"><b>With or without Firstmate.</b> On its own, you or your agent run these three lines. In a <a href="https://github.com/kunchenguid/firstmate">Firstmate</a> fleet, two files make every worker prove its task, plus an optional merge gate that checks that proof (<a href="#firstmate">how</a>).</p>
|
|
100
103
|
</div>
|
|
101
104
|
</div>
|
|
102
105
|
</header>
|
|
@@ -108,9 +111,8 @@ done-done doctor</code></pre>
|
|
|
108
111
|
<div class="tile"><span class="eyebrow">02 / CHECK</span><h3>Show the claim is true.</h3><p>A fresh judge reads each capture. Wrong output fails; uncertainty never becomes a pass.</p></div>
|
|
109
112
|
<div class="tile"><span class="eyebrow">03 / KEEP</span><h3>Save a receipt.</h3><p>A pass records the commit and file hashes. No judge pass, no receipt.</p></div>
|
|
110
113
|
</div>
|
|
111
|
-
<p><code>init</code> asks which coding agent and which judge
|
|
112
|
-
<p>
|
|
113
|
-
<pre><code>done-done verify --outcome "<what a user can now do>"</code></pre>
|
|
114
|
+
<p><code>init</code> asks which coding agent drives your app and which judge checks it, then writes your repo’s setup and a verify skill: how to launch and drive your app. You choose both agents and models. <code>verify</code> then drives the real product, has the judge check each capture, and writes a receipt on a pass; anything else stops at a gate with the judge’s reason.</p>
|
|
115
|
+
<p>No result? <code>doctor</code> checks setup, not the product: it names each missing piece and the command that fixes it. Commit <code>.done-done.yaml</code> and the verify skill so later runs and your teammates use the same recipe.</p>
|
|
114
116
|
<p class="note">Bring your own signed-in agent or API key. Captures go to the judge’s provider on your plan or key. Need no-sudo installation, non-interactive flags or a manual capture? Read the <a href="https://github.com/umeranjum17/done-done-axi/blob/main/docs/manual-cli.md">manual walkthrough</a>.</p>
|
|
115
117
|
<p>Prefer to ask your agent “is this really done?” Install the <a href="https://github.com/umeranjum17/done-done-axi#use-it-from-your-agent">agent skill</a>: <code>npx skills add umeranjum17/done-done-axi --skill done-done -a <agent> -y</code>.</p>
|
|
116
118
|
</section>
|
|
@@ -148,7 +150,7 @@ done-done doctor</code></pre>
|
|
|
148
150
|
<p class="lead">done-done works on its own: neither the CLI nor the skill needs another tool in this family. Each tool does one job and connects through small written contracts.</p>
|
|
149
151
|
<div class="sibs">
|
|
150
152
|
<div class="tile"><h3>done-done</h3><p class="what">Verification. This page.</p></div>
|
|
151
|
-
<div class="tile"><h3><a href="https://www.npmjs.com/package/sign-off-axi">sign-off</a></h3><p class="what">The board where agents ask you questions. <code>done-done signoff</code> writes a run's gate or evidence in its format; it
|
|
153
|
+
<div class="tile"><h3><a href="https://www.npmjs.com/package/sign-off-axi">sign-off</a></h3><p class="what">The board where agents ask you questions. <code>done-done signoff</code> writes a run's gate or evidence in its format; it posts only when you opt in with <code>--post</code>.</p></div>
|
|
152
154
|
<div class="tile"><h3><a href="https://www.npmjs.com/package/learn-from-mistakes-axi">learn-from-mistakes</a></h3><p class="what">Retros that find mistakes your agents keep repeating, with evidence.</p></div>
|
|
153
155
|
<div class="tile"><h3><a href="https://www.npmjs.com/package/first-draft-axi">first-draft</a></h3><p class="what">Turns a prompt into design mocks. Pass one to <code>done-done design --mock</code> to judge a screen next to it.</p></div>
|
|
154
156
|
</div>
|
package/docs/reference.md
CHANGED
|
@@ -16,7 +16,7 @@ done-done init
|
|
|
16
16
|
|
|
17
17
|
`init` writes a minimal `.done-done.yaml` only when one does not exist: just `app`, the repo directory name, so every checkout of the repo finds the same evidence (see [Where proof lives](#where-proof-lives)). It sets no `evidence.root`, so evidence goes outside the repo. When an existing config puts `evidence.root` inside the repo (an older `root: .proof`), `init` adds it to `.gitignore` when git does not already ignore it. A plain `init` never rewrites the config: re-running is a no-op and a customized `.done-done.yaml` is left byte-for-byte; `.gitignore` gains a line only when the ignore entry is missing. A malformed `.done-done.yaml` makes `init` exit 2 with the file named and untouched. Its result says whether the config was `written` or left `unchanged`, the evidence root, what it added to `.gitignore`, the verify skill outcome (`verify_skill`), and the next step.
|
|
18
18
|
|
|
19
|
-
`init` also has the agent the user set write the verify skill at `skill` (default `.agents/skills/done-done-verify-<app>`) from the repo's own files: plain code gives it the tracked file list and the text of the files that show how the repo proves itself (manifests, `.done-done.yaml` journeys, verify, e2e, smoke and demo scripts, `bin/` files, Dockerfiles, then source files (including Vue, Svelte, HTML, CSS and common compiled-language sources); JSON, YAML and Markdown are excluded except for manifests; at most 150,000 characters, and the prompt names what it left out; a `.env` or `.env.*` file, a file or folder whose path contains `secret`, `credential`, `token` or `password`, and `*.pem`, `*.key`, `*.p12` files never have their text sent by `init`, the agent is told not to open them, and no launch or feature can cite them; the tracked file list still names them), and one read-only call answers with the launch, readiness check, harness, evidence and up to five features (claude reads the repo with its read tools; every other agent gets only that text, and codex and gemini cannot run it tool-less). Launch prefers the repo's own safe proof path, a stub or demo engine with its own state, and when the repo shows none, `SKILL.md` opens Launch with `Not isolated:` and what the launch would touch, and `verify_skill` says so. `Safe:` is written only when the model marks the launch safe and its source is a tracked file that is not a manifest such as `package.json` or `README.md`; any other launch is `Not isolated:`, and its steps are not parsed. Each feature names the repo file it comes from. The prompt says a test suite, typecheck, lint or build is never a launch, a drive or the proof, asks for exactly one feature marked `core` (the core user journey) and, when a tracked file shows the app ships an installable artifact, one marked `upgrade` (see [Core journey and release path](#core-journey-and-release-path)), and plain code refuses an answer with no core feature, more than one, or no upgrade feature for such an app; `SKILL.md` gains a `Core journey` section (and a `Release proof` section for such an app) and `features/README.md` marks those entries `(core journey)` and `(upgrade journey)`. The prompt also says and plain code refuses an answer whose launch steps or a feature's drive is code-only ([Real-product evidence](#real-product-evidence)), as it refuses an answer that cites a file the repo does not track, a secret-looking file, or, for a tool-less agent, a tracked file whose text its prompt left out. It writes `SKILL.md`, plus `features/README.md` and one file per feature with numbered drive and capture steps; `features/README.md` says how many more were found and not written. With no agent set, no skill is written and `init` asks for the agent (exit 2); an agent that fails or answers with an uncited or missing field writes no skill, and `verify_skill` names the cause and the call record (exit 1). On a repo with nothing tracked yet (a fresh `git init`), no skill is written and no agent is asked — no answer could cite a tracked file — and `verify_skill` names the remedy: commit the repo's files, then re-run (exit 1). The write is all-or-nothing: when any target already exists — `SKILL.md`, `features/README.md` or a `features/<name>.md`, as a file or a directory — nothing is written and `verify_skill` says `skipped: <paths> already exist; delete the skill folder to regenerate`, so a re-run after a lost `SKILL.md` can never pair new answers with the old feature map; a finished skill is left byte-identical and a re-run asks the agent nothing. A symlink on the path skips the scaffold and reports why in `verify_skill`, with exit 0. Otherwise the files are written into a temporary directory inside the skill's parent and moved into place, so a failed write leaves no partial skill: the temporary directory and any moved file are removed, `init` exits 1, and `verify_skill` names the error and says to re-run `init`. The config is kept either way. A `skill` path with a `..` segment is refused (exit 2) before anything is written. A repo whose older `init` wrote `.agents/skills/verify-<app>` (and sets no `skill`) keeps working: every command reads that folder, and the next `init` moves it once to `.agents/skills/done-done-verify-<app>` (also updating the `name:` line in its `SKILL.md`) and reports `skill_renamed: <old> -> <new>`; when the prefixed folder already exists, the old one is left alone. Commit `.done-done.yaml` and the verify skill before the first capture: a capture from a tree with uncommitted changes fails `check`, and `init` says so in its help.
|
|
19
|
+
`init` also has the agent the user set write the verify skill at `skill` (default `.agents/skills/done-done-verify-<app>`) from the repo's own files: plain code gives it the tracked file list and the text of the files that show how the repo proves itself (manifests, `.done-done.yaml` journeys, verify, e2e, smoke and demo scripts, `bin/` files, Dockerfiles, then source files (including Vue, Svelte, HTML, CSS and common compiled-language sources); JSON, YAML and Markdown are excluded except for manifests; at most 150,000 characters, and the prompt names what it left out; a `.env` or `.env.*` file, a file or folder whose path contains `secret`, `credential`, `token` or `password`, and `*.pem`, `*.key`, `*.p12` files never have their text sent by `init`, the agent is told not to open them, and no launch or feature can cite them; the tracked file list still names them), and one read-only call answers with the launch, readiness check, harness, evidence and up to five features (claude reads the repo with its read tools; every other agent gets only that text, and codex and gemini cannot run it tool-less). Launch prefers the repo's own safe proof path, a stub or demo engine with its own state, and when the repo shows none, `SKILL.md` opens Launch with `Not isolated:` and what the launch would touch, and `verify_skill` says so. `Safe:` is written only when the model marks the launch safe and its source is a tracked file that is not a manifest such as `package.json` or `README.md`; any other launch is `Not isolated:`, and its steps are not parsed. Each feature names the repo file it comes from. The prompt says a test suite, typecheck, lint or build is never a launch, a drive or the proof, asks for exactly one feature marked `core` (the core user journey) and, when a tracked file shows the app ships an installable artifact, one marked `upgrade` (see [Core journey and release path](#core-journey-and-release-path)), and plain code refuses an answer with no core feature, more than one, or no upgrade feature for such an app unless the agent explicitly reports `first_release: true` (the repo says no earlier release yet; false when unknown). That first-release answer writes the skill with `Upgrade journey pending` in `Release proof`, so `maintain` and `affected` can use its core feature map without inventing an earlier version; it does not mark an upgrade journey or satisfy doctor's release check; `SKILL.md` gains a `Core journey` section (and a `Release proof` section for such an app) and `features/README.md` marks those entries `(core journey)` and `(upgrade journey)`. The prompt also says and plain code refuses an answer whose launch steps or a feature's drive is code-only ([Real-product evidence](#real-product-evidence)), as it refuses an answer that cites a file the repo does not track, a secret-looking file, or, for a tool-less agent, a tracked file whose text its prompt left out. It writes `SKILL.md`, plus `features/README.md` and one file per feature with numbered drive and capture steps; `features/README.md` says how many more were found and not written. With no agent set, no skill is written and `init` asks for the agent (exit 2); an agent that fails or answers with an uncited or missing field writes no skill, and `verify_skill` names the cause and the call record (exit 1). On a repo with nothing tracked yet (a fresh `git init`), no skill is written and no agent is asked — no answer could cite a tracked file — and `verify_skill` names the remedy: commit the repo's files, then re-run (exit 1). The write is all-or-nothing: when any target already exists — `SKILL.md`, `features/README.md` or a `features/<name>.md`, as a file or a directory — nothing is written and `verify_skill` says `skipped: <paths> already exist; delete the skill folder to regenerate`, so a re-run after a lost `SKILL.md` can never pair new answers with the old feature map; a finished skill is left byte-identical and a re-run asks the agent nothing. A symlink on the path skips the scaffold and reports why in `verify_skill`, with exit 0. Otherwise the files are written into a temporary directory inside the skill's parent and moved into place, so a failed write leaves no partial skill: the temporary directory and any moved file are removed, `init` exits 1, and `verify_skill` names the error and says to re-run `init`. The config is kept either way. A `skill` path with a `..` segment is refused (exit 2) before anything is written. A repo whose older `init` wrote `.agents/skills/verify-<app>` (and sets no `skill`) keeps working: every command reads that folder, and the next `init` moves it once to `.agents/skills/done-done-verify-<app>` (also updating the `name:` line in its `SKILL.md`) and reports `skill_renamed: <old> -> <new>`; when the prefixed folder already exists, the old one is left alone. Commit `.done-done.yaml` and the verify skill before the first capture: a capture from a tree with uncommitted changes fails `check`, and `init` says so in its help.
|
|
20
20
|
|
|
21
21
|
```sh
|
|
22
22
|
done-done init --agent opencode --model opencode-go/muse-spark-1.3-contributor
|
|
@@ -177,18 +177,23 @@ done-done never drives the browser: the recipe or journey writes one page-check
|
|
|
177
177
|
{"url": "http://127.0.0.1:4100/", "overflow": true, "overflowing": ["div#wide"], "console_errors": ["boom"], "failed_requests": [{"url": "http://127.0.0.1:4100/x.js", "status": 404}],
|
|
178
178
|
"squashed_text": [{"element": "h3.card-title", "text": "Approved", "chars": 1.6, "lines": 8, "words": 1}],
|
|
179
179
|
"clipped_text": [{"element": "h3.card-title", "text": "Answered: the release notes are approved", "by": "its own box"}],
|
|
180
|
-
"cut_off": [{"element": "div.clock", "text": "12:30", "edges": ["top"]}]
|
|
180
|
+
"cut_off": [{"element": "div.clock", "text": "12:30", "edges": ["top"]}],
|
|
181
|
+
"controls": [{"element": "button#request", "text": "Request changes", "native": true, "height": 21}],
|
|
182
|
+
"actions": [{"selector": "#request", "text": "Request changes", "failed": "changed nothing a user can read on the page (its text, elements or address) within 5s"}]}
|
|
181
183
|
```
|
|
182
184
|
|
|
185
|
+
`controls` and `actions` are optional too; [`done-done look`](#look-check) writes `controls` on every page and `actions` when it clicks one. With a [`look` block](#look-check), the check fails on each control in `controls` drawn as the browser's own, and on one taller than `max_control`; it fails on each `actions` item with a `failed` reason with or without one. A recipe that drives its own browser runs done-done's `CONTROLS` measurement (exported from `dom.js` next to `LOOK`) for `controls`, and `AIM` and `CHANGED` around each click for `actions`.
|
|
186
|
+
|
|
183
187
|
With your browser tool, for example chrome-devtools-axi: `overflow` and `overflowing` from `eval` (document `scrollWidth` beyond `clientWidth`, and the elements whose right edge passes it), `console_errors` from `console --type error`, `failed_requests` (status 400 or more, or 0 for no response) from `network`, `squashed_text` from `eval` of done-done's measurement function, run in each frame of the page (an embedded frame too). It lists every visible HTML element with its own text (SVG text has no layout box and is skipped; widths are untransformed layout pixels, so a scaled ancestor does not shrink them) whose box is under 8 characters wide (its width over the average glyph width of its font) while its text is wider than the box, or whose text sits on more lines (distinct line boxes) than it has words: a title wrapped one letter per line in a narrow column. `squashed_text`, `clipped_text` and `cut_off` are optional, so a record written without them stays valid; [`done-done look`](#look-check) writes all of them, and the check fails on each `clipped_text` and `cut_off` item as it does on `squashed_text`. The agent drive prompt carries the recipe, and a verify skill should carry it too. Verify's check and `done-done check` fail naming the capture and each issue not on an allow list, and fail when a capture has no record or a record is not this shape. No `dom` block keeps today's behaviour.
|
|
184
188
|
|
|
185
189
|
### Look check
|
|
186
190
|
|
|
187
|
-
`done-done look` is the look step a journey calls on a web page. It opens the page in headless Chrome at a 390x844 phone viewport (a mobile device at 1x scale), measures the look defects in the DOM, writes the screenshot with its page-check record next to it, and exits 1 on any hit, naming the element, the kind and its text. It is a measurement step, not a verdict: with no hit it ends `result: clean` (exit 0), never `pass`, and its capture is judged when a `check` or `verify` cites it.
|
|
191
|
+
`done-done look` is the look step a journey calls on a web page. It opens the page in headless Chrome at a 390x844 phone viewport (a mobile device at 1x scale), clicks each `--act` and checks it completes, measures the look defects in the DOM, writes the screenshot with its page-check record next to it, and exits 1 on any hit, naming the element, the kind and its text. It is a measurement step, not a verdict: with no hit it ends `result: clean` (exit 0), never `pass`, and its capture is judged when a `check` or `verify` cites it. Without `--mock` no model runs here, so the same page gives the same answer. A worker's claim that a phone UI works needs this 390 px capture of the real page, plus the page's scroll-container count, as proof; a mock or another width is not proof.
|
|
188
192
|
|
|
189
193
|
```sh
|
|
190
194
|
done-done look http://127.0.0.1:3000/inbox --out "$DONE_DONE_EVIDENCE_ROOT/inbox.png"
|
|
191
195
|
done-done look ./public/inbox.html # a path on disk opens as its file:// URL
|
|
196
|
+
done-done look "$DONE_DONE_BASE_URL/#decide" --act '#item-1 .approve' --mock design/decide.png --out "$DONE_DONE_EVIDENCE_ROOT/decide.png"
|
|
192
197
|
```
|
|
193
198
|
|
|
194
199
|
A journey in `.done-done.yaml` that calls it (one command per screen, `&&` between them, so the first hit stops the journey):
|
|
@@ -211,17 +216,31 @@ journeys:
|
|
|
211
216
|
| squashed text | done-done's [squashed-text measurement](#dom-assertions): a text block under 8 characters wide while its text does not fit, or on more lines than it has words |
|
|
212
217
|
| clipped text | part of an element's text shows and part runs past a container that hides its overflow (`overflow: hidden` or `clip`), with no `text-overflow: ellipsis` (sideways) or line clamp (downwards): a card title cut to one line |
|
|
213
218
|
| cut off | part of an element shows and part lies past the left, right or top edge of the viewport: a status-bar clock clipped at the top. The page is at its top, so content below the bottom edge is fine |
|
|
219
|
+
| action did not complete | an `--act <selector>` names nothing, a disabled control, or one covered where its centre is (a toast over a button), or its click changes nothing a user can read within 5 seconds: the page's visible text, its element count and its address all stay the same. A card that moves, a reply that shows as saved or a toast that says so is a change; a button that is only there is not |
|
|
220
|
+
| unstyled control | with a `look` block: a button, select, textarea or input drawn as the browser's own, its background and border the ones the browser gives the same control (same tag, input type and colour scheme) on a page with no styles. Checkbox, radio, range, colour and file inputs are not measured |
|
|
221
|
+
| oversized control | with `look.max_control`: a button, select, input or `role=button` taller than that many px. A textarea or list box is not measured |
|
|
222
|
+
| differs from the approved design | with `--mock <file>`: the [design check](#design-check)'s judge (the one the user set) names a defect on the capture next to that approved mock, the same request as `design --mock`, cached the same way |
|
|
214
223
|
|
|
215
224
|
Something entirely outside its container or the viewport is hidden on purpose (a closed drawer, a visually hidden label) and is not a hit; a cut element's children are not listed again. Content in a scrolling container (`overflow: auto` or `scroll`, such as a carousel) scrolls into view and is not clipped.
|
|
216
|
-
- **
|
|
225
|
+
- **The approved reference.** A `look` block in `.done-done.yaml` is the project's approved design reference; with none, look behaves as before:
|
|
226
|
+
|
|
227
|
+
```yaml
|
|
228
|
+
look:
|
|
229
|
+
tokens: web/tokens.css # the approved design tokens file, relative to the repo
|
|
230
|
+
max_control: --ctl-h # the tallest a control may be: px (44), or a token in tokens set in px, rem or em (a rem or em is 16 px)
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
The block alone turns on the unstyled-control check; `max_control` adds the size check. A token that is not set in the file, a tokens file that does not read or an unknown key is a config error (exit 2). The per-page reference is its mock: pass `--mock <file>` with the approved design of that screen. The `--act` clicks are real mouse clicks at each control's centre, in order, and the page they leave is the one measured and captured, so a journey drives the page to the state to check with them.
|
|
234
|
+
- **What it writes.** `--out <file.png>` (default `look.png` in `$DONE_DONE_EVIDENCE_ROOT` inside a journey, else in the current folder) gets the viewport screenshot, exactly the viewport size in pixels, and `<file>.dom.json` next to it the page-check record of [DOM assertions](#dom-assertions), with the page's console errors and failed requests too. look itself fails only on the kinds above; with a `dom` block, verify's check also reads the record and applies its allow lists to console errors and failed requests.
|
|
217
235
|
- **Chrome.** look starts its own headless Chrome or Chromium (`google-chrome`, `google-chrome-stable`, `chromium` or `chromium-browser` on PATH, or `DONE_DONE_CHROME` naming the binary) with a throwaway profile, and removes both when it ends. It runs Chrome without its sandbox so it starts in containers, so point it at your own app, not untrusted sites. No Chrome is a usage error (exit 2) that says how to install or point to one; a page that does not load (its network error, or no load event within 30 seconds) is exit 2 with the cause.
|
|
218
|
-
- **Limits.** Only the main frame is measured, not an embedded frame; a transform or `clip-path` that hides part of a box is not seen; nothing is
|
|
236
|
+
- **Limits.** Only the main frame is measured, not an embedded frame; a transform or `clip-path` that hides part of a box is not seen; nothing is scrolled first, and only `--act` clicks; drive any other state (typing, a URL per state) before you call look. Look at the capture too: a defect the DOM cannot see (a stretched image, a colour) is the [design check](#design-check)'s.
|
|
219
237
|
|
|
220
238
|
| Result | Exit |
|
|
221
239
|
|---|---|
|
|
222
240
|
| no hit | 0 |
|
|
223
241
|
| a hit of any kind | 1 |
|
|
224
|
-
| bad flags, no Chrome,
|
|
242
|
+
| bad flags, a bad `look` block, no Chrome, the page did not load, or `--mock` with no judge set | 2 |
|
|
243
|
+
| no hit, but the `--mock` judge is unsure | 3 |
|
|
225
244
|
|
|
226
245
|
### Lint check
|
|
227
246
|
|
|
@@ -248,7 +267,7 @@ A before/after change is judged as a pair. With `base_ref` set to a branch, tag
|
|
|
248
267
|
base_ref: origin/main
|
|
249
268
|
```
|
|
250
269
|
|
|
251
|
-
done-done checks the base ref out in a scratch git worktree of its own under the run folder (never your checkout) and removes it after. The same journey or agent drive runs there with the same `DONE_DONE_RUN_ID`, its own instance (when an `instance` block is set) and every matrix cell's env, and `DONE_DONE_EVIDENCE_ROOT` is the run folder's `before/`, so a journey must write its captures there, not to a path it builds itself. An agent drive is told it is at the base ref, where the outcome is not expected to hold. The run record keeps `base` (the ref and its commit). Each head capture pairs with the base capture of the same name, matrix cell suffix included (`home.dark.png` with `before/home.dark.png`): the judge gets
|
|
270
|
+
done-done checks the base ref out in a scratch git worktree of its own under the run folder (never your checkout) and removes it after. The same journey or agent drive runs there with the same `DONE_DONE_RUN_ID`, its own instance (when an `instance` block is set) and every matrix cell's env, and `DONE_DONE_EVIDENCE_ROOT` is the run folder's `before/`, so a journey must write its captures there, not to a path it builds itself. An agent drive is told it is at the base ref, where the outcome is not expected to hold. The run record keeps `base` (the ref and its commit). Each head capture pairs with the base capture of the same name, matrix cell suffix included (`home.dark.png` with `before/home.dark.png`): the judge gets all head/base pairs together in the check's one request (two images, or two texts for each text pair) and passes each only when its after capture shows the outcome as a change from before; a base capture is never judged against the outcome alone, and it may match its head capture byte for byte (the judge then sees no change). A head capture with no base capture of its name is reported (inconclusive, exit 3, with why the base drive failed when it did) and judged alone. Without `base_ref` (the default), each capture is judged alone in its question, alongside the check's other capture questions in the same request.
|
|
252
271
|
|
|
253
272
|
### Real-product evidence
|
|
254
273
|
|
|
@@ -288,7 +307,7 @@ A proof that drives some feature of the app does not show the app works when the
|
|
|
288
307
|
|
|
289
308
|
- The **core journey** is the one thing a user cannot live without, driven end to end on the real target the way a user does it: for a remote terminal app, pair, open a pane, type, see the output. A config names it with `journeys.<name>.core: true` (`init --journey <name> --core -- <command>`); a code-only journey never counts. A repo with no journeys names it in its verify skill's `features/README.md` with an entry line (`- ...`) that carries `(core journey)`.
|
|
290
309
|
- The **upgrade journey** is the release path users take: install the previous shipped version, upgrade over it in place (install the new APK over the old one, update the host or CLI package in place), then drive the core journey. It is needed when a tracked file shows the app ships an installable or upgradable artifact: a `package.json` with a `bin` that is not `"private": true`, an `eas.json`, an `AndroidManifest.xml`, a `build.gradle` with an `applicationId`, a `pyproject.toml` with `[project.scripts]` or `[tool.poetry.scripts]`, a `Cargo.toml` with `[[bin]]`, a `.goreleaser.yml`, or a systemd `*.service` unit. A config names it with `journeys.<name>.upgrade: true` (`init --journey upgrade --upgrade -- <command>`), or the feature map with `(upgrade journey)`. One journey may be both.
|
|
291
|
-
- `doctor`'s `core_journey` and `upgrade_journey` rows are required,
|
|
310
|
+
- `doctor`'s `core_journey` and `upgrade_journey` rows are required. `init` still requires exactly one core feature, but accepts `first_release: true` when the repo explicitly says no earlier release yet: it writes the verify skill and records upgrade proof as pending, not passed. `maintain` and `affected` then work from the generated map. `doctor` still reports the upgrade journey missing until one is named; after the first publication, add the in-place upgrade journey before the next release. Unknown release history is not a first-release exemption.
|
|
292
311
|
|
|
293
312
|
```yaml
|
|
294
313
|
journeys:
|
|
@@ -349,7 +368,7 @@ evidence: <paths of the evidence files, or "not proven: <cause>">
|
|
|
349
368
|
|---|---|
|
|
350
369
|
| Not empty | The file must contain bytes. |
|
|
351
370
|
| Valid media | The magic bytes match the extension. PNG and JPEG are complete (end markers present) and at least 64×64. PNG is not a blank frame. WebP and GIF are not truncated. MP4/MOV and WebM/MKV decode at least two video packets (`ffprobe`). Text and log files hold text; JSON parses. |
|
|
352
|
-
| Fresh | The file is in a run record whose commit is HEAD.
|
|
371
|
+
| Fresh | The file is in a run record whose commit is HEAD. Two exceptions: if the only commits since the run touch `ignore_paths`, the evidence still counts; and if the changed files touch no journey or feature the run covered (the same mapping as `done-done affected`, model-free), the proof holds and the result warns with both commits. A change that does touch the run's journey is stale. A receipt keeps each run's `commit` beside its own `head`. An empty commit does not count as touching only `ignore_paths`. |
|
|
353
372
|
| Clean | The run was not dirty, unless `evidence.allow_dirty: true`. The run command also exited 0. |
|
|
354
373
|
| Not reused | Fails when the files are byte-identical within one claim. Also fails when every file was already used to prove a different claim in this repo. A claim is named by `--claim-id <id>`, by default the current branch (with `--from <PR URL>`, the PR's branch; on a detached HEAD, the commit), so rewording the outcome is not reuse. One branch is one claim: several claims on one branch each need their own `--claim-id`, or they share one reuse scope. The ledger is `<root>/.done-done/ledger` in the evidence root, so each repo keeps its own (pin `app`, as `init` does, so two repos with the same directory name do not share one); `DONE_DONE_HOME` moves it only with the default evidence root. |
|
|
355
374
|
|
|
@@ -371,14 +390,14 @@ done-done verify --outcome "<what a user can now do>" # always judged
|
|
|
371
390
|
done-done check --from pr-body.md # always judged
|
|
372
391
|
```
|
|
373
392
|
|
|
374
|
-
- **Always on, done-done's own agent.** done-done gives no result without its judge: `check`, `receipt`, `verify`, `rerun` and `design` always start it, and there is no flag or key that turns it off. The checks above still run as steps, but they never give a result on their own: a check that fails stops the command with its causes before the judge (exit 2, no verdict), and a pass needs the judge to pass at least one capture, so a check that cites only files the judge does not read (video, other files) stops with exit 2 and no result, before the judge is asked. A judge that fails (a model error, a timeout, a used-up plan, an answer it cannot read) is an error with its cause: `check`, `receipt`, `design` and `verify` exit 2 with no result, and a `verify` run stops at its judge step with its cause in the run record. Neither writes a receipt. `judge.enabled` and the `--no-judge` flag are gone: the key is a config error (exit 2) that says to delete the line, and the flag is an unknown option. `check --judge <route>` (the form `judge.route` takes: `claude-code`, `answerer:opencode`, `answerer:pi`, `answerer:gemini` or `
|
|
393
|
+
- **Always on, done-done's own agent.** done-done gives no result without its judge: `check`, `receipt`, `verify`, `rerun` and `design` always start it, and there is no flag or key that turns it off. The checks above still run as steps, but they never give a result on their own: a check that fails stops the command with its causes before the judge (exit 2, no verdict), and a pass needs the judge to pass at least one capture, so a check that cites only files the judge does not read (video, other files) stops with exit 2 and no result, before the judge is asked. A judge that fails (a model error, a timeout, a used-up plan, an answer it cannot read) is an error with its cause: `check`, `receipt`, `design` and `verify` exit 2 with no result, and a `verify` run stops at its judge step with its cause in the run record. Neither writes a receipt. `judge.enabled` and the `--no-judge` flag are gone: the key is a config error (exit 2) that says to delete the line, and the flag is an unknown option. `check --judge <route>` (the form `judge.route` takes: `claude-code`, `answerer:opencode`, `answerer:pi`, `answerer:gemini`, `openai:key` or `minimax:key`) chooses the judge for that call only, over the config's; it saves nothing, and its model is that agent's default, or `judge.model` when the route is the configured one.
|
|
375
394
|
|
|
376
395
|
- **Set by the user, never picked.** The judge route is exactly `judge.route` from the global config: no route auto-pick or PATH detection. The model is `judge.model` when set, otherwise the agent's configured/default model (Pi defaults to `openai-codex/gpt-6-luna`). While none is set, a command that needs the judge asks in a terminal (a numbered list of judges, then of models) and saves the answer to the global config; anywhere else (an agent calling done-done) it exits 2 with `no judge is set: ask the user which judge to use (claude, opencode, pi, gemini); never pick one for them` and the command that saves the answer: `done-done init --judge <agent> --judge-model <model>` (`claude` is saved as `judge.route: claude-code`, `opencode`, `pi` and `gemini` as `answerer:<agent>`). An empty `judge.model` means that agent's `agent_config` model, else its cheap model; pi uses `openai-codex/gpt-6-luna` when no `agent_config.pi.model` is set. `judge.route: auto` from older configs is a config error (exit 2) naming the global config file; it says to delete the `judge.route` line in that file, then run that command (the file then loads, and the command saves the judge).
|
|
377
|
-
- **One question per capture.** Each cited capture that is in a done-done run record
|
|
396
|
+
- **One question per capture, one call per check.** Each cited capture that is in a done-done run record gets its own question in one judge call for the check, with the claim's `outcome:` line and no repo or paths. Its answer gives that capture's verdict and, where supported, the parts it shows. The judge reads PNG and JPEG screens as images, and text captures (`.txt`, `.log`, `.json`, `.jsonl`, `.out`, `.err`, `.stdout`, `.stderr`, `.md`, `.csv`, `.xml`, `.html`, `.toon`) as text, asking whether the text shows the outcome; a wrong text capture fails as `judge says wrong output`. A text capture over 64 KiB is never cut, and a screen that cannot be downscaled to the judge size is never sent: either stops the command with its cause (exit 2, no verdict) before any model is asked, so capture only the output the outcome is about. Video, WebP, GIF and other files are not judged, and the result lists them as `not_judged`. With `base_ref`, a head screen and its base screen are one question, and so are a head text capture and its base text capture (the judge sees both texts); a screen is never paired with text, see [Base-ref before](#base-ref-before).
|
|
378
397
|
- **Where a capture goes.** Each capture goes to the provider of the judge the user set, which may be a different provider than the author's; doctor's `judge_choice` row names it.
|
|
379
|
-
- **A fresh, tools-off call on your own plan, through BYOKit.**
|
|
398
|
+
- **A fresh, tools-off call on your own plan, through BYOKit.** The check's capture questions go together in one single-shot call through [`@byokit/decide`](https://www.npmjs.com/package/@byokit/decide) on a route set by `judge.route`. `openai:key` keeps its existing per-capture path because that route does not return the required answer fields:
|
|
380
399
|
- `claude-code`: your own `claude` binary (`agent_path_override.claude`, else PATH), run the way the [agent](#agents) runs it with your normal `claude` sign-in: a fresh session, every tool off, `--safe-mode` and `--setting-sources ''` keeping your CLAUDE.md, hooks, plugins and MCP servers out of the call, and no session persistence. done-done keeps no sign-in folder of its own and sets neither `HOME` nor `CLAUDE_CONFIG_DIR` for the call. A capture goes as an image content block (stream-json input). Your `agent_args_override.claude` flags and `agent_config.claude.effort` also apply to these judge calls, as they do to every claude call.
|
|
381
|
-
- `answerer:<agent>`: BYOKit's answerer calls that coding agent's CLI through the same adapter as the [agent](#agents) itself, in a scratch folder holding only
|
|
400
|
+
- `answerer:<agent>`: BYOKit's answerer calls that coding agent's CLI through the same adapter as the [agent](#agents) itself, in a scratch folder holding only this check's captures, with every tool off and the captures attached (for example `answerer:opencode` on `opencode-go/muse-spark-1.3-contributor`). It uses the agent's existing login. How each CLI runs tools-off:
|
|
382
401
|
- `opencode`: `OPENCODE_PERMISSION` denies every tool; the capture is an attached file (`-f`). OpenCode's free-tier models refuse every call made with every tool off (HTTP 403 "OpenCode's free tier can only be used from within OpenCode"), so the opencode judge needs a `judge.model` that is not on that free tier; the error names this fix.
|
|
383
402
|
- `pi`: `--no-tools`, and no skills, prompt templates or context files, so none of your own pi setup reaches the judge, and no extensions but the one that supplies the chosen model's provider; the capture is an `@<file>` argument. The model is `judge.model`, else `agent_config.pi.model`, else `openai-codex/gpt-6-luna`; it must be a model that reads images: a built-in pi provider's model (`pi --list-models` shows which do), or a claude-bridge model written as `claude-bridge/<model>`, so done-done loads the bridge extension for the call. A bare model id gets no extension.
|
|
384
403
|
- `gemini`: plan mode plus a deny-all `--policy` file, which replaces your own policy folder and removes every tool, MCP ones too; the capture is an `@<file>` in the prompt, which Gemini CLI's own `@`-command code reads (outside the tool policy, per its source; not yet proven live, since Gemini CLI was not signed in on the test machine), only inside the scratch folder.
|
|
@@ -390,9 +409,9 @@ done-done check --from pr-body.md # always judged
|
|
|
390
409
|
|
|
391
410
|
Apart from that file, done-done never reads, copies or stores a credential. The kit is a regular dependency, installed with done-done; it needs Node 22.18 or later. The `claude-code` route uses the `claude` you already signed in with (`claude auth login`, as usual); there is no second sign-in.
|
|
392
411
|
|
|
393
|
-
- **Bounded.** Captures longer than 1280 px on their long side are downscaled first to PNG (8-bit PNG or JPEG). One request
|
|
412
|
+
- **Bounded.** Captures longer than 1280 px on their long side are downscaled first to PNG (8-bit PNG or JPEG). One request covers all captures in a check (including base-ref pairs), except `openai:key`, which retains one request per capture and its 10-request cap; `agent_timeout` bounds each request. An answer that cannot be read (a question left out, or probabilities that do not sum to 1) is asked once more; one that still cannot be read is an error with its cause (no result) and is not cached.
|
|
394
413
|
- **Any judge the user sets runs.** The judge is a fresh session of its own, so it may be the author's own agent and model, and the fixer's too under `auto_fix`; done-done never refuses or warns about the user's choice. The author is the coding agent done-done starts (`agent:`), or `no agent set`. An `acp:<target>` author counts as the agent its target names (`acp:claude-code` is claude), and an author with no known model is never `model` apart. The result and doctor's `judge_choice` row name how far apart they are: `agent` (another agent), `model` (the same agent at another model) or `session-only` (the same agent and model, in a fresh session). `judge.allow_same` is gone: a config that still sets it is a config error (exit 2) that says to delete the line.
|
|
395
|
-
- **Cached and recorded.** Verdicts are kept in `$DONE_DONE_HOME/runs/<app>/judge/`, keyed by the capture
|
|
414
|
+
- **Cached and recorded.** Verdicts are kept in `$DONE_DONE_HOME/runs/<app>/judge/`, keyed by the full capture set (and each base capture's sha256, for a pair), the outcome text, the route, the model and `judge.floor`, so a re-run asks no model. Each answerer call writes its agent's 0600 call record (what ran and what came back; never the prompt text) to the run's `agents/` folder under `verify`, else to `$DONE_DONE_HOME/agents/`; the `claude-code` route writes one too, since it runs your `claude` the same way.
|
|
396
415
|
- **Only runs on a passing check.** When the deterministic checks already fail, no capture is judged, and a judge pass never turns a failed check into a pass.
|
|
397
416
|
|
|
398
417
|
| Verdict | Result | Exit |
|
|
@@ -404,7 +423,7 @@ done-done check --from pr-body.md # always judged
|
|
|
404
423
|
|
|
405
424
|
Under `verify`, a `fail` or `unsure` verdict does not end the run: it parks it at its [gate](#the-gate-and-respond).
|
|
406
425
|
|
|
407
|
-
The result gains a `judge` block with the route, the model, the independence, the author and, for each judged file, its verdict, a short reason and whether it came from the cache, and, when the outcome names several parts, the parts it lists and the positions of the ones it shows (`parts`, `covers`). For the `claude-code` route, a missing `claude` binary or an `agent_path_override.claude` that is not an executable file, or a Node older than 22.18 is a config error (exit 2) whose `help` line is the command to fix it. A `claude` that is not signed in (checked only when a call fails) is exit 2 too, with `claude auth login` as the `help` line, the same sign-in you use in your shell. done-done does not check which plan that sign-in is on, so `claude` uses whatever sign-in your environment gives it. The kit cannot tell a bad `judge.model` from a used-up plan, so the error names both. For `openai:key`, an unset `OPENAI_API_KEY` is a config error (exit 2) too, and a refused key or a bad `judge.model` is an error with the HTTP status, naming both. The judge's confidence is the model's own estimate, not a calibrated score. `judge.floor` (global config, a number from 0 to 1, default `0.7`) is the least probability the judge's pick may carry and still count: a pick below it is `unsure` with `not confident enough (confidence <n> below floor <f>)`, and never a pass. It governs every judge: check and receipt relevance, maintain triage, and the design check. An outcome that names several parts (widths, themes or frames, such as "at 390 and 1440 wide") needs each one shown by a passing capture:
|
|
426
|
+
The result gains a `judge` block with the route, the model, the independence, the author and, for each judged file, its verdict, a short reason and whether it came from the cache, and, when the outcome names several parts, the parts it lists and the positions of the ones it shows (`parts`, `covers`). For the `claude-code` route, a missing `claude` binary or an `agent_path_override.claude` that is not an executable file, or a Node older than 22.18 is a config error (exit 2) whose `help` line is the command to fix it. A `claude` that is not signed in (checked only when a call fails) is exit 2 too, with `claude auth login` as the `help` line, the same sign-in you use in your shell. done-done does not check which plan that sign-in is on, so `claude` uses whatever sign-in your environment gives it. The kit cannot tell a bad `judge.model` from a used-up plan, so the error names both. For `openai:key`, an unset `OPENAI_API_KEY` is a config error (exit 2) too, and a refused key or a bad `judge.model` is an error with the HTTP status, naming both. The judge's confidence is the model's own estimate, not a calibrated score. `judge.floor` (global config, a number from 0 to 1, default `0.7`) is the least probability the judge's pick may carry and still count: a pick below it is `unsure` with `not confident enough (confidence <n> below floor <f>)`, and never a pass. It governs every judge: check and receipt relevance, maintain triage, and the design check. An outcome that names several parts (widths, themes or frames, such as "at 390 and 1440 wide") needs each one shown by a passing capture: the single call returns the parts and the positions each capture shows, and the check matches them by name, never by position, so a call that lists them in another order or writes a label with other case, spacing or trailing punctuation still covers the same part; a part named in a reply that no passing capture shows makes the check inconclusive, naming that part. The lists are the judge kit's answer fields (`answerFields` on the question), so they arrive on the answer the kit resolves; the `openai:key` route is asked for no such field, so it requires no part, and on every other route a reply that leaves either list out, or names a position its own parts list has no part at, is asked once more, and one that still does is an error (exit 2). Each request asks the judge to name the element the outcome claims and quote what it sees before it decides, and a text capture (JSON, a log) is read entry by entry, every width and frame, before the judge answers; the reason shown is the model's one-sentence person reason, else its rationale; each judged file also shows the model's rationale, the quote it read, under `seen` when the model gave one. The floor is part of the verdict cache key, so a changed floor asks the judge again.
|
|
408
427
|
|
|
409
428
|
### Design check
|
|
410
429
|
|
|
@@ -442,6 +461,27 @@ done-done design shots/home.png --mock shots/home.png=design/home.png
|
|
|
442
461
|
|
|
443
462
|
The result lists each screen (failing first, then unsure, then passing; `--full` for more than five) with its `verdict`, its `mock` when it had one, its `defects` (each a `kind` and `where`: the element and where it is on the screen) and a `reason` when it is unsure, plus the `judge` block (route, model, independence and author) and the counts. A missing screen or mock file, or a `--mock` without `=`, is a usage error (exit 2). The judge reads only what is visible, so a wide logo mark that is meant to be wide can fail without a mock; pass its mock to settle it.
|
|
444
463
|
|
|
464
|
+
### Approved-design coverage
|
|
465
|
+
|
|
466
|
+
Commit the approved prototype's inventory as an `approved_design` manifest in `.done-done.yaml`. Export one row for **every screen × size × theme** in the approved shots; the manifest, not the files a claim happens to cite, is the build contract:
|
|
467
|
+
|
|
468
|
+
```yaml
|
|
469
|
+
approved_design:
|
|
470
|
+
- { screen: home, size: 390x844, theme: light, mock: design/home.phone.light.png, capture: home.phone.light.png }
|
|
471
|
+
- { screen: home, size: 1280x800, theme: dark, mock: design/home.desktop.dark.png, capture: home.desktop.dark.png }
|
|
472
|
+
- { screen: today, size: 390x844, theme: light, mock: design/today.phone.light.png, capture: today.phone.light.png }
|
|
473
|
+
```
|
|
474
|
+
|
|
475
|
+
`mock` is a repo-relative approved PNG/JPEG; `capture` is the unique PNG/JPEG **filename** the live recipe writes into its run folder. `size` is the raw image's width × height in pixels (`390x844`), not a device label. The list must be non-empty, tuples and capture filenames unique, and paths cannot contain `..`. Commit the manifest and approved mocks before capturing. Each mock must match its committed bytes (from `--trusted-ref` when supplied, otherwise HEAD); an uncommitted or edited reference cannot silently redefine approval. No folder-name guessing or model infers the inventory: the approver must enumerate all prototype shots, including navigation and responsive states that need their own screen.
|
|
476
|
+
|
|
477
|
+
Capture the **real product**, not the prototype, using `look --out "$DONE_DONE_EVIDENCE_ROOT/<capture>"` or the project's native capture recipe, then cite every required live image in `check` or `receipt` (or let `verify` cite the run). The existing run provenance, clean-tree, freshness, step-record and media checks still apply. Mocks need not be cited as live evidence.
|
|
478
|
+
|
|
479
|
+
When this manifest is configured, **every claim** is gated, including design-done and release claims; wording the outcome differently cannot bypass it. Each required filename must identify exactly one cited, run-bound live screen of the declared pixel size. Missing or ambiguous captures fail **before calling a judge**, with exit 1 and `result: fail`, naming the screen / size / theme. Unrecorded images cannot fill a missing screen. Existing invalid-evidence errors remain errors (exit 2).
|
|
480
|
+
|
|
481
|
+
After those checks pass, each live image is sent beside its approved mock through the existing [design judge](#design-check), in addition to the claim relevance judge. A mismatch (including missing navigation or elements) **or an unsure match** fails the claim (exit 1), naming its tuple. An unreadable mock, wrong-sized mock or judge error stops with its cause (exit 2); it never passes. The result's `design` list carries every tuple, live path and hash, mock path and hash, and per-screen verdict. Receipts keep that list under `judge.approved_design` and bind the approved mock bytes too, so `receipt --verify` detects an edited reference. `gate` validates the recorded passing comparisons without making a new model call, as with relevance verdicts.
|
|
482
|
+
|
|
483
|
+
With no manifest, claims behave as before. `design` and `look --mock` remain ad-hoc measurements of supplied screens, not coverage claims: use `check`, `receipt` or `verify` to prove the complete approved inventory. This checks the declared inventory; it cannot discover an approved shot omitted from the manifest or establish capture authenticity beyond the existing run record.
|
|
484
|
+
|
|
445
485
|
### Record a receipt
|
|
446
486
|
|
|
447
487
|
A receipt ties one passing check to one commit and to the exact bytes it cited, so a merge gate (or `gate` in CI) can re-verify the same proof later.
|
|
@@ -653,6 +693,8 @@ An `agent_config` entry merges with the defaults field by field, so `claude: { m
|
|
|
653
693
|
| `instance` | none; the app under test started per run, see [Run-instance harness](#run-instance-harness) |
|
|
654
694
|
| `matrix` | none, so each journey runs once; `theme`, `width`, `form` and `skip` set the [capture matrix](#capture-matrix) |
|
|
655
695
|
| `dom` | none; page-check records per capture, see [DOM assertions](#dom-assertions) |
|
|
696
|
+
| `look` | none; the look step's approved design reference (`tokens`, `max_control`), see [Look check](#look-check) |
|
|
697
|
+
| `approved_design` | none; committed screen × pixel size × theme manifest of mocks and required live filenames; gates all claims, see [Approved-design coverage](#approved-design-coverage) |
|
|
656
698
|
| `inputs` | `[]`; repo-relative paths or globs the drive must leave alone, see [Input guard](#input-guard) |
|
|
657
699
|
| `base_ref` | none; a ref `verify` also drives, pairing its captures with the head's, see [Base-ref before](#base-ref-before) |
|
|
658
700
|
| `evidence.root` | `<local_root>/<app>`, outside the repo. A relative path is under the repo; an absolute path is used as-is |
|
package/package.json
CHANGED
|
@@ -50,7 +50,11 @@ its findings and captures, and end it with `done-done respond --action fix` only
|
|
|
50
50
|
A journey that `spends` runs only after the user approves spending: `done-done run <journey>` refuses it, so ask the user and use `verify`.
|
|
51
51
|
|
|
52
52
|
For a web page, `done-done look <url>` is the look step a journey calls: it captures the page at a 390x844 phone viewport
|
|
53
|
-
with its page-check record and exits 1 on squashed, clipped or cut-off UI, naming each element.
|
|
53
|
+
with its page-check record and exits 1 on squashed, clipped or cut-off UI, naming each element. `--act <selector>` clicks a
|
|
54
|
+
control and fails when the page does not change (a card moves, a reply saves, a toast shows): a button that is only there
|
|
55
|
+
is not proof it works. A `look` block in `.done-done.yaml` (`tokens`, `max_control`) is the approved reference: look
|
|
56
|
+
then fails on a control drawn as the browser's own or taller than `max_control`, and `--mock <file>` has the judge compare
|
|
57
|
+
the page with its approved mock. A hit fails the journey;
|
|
54
58
|
never wrap it in `|| true` or log the exit and go on. A claim that a phone UI works needs a 390 px capture of the real
|
|
55
59
|
page and the page's scroll-container count as proof: a mock, a different width or a claim with no count is not proof. A journey in `.done-done.yaml` that calls it:
|
|
56
60
|
|