done-done-axi 0.4.0 → 0.4.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -1
- package/dist/cli.js +19 -3
- package/dist/config.js +10 -4
- package/dist/dom.js +2 -2
- package/dist/judge.js +25 -4
- package/docs/example-cli.md +38 -0
- package/docs/reference.md +5 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -159,7 +159,7 @@ Run `done-done` with no arguments to see the current config and the next step.
|
|
|
159
159
|
- ♻️ **Not reused**: Proof made only of files already used for a different claim fails, and so does citing the same file twice. A claim is named by `--claim-id <id>`, which defaults to your branch, so rewording it keeps its proof. Several claims on one branch each need their own `--claim-id`.
|
|
160
160
|
- 🧾 **Honest claims**: Reads a pull request's `## Checks` block — from a file, a URL, or stdin — and makes sure every line is filled in.
|
|
161
161
|
- 📌 **Verifiable later**: `receipt` records a passing check against the exact commit and the sha256 of every file it cited; `receipt --verify` re-checks it from any checkout of that commit on the same machine (a merge gate or a fresh clone), or in CI once the evidence folder is copied there ([how](docs/reference.md#where-proof-lives)).
|
|
162
|
-
- 👀 **Shows the outcome** (always): `check`, `receipt` and `verify` start done-done's own judge, a fresh, tools-off agent on your own plan or key (Claude Code, opencode, pi or Gemini CLI, or your own OpenAI API key), to say whether each capture is the right screen for the claim (and each text capture, such as a log, JSON or CLI output, the right output), and a wrong one fails. There is no way to turn it off: done-done gives no result without it. The checks above never pass on their own, and a check that cites only files the judge cannot read (a video) stops before the judge, with exit 2. You choose the judge route (`done-done init --judge <agent> --judge-model <model>`, or pick it from the list done-done shows in a terminal); for a Pi judge with no explicit model, done-done defaults to `openai-codex/gpt-6-luna` unless `agent_config.pi.model` is set. Any agent and model you choose runs, your coding agent's own too. A judge that fails or times out stops the command with its cause (exit 2; a `verify` run stops with exit 2 and its cause in the run record), and no receipt is written.
|
|
162
|
+
- 👀 **Shows the outcome** (always): `check`, `receipt` and `verify` start done-done's own judge, a fresh, tools-off agent on your own plan or key (Claude Code, opencode, pi or Gemini CLI, or your own OpenAI API key), to say whether each capture is the right screen for the claim (and each text capture, such as a log, JSON or CLI output, the right output), and a wrong one fails. There is no way to turn it off: done-done gives no result without it. The checks above never pass on their own, and a check that cites only files the judge cannot read (a video) stops before the judge, with exit 2. You choose the judge route (`done-done init --judge <agent> --judge-model <model>`, or pick it from the list done-done shows in a terminal; `done-done check --judge <route>` overrides it for one call); for a Pi judge with no explicit model, done-done defaults to `openai-codex/gpt-6-luna` unless `agent_config.pi.model` is set. Any agent and model you choose runs, your coding agent's own too. A judge that fails or times out stops the command with its cause (exit 2; a `verify` run stops with exit 2 and its cause in the run record), and no receipt is written.
|
|
163
163
|
- 🎨 **Looks right**: `done-done design <screens...> [--mock <screen>=<mock>]` has the same judge look at each captured screen, alone or next to its approved mock, and fails it on stretched or squashed images, clipped or squashed text, overlap, elements off-screen, blank or black strips and broken layout, naming each defect and where it is ([how](docs/reference.md#design-check)).
|
|
164
164
|
- 📐 **Fits the screen**: `done-done look <url>` opens a web page in headless Chrome at a 390x844 phone viewport, captures it, and fails the journey that calls it on squashed text, clipped text, elements cut off by an edge or a page wider than the screen, naming each element, with no model ([how](docs/reference.md#look-check)).
|
|
165
165
|
- 🧩 **Page is clean** (with a `dom` block): each captured screen also needs its page-check record. A page that overflows the viewport, logs a console error, has a failed request or shows squashed, clipped or cut-off UI fails, unless an allow list names it ([how](docs/reference.md#dom-assertions)).
|
|
@@ -188,6 +188,7 @@ Another fleet works the same way: add the include text to whatever your workers
|
|
|
188
188
|
## Learn more
|
|
189
189
|
|
|
190
190
|
- [Vision](VISION.md): the three-tool family, the rules they share, and where done-done goes next.
|
|
191
|
+
- [Worked example: a CLI project](docs/example-cli.md): set up, one journey, check and receipt on a command-line tool.
|
|
191
192
|
- [Full reference](docs/reference.md): every command and option, the Checks block rules, each file check, and both config files.
|
|
192
193
|
- [Report a problem or ask a question](https://github.com/umeranjum17/done-done-axi/issues)
|
|
193
194
|
|
package/dist/cli.js
CHANGED
|
@@ -10,7 +10,7 @@ import { parseArgs } from 'node:util';
|
|
|
10
10
|
import { encode } from '@toon-format/toon';
|
|
11
11
|
import { Document, parseDocument } from 'yaml';
|
|
12
12
|
import { ACPX_INSTALL, agentChoices, agentPath, agentRun, askAgent, editOnlyReason, findBin, JUDGES, modelFor, noAgent, noJudge, notStartable, routeOf, selectAgent, selectJudge, toollessReason } from './agents.js';
|
|
13
|
-
import { AGENTS, CHEAP, globalConfigText, home, inputPattern, isAgent, loadConfig, parseDuration, UsageError } from './config.js';
|
|
13
|
+
import { AGENTS, CHEAP, globalConfigText, home, inputPattern, isAgent, isJudgeRoute, loadConfig, parseDuration, UsageError } from './config.js';
|
|
14
14
|
import { backend, judge, judgeable, onPath, quote, screen } from './judge.js';
|
|
15
15
|
import { startInstance, stopInstance } from './instance.js';
|
|
16
16
|
import { attemptProblem, changedSince, commitFix, dirtyTracked, FIX_SCHEMA, inWorktree, maintainSteps, named, pathspec, readerProblem, repoState, triage } from './maintain.js';
|
|
@@ -93,6 +93,7 @@ const FLAGS = {
|
|
|
93
93
|
'--outcome "<claim>"': 'the outcome the evidence files prove; the files follow as arguments',
|
|
94
94
|
'--from <file | - | PR URL>': 'read the claim from the newest "## Checks" block of a file, stdin or a PR body',
|
|
95
95
|
'--trusted-ref <ref>': 'read .done-done.yaml from this git ref instead of the working tree (use in CI)',
|
|
96
|
+
'--judge <route>': 'check with this judge for this call instead of judge.route (same form: claude-code, answerer:opencode, answerer:pi, answerer:gemini or openai:key); its model is that agent\'s default, or judge.model when the route is the configured one',
|
|
96
97
|
'--citation-base <dir>': 'resolve evidence citations against this directory (default: cwd); absolute citations stay unchanged; does not change repo, config or run storage',
|
|
97
98
|
'--claim-id <id>': 'the claim the evidence proves; evidence already used for another claim id is reuse (default: the current branch, or the PR branch with --from <PR URL>; required on a detached HEAD)',
|
|
98
99
|
'--verify <head>': 'read-only: verify the receipt recorded for <head>',
|
|
@@ -154,8 +155,9 @@ const COMMANDS = {
|
|
|
154
155
|
usage: [
|
|
155
156
|
['done-done check --outcome "<claim>" <files...> [--claim-id <id>] [--trusted-ref <ref>] [--citation-base <dir>]', 'check evidence files against a claim'],
|
|
156
157
|
['done-done check --from <file | - | PR URL> [--claim-id <id>] [--trusted-ref <ref>] [--citation-base <dir>]', "check a claim's Checks block"],
|
|
158
|
+
['done-done check --judge answerer:pi --outcome "<claim>" <files...>', 'judge with this route for this call instead of judge.route'],
|
|
157
159
|
],
|
|
158
|
-
flags: ['--outcome "<claim>"', '--from <file | - | PR URL>', '--claim-id <id>', '--trusted-ref <ref>', '--citation-base <dir>'],
|
|
160
|
+
flags: ['--outcome "<claim>"', '--from <file | - | PR URL>', '--judge <route>', '--claim-id <id>', '--trusted-ref <ref>', '--citation-base <dir>'],
|
|
159
161
|
more: [CHECKS_TEXT],
|
|
160
162
|
},
|
|
161
163
|
receipt: {
|
|
@@ -589,9 +591,11 @@ function loadRecords(cfg) {
|
|
|
589
591
|
async function runCheck(argv, run) {
|
|
590
592
|
const { values, positionals } = parseArgs({
|
|
591
593
|
args: argv,
|
|
592
|
-
options: { outcome: { type: 'string' }, from: { type: 'string' }, 'claim-id': { type: 'string' }, 'trusted-ref': { type: 'string' }, 'citation-base': { type: 'string' } },
|
|
594
|
+
options: { outcome: { type: 'string' }, from: { type: 'string' }, judge: { type: 'string' }, 'claim-id': { type: 'string' }, 'trusted-ref': { type: 'string' }, 'citation-base': { type: 'string' } },
|
|
593
595
|
allowPositionals: true,
|
|
594
596
|
});
|
|
597
|
+
if (values.judge !== undefined && !isJudgeRoute(values.judge))
|
|
598
|
+
throw new UsageError(`--judge ${values.judge || '""'} is not a judge route; use claude-code, answerer:<agent> or a key route`, 'Same form as judge.route: claude-code, answerer:opencode, answerer:pi, answerer:gemini or openai:key');
|
|
595
599
|
if (values['citation-base']?.trim() === '')
|
|
596
600
|
throw new UsageError('--citation-base needs a non-empty directory');
|
|
597
601
|
if (!!values.outcome === !!values.from)
|
|
@@ -614,6 +618,9 @@ async function runCheck(argv, run) {
|
|
|
614
618
|
}
|
|
615
619
|
}
|
|
616
620
|
const cfg = run?.cfg ?? loadConfig(repo, projectText);
|
|
621
|
+
// --judge overrides judge.route for this call; a model set for the configured route is not the new route's, so it falls back to that agent's default.
|
|
622
|
+
if (values.judge !== undefined && values.judge !== cfg.judge.route)
|
|
623
|
+
cfg.judge = { ...cfg.judge, route: values.judge, model: '' };
|
|
617
624
|
const now = head(repo);
|
|
618
625
|
const findings = [];
|
|
619
626
|
let claim;
|
|
@@ -3026,6 +3033,15 @@ async function cmdInit(argv) {
|
|
|
3026
3033
|
throw new UsageError(`init --journey ${journey} needs a command after --`);
|
|
3027
3034
|
if (journey !== null && !/^[A-Za-z0-9][A-Za-z0-9_-]*$/.test(journey))
|
|
3028
3035
|
throw new UsageError(`invalid journey name ${journey}; use letters, digits, - or _`);
|
|
3036
|
+
// Only the judge named: that is a machine setting, so it changes the global config alone, in any directory.
|
|
3037
|
+
if (values.judge !== undefined && values.agent === undefined && values.hook === undefined && journey === null) {
|
|
3038
|
+
const cfg = loadConfig(process.cwd());
|
|
3039
|
+
const choice = { agent: values.judge, model: values['judge-model'] ?? (cfg.judge.route === routeOf(values.judge) ? cfg.judge.model : '') };
|
|
3040
|
+
checkJudge(cfg, choice.agent, choice.model);
|
|
3041
|
+
const globalConfig = writeGlobal();
|
|
3042
|
+
emit({ command: 'init', result: 'pass', global_config: globalConfig, judge: saveChoice('judge', choice), help: ['Set up a repo: cd <your app> && done-done init'] });
|
|
3043
|
+
return PASS;
|
|
3044
|
+
}
|
|
3029
3045
|
const repo = repoRoot();
|
|
3030
3046
|
const file = join(repo, '.done-done.yaml');
|
|
3031
3047
|
const existed = existsSync(file);
|
package/dist/config.js
CHANGED
|
@@ -20,6 +20,8 @@ export const ANSWERER_JUDGES = ['opencode', 'pi', 'gemini'];
|
|
|
20
20
|
// BYOKit key routes: a judge on the user's own API key, read from this environment variable, with no agent CLI.
|
|
21
21
|
export const KEY_ROUTES = { 'openai:key': 'OPENAI_API_KEY' };
|
|
22
22
|
// Or acp:<target>, any agent the user's acpx bridge can start; the target is one argv word, so it never reads as a flag.
|
|
23
|
+
const ROUTE_FORM = `claude-code (for claude), answerer:<agent> for one of ${ANSWERER_JUDGES.join(', ')}, or a key route (${Object.keys(KEY_ROUTES).join(', ')}); an acp:<target> agent cannot judge, since done-done cannot send it an image`;
|
|
24
|
+
export const isJudgeRoute = (v) => typeof v === 'string' && (v === 'claude-code' || Object.hasOwn(KEY_ROUTES, v) || ANSWERER_JUDGES.some((a) => v === `answerer:${a}`));
|
|
23
25
|
export const isAgent = (v) => AGENTS.includes(v) || (typeof v === 'string' && /^acp:[A-Za-z0-9][\w.-]*$/.test(v));
|
|
24
26
|
// The cost guard is this table: a default model for an agent here is always its cheap one, never a flagship,
|
|
25
27
|
// whatever the vendor; any other model runs only when the user names it. gemini's is the CLI's own
|
|
@@ -70,6 +72,7 @@ const KEYS = {
|
|
|
70
72
|
'judge.model': { scope: 'global', type: 'string' },
|
|
71
73
|
'judge.route': { scope: 'global', type: 'route' },
|
|
72
74
|
'judge.floor': { scope: 'global', type: 'number' },
|
|
75
|
+
'judge.env_file': { scope: 'global', type: 'string' },
|
|
73
76
|
};
|
|
74
77
|
const SECTIONS = new Set(['evidence', 'secrets', 'checks', 'judge']);
|
|
75
78
|
const MAPS = new Set(['auto_fix', 'agent_config', 'agent_args_override', 'agent_path_override']);
|
|
@@ -106,7 +109,7 @@ export function inputPattern(p) {
|
|
|
106
109
|
}
|
|
107
110
|
// A binary: a bare name looked up on PATH, or an absolute path (~ is the home folder).
|
|
108
111
|
const isBin = (v) => typeof v === 'string' && (/^[\w.-]+$/.test(v) || isAbsolute(tilde(v)));
|
|
109
|
-
const tilde = (p) => p.replace(/^~(?=\/|$)/, homedir());
|
|
112
|
+
export const tilde = (p) => p.replace(/^~(?=\/|$)/, homedir());
|
|
110
113
|
const isMap = (v) => !!v && typeof v === 'object' && !Array.isArray(v);
|
|
111
114
|
const isRegex = (v) => {
|
|
112
115
|
try {
|
|
@@ -151,7 +154,7 @@ function apply(target, text, file, scope) {
|
|
|
151
154
|
: spec.type === 'string[]' ? Array.isArray(value) && value.every((v) => typeof v === 'string')
|
|
152
155
|
: spec.type === 'inputs' ? Array.isArray(value) && value.every((v) => typeof v === 'string' && inputPattern(v) !== '' && !isAbsolute(v) && !v.split('/').includes('..'))
|
|
153
156
|
: spec.type === 'agent' ? isAgent(value)
|
|
154
|
-
: spec.type === 'route' ?
|
|
157
|
+
: spec.type === 'route' ? isJudgeRoute(value)
|
|
155
158
|
: spec.type === 'agent_config' ? isMap(value) && Object.entries(value).every(([a, c]) => isAgent(a) && isMap(c) && Object.entries(c).every(([k, m]) => ['model', 'effort'].includes(k) && typeof m === 'string'))
|
|
156
159
|
: spec.type === 'args' ? isMap(value) && Object.entries(value).every(([a, v]) => AGENTS.includes(a) && Array.isArray(v) && v.every((x) => typeof x === 'string'))
|
|
157
160
|
: spec.type === 'paths' ? isMap(value) && Object.entries(value).every(([a, v]) => AGENTS.includes(a) && isBin(v))
|
|
@@ -174,7 +177,7 @@ function apply(target, text, file, scope) {
|
|
|
174
177
|
number: 'a number from 0 to 1',
|
|
175
178
|
regexes: key === 'secrets.allow' ? 'a list of valid regexes' : 'a map of <name>: <valid regex>',
|
|
176
179
|
auto_fix: `a map of <step>: <attempts, 0 or more> for ${AUTO_FIX_STEPS.join(', ')}`,
|
|
177
|
-
route:
|
|
180
|
+
route: ROUTE_FORM,
|
|
178
181
|
inputs: 'a list of repo-relative paths or globs, not absolute and with no .. segment',
|
|
179
182
|
};
|
|
180
183
|
if (key === 'judge.route' && (value === 'auto' || value === ''))
|
|
@@ -271,7 +274,7 @@ function defaultConfig(repo) {
|
|
|
271
274
|
evidence: { local_root: join(home(), 'evidence'), root: '', allow_dirty: false, ignore_paths: [] },
|
|
272
275
|
secrets: { patterns: {}, allow: [] },
|
|
273
276
|
checks: { fix_title: '^(fix|bugfix|hotfix)\\b' },
|
|
274
|
-
judge: { route: '', model: '', floor: 0.7 },
|
|
277
|
+
judge: { route: '', model: '', floor: 0.7, env_file: '' },
|
|
275
278
|
trusted: false,
|
|
276
279
|
from: {},
|
|
277
280
|
};
|
|
@@ -353,5 +356,8 @@ judge:
|
|
|
353
356
|
model: ""
|
|
354
357
|
# Every judge's pick must reach this probability to count: below it the verdict is unsure (inconclusive), never a pass.
|
|
355
358
|
floor: 0.7
|
|
359
|
+
# A file of KEY=value lines (a provider API key) handed to the judge's agent, or to openai:key, for names your shell has not
|
|
360
|
+
# exported, so the judge works in any shell. The key stays in that file: done-done never copies or prints it.
|
|
361
|
+
# env_file: ~/.config/done-done/judge.env
|
|
356
362
|
`;
|
|
357
363
|
}
|
package/dist/dom.js
CHANGED
|
@@ -12,8 +12,7 @@ import { basename } from 'node:path';
|
|
|
12
12
|
// The measurement the recipe passes to its browser tool's eval, so no agent invents one: every visible element with its
|
|
13
13
|
// own non-empty text is squashed when its box is under 8 characters wide (box width / the average glyph width of its
|
|
14
14
|
// font) while its text is wider than the box, or when its text sits on more line boxes than it has words.
|
|
15
|
-
// ponytail: glyph width ignores
|
|
16
|
-
// label measures a little narrow and a wrapped paragraph of Chinese or Japanese is one word; count words per script if those show up.
|
|
15
|
+
// ponytail: glyph width ignores text-transform, so an uppercased label measures a little narrow, and words split on spaces, so a wrapped paragraph of Chinese or Japanese is one word; count words per script if those show up.
|
|
17
16
|
export const SQUASHED_TEXT = `() => {
|
|
18
17
|
const ctx = document.createElement('canvas').getContext('2d'); const out = [];
|
|
19
18
|
for (const e of document.querySelectorAll('body *')) {
|
|
@@ -22,6 +21,7 @@ export const SQUASHED_TEXT = `() => {
|
|
|
22
21
|
const s = getComputedStyle(e); const w = e.offsetWidth; const h = e.offsetHeight;
|
|
23
22
|
if (!(e instanceof HTMLElement) || !text || !e.getClientRects().length || ((w <= 1 || h <= 1) && s.overflow !== 'visible') || s.visibility !== 'visible') continue;
|
|
24
23
|
ctx.font = s.fontStyle + ' ' + s.fontWeight + ' ' + s.fontSize + ' ' + s.fontFamily;
|
|
24
|
+
ctx.letterSpacing = s.letterSpacing === 'normal' ? '0px' : s.letterSpacing;
|
|
25
25
|
const chars = w / (ctx.measureText('abcdefghijklmnopqrstuvwxyz').width / 26); const words = text.split(' ').length; const tops = new Set();
|
|
26
26
|
for (const n of nodes) { const r = document.createRange(); r.selectNodeContents(n); for (const b of r.getClientRects()) if (b.width) tops.add(Math.round(b.top)); }
|
|
27
27
|
if ((chars < 8 && ctx.measureText(text).width > w + 1) || tops.size > words)
|
package/dist/judge.js
CHANGED
|
@@ -3,7 +3,7 @@ import { accessSync, constants, mkdirSync, readFileSync, renameSync, statSync, w
|
|
|
3
3
|
import { tmpdir } from 'node:os';
|
|
4
4
|
import { delimiter, isAbsolute, join } from 'node:path';
|
|
5
5
|
import { agentRun, askAgent, notStartable } from './agents.js';
|
|
6
|
-
import { home, KEY_ROUTES, parseDuration, UsageError } from './config.js';
|
|
6
|
+
import { home, KEY_ROUTES, parseDuration, tilde, UsageError } from './config.js';
|
|
7
7
|
import { downscaleToPng } from './media.js';
|
|
8
8
|
const MAX_SIDE = 1280; // long edge sent to the model, in pixels
|
|
9
9
|
const MAX_CALLS = 10; // model requests per check; later captures are unsure
|
|
@@ -88,6 +88,27 @@ export function onPath(name) {
|
|
|
88
88
|
return '';
|
|
89
89
|
}
|
|
90
90
|
export const quote = (a) => (/^[\w@%+=:,./-]+$/.test(a) ? a : `'${a.replaceAll("'", `'\\''`)}'`);
|
|
91
|
+
// The variables in judge.env_file (KEY=value lines, `export ` and quotes allowed) that the shell has not set: a judge works
|
|
92
|
+
// without the provider key exported in every shell. Values are handed to the judge's call only, never printed or stored.
|
|
93
|
+
export function judgeEnv(cfg) {
|
|
94
|
+
const file = cfg.judge.env_file;
|
|
95
|
+
if (!file)
|
|
96
|
+
return {};
|
|
97
|
+
let text;
|
|
98
|
+
try {
|
|
99
|
+
text = readFileSync(tilde(file), 'utf8');
|
|
100
|
+
}
|
|
101
|
+
catch (e) {
|
|
102
|
+
throw new UsageError(`judge.env_file ${file} cannot be read (${e.code ?? 'error'})`, 'Fix judge.env_file in ~/.done-done/config.yaml, or create that file with KEY=value lines');
|
|
103
|
+
}
|
|
104
|
+
const env = {};
|
|
105
|
+
for (const line of text.split(/\r?\n/)) {
|
|
106
|
+
const m = /^\s*(?:export\s+)?([A-Za-z_]\w*)\s*=\s*(.*?)\s*$/.exec(line);
|
|
107
|
+
if (m && process.env[m[1]] === undefined)
|
|
108
|
+
env[m[1]] = m[2].replace(/^(["'])(.*)\1$/, '$2');
|
|
109
|
+
}
|
|
110
|
+
return env;
|
|
111
|
+
}
|
|
91
112
|
// The kit loads only when a capture is judged, so a command that judges nothing stays fast.
|
|
92
113
|
async function kit() {
|
|
93
114
|
const [major, minor] = process.versions.node.split('.').map(Number);
|
|
@@ -103,9 +124,9 @@ export async function backend(cfg, choice, record = () => '', questions = { rele
|
|
|
103
124
|
if (Object.hasOwn(KEY_ROUTES, choice.route)) {
|
|
104
125
|
const name = KEY_ROUTES[choice.route];
|
|
105
126
|
// The kit never reads the environment: done-done reads the key here and hands it only to the kit.
|
|
106
|
-
const key = process.env[name];
|
|
127
|
+
const key = process.env[name] ?? judgeEnv(cfg)[name];
|
|
107
128
|
if (!key)
|
|
108
|
-
throw new UsageError(`judge: ${choice.route} needs your own API key in ${name}, which is not set`, `Export ${name} with your key, then re-run this command`);
|
|
129
|
+
throw new UsageError(`judge: ${choice.route} needs your own API key in ${name}, which is not set`, `Export ${name} with your key, or list it in the file judge.env_file names, then re-run this command`);
|
|
109
130
|
const { openai, resolve } = await kit();
|
|
110
131
|
const ask = openai({ model: choice.model, key, supportsImages: true, maxRetries: 0 }).ask;
|
|
111
132
|
return { ask, resolve, lastError: () => '', detail: `${name} is set and the judge kit is present` };
|
|
@@ -129,7 +150,7 @@ export async function backend(cfg, choice, record = () => '', questions = { rele
|
|
|
129
150
|
requests++;
|
|
130
151
|
const attached = images.map((img) => ({ name: `${img.id}.${img.mime.split('/')[1]}`, bytes: Buffer.from(img.dataUrl.slice(img.dataUrl.indexOf(',') + 1), 'base64') }));
|
|
131
152
|
// A tool-less call runs in its own scratch folder, so the CLI sees the attached capture and nothing else.
|
|
132
|
-
const a = await askAgent({ ...run, prompt, schema: answerSchema(questions), cwd: tmpdir(), tools: 'none', safeMode: true, attempts, record: record(requests), images: attached });
|
|
153
|
+
const a = await askAgent({ ...run, prompt, schema: answerSchema(questions), cwd: tmpdir(), env: judgeEnv(cfg), tools: 'none', safeMode: true, attempts, record: record(requests), images: attached });
|
|
133
154
|
lastError = a.ok ? '' : a.error ?? 'no answer';
|
|
134
155
|
if (!a.ok)
|
|
135
156
|
throw new Error(lastError);
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Worked example: a CLI project
|
|
2
|
+
|
|
3
|
+
done-done on [learn-from-mistakes](https://github.com/umeranjum17/learn-from-mistakes), a command-line tool with no screen. The proof is a text capture: the output a user reads.
|
|
4
|
+
|
|
5
|
+
**1. Set up.** In a clone of the repo, name your agent and judge (`done-done` never picks them). `init` writes `.done-done.yaml`, and the agent writes a verify skill from the repo's own files:
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
done-done init --agent claude --model claude-haiku-5-5 --judge claude --judge-model claude-haiku-5-5
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
**2. One journey, the way a user starts.** It stages the tool's bundled example project in a throwaway folder, runs `init` and then `doctor` on it, and saves only `doctor`'s output into the run folder. Every capture in the run folder is judged against the claim on its own, so keep earlier steps (help, init) out of it:
|
|
12
|
+
|
|
13
|
+
```sh
|
|
14
|
+
done-done init --journey cli -- 'd=$(mktemp -d) && B=$PWD/bin/learn-from-mistakes-axi.mjs \
|
|
15
|
+
&& cp examples/defects.md examples/gate.sqlite examples/.learn-from-mistakes.yaml "$d" \
|
|
16
|
+
&& cd "$d" && git init -q && export LFM_HOME=$d/lfm \
|
|
17
|
+
&& node $B init --no-skill --agent claude --model claude-haiku-5-5 \
|
|
18
|
+
--judge-agent claude --judge-model claude-haiku-5-5 </dev/null >/dev/null \
|
|
19
|
+
&& node $B doctor > $DONE_DONE_EVIDENCE_ROOT/${RUN_ID}.doctor.txt; s=$?; rm -rf -- "$d"; exit $s'
|
|
20
|
+
git add -A && git commit -m "Add done-done"
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
**3. Run, check, receipt.** Commit first: `check` refuses proof from a dirty tree.
|
|
24
|
+
|
|
25
|
+
```sh
|
|
26
|
+
done-done run cli
|
|
27
|
+
done-done check --outcome "doctor, run on the example project, reports its sources and that the worker and judge agents are ready" ~/.done-done/evidence/learn-from-mistakes/*/*.doctor.txt
|
|
28
|
+
done-done receipt --outcome "<same claim>" <same files>
|
|
29
|
+
done-done receipt --verify "$(git rev-parse HEAD)"
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The judge reads the text and passes it (`result: pass`, exit 0); the receipt then verifies from any checkout of that commit.
|
|
33
|
+
|
|
34
|
+
What the example shows:
|
|
35
|
+
|
|
36
|
+
- A CLI needs no screenshot: a log or command output is a capture, and the judge reads it.
|
|
37
|
+
- Cite only files that show the claim. A claim spread over several captures (help, guide and doctor together) came back `inconclusive`, because each file must show it alone.
|
|
38
|
+
- A journey that makes a real model call spends from your plan; this one calls no model (`doctor` only checks that the agents are installed and signed in).
|
package/docs/reference.md
CHANGED
|
@@ -284,7 +284,7 @@ done-done verify --outcome "<what a user can now do>" # always judged
|
|
|
284
284
|
done-done check --from pr-body.md # always judged
|
|
285
285
|
```
|
|
286
286
|
|
|
287
|
-
- **Always on, done-done's own agent.** done-done gives no result without its judge: `check`, `receipt`, `verify`, `rerun` and `design` always start it, and there is no flag or key that turns it off. The checks above still run as steps, but they never give a result on their own: a check that fails stops the command with its causes before the judge (exit 2, no verdict), and a pass needs the judge to pass at least one capture, so a check that cites only files the judge does not read (video, other files) stops with exit 2 and no result, before the judge is asked. A judge that fails (a model error, a timeout, a used-up plan, an answer it cannot read) is an error with its cause: `check`, `receipt`, `design` and `verify` exit 2 with no result, and a `verify` run stops at its judge step with its cause in the run record. Neither writes a receipt. `judge.enabled` and the `--
|
|
287
|
+
- **Always on, done-done's own agent.** done-done gives no result without its judge: `check`, `receipt`, `verify`, `rerun` and `design` always start it, and there is no flag or key that turns it off. The checks above still run as steps, but they never give a result on their own: a check that fails stops the command with its causes before the judge (exit 2, no verdict), and a pass needs the judge to pass at least one capture, so a check that cites only files the judge does not read (video, other files) stops with exit 2 and no result, before the judge is asked. A judge that fails (a model error, a timeout, a used-up plan, an answer it cannot read) is an error with its cause: `check`, `receipt`, `design` and `verify` exit 2 with no result, and a `verify` run stops at its judge step with its cause in the run record. Neither writes a receipt. `judge.enabled` and the `--no-judge` flag are gone: the key is a config error (exit 2) that says to delete the line, and the flag is an unknown option. `check --judge <route>` (the form `judge.route` takes: `claude-code`, `answerer:opencode`, `answerer:pi`, `answerer:gemini` or `openai:key`) chooses the judge for that call only, over the config's; it saves nothing, and its model is that agent's default, or `judge.model` when the route is the configured one.
|
|
288
288
|
|
|
289
289
|
- **Set by the user, never picked.** The judge route is exactly `judge.route` from the global config: no route auto-pick or PATH detection. The model is `judge.model` when set, otherwise the agent's configured/default model (Pi defaults to `openai-codex/gpt-6-luna`). While none is set, a command that needs the judge asks in a terminal (a numbered list of judges, then of models) and saves the answer to the global config; anywhere else (an agent calling done-done) it exits 2 with `no judge is set: ask the user which judge to use (claude, opencode, pi, gemini); never pick one for them` and the command that saves the answer: `done-done init --judge <agent> --judge-model <model>` (`claude` is saved as `judge.route: claude-code`, `opencode`, `pi` and `gemini` as `answerer:<agent>`). An empty `judge.model` means that agent's `agent_config` model, else its cheap model; pi uses `openai-codex/gpt-6-luna` when no `agent_config.pi.model` is set. `judge.route: auto` from older configs is a config error (exit 2) naming the global config file; it says to delete the `judge.route` line in that file, then run that command (the file then loads, and the command saves the judge).
|
|
290
290
|
- **One question per capture.** Each cited capture that is in a done-done run record goes to the model with the claim's `outcome:` line and nothing else: no repo, no paths, no other files. The judge reads PNG and JPEG screens as images, and text captures (`.txt`, `.log`, `.json`, `.jsonl`, `.out`, `.err`, `.stdout`, `.stderr`, `.md`, `.csv`, `.xml`, `.html`, `.toon`) as text, asking whether the text shows the outcome; a wrong text capture fails as `judge says wrong output`. A text capture over 64 KiB is never cut, and a screen that cannot be downscaled to the judge size is never sent: either stops the command with its cause (exit 2, no verdict) before any model is asked, so capture only the output the outcome is about. Video, WebP, GIF and other files are not judged, and the result lists them as `not_judged`. With `base_ref`, a head screen and its base screen are one question, and so are a head text capture and its base text capture (the judge sees both texts); a screen is never paired with text, see [Base-ref before](#base-ref-before).
|
|
@@ -298,7 +298,9 @@ done-done check --from pr-body.md # always judged
|
|
|
298
298
|
- An `acp:<target>` agent cannot judge, since done-done cannot send it an image; doctor's `config` row says so when `judge.route` names one.
|
|
299
299
|
- `openai:key`: a key-only route with no agent CLI. BYOKit's OpenAI backend sends the capture to the OpenAI Responses API on your own API key, which done-done reads from `OPENAI_API_KEY` and hands only to the kit (never printed or stored). Set `judge.model` to a model your key can use that reads images; there is no default. API billing applies.
|
|
300
300
|
|
|
301
|
-
|
|
301
|
+
A provider key for an `answerer:<agent>` judge, or `OPENAI_API_KEY` for `openai:key`, is read from the environment of whoever runs done-done. To work in every shell, point `judge.env_file` (global config) at a file of `KEY=value` lines (`export` and quotes allowed); its names that the shell has not exported go to the judge's call only. done-done never writes a key into a config file and never prints one; `init --judge` on its own changes only the global config, in any directory.
|
|
302
|
+
|
|
303
|
+
Apart from that file, done-done never reads, copies or stores a credential. The kit is a regular dependency, installed with done-done; it needs Node 22.18 or later. The `claude-code` route uses the `claude` you already signed in with (`claude auth login`, as usual); there is no second sign-in.
|
|
302
304
|
|
|
303
305
|
- **Bounded.** Captures longer than 1280 px on their long side are downscaled first to PNG (8-bit PNG or JPEG). One request per capture (or per base-ref pair), at most 10 requests per check (later captures are unsure), `agent_timeout` per request. An answer that cannot be read (a question left out, or probabilities that do not sum to 1) is asked once more; one that still cannot be read is an error with its cause (no result) and is not cached.
|
|
304
306
|
- **Any judge the user sets runs.** The judge is a fresh session of its own, so it may be the author's own agent and model, and the fixer's too under `auto_fix`; done-done never refuses or warns about the user's choice. The author is the coding agent done-done starts (`agent:`), or `no agent set`. An `acp:<target>` author counts as the agent its target names (`acp:claude-code` is claude), and an author with no known model is never `model` apart. The result and doctor's `judge_choice` row name how far apart they are: `agent` (another agent), `model` (the same agent at another model) or `session-only` (the same agent and model, in a fresh session). `judge.allow_same` is gone: a config that still sets it is a config error (exit 2) that says to delete the line.
|
|
@@ -514,6 +516,7 @@ judge: # done-done's own judge; check, receipt, v
|
|
|
514
516
|
# route: claude-code # claude-code, answerer:<agent> (opencode, pi, gemini) or openai:key (reads OPENAI_API_KEY), set by you (`init --judge` saves the first three; `openai:key` is set by hand) and unset until then
|
|
515
517
|
model: '' # default: that agent's agent_config model, or openai-codex/gpt-6-luna for pi
|
|
516
518
|
floor: 0.7 # every judge's pick must reach this probability to count; below it the verdict is unsure (inconclusive), never a pass
|
|
519
|
+
# env_file: ~/.config/done-done/judge.env # KEY=value lines (a provider API key) handed to the judge for names your shell has not exported; the key stays in that file
|
|
517
520
|
```
|
|
518
521
|
|
|
519
522
|
An `agent_config` entry merges with the defaults field by field, so `claude: { model: claude-sonnet-5-5 }` keeps `effort: low`; the other maps merge entry by entry. `done-done doctor` shows which file set each agent setting.
|
package/package.json
CHANGED