@xsft/panels 0.8.1 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +58 -13
- package/bin/panels.js +8 -4
- package/lib/args.js +18 -4
- package/lib/auth.js +74 -0
- package/lib/judge.js +14 -4
- package/lib/panel.js +10 -9
- package/lib/report.js +12 -8
- package/lib/table.js +304 -386
- package/package.json +1 -2
- package/prompts/judge-system-prompt.txt +11 -13
- package/schema/report.json +7 -2
package/README.md
CHANGED
|
@@ -3,14 +3,17 @@
|
|
|
3
3
|
`panels run` runs a panel from cross.software/panels against a target. Each judge is a `claude` or `codex` session that scores one criterion of the panel. You need Node 20 or newer and one of the two harnesses.
|
|
4
4
|
|
|
5
5
|
```text
|
|
6
|
+
panels login | logout | whoami
|
|
7
|
+
|
|
6
8
|
panels run <panel> [target...] [flags]
|
|
7
9
|
|
|
8
|
-
<panel>
|
|
10
|
+
<panel> pnl_SID for latest; pnl_SID@ver_SID or pnl_SID@N to pin a version
|
|
11
|
+
owner/name and owner/name@N remain accepted
|
|
9
12
|
[target...] free words, joined with single spaces into one target
|
|
10
13
|
|
|
11
14
|
--harness <name> claude | codex default: claude if installed, else codex
|
|
12
15
|
--yolo the judges skip every permission question (claude --dangerously-skip-permissions, codex --yolo).
|
|
13
|
-
Without it, a judge asks its own questions in
|
|
16
|
+
Without it, a judge asks its own questions in the Agent pane. Without a terminal, the flag is required.
|
|
14
17
|
-- end of the flags; each later word joins the target
|
|
15
18
|
|
|
16
19
|
exit 0 pass: every criterion scored 4
|
|
@@ -18,43 +21,85 @@ exit 1 fail: every judge answered, and one or more criteria scored below 4
|
|
|
18
21
|
exit 2 error: bad flags, no target, panel not found, harness missing, or a judge failed
|
|
19
22
|
```
|
|
20
23
|
|
|
21
|
-
CAUTION: A judge gets no tool block and no sandbox from the CLI. Without `--yolo`, the harness asks its own permission questions in the
|
|
24
|
+
CAUTION: A judge gets no tool block and no sandbox from the CLI. Without `--yolo`, the harness asks its own permission questions in the Agent pane. With `--yolo`, nothing asks: the judge can read, write, run commands, and reach the network as you, and the system prompt (do not write or change files of the target, no forms, no posts) is the only guard. Do not run a panel on a repository that you do not trust.
|
|
22
25
|
|
|
23
26
|
```sh
|
|
24
27
|
npx @xsft/panels run xsft/marketing-site https://example.com --harness claude --yolo
|
|
25
28
|
npx @xsft/panels run xsft/marketing-site@8 . at http://localhost:4321 --harness codex
|
|
26
29
|
```
|
|
27
30
|
|
|
28
|
-
|
|
31
|
+
Copy the run command from the panel page to use its permanent panel ID. An SID has a type prefix and 32 lowercase hexadecimal digits. A version SID pins one version of that panel. The CLI still accepts `owner/name` and `owner/name@N`.
|
|
32
|
+
|
|
33
|
+
Run the command in the folder of the product. Plain-mode judges and terminal-mode `claude` start in that folder. Terminal-mode `codex` starts in its judge folder; its prompt names the product folder with an absolute path. The CLI does not test, fetch, or change the target.
|
|
34
|
+
|
|
35
|
+
Up to ten judges run at a time in terminal mode, or four in plain mode. Terminal mode marks a judge as failed after 20 minutes without an answer. Each plain-mode request gets 20 minutes, including the second request after an invalid answer.
|
|
36
|
+
|
|
37
|
+
The criteria of a panel say what to assess; the system prompt of the CLI (`prompts/judge-system-prompt.txt`) carries how to judge and the four levels: 4 Met, 3 Almost met, 2 Falls short, 1 Fails. Only 4 passes.
|
|
38
|
+
|
|
39
|
+
## Account access
|
|
40
|
+
|
|
41
|
+
Public panels need no login. Team panels require a current team membership. Private panels require the original author's account.
|
|
42
|
+
|
|
43
|
+
Run `npx -y @xsft/panels@latest login` before you run a team or private panel. Open the URL that the CLI prints. Log in, compare the code with your terminal, then approve the request. The request expires after ten minutes.
|
|
44
|
+
|
|
45
|
+
`panels whoami` shows the current account. `panels logout` revokes the CLI session and deletes the local token. CLI sessions expire after 30 days. A password change revokes every CLI session and approved login request.
|
|
46
|
+
|
|
47
|
+
The CLI stores its token in `~/.config/panels/auth.json`, outside the target folder. The file permits access only to its owner. The token applies only to the exact Panels API address. The CLI sends it only to that API and refuses HTTP redirects. It does not copy the token into judge prompts, environment variables, or reports.
|
|
29
48
|
|
|
30
49
|
## Terminal mode
|
|
31
50
|
|
|
32
|
-
On a terminal, `panels run` shows
|
|
51
|
+
On a terminal, `panels run` shows a criterion list and a detail pane. Each judge runs in its interactive harness inside a pseudo-terminal. Each judge writes `judges/N/answer.json` and stays open. The CLI writes the report when every judge has an answer or an error. When you quit, the command exits with the status from `report.json`.
|
|
52
|
+
|
|
53
|
+
The header shows the panel, target, version, harness, elapsed time, and key hints. At 100 columns or more, the list and detail appear side by side. At smaller widths, the list appears above the detail.
|
|
54
|
+
|
|
55
|
+
Each list row shows the rank, criterion name, score, elapsed time, and token count. Scores use bright green for 4/4, green for 3/4, yellow for 2/4, and red for 1/4. A pending score shows `·/4`. The total row shows the completed count, scores as quarters, longest judge time, and total tokens.
|
|
56
|
+
|
|
57
|
+
The detail pane has three tabs:
|
|
58
|
+
|
|
59
|
+
- **Verdict:** the score and verdict, or an error. Markdown preserves paragraphs and lists, shows bold text and inline code, and indents fenced code blocks.
|
|
60
|
+
- **Agent:** the live session of the selected judge, sized to the pane.
|
|
61
|
+
- **Criterion:** the description and Markdown body of the selected criterion.
|
|
62
|
+
|
|
63
|
+
Agent is the default while the verdict is pending. Verdict is the default after the judge completes or fails. A manual tab choice stays with the criterion until its pending or completed state changes.
|
|
64
|
+
|
|
65
|
+
| Focus | Key | Action |
|
|
66
|
+
|---|---|---|
|
|
67
|
+
| List | `↑` / `↓` | Select a criterion |
|
|
68
|
+
| List | `←` / `→` | Select Verdict, Agent, or Criterion |
|
|
69
|
+
| List | `Enter` | Focus the detail pane |
|
|
70
|
+
| Verdict or Criterion | `←` / `→` | Select a tab |
|
|
71
|
+
| Verdict or Criterion | `↑` / `↓` | Scroll one line |
|
|
72
|
+
| Verdict or Criterion | `PgUp` / `PgDn` | Scroll one page |
|
|
73
|
+
| Verdict or Criterion | `Esc` or `Enter` | Return to the list |
|
|
74
|
+
| Agent | `Ctrl-Q` | Return to the list |
|
|
75
|
+
| Agent | Any other key | Send the key to the judge |
|
|
76
|
+
| List | `h` | Choose the harness for the next restart |
|
|
77
|
+
| List | `r` | Confirm a restart of the selected judge |
|
|
78
|
+
| List or text pane | `q` | Quit; press twice if judges still run |
|
|
33
79
|
|
|
34
|
-
The
|
|
80
|
+
The harness modal offers `claude` and `codex`. The restart modal selects cancel by default. A confirmed restart discards the selected judge's answer and starts that judge with its selected harness. In either modal, `↑` and `↓` select an option, `Enter` confirms it, and `Esc` closes the modal.
|
|
35
81
|
|
|
36
|
-
|
|
82
|
+
The first judge starts alone so you can answer a folder trust prompt. With the first criterion and Agent selected, press `Enter` to answer the prompt. Press `Ctrl-Q` to return to the list. The other judges start after the trust prompt clears.
|
|
37
83
|
|
|
38
|
-
|
|
84
|
+
Without a terminal, judges run through `claude -p` or `codex exec`. Stdout prints `N/4 name` and each verdict, then the report path. Plain mode requires `--yolo`; without it, the command exits 2.
|
|
39
85
|
|
|
40
86
|
## Run folder
|
|
41
87
|
|
|
42
88
|
Each run writes to `.panels/<owner>-<name>/<UTC time>/`:
|
|
43
89
|
|
|
44
90
|
- `panel.json`: the panel as the api gave it.
|
|
45
|
-
- `report.json`: `{ panel: "owner
|
|
91
|
+
- `report.json`: `{ sid: "run_SID", panel: "pnl_SID@ver_SID", owner, name, version, target, status, criteria: [{ sid, name, score, verdict }] }`, in panel order. The owner, name, and numeric version identify the panel in text output. The run SID stays the same when a judge restarts. Older reports can omit these fields, the run SID, and criterion SIDs. `schema/report.json` is its schema, and its `criterion` definition is the shape of the answer of a judge.
|
|
46
92
|
- `judges/<k>/`: the prompt, the output schema, and the raw answer of judge k. In terminal mode, `answer.json` takes the place of the raw answer.
|
|
47
93
|
|
|
48
|
-
|
|
94
|
+
In plain mode, the last line on stdout is the path of `report.json`.
|
|
49
95
|
|
|
50
96
|
## Skipped
|
|
51
97
|
|
|
52
98
|
- Run upload to cross.software, and `--no-upload`.
|
|
53
99
|
- `--version`: `owner/name@N` pins a version. The CLI fetches the panel from the api on every run and never caches it.
|
|
54
100
|
- `--out`: the run folder is fixed.
|
|
55
|
-
- `--json`: stdout ends with the path of `report.json`.
|
|
56
|
-
- `--concurrency`: fixed at
|
|
101
|
+
- `--json`: plain-mode stdout ends with the path of `report.json`.
|
|
102
|
+
- `--concurrency`: fixed at ten in terminal mode and four in plain mode.
|
|
57
103
|
- `--model`: each harness uses its default.
|
|
58
104
|
- A panel URL as `<panel>`, a `PANELS_API` override, and a panel cache.
|
|
59
|
-
- A login: an owner cannot run an unpublished panel.
|
|
60
105
|
- Cost totals, and `cap` and `gap` fields.
|
package/bin/panels.js
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
+
import { randomBytes } from 'node:crypto';
|
|
2
3
|
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
|
|
3
4
|
import { delimiter, join } from 'node:path';
|
|
5
|
+
import { authCommand } from '../lib/auth.js';
|
|
4
6
|
import { parse, SYNOPSIS } from '../lib/args.js';
|
|
5
7
|
import { runJudge } from '../lib/judge.js';
|
|
6
8
|
import { fetchPanel } from '../lib/panel.js';
|
|
@@ -28,8 +30,8 @@ async function pool(items, size, fn) {
|
|
|
28
30
|
const EXIT = { pass: 0, fail: 1, error: 2 };
|
|
29
31
|
|
|
30
32
|
/** Runs the judges in the overview, and exits from report.json when the engineer quits. Prints nothing. */
|
|
31
|
-
async function runOnTerminal({ panel, runDir, harness, cwd, args }) {
|
|
32
|
-
await runTable({ panel, runDir, harness, cwd, target: args.target, yolo: args.yolo });
|
|
33
|
+
async function runOnTerminal({ sid, panel, runDir, harness, cwd, args }) {
|
|
34
|
+
await runTable({ sid, panel, runDir, harness, cwd, target: args.target, yolo: args.yolo });
|
|
33
35
|
const reportPath = join(runDir, 'report.json');
|
|
34
36
|
if (!existsSync(reportPath)) return 2;
|
|
35
37
|
return EXIT[JSON.parse(readFileSync(reportPath, 'utf8')).status];
|
|
@@ -49,6 +51,7 @@ async function main() {
|
|
|
49
51
|
console.error(`panels: ${args.error}`);
|
|
50
52
|
return 2;
|
|
51
53
|
}
|
|
54
|
+
if (args.command) return authCommand(args.command);
|
|
52
55
|
const harness = args.harness ?? (onPath('claude') ? 'claude' : 'codex');
|
|
53
56
|
if (!onPath(harness)) {
|
|
54
57
|
console.error(`panels: harness missing: ${harness} is not on PATH`);
|
|
@@ -65,12 +68,13 @@ async function main() {
|
|
|
65
68
|
return 2;
|
|
66
69
|
}
|
|
67
70
|
const { panel } = fetched;
|
|
71
|
+
const sid = `run_${randomBytes(16).toString('hex')}`;
|
|
68
72
|
const cwd = process.cwd();
|
|
69
73
|
const time = new Date().toISOString().replaceAll(':', '-');
|
|
70
74
|
const runDir = join(cwd, '.panels', `${panel.owner}-${panel.name}`, time);
|
|
71
75
|
mkdirSync(runDir, { recursive: true });
|
|
72
76
|
writeFileSync(join(runDir, 'panel.json'), JSON.stringify(panel, null, 2));
|
|
73
|
-
if (process.stdout.isTTY && process.stdin.isTTY) return runOnTerminal({ panel, runDir, harness, cwd, args });
|
|
77
|
+
if (process.stdout.isTTY && process.stdin.isTTY) return runOnTerminal({ sid, panel, runDir, harness, cwd, args });
|
|
74
78
|
const outcomes = await pool(panel.criteria, PARALLEL_JUDGES, (c, i) => runJudge({
|
|
75
79
|
dir: join(runDir, 'judges', String(i + 1)),
|
|
76
80
|
criterion: { rank: i + 1, ...c },
|
|
@@ -79,7 +83,7 @@ async function main() {
|
|
|
79
83
|
target: args.target,
|
|
80
84
|
yolo: true,
|
|
81
85
|
}));
|
|
82
|
-
const report = buildReport({ panel, target: args.target, outcomes });
|
|
86
|
+
const report = buildReport({ sid, panel, target: args.target, outcomes });
|
|
83
87
|
const reportPath = join(runDir, 'report.json');
|
|
84
88
|
writeFileSync(reportPath, `${JSON.stringify(report, null, 2)}\n`);
|
|
85
89
|
console.log(renderPlain(report).join('\n'));
|
package/lib/args.js
CHANGED
|
@@ -1,8 +1,11 @@
|
|
|
1
1
|
import { parseArgs } from 'node:util';
|
|
2
2
|
|
|
3
|
-
export const SYNOPSIS = `panels
|
|
3
|
+
export const SYNOPSIS = `panels login | logout | whoami
|
|
4
4
|
|
|
5
|
-
<panel>
|
|
5
|
+
panels run <panel> [target...] [flags]
|
|
6
|
+
|
|
7
|
+
<panel> pnl_SID for latest; pnl_SID@ver_SID or pnl_SID@N to pin a version
|
|
8
|
+
owner/name and owner/name@N remain accepted
|
|
6
9
|
[target...] free words, joined with single spaces into one target
|
|
7
10
|
|
|
8
11
|
--harness <name> claude | codex default: claude if installed, else codex
|
|
@@ -21,6 +24,7 @@ const options = {
|
|
|
21
24
|
};
|
|
22
25
|
|
|
23
26
|
const PANEL = /^([a-z0-9-]+)\/([a-z0-9-]+)(?:@([1-9]\d*))?$/;
|
|
27
|
+
const PANEL_SID = /^(pnl_[a-f0-9]{32})(?:@(ver_[a-f0-9]{32}|[1-9]\d*))?$/;
|
|
24
28
|
|
|
25
29
|
/** Returns the run options, { help: true }, { usage: true }, or { error } with a one-line message. */
|
|
26
30
|
export function parse(argv) {
|
|
@@ -32,13 +36,23 @@ export function parse(argv) {
|
|
|
32
36
|
}
|
|
33
37
|
if (values.help) return { help: true };
|
|
34
38
|
const [command, panel, ...words] = positionals;
|
|
39
|
+
if (['login', 'logout', 'whoami'].includes(command)) {
|
|
40
|
+
if (panel !== undefined || values.harness !== undefined || values.yolo !== undefined) return { usage: true };
|
|
41
|
+
return { command };
|
|
42
|
+
}
|
|
35
43
|
if (command !== 'run' || panel === undefined) return { usage: true };
|
|
36
44
|
const match = PANEL.exec(panel);
|
|
37
|
-
|
|
45
|
+
const sid = PANEL_SID.exec(panel);
|
|
46
|
+
const version = match?.[3] ?? (sid?.[2]?.startsWith('ver_') ? undefined : sid?.[2]);
|
|
47
|
+
if ((!match && !sid) || (version !== undefined && !Number.isSafeInteger(Number(version)))) {
|
|
48
|
+
return { error: `bad panel ${panel}: use pnl_SID[@ver_SID|@N] or owner/name[@N]` };
|
|
49
|
+
}
|
|
38
50
|
if (words.length === 0) return { error: 'no target: type what the panel reviews after the panel name, such as a URL, a path, or a branch' };
|
|
39
51
|
if (values.harness !== undefined && !['claude', 'codex'].includes(values.harness)) return { error: `--harness needs claude or codex, not ${values.harness}` };
|
|
40
52
|
return {
|
|
41
|
-
panel:
|
|
53
|
+
panel: sid
|
|
54
|
+
? { sid: sid[1], ...(sid[2]?.startsWith('ver_') ? { versionSid: sid[2] } : { version: version && Number(version) }) }
|
|
55
|
+
: { owner: match[1], name: match[2], version: version && Number(version) },
|
|
42
56
|
target: words.join(' '),
|
|
43
57
|
harness: values.harness,
|
|
44
58
|
yolo: Boolean(values.yolo),
|
package/lib/auth.js
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
import { randomUUID } from 'node:crypto';
|
|
2
|
+
import { chmod, mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises';
|
|
3
|
+
import { homedir } from 'node:os';
|
|
4
|
+
import { dirname, join } from 'node:path';
|
|
5
|
+
import { setTimeout } from 'node:timers/promises';
|
|
6
|
+
|
|
7
|
+
export const API = 'https://cross.software/panels/api';
|
|
8
|
+
const credentials = () => join(homedir(), '.config', 'panels', 'auth.json');
|
|
9
|
+
|
|
10
|
+
export async function readToken(api = API, file = credentials()) {
|
|
11
|
+
const value = await readFile(file, 'utf8').then(JSON.parse, (error) => {
|
|
12
|
+
if (error.code === 'ENOENT') return null;
|
|
13
|
+
throw error;
|
|
14
|
+
});
|
|
15
|
+
return value?.api === api ? value.token : undefined;
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
export async function saveToken(token, api = API, file = credentials()) {
|
|
19
|
+
await mkdir(dirname(file), { recursive: true, mode: 0o700 });
|
|
20
|
+
await chmod(dirname(file), 0o700);
|
|
21
|
+
const temporary = `${file}.${randomUUID()}`;
|
|
22
|
+
await writeFile(temporary, `${JSON.stringify({ api, token })}\n`, { mode: 0o600, flag: 'wx' });
|
|
23
|
+
await rename(temporary, file);
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
export async function authFetch(path, { api = API, token, ...init } = {}) {
|
|
27
|
+
return fetch(`${api}${path}`, {
|
|
28
|
+
...init,
|
|
29
|
+
redirect: 'error',
|
|
30
|
+
cache: 'no-store',
|
|
31
|
+
headers: { ...init.headers, ...(token ? { Authorization: `Bearer ${token}` } : {}) },
|
|
32
|
+
});
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
export async function authCommand(command, { api = API, file = credentials(), log = console.log, wait = setTimeout } = {}) {
|
|
36
|
+
if (command === 'login') {
|
|
37
|
+
const response = await authFetch('/cli/device', { api, method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{}' });
|
|
38
|
+
if (!response.ok) throw new Error(`login failed: ${response.status}`);
|
|
39
|
+
const device = await response.json();
|
|
40
|
+
log(`Open ${device.verificationUri}`);
|
|
41
|
+
log(`Check code ${device.userCode}, then approve the login. This request expires in 10 minutes.`);
|
|
42
|
+
const expiresAt = Date.now() + device.expiresIn * 1000;
|
|
43
|
+
while (Date.now() < expiresAt) {
|
|
44
|
+
await wait(device.interval * 1000);
|
|
45
|
+
const answer = await authFetch('/cli/token', {
|
|
46
|
+
api, method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ deviceCode: device.deviceCode }),
|
|
47
|
+
});
|
|
48
|
+
if (answer.status === 202) continue;
|
|
49
|
+
if (!answer.ok) throw new Error('login expired or failed; run panels login again');
|
|
50
|
+
const { token, user } = await answer.json();
|
|
51
|
+
await saveToken(token, api, file);
|
|
52
|
+
log(`Logged in as ${user.handle}.`);
|
|
53
|
+
return 0;
|
|
54
|
+
}
|
|
55
|
+
throw new Error('login expired; run panels login again');
|
|
56
|
+
}
|
|
57
|
+
const token = await readToken(api, file);
|
|
58
|
+
if (!token) {
|
|
59
|
+
log('Not logged in. Run panels login.');
|
|
60
|
+
return command === 'logout' ? 0 : 2;
|
|
61
|
+
}
|
|
62
|
+
if (command === 'logout') {
|
|
63
|
+
const response = await authFetch('/cli/session', { api, token, method: 'DELETE', headers: { 'Content-Type': 'application/json' } });
|
|
64
|
+
if (!response.ok && response.status !== 401) throw new Error(`logout failed: ${response.status}`);
|
|
65
|
+
await rm(file, { force: true });
|
|
66
|
+
log('Logged out.');
|
|
67
|
+
return 0;
|
|
68
|
+
}
|
|
69
|
+
const response = await authFetch('/viewer', { api, token });
|
|
70
|
+
if (!response.ok) throw new Error(`account fetch failed: ${response.status}`);
|
|
71
|
+
const user = await response.json();
|
|
72
|
+
log(user ? `${user.handle} (${user.email})` : 'Login expired. Run panels login.');
|
|
73
|
+
return user ? 0 : 2;
|
|
74
|
+
}
|
package/lib/judge.js
CHANGED
|
@@ -4,6 +4,16 @@ import { join } from 'node:path';
|
|
|
4
4
|
|
|
5
5
|
const { criterion: CRITERION } = JSON.parse(readFileSync(new URL('../schema/report.json', import.meta.url), 'utf8')).$defs;
|
|
6
6
|
const SYSTEM_PROMPT = readFileSync(new URL('../prompts/judge-system-prompt.txt', import.meta.url), 'utf8');
|
|
7
|
+
/** The words of Navid to every judge of the terminal mode, as he wrote them on 2026-10-18. */
|
|
8
|
+
export const GUIDANCE = `You give a score from 1 to 4 and a verdict. Anyone may read it. Write so they understand it the first time and know what to do next.
|
|
9
|
+
|
|
10
|
+
A problem is useful only if someone can fix it. "This feels weak" cannot be fixed. "The page never says what goes wrong without the product" can. So say what is missing, where it should be, and show the words that are there now.
|
|
11
|
+
|
|
12
|
+
Find the problems first. Then score. 4 means you would put this site live today. Only 4 is a pass. The closer a problem is to what the visitor sees first, the lower the score.
|
|
13
|
+
|
|
14
|
+
Keep it short. A long paragraph that shows you looked at everything hides the reason for the score. If you skipped something, write "Not checked:" and name it. That is enough.
|
|
15
|
+
|
|
16
|
+
You will want to write down everything you saw, good parts first. Do not. Write what is missing.`;
|
|
7
17
|
export const MINUTES_PER_CRITERION = 20;
|
|
8
18
|
// A judge gets no tool list, no permission mode, and no sandbox flag: the harness defaults, and its own questions
|
|
9
19
|
// in its tab. With --yolo, the judge skips every question: no sandbox, nothing is denied, nothing asks. The system
|
|
@@ -35,7 +45,7 @@ export function check(answer, criterion) {
|
|
|
35
45
|
const errors = [];
|
|
36
46
|
if (answer.name !== criterion.name) errors.push(`name must be "${criterion.name}".`);
|
|
37
47
|
if (!Number.isInteger(answer.score) || answer.score < 1 || answer.score > 4) errors.push('score must be a whole number from 1 to 4.');
|
|
38
|
-
if (typeof answer.verdict !== 'string' || !answer.verdict.trim()) errors.push('verdict must be
|
|
48
|
+
if (typeof answer.verdict !== 'string' || !answer.verdict.trim()) errors.push('verdict must be text.');
|
|
39
49
|
return errors;
|
|
40
50
|
}
|
|
41
51
|
|
|
@@ -130,9 +140,9 @@ export const tomlString = (s) => `"${s.replace(/["\\\x00-\x1f\x7f]/g, (c) => TOM
|
|
|
130
140
|
*/
|
|
131
141
|
export function writeJudge({ dir, criterion, harness, cwd, target, yolo = false }) {
|
|
132
142
|
mkdirSync(dir, { recursive: true });
|
|
133
|
-
//
|
|
134
|
-
// work to a sub-agent still leaves the file at the exact path.
|
|
135
|
-
const closing =
|
|
143
|
+
// Navid's words to the judge (2026-10-18), verbatim, then the answer file rule. Both sit at the end of the user
|
|
144
|
+
// prompt and in the system prompt, so a judge that hands the work to a sub-agent still leaves the file at the exact path.
|
|
145
|
+
const closing = `${GUIDANCE}\n\nThe engineer ran the command in ${cwd}. Your answer counts only as the file ${join(dir, 'answer.json')}: write it with that exact absolute path, as one JSON object { name, score, verdict } that matches ${join(dir, 'schema.json')}. JSON in your reply does not count. Whoever does the work, that file must exist at that path when you stop. These two files are the only exceptions to the rules on writing files and on the .panels folder.\n`;
|
|
136
146
|
const prompt = `${userMessage(criterion, { target })}\n${closing}`;
|
|
137
147
|
const system = `${SYSTEM_PROMPT}\n${closing}`;
|
|
138
148
|
writeFileSync(join(dir, 'prompt.md'), prompt);
|
package/lib/panel.js
CHANGED
|
@@ -1,13 +1,14 @@
|
|
|
1
|
-
|
|
1
|
+
import { API, authFetch, readToken } from './auth.js';
|
|
2
2
|
|
|
3
|
-
/**
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
if (res.status ===
|
|
3
|
+
/** Reads the current panel or a pinned version. The API checks the account on every request. */
|
|
4
|
+
export async function fetchPanel({ owner, name, sid, version, versionSid }, { api = API, file } = {}) {
|
|
5
|
+
const token = await readToken(api, file);
|
|
6
|
+
const reference = sid ?? `${owner}/${name}`;
|
|
7
|
+
const pin = versionSid ? `/versions/${versionSid}` : version ? `/v/${version}` : '';
|
|
8
|
+
const res = await authFetch(`/panels/${reference}${pin}`, { api, token });
|
|
9
|
+
if (res.status === 401) return { error: 'login required: run panels login' };
|
|
10
|
+
if (res.status === 403) return { error: 'this account cannot access the panel' };
|
|
11
|
+
if (res.status === 404) return { error: token ? 'panel not found or this account has no access' : 'panel not found; for a team or private panel, run panels login' };
|
|
11
12
|
if (!res.ok) return { error: `panel fetch failed: ${res.status}` };
|
|
12
13
|
return { panel: await res.json() };
|
|
13
14
|
}
|
package/lib/report.js
CHANGED
|
@@ -4,18 +4,20 @@ import { readFileSync } from 'node:fs';
|
|
|
4
4
|
export const VERSION = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version;
|
|
5
5
|
|
|
6
6
|
/**
|
|
7
|
-
*
|
|
8
|
-
*
|
|
9
|
-
* gets score null and the error as its verdict, and the run status is error. The panel passes only when
|
|
10
|
-
* every criterion scores 4.
|
|
7
|
+
* Each outcome belongs to the criterion at the same index in panel.criteria.
|
|
8
|
+
* A failed judge gives score null and status error. Every criterion must score 4 for a pass.
|
|
11
9
|
*/
|
|
12
|
-
export function buildReport({ panel, target, outcomes }) {
|
|
10
|
+
export function buildReport({ sid, panel, target, outcomes }) {
|
|
13
11
|
const criteria = panel.criteria.map((c, i) => {
|
|
14
12
|
const { answer, error } = outcomes[i];
|
|
15
|
-
return
|
|
13
|
+
return { ...(c.sid ? { sid: c.sid } : {}), name: c.name, score: answer ? answer.score : null, verdict: answer ? answer.verdict : `error: ${error}` };
|
|
16
14
|
});
|
|
17
15
|
const status = criteria.some((c) => c.score === null) ? 'error' : criteria.every((c) => c.score === 4) ? 'pass' : 'fail';
|
|
18
|
-
return {
|
|
16
|
+
return {
|
|
17
|
+
sid,
|
|
18
|
+
panel: panel.sid ? `${panel.sid}@${panel.versionSid}` : `${panel.owner}/${panel.name}@${panel.version}`,
|
|
19
|
+
owner: panel.owner, name: panel.name, version: panel.version, target, status, criteria,
|
|
20
|
+
};
|
|
19
21
|
}
|
|
20
22
|
|
|
21
23
|
/**
|
|
@@ -23,6 +25,8 @@ export function buildReport({ panel, target, outcomes }) {
|
|
|
23
25
|
* "N/4 name" and the verdict under it, for each criterion.
|
|
24
26
|
*/
|
|
25
27
|
export function renderPlain(report) {
|
|
26
|
-
const [
|
|
28
|
+
const [reference, pin] = report.panel.split('@');
|
|
29
|
+
const name = report.owner ? `${report.owner}/${report.name}` : reference;
|
|
30
|
+
const version = report.version ?? pin;
|
|
27
31
|
return [`panels run ${name} v${version} ${report.target} ${VERSION}`, '', ...report.criteria.flatMap((c) => [`${c.score === null ? 'err' : `${c.score}/4`} ${c.name}`, ` ${c.verdict}`, ''])];
|
|
28
32
|
}
|