humanish 0.110.0 → 0.110.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +14 -11
- package/CHANGELOG.md +13 -0
- package/CONTEXT.md +8 -6
- package/README.md +96 -47
- package/dist/actors/computer-use/loop/ending.js +4 -4
- package/dist/actors/local-agent/claude-participant.d.ts +74 -0
- package/dist/actors/local-agent/claude-participant.js +243 -0
- package/dist/actors/local-agent/claude-session.d.ts +16 -3
- package/dist/actors/local-agent/claude-session.js +64 -23
- package/dist/actors/local-agent/cli.d.ts +11 -0
- package/dist/actors/local-agent/cli.js +103 -24
- package/dist/actors/local-agent/readiness.d.ts +2 -2
- package/dist/actors/local-agent/readiness.js +18 -11
- package/dist/cli/commands/analyze.js +21 -5
- package/dist/cli/commands/run-signals.d.ts +2 -0
- package/dist/cli/commands/run-signals.js +96 -44
- package/dist/cli/commands/runs.d.ts +5 -0
- package/dist/cli/commands/runs.js +86 -70
- package/dist/cli/commands/study-route-computer-use.js +2 -0
- package/dist/cli/commands/study-route-scripted.js +2 -0
- package/dist/cli/commands/study-route-shared-world.js +2 -0
- package/dist/cli/commands/study-route-terminal.js +2 -0
- package/dist/cli/doctor.d.ts +4 -1
- package/dist/cli/doctor.js +42 -4
- package/dist/cli/findings.d.ts +103 -0
- package/dist/cli/findings.js +316 -0
- package/dist/cli/first-run-path.d.ts +5 -4
- package/dist/cli/first-run-path.js +14 -7
- package/dist/cli/io.js +2 -1
- package/dist/cli/product-sentence.d.ts +3 -3
- package/dist/cli/product-sentence.js +3 -3
- package/dist/keys/env-file.d.ts +4 -0
- package/dist/keys/env-file.js +17 -0
- package/dist/observer/data.d.ts +14 -1
- package/dist/observer/data.js +19 -9
- package/dist/observer/library.d.ts +4 -0
- package/dist/observer/library.js +18 -15
- package/dist/observer/render.js +8 -2
- package/dist/observer/run-routes.d.ts +2 -0
- package/dist/observer/run-routes.js +43 -34
- package/dist/observer-app.html +3 -3
- package/dist/routes/computer-use/bundle.js +3 -1
- package/dist/routes/computer-use/fanout-records.js +1 -1
- package/dist/routes/computer-use/participant-execution.js +3 -1
- package/dist/routes/computer-use/participant-model.js +5 -2
- package/dist/routes/computer-use/plan.js +4 -4
- package/dist/routes/computer-use/preflight.js +1 -1
- package/dist/routes/computer-use/result.js +17 -28
- package/dist/routes/computer-use/single-bundle.js +7 -1
- package/dist/routes/scripted/bundle.js +4 -2
- package/dist/routes/scripted/plan.js +2 -2
- package/dist/routes/scripted/result.d.ts +3 -5
- package/dist/routes/scripted/result.js +16 -22
- package/dist/routes/scripted/scenario.js +4 -4
- package/dist/routes/shared-world/bundle-records.js +6 -1
- package/dist/routes/shared-world/bundle.js +2 -0
- package/dist/routes/shared-world/plan.js +1 -1
- package/dist/routes/shared-world/result.js +22 -22
- package/dist/routes/terminal/bundle.js +4 -2
- package/dist/routes/terminal/dry-run.js +8 -12
- package/dist/routes/terminal/ledger.d.ts +3 -3
- package/dist/routes/terminal/ledger.js +6 -6
- package/dist/routes/terminal/live-finish.js +15 -13
- package/dist/routes/terminal/plan.js +4 -4
- package/dist/routes/terminal/result.d.ts +6 -7
- package/dist/routes/terminal/result.js +6 -22
- package/dist/routes/terminal/session.js +1 -1
- package/dist/run/bundle-shape.d.ts +3 -1
- package/dist/run/bundle-shape.js +25 -1
- package/dist/run/bundle.d.ts +24 -0
- package/dist/run/display.d.ts +59 -0
- package/dist/run/display.js +106 -0
- package/dist/run/dry-run.js +11 -7
- package/dist/run/judge.d.ts +3 -2
- package/dist/run/judge.js +10 -0
- package/dist/run/pricing.js +1 -1
- package/dist/run/projection.d.ts +0 -2
- package/dist/run/projection.js +0 -11
- package/dist/run/reclaim-outcomes.d.ts +89 -0
- package/dist/run/reclaim-outcomes.js +213 -0
- package/dist/run/reclaim.d.ts +18 -17
- package/dist/run/reclaim.js +180 -113
- package/dist/run/results.d.ts +10 -20
- package/dist/run/results.js +2 -2
- package/dist/run/run-index.d.ts +11 -1
- package/dist/run/run-index.js +11 -0
- package/dist/run/run.d.ts +34 -18
- package/dist/run/run.js +166 -67
- package/dist/run/sandbox-creates.d.ts +59 -0
- package/dist/run/sandbox-creates.js +138 -0
- package/dist/run/sandbox-receipts.d.ts +14 -0
- package/dist/run/sandbox-receipts.js +44 -0
- package/dist/run/stats.d.ts +5 -0
- package/dist/run/stats.js +8 -4
- package/dist/run/status.d.ts +6 -9
- package/dist/run/status.js +7 -23
- package/dist/run/stored-runs.d.ts +5 -14
- package/dist/run/stored-runs.js +31 -146
- package/dist/run/streams.d.ts +3 -2
- package/dist/study/doctor.d.ts +1 -1
- package/dist/study/doctor.js +5 -2
- package/dist/study/init.js +14 -3
- package/dist/study/keys.d.ts +12 -0
- package/dist/study/keys.js +32 -0
- package/dist/study/persona.d.ts +4 -0
- package/dist/study/persona.js +4 -2
- package/dist/substrates/e2b/peer-install.d.ts +51 -0
- package/dist/substrates/e2b/peer-install.js +161 -0
- package/dist/substrates/e2b/sandbox.d.ts +34 -6
- package/dist/substrates/e2b/sandbox.js +128 -35
- package/dist/substrates/e2b/sdk.d.ts +33 -0
- package/dist/substrates/e2b/sdk.js +21 -21
- package/dist/substrates/shell.d.ts +2 -0
- package/dist/substrates/shell.js +4 -0
- package/dist/tui-app.js +133 -133
- package/dist/verify/liveness.d.ts +19 -3
- package/dist/verify/liveness.js +46 -25
- package/dist/verify/verify.d.ts +3 -0
- package/dist/verify/verify.js +6 -2
- package/docs/architecture/actor-contract.md +2 -1
- package/docs/architecture/observer.md +22 -7
- package/docs/architecture/task-protocol-support.md +13 -13
- package/docs/architecture/terminal-product-route.md +1 -1
- package/docs/contracts/run-bundle.md +75 -6
- package/docs/contracts/schemas.md +39 -22
- package/docs/contracts/study-analysis.md +45 -0
- package/docs/contracts/study-costs.md +8 -0
- package/docs/decisions/0001-run-bundle-is-the-source-of-truth.md +3 -0
- package/docs/decisions/0003-contained-paths-and-cleanup-authority.md +9 -3
- package/docs/principles/invariants-and-defaults.md +11 -8
- package/package.json +4 -3
- package/skills/humanish/SKILL.md +16 -2
|
@@ -1,7 +1,23 @@
|
|
|
1
1
|
import type { RunBundle } from "../run/bundle.js";
|
|
2
2
|
import type { PreparedRunArtifactPaths } from "../run/paths.js";
|
|
3
|
+
/** What verify can say about a run that is not finished. */
|
|
4
|
+
export interface UnfinishedRun {
|
|
5
|
+
liveness: "running" | "interrupted";
|
|
6
|
+
/**
|
|
7
|
+
* clean: the run's reclaim receipt records every sandbox gone and a finished search of E2B by
|
|
8
|
+
* the run's tags. unconfirmed: it records a sandbox not confirmed gone, or the run journaled one
|
|
9
|
+
* it does not cover. unknown: there is no reclaim receipt, or it records no finished tag search
|
|
10
|
+
* (written by humanish 0.110.0 or earlier, or by a reclaim that did not finish), so a sandbox whose
|
|
11
|
+
* id never reached a receipt could still run.
|
|
12
|
+
*/
|
|
13
|
+
sandboxes: "clean" | "unconfirmed" | "unknown";
|
|
14
|
+
}
|
|
3
15
|
/**
|
|
4
|
-
* One warning when the run is not finished, by the run index's
|
|
5
|
-
* has a usable one, else a participant still `running` in its
|
|
16
|
+
* One warning, and the unfinished-run facts, when the run is not finished, by the run index's
|
|
17
|
+
* rule: its status record when it has a usable one, else a participant still `running` in its
|
|
18
|
+
* bundle's `simulations[]`. Nothing for a finished run.
|
|
6
19
|
*/
|
|
7
|
-
export declare function
|
|
20
|
+
export declare function runNotFinished(runPaths: PreparedRunArtifactPaths, bundle: RunBundle, nowMs?: number): Promise<{
|
|
21
|
+
warnings: string[];
|
|
22
|
+
unfinished?: UnfinishedRun;
|
|
23
|
+
}>;
|
package/dist/verify/liveness.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
|
-
// `humanish verify`'s RUN_NOT_FINISHED warning
|
|
2
|
-
// and a run killed mid-way leaves an
|
|
3
|
-
//
|
|
1
|
+
// `humanish verify`'s RUN_NOT_FINISHED warning and the unfinished-run facts its one-line output
|
|
2
|
+
// leads with. Verify checks the integrity of what a run wrote, and a run killed mid-way leaves an
|
|
3
|
+
// in-progress bundle that passes those checks. The warning says what verify saw on disk, so
|
|
4
|
+
// `ok: true` is not read as "the run finished" or "its sandboxes stopped". It never flips `ok`.
|
|
4
5
|
import { sandboxIdDigest } from "../evidence/redaction.js";
|
|
5
6
|
import path from "node:path";
|
|
6
7
|
import { readRunJsonIfExists, readRunTextIfExists } from "../run/locate.js";
|
|
@@ -9,19 +10,21 @@ import { RECLAIM_RECEIPT_ARTIFACT } from "../run/reclaim.js";
|
|
|
9
10
|
import { runLiveness } from "../run/run-index.js";
|
|
10
11
|
import { parseSandboxReceipts, SANDBOX_RECEIPTS_ARTIFACT } from "../run/sandbox-receipts.js";
|
|
11
12
|
import { RUN_STATUS_FILE } from "../run/status.js";
|
|
13
|
+
import { plural } from "../run/text.js";
|
|
12
14
|
/** The stable code every RUN_NOT_FINISHED warning starts with. */
|
|
13
15
|
const RUN_NOT_FINISHED = "RUN_NOT_FINISHED";
|
|
14
16
|
const GONE_STATES = new Set(["killed", "already-gone"]);
|
|
15
17
|
/**
|
|
16
|
-
* One warning when the run is not finished, by the run index's
|
|
17
|
-
* has a usable one, else a participant still `running` in its
|
|
18
|
+
* One warning, and the unfinished-run facts, when the run is not finished, by the run index's
|
|
19
|
+
* rule: its status record when it has a usable one, else a participant still `running` in its
|
|
20
|
+
* bundle's `simulations[]`. Nothing for a finished run.
|
|
18
21
|
*/
|
|
19
|
-
export async function
|
|
22
|
+
export async function runNotFinished(runPaths, bundle, nowMs = Date.now()) {
|
|
20
23
|
const runId = path.basename(runPaths.absoluteRunRoot);
|
|
21
24
|
const status = await readRunJsonIfExists(runPaths, RUN_STATUS_FILE);
|
|
22
25
|
const { liveness, record } = runLiveness(runId, status, bundle, nowMs);
|
|
23
26
|
if (liveness === "finished")
|
|
24
|
-
return [];
|
|
27
|
+
return { warnings: [] };
|
|
25
28
|
const seen = [];
|
|
26
29
|
if (record === undefined) {
|
|
27
30
|
const running = bundle.simulations.filter((simulation) => simulation.status === "running");
|
|
@@ -39,30 +42,48 @@ export async function runNotFinishedWarnings(runPaths, bundle, nowMs = Date.now(
|
|
|
39
42
|
const runningStreams = bundle.streams.filter((stream) => stream.status === "running").length;
|
|
40
43
|
seen.push(`${runningStreams} of ${bundle.streams.length} streams are still running`);
|
|
41
44
|
const reclaim = await reclaimState(runPaths, runId);
|
|
42
|
-
return
|
|
43
|
-
|
|
44
|
-
|
|
45
|
+
return {
|
|
46
|
+
warnings: [
|
|
47
|
+
`${RUN_NOT_FINISHED}: ${seen.join("; ")}. ${reclaim.text} Verify ok covers the integrity of what was written; it does not mean the run finished.`,
|
|
48
|
+
],
|
|
49
|
+
unfinished: { liveness, sandboxes: reclaim.sandboxes },
|
|
50
|
+
};
|
|
45
51
|
}
|
|
46
|
-
/** What the reclaim receipt says about
|
|
52
|
+
/** What the reclaim receipt says about this run's sandboxes, and what to run. */
|
|
47
53
|
async function reclaimState(runPaths, runId) {
|
|
48
54
|
const journal = await readRunTextIfExists(runPaths, SANDBOX_RECEIPTS_ARTIFACT);
|
|
49
55
|
// Matched by digest: a reclaim receipt names each sandbox by digest, and one written before it
|
|
50
56
|
// did holds the raw id, which is digested here.
|
|
51
57
|
const journaled = new Set((journal === null ? [] : parseSandboxReceipts(journal)).map((receipt) => sandboxIdDigest(receipt.sandboxId)));
|
|
52
|
-
const
|
|
53
|
-
const
|
|
54
|
-
if (
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
}
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
58
|
+
const command = `\`humanish reclaim --run ${runId}\``;
|
|
59
|
+
const receipt = reclaimReceipt(await readRunJsonIfExists(runPaths, RECLAIM_RECEIPT_ARTIFACT), runId);
|
|
60
|
+
if (receipt === undefined)
|
|
61
|
+
return {
|
|
62
|
+
sandboxes: "unknown",
|
|
63
|
+
text: `Sandboxes unknown: it has no ${RECLAIM_RECEIPT_ARTIFACT} and journaled ${plural(journaled.size, "sandbox", "sandboxes")}; ${command} stops those and searches E2B by this run's tags for any whose id never reached a receipt.`,
|
|
64
|
+
};
|
|
65
|
+
const { outcomes } = receipt;
|
|
66
|
+
const gone = outcomes.filter((outcome) => GONE_STATES.has(outcome.state)).length;
|
|
67
|
+
const reclaimed = new Set(outcomes.map((outcome) => outcome.digest));
|
|
68
|
+
const missing = [...journaled].filter((id) => !reclaimed.has(id)).length;
|
|
69
|
+
const recorded = `${RECLAIM_RECEIPT_ARTIFACT} records ${gone} of ${outcomes.length} sandboxes gone${missing === 0 ? "" : `, and ${missing} journaled sandboxes are not in it`}`;
|
|
70
|
+
if (gone < outcomes.length || missing > 0)
|
|
71
|
+
return {
|
|
72
|
+
sandboxes: "unconfirmed",
|
|
73
|
+
text: `Sandboxes unconfirmed: ${recorded}; ${command} retries.`,
|
|
74
|
+
};
|
|
75
|
+
if (receipt.state === "clean")
|
|
76
|
+
return {
|
|
77
|
+
sandboxes: "clean",
|
|
78
|
+
text: `Sandboxes clean: ${recorded}, and E2B listed none still tagged with this run.`,
|
|
79
|
+
};
|
|
80
|
+
return {
|
|
81
|
+
sandboxes: "unknown",
|
|
82
|
+
text: `Sandboxes unknown: ${recorded}, but it records no finished search of E2B by this run's tags; ${command} searches.`,
|
|
83
|
+
};
|
|
63
84
|
}
|
|
64
|
-
/**
|
|
65
|
-
function
|
|
85
|
+
/** A reclaim receipt written for this run, or undefined when there is none. */
|
|
86
|
+
function reclaimReceipt(receipt, runId) {
|
|
66
87
|
if (!isRecord(receipt) || receipt.runId !== runId || !Array.isArray(receipt.outcomes))
|
|
67
88
|
return undefined;
|
|
68
89
|
const outcomes = [];
|
|
@@ -75,5 +96,5 @@ function reclaimOutcomes(receipt, runId) {
|
|
|
75
96
|
const digest = typeof recorded === "string" ? recorded : sandboxIdDigest(sandboxId);
|
|
76
97
|
outcomes.push({ digest, state });
|
|
77
98
|
}
|
|
78
|
-
return outcomes;
|
|
99
|
+
return typeof receipt.state === "string" ? { state: receipt.state, outcomes } : { outcomes };
|
|
79
100
|
}
|
package/dist/verify/verify.d.ts
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { type PreparedRunArtifactPaths } from "../run/paths.js";
|
|
2
|
+
import { type UnfinishedRun } from "./liveness.js";
|
|
2
3
|
export declare const VERIFY_SCHEMA = "humanish.verify-result.v1";
|
|
3
4
|
export interface VerifyResult {
|
|
4
5
|
schema: typeof VERIFY_SCHEMA;
|
|
@@ -21,6 +22,8 @@ export interface VerifyResult {
|
|
|
21
22
|
}>;
|
|
22
23
|
};
|
|
23
24
|
warnings: string[];
|
|
25
|
+
/** Present when the run is not finished: its liveness and what is known of its sandboxes. */
|
|
26
|
+
unfinished?: UnfinishedRun;
|
|
24
27
|
error?: {
|
|
25
28
|
code: "HUMANISH_RUN_NOT_FOUND" | "HUMANISH_INVALID_RUN_BUNDLE";
|
|
26
29
|
message: string;
|
package/dist/verify/verify.js
CHANGED
|
@@ -13,7 +13,7 @@ import { runNotFoundMessage } from "../run/run-not-found.js";
|
|
|
13
13
|
import { actorVerdictConsistencyFindings, noEngagementActorFindings, validateCodexAppServerEvidence, validateTerminalProductEvidence, } from "./actor.js";
|
|
14
14
|
import { invalidRunEvidenceReferences, MAX_REPORTED_FINDINGS, missingLocalEvidenceArtifacts, rawScreenshotPostureWarnings, rawScreenshotStreamIds, redactedShapeFramePaths, scanRunPublicSafetyArtifacts, streamScreenshotPaths, } from "./artifacts.js";
|
|
15
15
|
import { flatContextWarnings } from "./context-growth.js";
|
|
16
|
-
import {
|
|
16
|
+
import { runNotFinished } from "./liveness.js";
|
|
17
17
|
import { costAndReceiptFindings } from "./costs.js";
|
|
18
18
|
import { rerunLineageFindings } from "./rerun.js";
|
|
19
19
|
import { sharedWorldEvidenceFindings } from "./shared-world.js";
|
|
@@ -137,13 +137,16 @@ export async function verifyResolvedRun(cwd, runInput, runPaths) {
|
|
|
137
137
|
message: "Derived analysis contains sensitive text or unsafe artifact paths; sharing is blocked, original recording remains independently verifiable.",
|
|
138
138
|
});
|
|
139
139
|
const ok = checks.every((check) => check.ok);
|
|
140
|
+
const notFinished = isRunBundle(bundle)
|
|
141
|
+
? await runNotFinished(runPaths, bundle)
|
|
142
|
+
: { warnings: [], unfinished: undefined };
|
|
140
143
|
const warnings = isRunBundle(bundle)
|
|
141
144
|
? [
|
|
142
145
|
...rawScreenshotPostureWarnings(bundle, redactedShapeFrames),
|
|
143
146
|
...undeclaredSubjectStateWarnings(bundle),
|
|
144
147
|
...desktopGeometryWarnings(bundle),
|
|
145
148
|
...flatContextWarnings(bundle),
|
|
146
|
-
...
|
|
149
|
+
...notFinished.warnings,
|
|
147
150
|
]
|
|
148
151
|
: [];
|
|
149
152
|
const shareSafety = isRunBundle(bundle)
|
|
@@ -175,6 +178,7 @@ export async function verifyResolvedRun(cwd, runInput, runPaths) {
|
|
|
175
178
|
checks,
|
|
176
179
|
shareSafety,
|
|
177
180
|
warnings,
|
|
181
|
+
...(notFinished.unfinished === undefined ? {} : { unfinished: notFinished.unfinished }),
|
|
178
182
|
...(ok
|
|
179
183
|
? {}
|
|
180
184
|
: {
|
|
@@ -444,7 +444,8 @@ traits: { patience, skill, accessibilityNeeds? }, constraints[], sourceDigest }`
|
|
|
444
444
|
receives them as thread/start `baseInstructions` (`threadStartParams` in
|
|
445
445
|
`src/actors/codex/restricted-launch.ts`), on an ephemeral read-only thread whose only
|
|
446
446
|
dynamic tool is `humanish_ui`. A local-agent Claude Code runs one `claude -p`
|
|
447
|
-
stream-json session
|
|
447
|
+
stream-json session restricted to Read in its folder (`claudeParticipantFlags` in
|
|
448
|
+
`src/actors/local-agent/claude-participant.ts`); the instructions open its first user
|
|
448
449
|
message (`promptFor` in `src/actors/local-agent/cli.ts`), and later turns rely on the
|
|
449
450
|
session's memory. `HUMANISH_LOCAL_AGENT_ONE_SHOT` instead spawns one `claude -p` per
|
|
450
451
|
turn, which resends the instructions every turn.
|
|
@@ -93,10 +93,23 @@ Served data may also include `runtime` with `state`, `observedAt`, and
|
|
|
93
93
|
`source: "local-run-status"`. This is a current observation of a contained,
|
|
94
94
|
matching `status.json`; it never changes the run's recorded verdict or participant
|
|
95
95
|
outcomes. A fresh heartbeat means running, an explicitly finalized record means
|
|
96
|
-
finished,
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
and export do not create this
|
|
96
|
+
finished, a record the CLI's signal handler wrote means interrupted, and stale or
|
|
97
|
+
invalid timing means unknown. Missing, malformed, or mismatched records omit the
|
|
98
|
+
observation. A stale heartbeat alone does not prove interruption, and stored PIDs
|
|
99
|
+
are neither probed nor returned. Static rendering and export do not create this
|
|
100
|
+
served-only observation.
|
|
101
|
+
|
|
102
|
+
`run.display` says whether the run passed: `{ state, label, tone, reason? }` from
|
|
103
|
+
`runDisplay` (`src/run/display.ts`), the function every surface that shows a run's
|
|
104
|
+
result calls. `state` is `passed` only when `run.status` (the verdict) is `pass` and
|
|
105
|
+
`run.json`'s `outcome` does not record the run as failed. A run whose execution
|
|
106
|
+
failed under a pass verdict is `failed`, with the first failure in `reason`.
|
|
107
|
+
`blocked`, `timed_out`, `no_verdict`, `interrupted`, `running`, `dry_run` and
|
|
108
|
+
`unknown` are the other states. For a `run.json` without an `outcome`, the served
|
|
109
|
+
Observer and `observe` read `status.json` for liveness and, on a run recorded
|
|
110
|
+
before `run.json` carried an outcome, for its ok. The sidebar's dot is the pass
|
|
111
|
+
accent only for `passed`. observer-data.json written before `display` existed
|
|
112
|
+
keeps the verdict reading.
|
|
100
113
|
|
|
101
114
|
Watch is deliberately distinct from `humanish observe --all`. Watch serves one
|
|
102
115
|
attached run, and the process that created it may inject runtime stream URLs
|
|
@@ -167,9 +180,11 @@ origin-dependent scripts and modules in those artifacts may require independent
|
|
|
167
180
|
hosting. See the [CSP sandbox standard](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Content-Security-Policy/sandbox).
|
|
168
181
|
|
|
169
182
|
History entries may include `runtimeState` from the same contained local status
|
|
170
|
-
read as the Observer
|
|
171
|
-
|
|
172
|
-
|
|
183
|
+
read as the Observer, and `display` from the run's Observer data. Their existing
|
|
184
|
+
`status` remains the recorded verdict. The library's pip color and the sidebar's
|
|
185
|
+
label read `display`. Running filters should use runtime state when present,
|
|
186
|
+
preserving the difference between an active study and its provisional evidence
|
|
187
|
+
outcome.
|
|
173
188
|
|
|
174
189
|
## UI Shape
|
|
175
190
|
|
|
@@ -5,20 +5,20 @@ participant; hidden success criteria go only to the observation tracker. A route
|
|
|
5
5
|
that cannot carry both halves refuses the declaration before execution. Removing
|
|
6
6
|
`tasks` is an explicit choice to run a mission-only study, not an automatic fallback.
|
|
7
7
|
|
|
8
|
-
| Execution path | Task support | Mechanism
|
|
9
|
-
| ----------------------------------------------------- | ------------ |
|
|
10
|
-
| CUA, app-url, one participant or fan-out | Supported | The participant prompt composer renders goals; the CUA loop tracks observations
|
|
11
|
-
| CUA, provisioned clone or local-tree, separate worlds | Supported | Same prompt composer and loop
|
|
12
|
-
| CUA, local-app with caller executor/provider | Supported | Same loop; the caller supplies observations
|
|
13
|
-
| CUA, desktop-cli | Supported | Same prompt and tracker; unavailable criterion inputs remain unmeasured
|
|
14
|
-
| Shared-world, provisioned or external-public | Rejected | Actor specs omit the protocol before CUA session dispatch
|
|
15
|
-
| Terminal-product | Rejected | Terminal prompt and result contract do not implement tasks
|
|
16
|
-
| Scripted-browser, local or provisioned | Rejected | Scenario steps drive the participant; no task protocol is consumed
|
|
17
|
-
|
|
|
18
|
-
| Any second or later `actors[]` entry | Rejected | Current runners consume only the first actor; declare every participant under the first actor |
|
|
8
|
+
| Execution path | Task support | Mechanism |
|
|
9
|
+
| ----------------------------------------------------- | ------------ | ------------------------------------------------------------------------------- |
|
|
10
|
+
| CUA, app-url, one participant or fan-out | Supported | The participant prompt composer renders goals; the CUA loop tracks observations |
|
|
11
|
+
| CUA, provisioned clone or local-tree, separate worlds | Supported | Same prompt composer and loop |
|
|
12
|
+
| CUA, local-app with caller executor/provider | Supported | Same loop; the caller supplies observations |
|
|
13
|
+
| CUA, desktop-cli | Supported | Same prompt and tracker; unavailable criterion inputs remain unmeasured |
|
|
14
|
+
| Shared-world, provisioned or external-public | Rejected | Actor specs omit the protocol before CUA session dispatch |
|
|
15
|
+
| Terminal-product | Rejected | Terminal prompt and result contract do not implement tasks |
|
|
16
|
+
| Scripted-browser, local or provisioned | Rejected | Scenario steps drive the participant; no task protocol is consumed |
|
|
17
|
+
| Preview (`this-repo`) | Rejected | Study dispatch does not pass task declarations to the dry-run engine |
|
|
19
18
|
|
|
20
|
-
|
|
21
|
-
|
|
19
|
+
A study has one `actor`, and its participants go in `participants`, so every participant
|
|
20
|
+
gets the same tasks. Both registered CUA actors (`openai-computer-use` and `local-agent`) share
|
|
21
|
+
the CUA session loop. Their task support does not depend on which provider chooses actions.
|
|
22
22
|
Custom session hooks remain caller-owned implementations of that same contract;
|
|
23
23
|
this preflight does not certify arbitrary hook behavior.
|
|
24
24
|
|
|
@@ -33,7 +33,7 @@ fail-closed cross-validation, and forward-declared warnings.
|
|
|
33
33
|
| `execution.runtimeAuth` | `openai-env` (default) or opt-in `openai-egress`; names-only durable evidence |
|
|
34
34
|
| `execution.runtime.version` | Optional exact `@openai/codex` version; observed before keyed execution |
|
|
35
35
|
| `actor.model` / `reasoningEffort` | Passed to Codex as `--model` (default `gpt-5.6-sol`) and `-c model_reasoning_effort`; declarations, not observed provider identity |
|
|
36
|
-
| `caps` | `{ maxUsd, maxJobs, maxMinutes }`: the blast-radius budget; `maxUsd > 0` is refused
|
|
36
|
+
| `caps` | `{ maxUsd, maxJobs, maxMinutes }`: the blast-radius budget; `maxUsd > 0` is refused: no adopter cost source exists until issue 347 lands |
|
|
37
37
|
| `policies` | `allowPrivateRepoAccess` / `allowProviderCredentials` / `allowPaymentCredentials` / `allowGitHubMutation`, all default false |
|
|
38
38
|
| `actor.type` | `codex-exec`: a registered terminal actor (`keyPlacement: in-sandbox-command-scoped`) |
|
|
39
39
|
| route | `terminal` → `runTerminalProductStudy` ([`src/routes/terminal/route.ts`](https://github.com/danielgwilson/humanish/blob/main/src/routes/terminal/route.ts)) |
|
|
@@ -70,6 +70,15 @@ artifacts:
|
|
|
70
70
|
review:
|
|
71
71
|
schema: humanish.review.v1
|
|
72
72
|
verdict: "contract_proof_only|pass|fail|blocked|timed_out"
|
|
73
|
+
outcome: # optional, additive: how the run ended (Run Outcome below)
|
|
74
|
+
state: "finished"
|
|
75
|
+
ok: false
|
|
76
|
+
execution:
|
|
77
|
+
succeeded: false
|
|
78
|
+
failures:
|
|
79
|
+
- kind: "harness|provider-cleanup|provider-policy|sandbox-cleanup|evidence|cap|run"
|
|
80
|
+
message: "<public-safe message>"
|
|
81
|
+
warnings: [] # omitted when empty
|
|
73
82
|
redaction:
|
|
74
83
|
status: "passed"
|
|
75
84
|
notes: "<public-safe note>"
|
|
@@ -100,6 +109,59 @@ Persisted `run.json` files must not contain absolute local target paths. Runtime
|
|
|
100
109
|
commands may return the caller's working directory in process-local JSON
|
|
101
110
|
responses, but durable run bundles use the public-safe `[target-cwd]` marker.
|
|
102
111
|
|
|
112
|
+
## Run Outcome
|
|
113
|
+
|
|
114
|
+
`outcome` records how the run ended as an execution. `review.verdict` is what the
|
|
115
|
+
participants experienced; `outcome.ok` is the route result's `ok`, which
|
|
116
|
+
`humanish run --json` prints as `runOk`. The two differ: a terminal run whose agent passed but whose sandbox
|
|
117
|
+
teardown is unproven has verdict `pass` and `ok: false`.
|
|
118
|
+
|
|
119
|
+
```yaml
|
|
120
|
+
# Run.finish published the final bundle.
|
|
121
|
+
outcome:
|
|
122
|
+
state: finished
|
|
123
|
+
ok: true
|
|
124
|
+
execution:
|
|
125
|
+
succeeded: true
|
|
126
|
+
failures: []
|
|
127
|
+
warnings: # omitted when empty
|
|
128
|
+
- kind: sandbox-cleanup
|
|
129
|
+
message: "lane-01: Sandbox release is unconfirmed. Reclaim it by recorded id with `humanish reclaim --run <id>`."
|
|
130
|
+
---
|
|
131
|
+
# The CLI's signal handler stopped the run before it finished.
|
|
132
|
+
outcome:
|
|
133
|
+
state: interrupted
|
|
134
|
+
ok: false
|
|
135
|
+
signal: SIGINT # or SIGTERM, SIGHUP
|
|
136
|
+
at: "<ISO timestamp>"
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
`Run.finish` (`src/run/run.ts`) writes `finished` with the route's `ok` and its
|
|
140
|
+
execution outcome (`judgeExecution` in `src/run/judge.ts`). Each route's policy in
|
|
141
|
+
`OUTCOME_POLICIES` decides which failure kinds fail the run and which only warn.
|
|
142
|
+
An Observer that does not render is an `evidence` failure known only after
|
|
143
|
+
`run.json` is on disk: `FinishedRun.renderObserver` adds it to `outcome` and to
|
|
144
|
+
`status.json` before the route returns, and no Observer page exists that could
|
|
145
|
+
show the run without it. Messages pass the same redaction and sandbox id scrub
|
|
146
|
+
as the rest of the bundle.
|
|
147
|
+
|
|
148
|
+
When a signal stops a live run, the handler writes `interrupted` into the last
|
|
149
|
+
flushed `run.json`, and every later write of that run keeps it. A run stopped
|
|
150
|
+
before its first flush has no `run.json`; its `status.json` is the record.
|
|
151
|
+
|
|
152
|
+
`outcome` is absent while a run is in progress, when its process died without
|
|
153
|
+
writing it, and on runs recorded before the field existed. For those, readers
|
|
154
|
+
take liveness from `status.json` and, for an older finished run, its `ok` and
|
|
155
|
+
`execution` from `status.json`'s `outcome`. That record is the only copy of an
|
|
156
|
+
older run's `ok`; for a run with `outcome` in `run.json`, `status.json` copies it.
|
|
157
|
+
|
|
158
|
+
Every surface that shows whether a run passed calls `runDisplay`
|
|
159
|
+
(`src/run/display.ts`) with the run's liveness, verdict and `ok`: the Observer
|
|
160
|
+
(`observer-data.json` `run.display`), its run library, the served Observer,
|
|
161
|
+
`humanish runs`, `humanish review`, `humanish stats`, the TUI and `review.md`'s
|
|
162
|
+
`outcome` line. A run
|
|
163
|
+
shows as `passed` only when its verdict is `pass` and its `ok` is not false.
|
|
164
|
+
|
|
103
165
|
## Recorded session stop causes
|
|
104
166
|
|
|
105
167
|
An actor trace may include `stopCause` alongside its unchanged `status`,
|
|
@@ -543,12 +605,19 @@ writing; an older one reads as interrupted. The warning names what verify saw:
|
|
|
543
605
|
- the record's state and its last `updatedAt`, or the running `simulations[]` entries when
|
|
544
606
|
there is no usable record;
|
|
545
607
|
- how many streams are still `running`;
|
|
546
|
-
- what `reclaim-receipt.json` records: how many of the run's
|
|
547
|
-
(killed or already gone), and how many journaled sandboxes it does
|
|
548
|
-
matches its outcomes to the receipts by `sandboxIdDigest`; a reclaim
|
|
549
|
-
before 0.110 names raw ids, which are digested to match;
|
|
550
|
-
- with
|
|
551
|
-
`humanish reclaim --run <id>`.
|
|
608
|
+
- the sandbox state, and what `reclaim-receipt.json` records: how many of the run's
|
|
609
|
+
sandboxes are gone (killed or already gone), and how many journaled sandboxes it does
|
|
610
|
+
not cover. It matches its outcomes to the receipts by `sandboxIdDigest`; a reclaim
|
|
611
|
+
receipt from before 0.110 names raw ids, which are digested to match;
|
|
612
|
+
- with no reclaim receipt, the command that stops the journaled sandboxes and searches
|
|
613
|
+
E2B by the run's owner tags: `humanish reclaim --run <id>`.
|
|
614
|
+
|
|
615
|
+
The result also carries `unfinished: { liveness, sandboxes }`, and the one-line output
|
|
616
|
+
leads with it, for example `verified <id> (interrupted, sandboxes unknown) · share_ready`.
|
|
617
|
+
`sandboxes` is `clean` only when the reclaim receipt records every sandbox gone and its
|
|
618
|
+
`state` is `clean`, which means the search by tag finished. It is `unconfirmed` when the
|
|
619
|
+
receipt records a sandbox not gone or misses a journaled one, and `unknown` otherwise,
|
|
620
|
+
including a receipt from humanish 0.110.0 or earlier, which did not search by tag.
|
|
552
621
|
|
|
553
622
|
The local-evidence check includes screenshots declared only by
|
|
554
623
|
`streams[].actor.items[].screenshotRef` or `streams[].liveActor.items[].screenshotRef`,
|
|
@@ -37,6 +37,7 @@ workflow without leaking private upstream truth into core.
|
|
|
37
37
|
| Review | `humanish.review.v1` | `contract-proof-review` |
|
|
38
38
|
| Study analysis | `humanish.study-analysis.v1` | see [study analysis](study-analysis.md) and synthetic analysis fixtures in `tests/` |
|
|
39
39
|
| Study analysis correction | `humanish.study-analysis-correction.v1` | see [study analysis](study-analysis.md#human-review-and-sharing) |
|
|
40
|
+
| Analysis findings view | `humanish.analysis-findings.v1` (`review --json` field `analysis`; read-only projection of the selected analysis) | see [study analysis](study-analysis.md#reading-findings) |
|
|
40
41
|
| Analysis execution receipt | `humanish.analysis-execution.v1` | see [study analysis](study-analysis.md#durable-records) |
|
|
41
42
|
| Analysis execution start | `humanish.analysis-execution-start.v1` | see [study analysis](study-analysis.md#durable-records) |
|
|
42
43
|
| Verification | `humanish.verify-result.v1` | `five-check-verify` |
|
|
@@ -300,10 +301,11 @@ instruction?, target?, entry?, host?, reasoningEffort?, stopWhen?, dwell? }` bec
|
|
|
300
301
|
wall-clock kill; `maxUsd`/`maxJobs` are enforced fail-closed against the cost
|
|
301
302
|
ledger after the session (a run whose known spend exceeds the cap fails closed,
|
|
302
303
|
`HUMANISH_TERMINAL_CAPS_EXCEEDED`). Core records Codex provider spend as
|
|
303
|
-
unpriced tokens,
|
|
304
|
-
a live run refuses `maxUsd > 0` before creating a sandbox
|
|
305
|
-
(`HUMANISH_TERMINAL_UNPRICED_CAP`)
|
|
306
|
-
|
|
304
|
+
unpriced tokens and measures no product, media or payment spend, so no line can trip a
|
|
305
|
+
positive `maxUsd`: a live run refuses `maxUsd > 0` before creating a sandbox
|
|
306
|
+
(`HUMANISH_TERMINAL_UNPRICED_CAP`). A terminal `maxUsd` cap has no adopter cost source
|
|
307
|
+
until [issue 347](https://github.com/danielgwilson/humanish/issues/347) lands;
|
|
308
|
+
`StudyDeps.costProbe` is a test seam. The sandbox's server-side timeout is the steps before the
|
|
307
309
|
codex command (Node bootstrap, runtime version check, and product setup when
|
|
308
310
|
`subject.product.install` is declared) plus `maxMinutes` plus a 5-minute
|
|
309
311
|
teardown buffer; a live run refuses a `maxMinutes` that would take it past
|
|
@@ -1088,8 +1090,8 @@ discipline** that distinguishes three states and never conflates them:
|
|
|
1088
1090
|
and is never coerced to `0`); `fullyMeasured` is true only when no line is null.
|
|
1089
1091
|
Core meters only the `provider` line, populated from the actor trace's
|
|
1090
1092
|
`tokenUsage.costUsd` when present (else `null`); `product`/`media`/`payment`
|
|
1091
|
-
remain `null
|
|
1092
|
-
|
|
1093
|
+
remain `null`, because no adopter can supply those signals until
|
|
1094
|
+
[issue 347](https://github.com/danielgwilson/humanish/issues/347) lands. The terminal route records the model as `codex` and does not pin Codex's
|
|
1093
1095
|
model, so a measured token count stays `usd: null` with `source: unpriced-token-usage`, and
|
|
1094
1096
|
`knownTotalUsd: 0` with `fullyMeasured: false` means no line of this ledger was priced. E2B
|
|
1095
1097
|
time for the terminal sandbox is not a line of this ledger, whose lines are checked against
|
|
@@ -1149,9 +1151,13 @@ finished | interrupted, mode, study?, pid, startedAt, updatedAt, completedAt?,
|
|
|
1149
1151
|
signal?, outcome? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
|
|
1150
1152
|
`study`, and readers accept it.
|
|
1151
1153
|
`outcome` carries the bundle's `verdict`, `participants`, `estimatedCostUsd` and
|
|
1152
|
-
`estimatedCostComplete` when the run finishes,
|
|
1153
|
-
failures: [{ kind, message }], warnings? }`
|
|
1154
|
-
|
|
1154
|
+
`estimatedCostComplete` when the run finishes, with `ok` and `execution: { succeeded,
|
|
1155
|
+
failures: [{ kind, message }], warnings? }` copied from `run.json`'s `outcome`
|
|
1156
|
+
([Run Outcome](run-bundle.md#run-outcome)) in the same publication. An Observer
|
|
1157
|
+
that does not render adds its `evidence` failure to both files. A record written
|
|
1158
|
+
before `run.json` carried an outcome got `ok` and `execution` after the Observer
|
|
1159
|
+
rendered and is the only copy of that run's `ok`; readers use it for such a run
|
|
1160
|
+
only. The verdict is what the participants experienced; `execution` is whether the run
|
|
1155
1161
|
worked as an execution (`kind` is `harness`, `provider-cleanup`,
|
|
1156
1162
|
`provider-policy`, `sandbox-cleanup`, `evidence`, `cap` or `run`), and `ok`
|
|
1157
1163
|
reads both under the route's policy (`OUTCOME_POLICIES` in `src/run/judge.ts`).
|
|
@@ -1177,10 +1183,13 @@ and the estimate is a lower bound. A record without it, as every record before
|
|
|
1177
1183
|
|
|
1178
1184
|
`interrupted` with `signal` (`SIGINT`, `SIGTERM` or `SIGHUP`) is written by
|
|
1179
1185
|
the CLI's run command when that signal stops a live run outside post-run
|
|
1180
|
-
analysis. It
|
|
1181
|
-
|
|
1182
|
-
|
|
1183
|
-
`
|
|
1186
|
+
analysis. It first prints one line saying it is stopping and refuses any further sandbox
|
|
1187
|
+
create. It then kills the E2B sandboxes the run's `sandbox-receipts.ndjson` names, each sandbox
|
|
1188
|
+
a create in flight reports as its id arrives, and each one E2B lists with the run's owner tags,
|
|
1189
|
+
writes `reclaim-receipt.json` as `humanish reclaim` does, and exits
|
|
1190
|
+
128+n. The bundle is not finished: run.json keeps its last live flush, with
|
|
1191
|
+
`outcome: { state: interrupted, ok: false, signal, at }` when the run had
|
|
1192
|
+
flushed one, and status.json gets no `outcome`. A `running` record that stopped refreshing still reads as
|
|
1184
1193
|
interrupted once stale. A library caller of `runStudy` gets no signal handling. A
|
|
1185
1194
|
humanish older than this field rejects the state and falls back to the bundle's
|
|
1186
1195
|
own liveness.
|
|
@@ -1192,8 +1201,10 @@ nothing at all until they completed.
|
|
|
1192
1201
|
|
|
1193
1202
|
It is a derived index, not evidence. `run.json` remains the evidence-of-record;
|
|
1194
1203
|
`verify` never gates on `status.json`, nothing in it is a claim about what a
|
|
1195
|
-
participant did, and when the two disagree the bundle wins
|
|
1196
|
-
|
|
1204
|
+
participant did, and when the two disagree the bundle wins. Fields that cannot be
|
|
1205
|
+
rebuilt from `run.json`: `pid`, `startedAt`, `updatedAt` and `completedAt`, which
|
|
1206
|
+
no bundle records, and, on a record written before `run.json` carried an
|
|
1207
|
+
`outcome`, `outcome.ok` and `outcome.execution`. A `running` record whose `updatedAt` is older than three
|
|
1197
1208
|
touch intervals is interrupted, not alive. A dropped connection or a killed
|
|
1198
1209
|
terminal leaves exactly that shape, so the record reads as interrupted
|
|
1199
1210
|
(`classifyRunStatus` is the one shared definition). Fields are
|
|
@@ -1210,16 +1221,20 @@ them is a claim about what a participant did.
|
|
|
1210
1221
|
`humanish.run-index.v1` is the listing projection. One entry per run, read
|
|
1211
1222
|
cheapest-source-first: the `status.json` record, else the bundle, else the run
|
|
1212
1223
|
directory alone. `{ runId, derivedFrom: status | bundle | directory, liveness,
|
|
1213
|
-
mode?, pid?, study?, startedAt?, updatedAt?, completedAt?, verdict?,
|
|
1214
|
-
participants?, estimatedCostUsd?, estimatedCostComplete?,
|
|
1215
|
-
durationMs? }`.
|
|
1224
|
+
mode?, pid?, study?, startedAt?, updatedAt?, completedAt?, verdict?, ok?,
|
|
1225
|
+
failure?, participants?, estimatedCostUsd?, estimatedCostComplete?,
|
|
1226
|
+
analysisCost?, durationMs? }`. `ok` and `failure` (the first execution failure)
|
|
1227
|
+
are the run's own outcome; `runDisplay` (`src/run/display.ts`) reads them with
|
|
1228
|
+
`liveness` and `verdict`, so the TUI and `humanish stats` show a run with
|
|
1229
|
+
`ok: false` as failed whatever its verdict. The point is cost: walking
|
|
1216
1230
|
every run tree and parsing every bundle measured 167ms on a 25-run project,
|
|
1217
1231
|
against 16ms cold and 2.8ms warm here, which is what makes a surface that
|
|
1218
1232
|
refreshes on a cadence affordable. `derivedFrom` is reported so a surprising row
|
|
1219
1233
|
can be traced to the file it came from. A run with receipts and no outcome is
|
|
1220
1234
|
`interrupted` (the shape a dropped connection leaves), and so is an
|
|
1221
1235
|
in-progress bundle reached without a status record: there is no freshness to
|
|
1222
|
-
judge, and nothing in it says the run finished.
|
|
1236
|
+
judge, and nothing in it says the run finished. A bundle with an `outcome` reached
|
|
1237
|
+
without a status record takes its liveness from `outcome.state`.
|
|
1223
1238
|
|
|
1224
1239
|
`estimatedCostUsd` is the participants-and-desktops figure from the status
|
|
1225
1240
|
record or bundle and excludes analysis. `estimatedCostComplete` says whether that
|
|
@@ -1378,7 +1393,7 @@ usage is known or bounded:
|
|
|
1378
1393
|
- A request lost without a reply (it stalled, or the OpenAI provider's
|
|
1379
1394
|
transport threw) is booked at its worst case before it is sent again: the
|
|
1380
1395
|
latest reported request's input at the model's highest input rate plus
|
|
1381
|
-
`
|
|
1396
|
+
`actor.maxOutputTokens` at its output rate, both from `src/run/pricing.ts`.
|
|
1382
1397
|
The reply to the resent request replaces that input with its own. The
|
|
1383
1398
|
request is sent again only when the estimate with that charge stays within
|
|
1384
1399
|
`maxUsd` and the study budget. Otherwise the session ends `budget_reached`
|
|
@@ -1566,7 +1581,7 @@ of it fails.
|
|
|
1566
1581
|
| `SubjectPhaseEvent` | the `subject-phase` `StudyEvent` through `onEvent`. Its `type` is `"subject-phase"`; the phase that was `type` is `name`, and `target` is `{ kind: "subject" }` or `{ kind: "participant", participant }`. `message`, `at`, `ok` and `durationMs` are unchanged |
|
|
1567
1582
|
| `StudyOutcome.backend`, `LabBackend` | `outcome.route` |
|
|
1568
1583
|
| `selectLabBackend`, the `routesTo*` predicates, `actorResolvesToTerminal` | `routeOf(config)` |
|
|
1569
|
-
| `resolveStudyDryRun` | `RunStudyOptions.dryRun`; without it, `
|
|
1584
|
+
| `resolveStudyDryRun` | `RunStudyOptions.dryRun`; without it, `mode` decides |
|
|
1570
1585
|
| `cuaLaneCount` | the `plan` `StudyEvent`, which lists every participant |
|
|
1571
1586
|
| `resolveSeatUrl` | `parseStudy`, which refuses a participant entry that is not same-origin loopback; build the URL with `new URL(entry, serveUrl)` |
|
|
1572
1587
|
| `cuaLaneValidationReason`, `sharedWorldValidationReason`, `concurrentSharedWorldValidationReason`, `externalPublicSharedWorldValidationReason` | `parseStudy`, which returns the same reason |
|
|
@@ -1682,7 +1697,9 @@ the stream shape, the meaningful-use rubric, and hard-failure rules.
|
|
|
1682
1697
|
Review summarizes whether evidence supports the claim. It does not replace
|
|
1683
1698
|
verification or maintainer acceptance.
|
|
1684
1699
|
|
|
1685
|
-
`verdict` is the
|
|
1700
|
+
`verdict` is what the participants experienced, collapsed to one word. Whether
|
|
1701
|
+
the run passed also reads `run.json`'s `outcome.ok` ([Run Outcome](run-bundle.md#run-outcome)):
|
|
1702
|
+
a run whose execution failed shows as failed beside a `pass` verdict. `participants.reachedGoal` retains the count of
|
|
1686
1703
|
recorded successful sessions; it is not an independent adjudication of their
|
|
1687
1704
|
claims. Computer-use review and Observer labels distinguish a participant's
|
|
1688
1705
|
reported completion from a recorded `stopWhen` or dwell condition match. A
|
|
@@ -288,6 +288,51 @@ companion project validated records; generic HTTP serving of producer-owned raw
|
|
|
288
288
|
analysis/correction JSON, execution receipts, atomic write temporaries and locks
|
|
289
289
|
is disabled. Existing adapter captures and logs retain their contained routes.
|
|
290
290
|
|
|
291
|
+
## Reading findings
|
|
292
|
+
|
|
293
|
+
`humanish review` prints a run's analysis findings after its participant review, and
|
|
294
|
+
`humanish analyze show` prints them alone. Each finding shows its title, impact, confidence and
|
|
295
|
+
recovery, the participants it affected out of those exposed, the frames its evidence cites with
|
|
296
|
+
their time since the first retained capture, the capture files, its next step and the latest
|
|
297
|
+
human review note. A live run's human output ends with up to three findings, one line each, and
|
|
298
|
+
the `review` command that prints all of them. `analyze show --json` prints the validated
|
|
299
|
+
analysis record and its corrections, as before. Reading findings never starts an analysis.
|
|
300
|
+
|
|
301
|
+
`review --json` carries the same view as its `analysis` field, schema
|
|
302
|
+
`humanish.analysis-findings.v1`, built in `src/cli/findings.ts`:
|
|
303
|
+
|
|
304
|
+
| Field | Meaning |
|
|
305
|
+
| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
306
|
+
| `schema` | `humanish.analysis-findings.v1`. |
|
|
307
|
+
| `runId` | The run the view reads. |
|
|
308
|
+
| `state` | `ready`, `stale`, `running`, `none`, `skipped`, `failed`, `dry_run` or `unavailable`. Only `ready` and `stale` carry findings. |
|
|
309
|
+
| `reason` | The stable code behind a state other than `ready`, or null: an automatic analysis reason such as `AUTOMATIC_ANALYSIS_KEY_MISSING`, a failed attempt's error code, or a load warning such as `ANALYSIS_SOURCE_CHANGED`. |
|
|
310
|
+
| `message` | What happened, in words. |
|
|
311
|
+
| `next` | The command that gets findings for this run, or null when no command can, as for a dry run or a run whose participants left no evidence. |
|
|
312
|
+
| `analysisId`, `status`, `provider`, `model`, `completedAt`, `estimatedCostUsd` | The analysis version `analyze show` selects: the newest ready one, else the newest stale one. Null without one. |
|
|
313
|
+
| `runPath`, `path` | The run directory and the version's `analysis.json`, relative to the project directory. |
|
|
314
|
+
| `summary`, `limitations` | The analysis's own summary and limitations. |
|
|
315
|
+
| `findings[]` | Highest priority first: `id`, `title`, `summary`, `impact` (`blocked_task`, `friction`, `recovery`, `uncertain`), `confidence` (`low`, `medium`, `high`), `recovery` (`recovered`, `not_observed`, `unknown`), `affected[]` (`streamId`, `label`), `exposedCount`, `evidence[]`, `nextStep` and `correction`. |
|
|
316
|
+
| `findings[].evidence[]` | Each cited item once: `id`, `streamId`, `kind`, `bases` (how the observations used it), `frame` (the latest retained capture's index from 0, null before any capture), `elapsedMs` (time since the first retained capture, not a video offset), `at` and `capture` (the file relative to the project directory, null for nonvisual evidence). |
|
|
317
|
+
| `findings[].correction` | The latest `analyze correct` note on that finding version (`status`, `reason`, `replacementClaim`), or null. |
|
|
318
|
+
| `warnings` | Load warnings for the selected version. |
|
|
319
|
+
|
|
320
|
+
The states without findings:
|
|
321
|
+
|
|
322
|
+
- `running`: the automatic analysis is queued or running. `next` is `review` again.
|
|
323
|
+
- `skipped`: the automatic analysis was skipped or refused, for example without
|
|
324
|
+
`OPENAI_API_KEY`. `next` is `analyze --max-cost 3`.
|
|
325
|
+
- `failed`: the latest attempt failed or was cancelled and no usable version exists. `next`
|
|
326
|
+
reruns it with the same analyst, `--provider codex` for the Codex account analyst.
|
|
327
|
+
- `none`: no analysis ran for this live run, for example with `review.analysis: false`.
|
|
328
|
+
- `dry_run`: a dry run has no participant evidence to analyze.
|
|
329
|
+
- `unavailable`: the run or its analysis records could not be read or checked, or the analysis
|
|
330
|
+
text matched a sensitive-text pattern (`ANALYSIS_SENSITIVE_TEXT_QUARANTINED`), which the
|
|
331
|
+
Observer withholds too.
|
|
332
|
+
|
|
333
|
+
`stale` lists the findings with a warning: the run's evidence changed after the analysis, and
|
|
334
|
+
`next` requests a new one. `analyze show` without `--json` exits 2 when the JSON form would.
|
|
335
|
+
|
|
291
336
|
## Human review and sharing
|
|
292
337
|
|
|
293
338
|
```bash
|