humanish 0.110.0 → 0.110.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +15 -12
- package/CHANGELOG.md +26 -0
- package/CONTEXT.md +8 -6
- package/README.md +103 -53
- package/dist/actors/computer-use/loop/ending.js +4 -4
- package/dist/actors/local-agent/claude-participant.d.ts +74 -0
- package/dist/actors/local-agent/claude-participant.js +243 -0
- package/dist/actors/local-agent/claude-session.d.ts +16 -3
- package/dist/actors/local-agent/claude-session.js +64 -23
- package/dist/actors/local-agent/cli.d.ts +17 -0
- package/dist/actors/local-agent/cli.js +111 -24
- package/dist/actors/local-agent/readiness.d.ts +2 -2
- package/dist/actors/local-agent/readiness.js +20 -12
- package/dist/cli/commands/analyze.js +21 -5
- package/dist/cli/commands/observe.js +15 -2
- package/dist/cli/commands/run-signals.d.ts +2 -0
- package/dist/cli/commands/run-signals.js +106 -45
- package/dist/cli/commands/runs.d.ts +5 -0
- package/dist/cli/commands/runs.js +113 -74
- package/dist/cli/commands/setup.js +23 -14
- package/dist/cli/commands/study-route-computer-use.js +9 -3
- package/dist/cli/commands/study-route-open.js +1 -0
- package/dist/cli/commands/study-route-preview.js +1 -0
- package/dist/cli/commands/study-route-scripted.js +2 -0
- package/dist/cli/commands/study-route-shared-world.js +6 -2
- package/dist/cli/commands/study-route-terminal.js +2 -0
- package/dist/cli/commands/watch.js +14 -4
- package/dist/cli/doctor.d.ts +4 -1
- package/dist/cli/doctor.js +51 -10
- package/dist/cli/findings.d.ts +103 -0
- package/dist/cli/findings.js +317 -0
- package/dist/cli/first-run-path.d.ts +32 -11
- package/dist/cli/first-run-path.js +113 -69
- package/dist/cli/invocation.d.ts +7 -0
- package/dist/cli/invocation.js +32 -0
- package/dist/cli/io.d.ts +3 -1
- package/dist/cli/io.js +8 -15
- package/dist/cli/observer-follow.d.ts +20 -0
- package/dist/cli/observer-follow.js +36 -1
- package/dist/cli/orientation.js +16 -14
- package/dist/cli/product-sentence.d.ts +3 -3
- package/dist/cli/product-sentence.js +3 -3
- package/dist/cli/version.d.ts +1 -0
- package/dist/cli/version.js +11 -0
- package/dist/evidence/encoded-text.d.ts +9 -6
- package/dist/evidence/encoded-text.js +84 -19
- package/dist/evidence/redaction.d.ts +2 -0
- package/dist/evidence/redaction.js +123 -21
- package/dist/feedback/export-bundle.js +2 -2
- package/dist/keys/env-file.d.ts +4 -0
- package/dist/keys/env-file.js +17 -0
- package/dist/keys/key-resolution.js +6 -5
- package/dist/observer/data.d.ts +14 -1
- package/dist/observer/data.js +19 -9
- package/dist/observer/exposure.js +1 -1
- package/dist/observer/library.d.ts +4 -0
- package/dist/observer/library.js +21 -17
- package/dist/observer/render.js +10 -3
- package/dist/observer/run-routes.d.ts +2 -0
- package/dist/observer/run-routes.js +51 -35
- package/dist/observer/serve.js +2 -1
- package/dist/observer-app.html +3 -3
- package/dist/routes/computer-use/bundle.js +3 -1
- package/dist/routes/computer-use/fanout-records.js +1 -1
- package/dist/routes/computer-use/participant-execution.js +3 -1
- package/dist/routes/computer-use/participant-model.js +5 -2
- package/dist/routes/computer-use/plan.js +9 -4
- package/dist/routes/computer-use/preflight.js +1 -1
- package/dist/routes/computer-use/result.js +17 -28
- package/dist/routes/computer-use/single-bundle.js +7 -1
- package/dist/routes/scripted/bundle.js +4 -2
- package/dist/routes/scripted/plan.js +2 -2
- package/dist/routes/scripted/result.d.ts +3 -5
- package/dist/routes/scripted/result.js +16 -22
- package/dist/routes/scripted/scenario.js +4 -4
- package/dist/routes/shared-world/bundle-records.js +6 -1
- package/dist/routes/shared-world/bundle.js +2 -0
- package/dist/routes/shared-world/plan.js +6 -2
- package/dist/routes/shared-world/result.js +22 -22
- package/dist/routes/terminal/bundle.js +6 -3
- package/dist/routes/terminal/credentials.d.ts +5 -5
- package/dist/routes/terminal/credentials.js +14 -6
- package/dist/routes/terminal/dry-run.js +8 -12
- package/dist/routes/terminal/ledger.d.ts +3 -3
- package/dist/routes/terminal/ledger.js +6 -6
- package/dist/routes/terminal/live-finish.js +15 -13
- package/dist/routes/terminal/plan.js +12 -4
- package/dist/routes/terminal/result.d.ts +6 -7
- package/dist/routes/terminal/result.js +6 -22
- package/dist/routes/terminal/route.js +5 -4
- package/dist/routes/terminal/runtime-auth.d.ts +6 -0
- package/dist/routes/terminal/runtime-auth.js +5 -0
- package/dist/routes/terminal/session.js +1 -1
- package/dist/run/bundle-shape.d.ts +3 -1
- package/dist/run/bundle-shape.js +25 -1
- package/dist/run/bundle.d.ts +24 -0
- package/dist/run/display.d.ts +59 -0
- package/dist/run/display.js +106 -0
- package/dist/run/dry-run.js +13 -8
- package/dist/run/judge.d.ts +3 -2
- package/dist/run/judge.js +12 -1
- package/dist/run/local-only-files.d.ts +13 -0
- package/dist/run/local-only-files.js +35 -0
- package/dist/run/pricing.js +1 -1
- package/dist/run/projection.d.ts +0 -2
- package/dist/run/projection.js +0 -11
- package/dist/run/reclaim-outcomes.d.ts +89 -0
- package/dist/run/reclaim-outcomes.js +213 -0
- package/dist/run/reclaim.d.ts +23 -17
- package/dist/run/reclaim.js +213 -114
- package/dist/run/results.d.ts +10 -20
- package/dist/run/results.js +2 -2
- package/dist/run/run-index.d.ts +11 -1
- package/dist/run/run-index.js +11 -0
- package/dist/run/run-not-found.js +3 -2
- package/dist/run/run.d.ts +34 -18
- package/dist/run/run.js +166 -67
- package/dist/run/sandbox-creates.d.ts +59 -0
- package/dist/run/sandbox-creates.js +140 -0
- package/dist/run/sandbox-receipts.d.ts +20 -5
- package/dist/run/sandbox-receipts.js +56 -7
- package/dist/run/stats.d.ts +5 -0
- package/dist/run/stats.js +8 -4
- package/dist/run/status.d.ts +6 -9
- package/dist/run/status.js +7 -23
- package/dist/run/stored-runs.d.ts +5 -14
- package/dist/run/stored-runs.js +31 -146
- package/dist/run/streams.d.ts +3 -2
- package/dist/study/agents-section.d.ts +21 -0
- package/dist/study/agents-section.js +86 -0
- package/dist/study/doctor.d.ts +1 -1
- package/dist/study/doctor.js +8 -4
- package/dist/study/init-templates.js +0 -95
- package/dist/study/init.js +45 -37
- package/dist/study/keys.d.ts +12 -0
- package/dist/study/keys.js +32 -0
- package/dist/study/local-browser-host.d.ts +11 -0
- package/dist/study/local-browser-host.js +42 -0
- package/dist/study/parse/actors.js +9 -1
- package/dist/study/parse/subject.js +12 -0
- package/dist/study/parse/url-credentials.d.ts +9 -0
- package/dist/study/parse/url-credentials.js +50 -0
- package/dist/study/persona.d.ts +4 -0
- package/dist/study/persona.js +4 -2
- package/dist/study/preflight-probes.js +2 -1
- package/dist/study/types.d.ts +8 -6
- package/dist/study/url-credentials.d.ts +6 -0
- package/dist/study/url-credentials.js +34 -0
- package/dist/substrates/e2b/peer-install.d.ts +51 -0
- package/dist/substrates/e2b/peer-install.js +161 -0
- package/dist/substrates/e2b/sandbox.d.ts +34 -6
- package/dist/substrates/e2b/sandbox.js +141 -30
- package/dist/substrates/e2b/sdk.d.ts +33 -0
- package/dist/substrates/e2b/sdk.js +21 -21
- package/dist/substrates/local/firecracker-desktop.js +3 -2
- package/dist/substrates/local/runtime-host.js +5 -4
- package/dist/substrates/local/runtime.js +5 -4
- package/dist/substrates/shell.d.ts +2 -0
- package/dist/substrates/shell.js +4 -0
- package/dist/tui/contract.js +5 -4
- package/dist/tui-app.js +133 -133
- package/dist/verify/liveness.d.ts +19 -3
- package/dist/verify/liveness.js +47 -25
- package/dist/verify/verify.d.ts +3 -0
- package/dist/verify/verify.js +8 -3
- package/docs/architecture/actor-contract.md +2 -1
- package/docs/architecture/observer.md +22 -7
- package/docs/architecture/task-protocol-support.md +13 -13
- package/docs/architecture/terminal-product-route.md +15 -13
- package/docs/contracts/run-bundle.md +95 -10
- package/docs/contracts/schemas.md +44 -26
- package/docs/contracts/study-analysis.md +45 -0
- package/docs/contracts/study-costs.md +8 -0
- package/docs/decisions/0001-run-bundle-is-the-source-of-truth.md +3 -0
- package/docs/decisions/0003-contained-paths-and-cleanup-authority.md +9 -3
- package/docs/principles/invariants-and-defaults.md +11 -8
- package/package.json +6 -3
- package/skills/humanish/SKILL.md +38 -20
|
@@ -37,6 +37,7 @@ workflow without leaking private upstream truth into core.
|
|
|
37
37
|
| Review | `humanish.review.v1` | `contract-proof-review` |
|
|
38
38
|
| Study analysis | `humanish.study-analysis.v1` | see [study analysis](study-analysis.md) and synthetic analysis fixtures in `tests/` |
|
|
39
39
|
| Study analysis correction | `humanish.study-analysis-correction.v1` | see [study analysis](study-analysis.md#human-review-and-sharing) |
|
|
40
|
+
| Analysis findings view | `humanish.analysis-findings.v1` (`review --json` field `analysis`; read-only projection of the selected analysis) | see [study analysis](study-analysis.md#reading-findings) |
|
|
40
41
|
| Analysis execution receipt | `humanish.analysis-execution.v1` | see [study analysis](study-analysis.md#durable-records) |
|
|
41
42
|
| Analysis execution start | `humanish.analysis-execution-start.v1` | see [study analysis](study-analysis.md#durable-records) |
|
|
42
43
|
| Verification | `humanish.verify-result.v1` | `five-check-verify` |
|
|
@@ -278,7 +279,7 @@ instruction?, target?, entry?, host?, reasoningEffort?, stopWhen?, dwell? }` bec
|
|
|
278
279
|
an interactive PTY would overstate the mechanism (a duplex PTY
|
|
279
280
|
transport does not ship). `terminal.stdin` defaults to `disabled`
|
|
280
281
|
(`sent`/assisted input is rejected until the interventions ledger + a
|
|
281
|
-
non-comparable marker exist). `runtimeAuth` (`openai-
|
|
282
|
+
non-comparable marker exist). `runtimeAuth` (`openai-egress`, the default, or `openai-env`)
|
|
282
283
|
declares the agent's runtime-auth channel, recorded as names only. On a live run, the engine
|
|
283
284
|
resolves the registered terminal descriptor and requires
|
|
284
285
|
`keyPlacement: in-sandbox-command-scoped` before creating a sandbox. With `openai-env`
|
|
@@ -300,10 +301,11 @@ instruction?, target?, entry?, host?, reasoningEffort?, stopWhen?, dwell? }` bec
|
|
|
300
301
|
wall-clock kill; `maxUsd`/`maxJobs` are enforced fail-closed against the cost
|
|
301
302
|
ledger after the session (a run whose known spend exceeds the cap fails closed,
|
|
302
303
|
`HUMANISH_TERMINAL_CAPS_EXCEEDED`). Core records Codex provider spend as
|
|
303
|
-
unpriced tokens,
|
|
304
|
-
a live run refuses `maxUsd > 0` before creating a sandbox
|
|
305
|
-
(`HUMANISH_TERMINAL_UNPRICED_CAP`)
|
|
306
|
-
|
|
304
|
+
unpriced tokens and measures no product, media or payment spend, so no line can trip a
|
|
305
|
+
positive `maxUsd`: a live run refuses `maxUsd > 0` before creating a sandbox
|
|
306
|
+
(`HUMANISH_TERMINAL_UNPRICED_CAP`). A terminal `maxUsd` cap has no adopter cost source
|
|
307
|
+
until [issue 347](https://github.com/danielgwilson/humanish/issues/347) lands;
|
|
308
|
+
`StudyDeps.costProbe` is a test seam. The sandbox's server-side timeout is the steps before the
|
|
307
309
|
codex command (Node bootstrap, runtime version check, and product setup when
|
|
308
310
|
`subject.product.install` is declared) plus `maxMinutes` plus a 5-minute
|
|
309
311
|
teardown buffer; a live run refuses a `maxMinutes` that would take it past
|
|
@@ -1066,9 +1068,9 @@ contract.
|
|
|
1066
1068
|
## Terminal Cost Ledger And No-Spend Proof
|
|
1067
1069
|
|
|
1068
1070
|
The terminal-product route (`src/routes/terminal/route.ts`), under the default
|
|
1069
|
-
`runtimeAuth: openai-
|
|
1070
|
-
|
|
1071
|
-
|
|
1071
|
+
`runtimeAuth: openai-egress`, gives the in-sandbox agent command a placeholder and
|
|
1072
|
+
keeps the key in an external E2B header transform; `openai-env` passes a real
|
|
1073
|
+
provider key only to that command. Neither mode puts it in sandbox-global env or
|
|
1072
1074
|
metadata, so the no-spend claim must be derived from a ledger, never asserted. The
|
|
1073
1075
|
live run writes both to `terminal-ledgers.json` (a `cost` block + a
|
|
1074
1076
|
`noSpendProof` block, additive to `humanish.terminal-ledgers.v1`).
|
|
@@ -1088,8 +1090,8 @@ discipline** that distinguishes three states and never conflates them:
|
|
|
1088
1090
|
and is never coerced to `0`); `fullyMeasured` is true only when no line is null.
|
|
1089
1091
|
Core meters only the `provider` line, populated from the actor trace's
|
|
1090
1092
|
`tokenUsage.costUsd` when present (else `null`); `product`/`media`/`payment`
|
|
1091
|
-
remain `null
|
|
1092
|
-
|
|
1093
|
+
remain `null`, because no adopter can supply those signals until
|
|
1094
|
+
[issue 347](https://github.com/danielgwilson/humanish/issues/347) lands. The terminal route records the model as `codex` and does not pin Codex's
|
|
1093
1095
|
model, so a measured token count stays `usd: null` with `source: unpriced-token-usage`, and
|
|
1094
1096
|
`knownTotalUsd: 0` with `fullyMeasured: false` means no line of this ledger was priced. E2B
|
|
1095
1097
|
time for the terminal sandbox is not a line of this ledger, whose lines are checked against
|
|
@@ -1149,9 +1151,13 @@ finished | interrupted, mode, study?, pid, startedAt, updatedAt, completedAt?,
|
|
|
1149
1151
|
signal?, outcome? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
|
|
1150
1152
|
`study`, and readers accept it.
|
|
1151
1153
|
`outcome` carries the bundle's `verdict`, `participants`, `estimatedCostUsd` and
|
|
1152
|
-
`estimatedCostComplete` when the run finishes,
|
|
1153
|
-
failures: [{ kind, message }], warnings? }`
|
|
1154
|
-
|
|
1154
|
+
`estimatedCostComplete` when the run finishes, with `ok` and `execution: { succeeded,
|
|
1155
|
+
failures: [{ kind, message }], warnings? }` copied from `run.json`'s `outcome`
|
|
1156
|
+
([Run Outcome](run-bundle.md#run-outcome)) in the same publication. An Observer
|
|
1157
|
+
that does not render adds its `evidence` failure to both files. A record written
|
|
1158
|
+
before `run.json` carried an outcome got `ok` and `execution` after the Observer
|
|
1159
|
+
rendered and is the only copy of that run's `ok`; readers use it for such a run
|
|
1160
|
+
only. The verdict is what the participants experienced; `execution` is whether the run
|
|
1155
1161
|
worked as an execution (`kind` is `harness`, `provider-cleanup`,
|
|
1156
1162
|
`provider-policy`, `sandbox-cleanup`, `evidence`, `cap` or `run`), and `ok`
|
|
1157
1163
|
reads both under the route's policy (`OUTCOME_POLICIES` in `src/run/judge.ts`).
|
|
@@ -1177,10 +1183,14 @@ and the estimate is a lower bound. A record without it, as every record before
|
|
|
1177
1183
|
|
|
1178
1184
|
`interrupted` with `signal` (`SIGINT`, `SIGTERM` or `SIGHUP`) is written by
|
|
1179
1185
|
the CLI's run command when that signal stops a live run outside post-run
|
|
1180
|
-
analysis. It
|
|
1181
|
-
|
|
1182
|
-
|
|
1183
|
-
`
|
|
1186
|
+
analysis. It first prints one line saying it is stopping and refuses any further sandbox
|
|
1187
|
+
create. It then kills the E2B sandboxes the run's `sandbox-receipts.ndjson` names, each sandbox
|
|
1188
|
+
a create in flight reports as its id arrives, and each one E2B lists with the run's owner tags,
|
|
1189
|
+
writes `reclaim-receipt.json` as `humanish reclaim` does, rewrites each run file that names one of
|
|
1190
|
+
the run's raw sandbox ids with the marker and digest, as a finished run's publish does, and exits
|
|
1191
|
+
128+n. The bundle is not finished: run.json keeps its last live flush, with
|
|
1192
|
+
`outcome: { state: interrupted, ok: false, signal, at }` when the run had
|
|
1193
|
+
flushed one, and status.json gets no `outcome`. A `running` record that stopped refreshing still reads as
|
|
1184
1194
|
interrupted once stale. A library caller of `runStudy` gets no signal handling. A
|
|
1185
1195
|
humanish older than this field rejects the state and falls back to the bundle's
|
|
1186
1196
|
own liveness.
|
|
@@ -1192,8 +1202,10 @@ nothing at all until they completed.
|
|
|
1192
1202
|
|
|
1193
1203
|
It is a derived index, not evidence. `run.json` remains the evidence-of-record;
|
|
1194
1204
|
`verify` never gates on `status.json`, nothing in it is a claim about what a
|
|
1195
|
-
participant did, and when the two disagree the bundle wins
|
|
1196
|
-
|
|
1205
|
+
participant did, and when the two disagree the bundle wins. Fields that cannot be
|
|
1206
|
+
rebuilt from `run.json`: `pid`, `startedAt`, `updatedAt` and `completedAt`, which
|
|
1207
|
+
no bundle records, and, on a record written before `run.json` carried an
|
|
1208
|
+
`outcome`, `outcome.ok` and `outcome.execution`. A `running` record whose `updatedAt` is older than three
|
|
1197
1209
|
touch intervals is interrupted, not alive. A dropped connection or a killed
|
|
1198
1210
|
terminal leaves exactly that shape, so the record reads as interrupted
|
|
1199
1211
|
(`classifyRunStatus` is the one shared definition). Fields are
|
|
@@ -1210,16 +1222,20 @@ them is a claim about what a participant did.
|
|
|
1210
1222
|
`humanish.run-index.v1` is the listing projection. One entry per run, read
|
|
1211
1223
|
cheapest-source-first: the `status.json` record, else the bundle, else the run
|
|
1212
1224
|
directory alone. `{ runId, derivedFrom: status | bundle | directory, liveness,
|
|
1213
|
-
mode?, pid?, study?, startedAt?, updatedAt?, completedAt?, verdict?,
|
|
1214
|
-
participants?, estimatedCostUsd?, estimatedCostComplete?,
|
|
1215
|
-
durationMs? }`.
|
|
1225
|
+
mode?, pid?, study?, startedAt?, updatedAt?, completedAt?, verdict?, ok?,
|
|
1226
|
+
failure?, participants?, estimatedCostUsd?, estimatedCostComplete?,
|
|
1227
|
+
analysisCost?, durationMs? }`. `ok` and `failure` (the first execution failure)
|
|
1228
|
+
are the run's own outcome; `runDisplay` (`src/run/display.ts`) reads them with
|
|
1229
|
+
`liveness` and `verdict`, so the TUI and `humanish stats` show a run with
|
|
1230
|
+
`ok: false` as failed whatever its verdict. The point is cost: walking
|
|
1216
1231
|
every run tree and parsing every bundle measured 167ms on a 25-run project,
|
|
1217
1232
|
against 16ms cold and 2.8ms warm here, which is what makes a surface that
|
|
1218
1233
|
refreshes on a cadence affordable. `derivedFrom` is reported so a surprising row
|
|
1219
1234
|
can be traced to the file it came from. A run with receipts and no outcome is
|
|
1220
1235
|
`interrupted` (the shape a dropped connection leaves), and so is an
|
|
1221
1236
|
in-progress bundle reached without a status record: there is no freshness to
|
|
1222
|
-
judge, and nothing in it says the run finished.
|
|
1237
|
+
judge, and nothing in it says the run finished. A bundle with an `outcome` reached
|
|
1238
|
+
without a status record takes its liveness from `outcome.state`.
|
|
1223
1239
|
|
|
1224
1240
|
`estimatedCostUsd` is the participants-and-desktops figure from the status
|
|
1225
1241
|
record or bundle and excludes analysis. `estimatedCostComplete` says whether that
|
|
@@ -1378,7 +1394,7 @@ usage is known or bounded:
|
|
|
1378
1394
|
- A request lost without a reply (it stalled, or the OpenAI provider's
|
|
1379
1395
|
transport threw) is booked at its worst case before it is sent again: the
|
|
1380
1396
|
latest reported request's input at the model's highest input rate plus
|
|
1381
|
-
`
|
|
1397
|
+
`actor.maxOutputTokens` at its output rate, both from `src/run/pricing.ts`.
|
|
1382
1398
|
The reply to the resent request replaces that input with its own. The
|
|
1383
1399
|
request is sent again only when the estimate with that charge stays within
|
|
1384
1400
|
`maxUsd` and the study budget. Otherwise the session ends `budget_reached`
|
|
@@ -1566,7 +1582,7 @@ of it fails.
|
|
|
1566
1582
|
| `SubjectPhaseEvent` | the `subject-phase` `StudyEvent` through `onEvent`. Its `type` is `"subject-phase"`; the phase that was `type` is `name`, and `target` is `{ kind: "subject" }` or `{ kind: "participant", participant }`. `message`, `at`, `ok` and `durationMs` are unchanged |
|
|
1567
1583
|
| `StudyOutcome.backend`, `LabBackend` | `outcome.route` |
|
|
1568
1584
|
| `selectLabBackend`, the `routesTo*` predicates, `actorResolvesToTerminal` | `routeOf(config)` |
|
|
1569
|
-
| `resolveStudyDryRun` | `RunStudyOptions.dryRun`; without it, `
|
|
1585
|
+
| `resolveStudyDryRun` | `RunStudyOptions.dryRun`; without it, `mode` decides |
|
|
1570
1586
|
| `cuaLaneCount` | the `plan` `StudyEvent`, which lists every participant |
|
|
1571
1587
|
| `resolveSeatUrl` | `parseStudy`, which refuses a participant entry that is not same-origin loopback; build the URL with `new URL(entry, serveUrl)` |
|
|
1572
1588
|
| `cuaLaneValidationReason`, `sharedWorldValidationReason`, `concurrentSharedWorldValidationReason`, `externalPublicSharedWorldValidationReason` | `parseStudy`, which returns the same reason |
|
|
@@ -1682,7 +1698,9 @@ the stream shape, the meaningful-use rubric, and hard-failure rules.
|
|
|
1682
1698
|
Review summarizes whether evidence supports the claim. It does not replace
|
|
1683
1699
|
verification or maintainer acceptance.
|
|
1684
1700
|
|
|
1685
|
-
`verdict` is the
|
|
1701
|
+
`verdict` is what the participants experienced, collapsed to one word. Whether
|
|
1702
|
+
the run passed also reads `run.json`'s `outcome.ok` ([Run Outcome](run-bundle.md#run-outcome)):
|
|
1703
|
+
a run whose execution failed shows as failed beside a `pass` verdict. `participants.reachedGoal` retains the count of
|
|
1686
1704
|
recorded successful sessions; it is not an independent adjudication of their
|
|
1687
1705
|
claims. Computer-use review and Observer labels distinguish a participant's
|
|
1688
1706
|
reported completion from a recorded `stopWhen` or dwell condition match. A
|
|
@@ -288,6 +288,51 @@ companion project validated records; generic HTTP serving of producer-owned raw
|
|
|
288
288
|
analysis/correction JSON, execution receipts, atomic write temporaries and locks
|
|
289
289
|
is disabled. Existing adapter captures and logs retain their contained routes.
|
|
290
290
|
|
|
291
|
+
## Reading findings
|
|
292
|
+
|
|
293
|
+
`humanish review` prints a run's analysis findings after its participant review, and
|
|
294
|
+
`humanish analyze show` prints them alone. Each finding shows its title, impact, confidence and
|
|
295
|
+
recovery, the participants it affected out of those exposed, the frames its evidence cites with
|
|
296
|
+
their time since the first retained capture, the capture files, its next step and the latest
|
|
297
|
+
human review note. A live run's human output ends with up to three findings, one line each, and
|
|
298
|
+
the `review` command that prints all of them. `analyze show --json` prints the validated
|
|
299
|
+
analysis record and its corrections, as before. Reading findings never starts an analysis.
|
|
300
|
+
|
|
301
|
+
`review --json` carries the same view as its `analysis` field, schema
|
|
302
|
+
`humanish.analysis-findings.v1`, built in `src/cli/findings.ts`:
|
|
303
|
+
|
|
304
|
+
| Field | Meaning |
|
|
305
|
+
| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
306
|
+
| `schema` | `humanish.analysis-findings.v1`. |
|
|
307
|
+
| `runId` | The run the view reads. |
|
|
308
|
+
| `state` | `ready`, `stale`, `running`, `none`, `skipped`, `failed`, `dry_run` or `unavailable`. Only `ready` and `stale` carry findings. |
|
|
309
|
+
| `reason` | The stable code behind a state other than `ready`, or null: an automatic analysis reason such as `AUTOMATIC_ANALYSIS_KEY_MISSING`, a failed attempt's error code, or a load warning such as `ANALYSIS_SOURCE_CHANGED`. |
|
|
310
|
+
| `message` | What happened, in words. |
|
|
311
|
+
| `next` | The command that gets findings for this run, or null when no command can, as for a dry run or a run whose participants left no evidence. |
|
|
312
|
+
| `analysisId`, `status`, `provider`, `model`, `completedAt`, `estimatedCostUsd` | The analysis version `analyze show` selects: the newest ready one, else the newest stale one. Null without one. |
|
|
313
|
+
| `runPath`, `path` | The run directory and the version's `analysis.json`, relative to the project directory. |
|
|
314
|
+
| `summary`, `limitations` | The analysis's own summary and limitations. |
|
|
315
|
+
| `findings[]` | Highest priority first: `id`, `title`, `summary`, `impact` (`blocked_task`, `friction`, `recovery`, `uncertain`), `confidence` (`low`, `medium`, `high`), `recovery` (`recovered`, `not_observed`, `unknown`), `affected[]` (`streamId`, `label`), `exposedCount`, `evidence[]`, `nextStep` and `correction`. |
|
|
316
|
+
| `findings[].evidence[]` | Each cited item once: `id`, `streamId`, `kind`, `bases` (how the observations used it), `frame` (the latest retained capture's index from 0, null before any capture), `elapsedMs` (time since the first retained capture, not a video offset), `at` and `capture` (the file relative to the project directory, null for nonvisual evidence). |
|
|
317
|
+
| `findings[].correction` | The latest `analyze correct` note on that finding version (`status`, `reason`, `replacementClaim`), or null. |
|
|
318
|
+
| `warnings` | Load warnings for the selected version. |
|
|
319
|
+
|
|
320
|
+
The states without findings:
|
|
321
|
+
|
|
322
|
+
- `running`: the automatic analysis is queued or running. `next` is `review` again.
|
|
323
|
+
- `skipped`: the automatic analysis was skipped or refused, for example without
|
|
324
|
+
`OPENAI_API_KEY`. `next` is `analyze --max-cost 3`.
|
|
325
|
+
- `failed`: the latest attempt failed or was cancelled and no usable version exists. `next`
|
|
326
|
+
reruns it with the same analyst, `--provider codex` for the Codex account analyst.
|
|
327
|
+
- `none`: no analysis ran for this live run, for example with `review.analysis: false`.
|
|
328
|
+
- `dry_run`: a dry run has no participant evidence to analyze.
|
|
329
|
+
- `unavailable`: the run or its analysis records could not be read or checked, or the analysis
|
|
330
|
+
text matched a sensitive-text pattern (`ANALYSIS_SENSITIVE_TEXT_QUARANTINED`), which the
|
|
331
|
+
Observer withholds too.
|
|
332
|
+
|
|
333
|
+
`stale` lists the findings with a warning: the run's evidence changed after the analysis, and
|
|
334
|
+
`next` requests a new one. `analyze show` without `--json` exits 2 when the JSON form would.
|
|
335
|
+
|
|
291
336
|
## Human review and sharing
|
|
292
337
|
|
|
293
338
|
```bash
|
|
@@ -29,6 +29,14 @@ They do not suddenly include a separate analysis request. The bundle's
|
|
|
29
29
|
`cost.estimatedTotalUsd`, `status.json`'s `outcome.estimatedCostUsd` and the run
|
|
30
30
|
index's `estimatedCostUsd` stay participants and desktops only.
|
|
31
31
|
|
|
32
|
+
Each study's `passed` counts the runs that show as passed on every surface
|
|
33
|
+
(`runDisplay` in `src/run/display.ts`): a `pass` verdict and a run whose own `ok`
|
|
34
|
+
is not false. `passRate` divides it by `judged`, the runs with a verdict.
|
|
35
|
+
`totals.verdicts` counts the participants' verdicts; `totals.outcomes` counts
|
|
36
|
+
every selected run by its display state (`passed`, `failed`, `blocked`,
|
|
37
|
+
`timed_out`, `no_verdict`, `dry_run`, `interrupted`, `running`, `unknown`), and
|
|
38
|
+
the human output prints that line.
|
|
39
|
+
|
|
32
40
|
## One run's cost
|
|
33
41
|
|
|
34
42
|
A run's analysis spend has one reader, `readAnalysisAccounting` in
|
|
@@ -18,6 +18,9 @@ the evidence. Everything else is a projection rebuilt from them:
|
|
|
18
18
|
- `status.json` (`humanish.run-status.v1`) is a liveness index. When it disagrees with `run.json`,
|
|
19
19
|
`run.json` wins. `verify` reads it, with the sandbox and reclaim receipts, only to say a run
|
|
20
20
|
did not finish (`RUN_NOT_FINISHED`). That warning changes no check and no grade.
|
|
21
|
+
- A run's `ok` and execution outcome are in `run.json`'s `outcome`, and `status.json` copies them.
|
|
22
|
+
A `status.json` written before `run.json` carried an outcome holds the only copy of that run's
|
|
23
|
+
`ok`, so readers fall back to it for such a run only.
|
|
21
24
|
- Analysis findings cite turns and artifacts in the bundle.
|
|
22
25
|
|
|
23
26
|
## Consequences
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# 0003: Managed paths bind to physical identities; cleanup uses create-time receipts
|
|
2
2
|
|
|
3
|
-
Accepted in 0.15.1, extended in 0.36.0 (`humanish reclaim`).
|
|
3
|
+
Accepted in 0.15.1, extended in 0.36.0 (`humanish reclaim`) and after 0.110.0 (search by owner
|
|
4
|
+
tags).
|
|
4
5
|
|
|
5
6
|
## Context
|
|
6
7
|
|
|
@@ -18,11 +19,16 @@ Separately, an account-wide provider operation once destroyed unrelated infrastr
|
|
|
18
19
|
- The process that creates a resource cleans it up with the handle it holds. After a crash,
|
|
19
20
|
`humanish reclaim` kills sandboxes by the exact ids journaled at create time in the run's
|
|
20
21
|
`sandbox-receipts.ndjson`, and reports each outcome.
|
|
21
|
-
-
|
|
22
|
+
- Nothing lists an account's resources unfiltered. Every sandbox carries its run's owner tags
|
|
23
|
+
(`tool`, `runId`, `runKey`), and reclaim lists only the sandboxes E2B matches to all three,
|
|
24
|
+
checks each tag again before it acts, and stops after three pages. A sandbox whose id never
|
|
25
|
+
reached a receipt, from a create that threw after E2B allocated or a process that died
|
|
26
|
+
mid-create, is otherwise unreachable until its timeout.
|
|
22
27
|
|
|
23
28
|
## Consequences
|
|
24
29
|
|
|
25
|
-
- A shared operator key is safe to use: humanish reaches only resources it created.
|
|
30
|
+
- A shared operator key is safe to use: humanish reaches only resources it created. `runKey` keeps
|
|
31
|
+
two projects that reuse a run id from matching each other's sandboxes.
|
|
26
32
|
- An exported bundle does not carry cleanup authority; `export` omits the receipt journal.
|
|
27
33
|
- Unverified git metadata (arbitrary `gitdir:` redirects) is reported as unavailable instead of
|
|
28
34
|
followed.
|
|
@@ -66,14 +66,17 @@ certify (see the conformance suite).
|
|
|
66
66
|
6. **Claims match mechanism.** A config field that is parsed but not consumed warns; a
|
|
67
67
|
document that overstates behavior is a defect; an evidence artifact never claims a
|
|
68
68
|
stronger evidence class than its actor and scenario can support.
|
|
69
|
-
7. **Reclamation is by exact created id, never account-wide
|
|
70
|
-
creates a provider resource (an E2B sandbox, for example) tracks
|
|
71
|
-
reclaims and proves cleanup against that id alone (a returned
|
|
72
|
-
getInfo-by-id NotFound check). No route ever calls a provider's
|
|
73
|
-
discover or verify its own cleanup, because that
|
|
74
|
-
including resources humanish did not create.
|
|
75
|
-
|
|
76
|
-
|
|
69
|
+
7. **Reclamation is by exact created id or the run's exact owner tags, never account-wide
|
|
70
|
+
enumeration.** A route that creates a provider resource (an E2B sandbox, for example) tracks
|
|
71
|
+
the exact id it created and reclaims and proves cleanup against that id alone (a returned
|
|
72
|
+
found-and-killed boolean, a getInfo-by-id NotFound check). No route ever calls a provider's
|
|
73
|
+
list endpoint unfiltered to discover or verify its own cleanup, because that enumerates the
|
|
74
|
+
whole account/team, including resources humanish did not create. The one listing is reclaim's
|
|
75
|
+
search for a sandbox whose id never reached the run: every sandbox carries its run's owner
|
|
76
|
+
tags (`tool`, `runId`, `runKey`), E2B filters by all three, reclaim checks each tag again
|
|
77
|
+
before it acts, and it reads at most three pages. This is what makes a single shared operator
|
|
78
|
+
key safe: humanish only ever reaches a resource it created, never anything else the key's
|
|
79
|
+
owner is running.
|
|
77
80
|
The one narrow exception is an explicit, maintainer-run orphan sweep (never a default, never
|
|
78
81
|
reachable by a normal run) that is itself opt-in gated (an explicit flag or env var) and
|
|
79
82
|
documented as such at its call site.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "humanish",
|
|
3
|
-
"version": "0.110.
|
|
3
|
+
"version": "0.110.2",
|
|
4
4
|
"description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"cli",
|
|
@@ -56,10 +56,12 @@
|
|
|
56
56
|
},
|
|
57
57
|
"scripts": {
|
|
58
58
|
"build": "tsc -p tsconfig.build.json && pnpm --filter humanish-observer build && pnpm --filter humanish-tui build && node scripts/finish-build.mjs",
|
|
59
|
-
"check": "pnpm format:check && pnpm lint && pnpm knip && pnpm prose:check && pnpm vocabulary:check && pnpm site-css:check && pnpm typecheck && pnpm test && pnpm tui:test && pnpm build && pnpm guest-media:startup:proof && pnpm cli:startup:test && pnpm cli:preflight:test && pnpm cli:dotenv:test && pnpm tui:smoke",
|
|
59
|
+
"check": "pnpm format:check && pnpm lint && pnpm knip && pnpm prose:check && pnpm vocabulary:check && pnpm docs:check && pnpm site-css:check && pnpm typecheck && pnpm test && pnpm tui:test && pnpm build && pnpm guest-media:startup:proof && pnpm cli:startup:test && pnpm cli:preflight:test && pnpm cli:dotenv:test && pnpm cli:peer:test && pnpm cli:skill:test && pnpm tui:smoke",
|
|
60
60
|
"cli:startup:test": "node scripts/terminal-startup-exit-proof.mjs",
|
|
61
61
|
"cli:preflight:test": "node scripts/task-route-preflight-proof.mjs",
|
|
62
62
|
"cli:dotenv:test": "node scripts/dotenv-proof.mjs",
|
|
63
|
+
"cli:peer:test": "node scripts/peer-install-proof.mjs",
|
|
64
|
+
"cli:skill:test": "node scripts/skill-first-run-proof.mjs",
|
|
63
65
|
"humanish": "tsx src/cli.ts",
|
|
64
66
|
"pack:dry-run": "npm pack --dry-run",
|
|
65
67
|
"prepack": "pnpm build",
|
|
@@ -73,9 +75,10 @@
|
|
|
73
75
|
"tui:smoke": "node scripts/tui-smoke.mjs",
|
|
74
76
|
"tui:test": "pnpm --filter humanish-tui test",
|
|
75
77
|
"release:dogfood": "node scripts/release-dogfood.mjs",
|
|
78
|
+
"bench": "tsx bench/run.ts",
|
|
76
79
|
"browser-control:proof": "node scripts/browser-control-proof.mjs",
|
|
77
80
|
"docs:generate": "tsx scripts/generate-cli-docs.ts",
|
|
78
|
-
"docs:check": "tsx scripts/generate-cli-docs.ts --check && tsx scripts/check-doc-paths.ts && tsx scripts/check-doc-symbols.ts && tsx scripts/check-doc-comments.ts",
|
|
81
|
+
"docs:check": "tsx scripts/generate-cli-docs.ts --check && tsx scripts/check-doc-paths.ts && tsx scripts/check-doc-symbols.ts && tsx scripts/check-doc-comments.ts && tsx scripts/check-doc-study-fields.ts",
|
|
79
82
|
"observer:browser:proof": "node scripts/observer-browser-proof.mjs",
|
|
80
83
|
"observer:iframe:proof": "node --import tsx scripts/observer-iframe-proof.mjs",
|
|
81
84
|
"observer:chrome:proof": "node scripts/observer-chrome-proof.mjs",
|
package/skills/humanish/SKILL.md
CHANGED
|
@@ -35,7 +35,7 @@ Everything it shows has a machine-readable equivalent, which is what you want:
|
|
|
35
35
|
| browsing studies | `npx humanish study list --json` |
|
|
36
36
|
| browsing runs | `npx humanish runs --json` |
|
|
37
37
|
| starting a run | `npx humanish run <study> --json --no-open` |
|
|
38
|
-
| a run's
|
|
38
|
+
| a run's findings | `npx humanish review --run <id> --json` |
|
|
39
39
|
| communication setup | `npx humanish comms providers --json` and `npx humanish comms connections list --json` |
|
|
40
40
|
|
|
41
41
|
If a human asks you to "open the TUI", tell them the command to type; do not run
|
|
@@ -289,7 +289,6 @@ Useful commands:
|
|
|
289
289
|
```bash
|
|
290
290
|
npx humanish study list
|
|
291
291
|
npx humanish study show first-run
|
|
292
|
-
npx humanish watch first-run
|
|
293
292
|
npx humanish run first-run --json --no-open
|
|
294
293
|
```
|
|
295
294
|
|
|
@@ -377,25 +376,30 @@ inbox page. See `docs/contracts/schemas.md` for the full `comms:` shape and
|
|
|
377
376
|
## First Proof Run
|
|
378
377
|
|
|
379
378
|
Run the no-credentials path first. It is a dry run: humanish writes a synthetic
|
|
380
|
-
run bundle and tests no target app behavior
|
|
379
|
+
run bundle and tests no target app behavior. Every command in this block
|
|
380
|
+
finishes and exits, needs no key and spends nothing; lines marked
|
|
381
|
+
`# live only` need keys and spend money, so run them only when the user has
|
|
382
|
+
approved a live run. `run` prints the run id; use it in place of `<id>`.
|
|
381
383
|
|
|
382
384
|
```bash
|
|
383
|
-
npx humanish doctor
|
|
384
|
-
npx humanish watch
|
|
385
|
-
npx humanish verify --run latest --json
|
|
386
|
-
npx humanish feedback issue --run latest --repo example/app --format markdown
|
|
387
|
-
```
|
|
388
|
-
|
|
389
|
-
For CI or non-interactive proof:
|
|
390
|
-
|
|
391
|
-
```bash
|
|
392
|
-
npx humanish watch --json --no-open
|
|
385
|
+
npx humanish doctor --json
|
|
393
386
|
npx humanish run first-run --json --no-open
|
|
387
|
+
npx humanish verify --run <id> --json
|
|
388
|
+
npx humanish review --run <id> --json
|
|
389
|
+
npx humanish observe --run <id>
|
|
390
|
+
npx humanish run try-live --json --no-open # live only
|
|
391
|
+
npx humanish feedback issue --run <id> --repo example/app --format markdown # live only
|
|
394
392
|
```
|
|
395
393
|
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
394
|
+
`observe` prints the path of the run's Observer page for the person you are
|
|
395
|
+
working for. Without an interactive terminal, `observe` and `watch` print the
|
|
396
|
+
result and exit; at a person's terminal they serve the Observer until Ctrl-C.
|
|
397
|
+
|
|
398
|
+
`feedback issue` drafts from a live run only and refuses a dry run with
|
|
399
|
+
`HUMANISH_FEEDBACK_REQUIRES_LIVE_RUN`. Its Markdown draft is checked for
|
|
400
|
+
secret, key, token and local-path shapes; it is not checked for personal data. It
|
|
401
|
+
must not call the GitHub API, require a token, update Projects, use provider
|
|
402
|
+
credits, or claim product behavior proof from a dry run.
|
|
399
403
|
|
|
400
404
|
When the target app can run locally, prove a known browser path with a
|
|
401
405
|
scripted-browser study after starting the app on loopback. The study replays a
|
|
@@ -414,8 +418,8 @@ no browser.
|
|
|
414
418
|
# in another terminal, start the target app on 127.0.0.1 or localhost
|
|
415
419
|
npx humanish study show <study> --json
|
|
416
420
|
npx humanish run <study> --json --no-open
|
|
417
|
-
npx humanish verify --run
|
|
418
|
-
npx humanish observe --run
|
|
421
|
+
npx humanish verify --run <id> --json
|
|
422
|
+
npx humanish observe --run <id> --open # for a person; at their terminal it serves until Ctrl-C
|
|
419
423
|
```
|
|
420
424
|
|
|
421
425
|
Do not use `humanish watch --count ...` as a substitute for a scripted-browser
|
|
@@ -442,7 +446,7 @@ Do not paste values into files, prompts, run bundles, issue drafts, or logs.
|
|
|
442
446
|
Load local values only at invocation time:
|
|
443
447
|
|
|
444
448
|
```bash
|
|
445
|
-
npx humanish
|
|
449
|
+
npx humanish run .humanish/studies/local-live.yaml --dotenv .humanish/local/provider.env --json --no-open
|
|
446
450
|
```
|
|
447
451
|
|
|
448
452
|
When choosing dogfood targets, prefer apps, CLIs, or agent-facing tools with a
|
|
@@ -454,7 +458,21 @@ publish their names, screenshots, logs, source snippets, or operational details.
|
|
|
454
458
|
|
|
455
459
|
## Reporting Back
|
|
456
460
|
|
|
457
|
-
|
|
461
|
+
After a live run, read its findings before anything else:
|
|
462
|
+
|
|
463
|
+
```bash
|
|
464
|
+
npx humanish review --run <id> # the verdict, then each analysis finding
|
|
465
|
+
npx humanish review --run <id> --json # the same; findings are under `analysis`
|
|
466
|
+
```
|
|
467
|
+
|
|
468
|
+
Report each finding to the user: its title, impact, confidence and recovery, the participants it
|
|
469
|
+
affected, and the capture files it cites. These come from the post-run analysis of the evidence.
|
|
470
|
+
Label the participant's own report in the review summary as the participant's account. When
|
|
471
|
+
`analysis.state` is not `ready`, report `analysis.message` and the `analysis.next` command in
|
|
472
|
+
place of findings; `analysis.next` is null when no command can produce them. `latest` moves with
|
|
473
|
+
every run, including dry runs, so use the run id the run printed.
|
|
474
|
+
|
|
475
|
+
Then report:
|
|
458
476
|
|
|
459
477
|
- files changed in the target repo;
|
|
460
478
|
- exact proof commands run;
|