humanish 0.139.1 → 0.141.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +23 -0
- package/README.md +5 -4
- package/dist/actors/codex/restricted-participant.js +1 -1
- package/dist/actors/computer-use/external-handoff.d.ts +22 -0
- package/dist/actors/computer-use/external-handoff.js +83 -0
- package/dist/actors/computer-use/loop/observation.d.ts +7 -0
- package/dist/actors/computer-use/loop/observation.js +17 -1
- package/dist/actors/computer-use/loop/types.d.ts +6 -0
- package/dist/analysis/analysis-limits.d.ts +6 -0
- package/dist/analysis/analysis-limits.js +6 -0
- package/dist/analysis/analysis-lock.d.ts +8 -0
- package/dist/analysis/analysis-lock.js +129 -0
- package/dist/analysis/automatic-completion.js +31 -0
- package/dist/analysis/automatic-config.d.ts +15 -6
- package/dist/analysis/automatic-config.js +25 -9
- package/dist/analysis/execute.js +1 -1
- package/dist/analysis/job.d.ts +1 -0
- package/dist/analysis/job.js +6 -2
- package/dist/analysis/recording-ended.d.ts +16 -0
- package/dist/analysis/recording-ended.js +29 -0
- package/dist/analysis/service.d.ts +1 -3
- package/dist/analysis/service.js +23 -50
- package/dist/analysis/source.d.ts +1 -1
- package/dist/browser-control/protocol.d.ts +8 -0
- package/dist/browser-control/protocol.js +21 -0
- package/dist/cli/commands/analysis-signals.d.ts +10 -4
- package/dist/cli/commands/analysis-signals.js +28 -6
- package/dist/cli/commands/runs.js +8 -2
- package/dist/cli/commands/study-route-computer-use.js +4 -5
- package/dist/cli/commands/study-route-scripted.js +5 -6
- package/dist/cli/commands/study-route-shared-world.js +4 -5
- package/dist/cli/commands/study-route-terminal.js +5 -6
- package/dist/cli/commands/study-run.js +1 -1
- package/dist/cli/doctor.d.ts +2 -0
- package/dist/cli/doctor.js +3 -0
- package/dist/cli/findings.d.ts +40 -14
- package/dist/cli/findings.js +117 -35
- package/dist/cli/io.d.ts +5 -1
- package/dist/cli/io.js +12 -2
- package/dist/cli/run-findings.d.ts +19 -0
- package/dist/cli/run-findings.js +106 -0
- package/dist/evidence/browser-hygiene.d.ts +12 -1
- package/dist/evidence/browser-hygiene.js +45 -2
- package/dist/feedback/export-bundle.js +1 -1
- package/dist/guest/bootstrap.d.ts +4 -2
- package/dist/guest/bootstrap.js +15 -7
- package/dist/guest/display-config.d.ts +31 -0
- package/dist/guest/display-config.js +35 -0
- package/dist/guest/external-links.d.ts +14 -0
- package/dist/guest/external-links.js +48 -0
- package/dist/guest/runtime-desktop.d.ts +9 -3
- package/dist/guest/runtime-desktop.js +64 -27
- package/dist/guest/runtime.d.ts +2 -1
- package/dist/guest/runtime.js +1 -1
- package/dist/guest-runtime-main.js +2 -1
- package/dist/observer/data.d.ts +4 -0
- package/dist/observer/run-routes.js +12 -3
- package/dist/observer-app.html +2 -2
- package/dist/routes/computer-use/e2b-desktop/desktop.js +27 -17
- package/dist/routes/computer-use/e2b-desktop/fidelity.js +10 -17
- package/dist/routes/computer-use/e2b-desktop/teardown.js +1 -1
- package/dist/routes/computer-use/in-process-desktop.js +2 -2
- package/dist/routes/computer-use/live-phase.js +1 -1
- package/dist/routes/computer-use/local-vm.js +34 -1
- package/dist/routes/computer-use/plan.js +1 -1
- package/dist/routes/computer-use/setup.js +2 -2
- package/dist/routes/scripted/plan.js +1 -1
- package/dist/routes/shared-world/plan.js +1 -1
- package/dist/routes/shared-world/route.js +1 -1
- package/dist/routes/terminal/plan.js +1 -1
- package/dist/run/display.d.ts +6 -1
- package/dist/run/display.js +11 -0
- package/dist/run/host-suspension.js +4 -1
- package/dist/run/preflight-receipts.d.ts +3 -12
- package/dist/run/preflight-receipts.js +7 -67
- package/dist/run/process-owner.d.ts +28 -0
- package/dist/run/process-owner.js +83 -0
- package/dist/run/projection.js +1 -1
- package/dist/run/reclaim.d.ts +6 -0
- package/dist/run/reclaim.js +32 -2
- package/dist/run/route-shell.js +2 -2
- package/dist/run/run-index.d.ts +3 -1
- package/dist/run/run-index.js +23 -11
- package/dist/run/run.d.ts +16 -5
- package/dist/run/run.js +25 -10
- package/dist/run/status.d.ts +57 -2
- package/dist/run/status.js +87 -1
- package/dist/run/stored-runs.d.ts +3 -0
- package/dist/run/stored-runs.js +9 -4
- package/dist/study/doctor.js +11 -6
- package/dist/study/init-templates.js +9 -3
- package/dist/study/mobile-emulation.d.ts +27 -0
- package/dist/study/mobile-emulation.js +60 -0
- package/dist/study/plan.js +1 -1
- package/dist/study/preflight-result.js +1 -1
- package/dist/study/run-study-events.d.ts +7 -1
- package/dist/study/summary.js +1 -1
- package/dist/study/warnings.js +5 -2
- package/dist/substrates/e2b/cdp-probe.d.ts +2 -6
- package/dist/substrates/e2b/desktop-browser.js +3 -1
- package/dist/substrates/e2b/desktop-cdp.d.ts +0 -2
- package/dist/substrates/e2b/desktop-cdp.js +0 -2
- package/dist/substrates/e2b/desktop-executor.d.ts +7 -0
- package/dist/substrates/e2b/desktop-executor.js +3 -0
- package/dist/substrates/e2b/desktop-handoff.d.ts +22 -0
- package/dist/substrates/e2b/desktop-handoff.js +86 -0
- package/dist/substrates/local/firecracker-desktop.d.ts +5 -0
- package/dist/substrates/local/firecracker-desktop.js +7 -1
- package/dist/substrates/local/runtime-config.d.ts +12 -0
- package/dist/substrates/local/runtime-config.js +55 -13
- package/dist/substrates/local/runtime-release.js +8 -8
- package/dist/substrates/local/runtime.js +1 -0
- package/dist/tui-app.js +115 -115
- package/docs/architecture/observer.md +7 -3
- package/docs/contracts/schemas.md +24 -3
- package/docs/contracts/study-analysis.md +15 -9
- package/docs/principles/invariants-and-defaults.md +1 -1
- package/docs/product/automatic-analysis.md +75 -18
- package/package.json +1 -1
- package/skills/humanish/SKILL.md +153 -31
|
@@ -159,9 +159,13 @@ allowlist, contained file reads, read-only routes and no-store security headers.
|
|
|
159
159
|
Served data may also include `runtime` with `state`, `observedAt`, and
|
|
160
160
|
`source: "local-run-status"`. This is a current observation of a contained,
|
|
161
161
|
matching `status.json`; it never changes the run's recorded verdict or participant
|
|
162
|
-
outcomes. A fresh heartbeat means running,
|
|
163
|
-
finished, a record the CLI's signal handler
|
|
164
|
-
|
|
162
|
+
outcomes. A fresh heartbeat means running, a record its process wrote as `finished` means
|
|
163
|
+
finished, a record written as `interrupted` (by the CLI's signal handler, by a run that ended
|
|
164
|
+
without publishing, or by `humanish reclaim`) means interrupted, and stale or
|
|
165
|
+
invalid timing means unknown. While the run's own process analyzes it after the run ended,
|
|
166
|
+
`runtime.analysis` is `{ startedAt }`: the status band reads **Writing findings** and the
|
|
167
|
+
Findings view shows the analysis as pending, with how long ago it started, before the analysis
|
|
168
|
+
job exists. Missing, malformed, or mismatched records omit the
|
|
165
169
|
observation. A stale heartbeat alone does not prove interruption, and stored PIDs
|
|
166
170
|
are neither probed nor returned. Static rendering and export do not create this
|
|
167
171
|
served-only observation.
|
|
@@ -1411,9 +1411,9 @@ proof claims zero on a `null` line, or when known spend exceeds the declared cap
|
|
|
1411
1411
|
|
|
1412
1412
|
`humanish.run-status.v1` is `status.json`, written inside each run directory by
|
|
1413
1413
|
every route at run start, refreshed on a fixed cadence while the run is
|
|
1414
|
-
alive, and
|
|
1414
|
+
alive, and rewritten when the run ends: `{ schema, runId, state: running |
|
|
1415
1415
|
finished | interrupted, mode, study?, sandboxes?, pid, startedAt, updatedAt, completedAt?,
|
|
1416
|
-
signal?, outcome? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
|
|
1416
|
+
signal?, reason?, outcome?, analysis? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
|
|
1417
1417
|
`study`, and readers accept it. `sandboxes: none` is written at run start by a route that has no
|
|
1418
1418
|
way to create a sandbox for the run: a scripted run against an `app-url` subject. A run that may
|
|
1419
1419
|
create one has no `sandboxes` field. `humanish reclaim` reports such a run `clean` with
|
|
@@ -1453,6 +1453,24 @@ with it at publish: false means some participant or desktop usage has no price
|
|
|
1453
1453
|
and the estimate is a lower bound. A record without it, as every record before
|
|
1454
1454
|
0.109, says nothing about completeness, and readers treat it as unknown.
|
|
1455
1455
|
|
|
1456
|
+
`finished` means the run published its bundle. A process that leaves the run without publishing
|
|
1457
|
+
one, as a route that throws or returns a refusal after the run started does, writes
|
|
1458
|
+
`interrupted` with `reason: no-publish` and no `signal`; humanish 0.139.1 and earlier wrote
|
|
1459
|
+
`finished` with no `outcome` there. `humanish reclaim --run <id>` (not `--check`) rewrites a
|
|
1460
|
+
`running` record whose heartbeat is stale as `interrupted` with `reason: reclaimed` and
|
|
1461
|
+
`completedAt` set to its last `updatedAt`, and reports `markedInterrupted: true`. Either way the
|
|
1462
|
+
run is excluded from a study's duration and cost medians, which count only finished runs. Readers
|
|
1463
|
+
older than `reason` read such a record as interrupted.
|
|
1464
|
+
|
|
1465
|
+
`analysis` is added to a `finished` record when automatic analysis starts in the run's own
|
|
1466
|
+
process: `{ startedAt, endedAt?, state? }`. While it runs, the heartbeat refreshes `updatedAt`
|
|
1467
|
+
again; `completedAt` stays when the participants ended, so a run's duration excludes its
|
|
1468
|
+
analysis. When analysis ends, `endedAt` and `state` (the automatic analysis outcome's state:
|
|
1469
|
+
`complete`, `partial`, `failed`, `cancelled`, `skipped` or `unknown`) are written. An `analysis`
|
|
1470
|
+
without `endedAt` whose `updatedAt` is older than three touch intervals belonged to a process that
|
|
1471
|
+
stopped during analysis. `runAnalysisPhase` (`src/run/status.ts`) reads it as `running`,
|
|
1472
|
+
`stopped` or `ended`. Readers that predate the field ignore it.
|
|
1473
|
+
|
|
1456
1474
|
`interrupted` with `signal` (`SIGINT`, `SIGTERM` or `SIGHUP`) is written by
|
|
1457
1475
|
the CLI's run command when that signal stops a live run outside post-run
|
|
1458
1476
|
analysis. It first prints one line saying it is stopping and refuses any further sandbox
|
|
@@ -1496,7 +1514,10 @@ cheapest-source-first: the `status.json` record, else the bundle, else the run
|
|
|
1496
1514
|
directory alone. `{ runId, derivedFrom: status | bundle | directory, liveness,
|
|
1497
1515
|
mode?, pid?, study?, startedAt?, updatedAt?, completedAt?, verdict?, ok?,
|
|
1498
1516
|
failure?, participants?, estimatedCostUsd?, estimatedCostComplete?,
|
|
1499
|
-
analysisCost?, durationMs? }`. `
|
|
1517
|
+
analysisCost?, durationMs?, analysis? }`. `analysis` is the status record's automatic analysis as
|
|
1518
|
+
`runAnalysisPhase` reads it, `{ state: running | stopped | ended, startedAt, endedAt?, outcome? }`;
|
|
1519
|
+
`humanish runs` and the TUI add `analyzing` or `analysis stopped` to the run's display from it, and
|
|
1520
|
+
`humanish runs --json` carries it on each run. `ok` and `failure` (the first execution failure)
|
|
1500
1521
|
are the run's own outcome; `runDisplay` (`src/run/display.ts`) reads them with
|
|
1501
1522
|
`liveness` and `verdict`, so the TUI and `humanish stats` show a run with
|
|
1502
1523
|
`ok: false` as failed whatever its verdict. The point is cost: walking
|
|
@@ -3,15 +3,16 @@
|
|
|
3
3
|
Study analysis is an independent interpretation of retained participant evidence.
|
|
4
4
|
It is separate from the participant's account, recorded outcome, and the run's
|
|
5
5
|
deterministic review verdict. Opening an Observer never starts a provider request.
|
|
6
|
-
Supported live runs request analysis on completion by default
|
|
7
|
-
|
|
8
|
-
|
|
6
|
+
Supported live runs request analysis on completion by default. A study that declares no analyst
|
|
7
|
+
gets the Codex account analyst when its participant is a Codex `local-agent`, and the OpenAI
|
|
8
|
+
analyst with a separate $10 cap on its expected cost otherwise. Set `review.analysis: false` to
|
|
9
|
+
disable that request; see [automatic analysis](../product/automatic-analysis.md).
|
|
9
10
|
|
|
10
11
|
## Invocation
|
|
11
12
|
|
|
12
13
|
```bash
|
|
13
|
-
humanish analyze --run latest --max-cost
|
|
14
|
-
humanish analyze --run latest --max-cost
|
|
14
|
+
humanish analyze --run latest --max-cost 10 --dry-run --json
|
|
15
|
+
humanish analyze --run latest --max-cost 10 --json
|
|
15
16
|
humanish observe --run latest
|
|
16
17
|
humanish analyze list --run latest --json
|
|
17
18
|
humanish analyze show --run latest --json
|
|
@@ -116,9 +117,11 @@ the current merge instructions, billed $1.92, most of its input served from the
|
|
|
116
117
|
provider's prompt cache, and each of its 466 citations resolved. Two runs do not
|
|
117
118
|
calibrate the merge request's estimate.
|
|
118
119
|
|
|
119
|
-
##
|
|
120
|
+
## Codex account analysis
|
|
120
121
|
|
|
121
|
-
|
|
122
|
+
A study whose participant is a Codex `local-agent` uses this analyst unless it declares another,
|
|
123
|
+
and any study can select it with `review.analysis.provider: codex`. An existing completed
|
|
124
|
+
recording can use it too:
|
|
122
125
|
|
|
123
126
|
```bash
|
|
124
127
|
humanish analyze --run latest --provider codex --dry-run --json
|
|
@@ -421,7 +424,7 @@ It contains only the attempt/run IDs, input/config/source digests, prompt versio
|
|
|
421
424
|
and timestamp. The live caller retains a binding to that exact directory and
|
|
422
425
|
publishes the final receipt once. A start without usable final accounting means
|
|
423
426
|
dispatch and spend remain unresolved; it is not proof that a provider charged.
|
|
424
|
-
A cancellation before transport can
|
|
427
|
+
A cancellation before transport can end with `usage.dispatched: false`.
|
|
425
428
|
Admission refusals, missing credentials and rejected dispatch guards do not
|
|
426
429
|
create a potentially paid attempt. Reuse does not create a new start.
|
|
427
430
|
|
|
@@ -486,7 +489,10 @@ The states without findings:
|
|
|
486
489
|
- `skipped`: the automatic analysis was skipped or refused, for example without
|
|
487
490
|
`OPENAI_API_KEY`, or the run's study file sets `review.analysis: false`
|
|
488
491
|
(`AUTOMATIC_ANALYSIS_DISABLED`, read from the file when the view is built). `next` is
|
|
489
|
-
`analyze --max-cost
|
|
492
|
+
`analyze --max-cost 10`, which runs an analysis anyway. For a skip without `OPENAI_API_KEY`
|
|
493
|
+
on a machine where Codex is signed in with a ChatGPT account (read when the view is built),
|
|
494
|
+
`next` is `analyze --provider codex` and the message names `review.analysis.provider: codex`
|
|
495
|
+
in the run's study file; otherwise the message says to set the key. For an analysis refused for its
|
|
490
496
|
cost, the message gives the expected cost, the worst case and the cap, and `next` sets
|
|
491
497
|
`--max-cost` to the worst case rounded up.
|
|
492
498
|
- `failed`: the latest attempt failed or was cancelled and no usable version exists. `next`
|
|
@@ -90,7 +90,7 @@ silently drifting from one is not.
|
|
|
90
90
|
| Default | Why it is the default | Legitimate override |
|
|
91
91
|
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
92
92
|
| Dry-run | Spend safety (the live-spend rule sets the floor; dry-run keeps it far away) | `mode: live` |
|
|
93
|
-
| Post-run analysis on supported live studies | Findings accompany the recording; a separate $
|
|
93
|
+
| Post-run analysis on supported live studies | Findings accompany the recording; a Codex participant's run is analyzed by the same account, and otherwise a separate $10 cap on its expected cost is visible before execution | `review.analysis: false` disables it; an explicit mapping sets another analysis budget. Dry-run and unsupported routes never dispatch; missing default credentials produce a recorded skip |
|
|
94
94
|
| Separate worlds (`per-lane-worlds`) | Isolation, attribution, reproducibility | `route: shared-world`: N participants against one mutable plane for scenarios that are about interaction between participants. Participants run at once, as many as `execution.concurrency` (two or more) and the E2B plan's limit allow, unless the study declares when each one starts (`participants[].startAfterMs`). The plane is one getHost-exposed subject sandbox with synthetic seeded data, or an operator-owned public deployment (external-public), driven by N actor sandboxes. The bundle declares the weaker `attributionClass: shared-world` and a verify-enforced `attributionLimits` ceiling (`best-effort-causal-attribution` and the rest of the concurrent set). Sequential turns at `execution.concurrency: 1` were removed in 0.106.0. |
|
|
95
95
|
| External key placement | Smallest blast radius: when the keyed process (e.g. a computer-use provider loop) runs outside the sandbox, its key never enters | In-sandbox placement when the keyed process runs inside (an agent harness under test); declared per actor type, with a spend budget |
|
|
96
96
|
| Loopback entry URLs | Public-safety: never drive third-party sites unbidden | `policies.allowPublicTargets` for an owner-declared deployment/preview (a Vercel preview of your own app). Multi-participant public/preview fan-out needs explicit `participants[].target` for every participant, so the adapter-owned topology is declared rather than inferred. Provisioned clone subjects always serve in-sandbox on loopback |
|
|
@@ -3,7 +3,14 @@
|
|
|
3
3
|
Supported live studies automatically request analysis after each recording finishes. Findings remain
|
|
4
4
|
separate from participant feedback and the recorded study verdict.
|
|
5
5
|
|
|
6
|
-
|
|
6
|
+
A study that declares no `review.analysis` gets its analyst from its participant (`actor`). A
|
|
7
|
+
Codex `local-agent` participant, on a local or hosted desktop, gets the Codex account analyst
|
|
8
|
+
described below, so a study run on a signed-in Codex needs no OpenAI API key for its findings.
|
|
9
|
+
Every other participant, including a Claude Code `local-agent` participant, gets the OpenAI
|
|
10
|
+
analyst. humanish never moves a study to the other analyst because a key is missing: the account
|
|
11
|
+
that pays for analysis is the one the study names or its participant implies.
|
|
12
|
+
|
|
13
|
+
The OpenAI default is `gpt-6-astra` with high reasoning effort, a separate $10 cap on its
|
|
7
14
|
expected cost and a 600-second timeout. When no output limit is specified,
|
|
8
15
|
humanish selects 32,768 tokens if the exact input's expected cost still fits that
|
|
9
16
|
cap; otherwise it keeps the established 16,384-token allowance. This preserves
|
|
@@ -12,7 +19,7 @@ previously admitted studies without increasing their spending limit. To customiz
|
|
|
12
19
|
```yaml
|
|
13
20
|
review:
|
|
14
21
|
analysis:
|
|
15
|
-
maxCostUsd:
|
|
22
|
+
maxCostUsd: 10
|
|
16
23
|
# Optional overrides. Omit maxOutputTokens for budget-aware selection.
|
|
17
24
|
model: gpt-6-astra
|
|
18
25
|
timeoutMs: 600000
|
|
@@ -20,7 +27,8 @@ review:
|
|
|
20
27
|
# question: Where did participants need to recover?
|
|
21
28
|
```
|
|
22
29
|
|
|
23
|
-
Omitting `review.analysis` uses these defaults
|
|
30
|
+
Omitting `review.analysis` uses these defaults for any participant other than a Codex
|
|
31
|
+
`local-agent`. Set `review.analysis: false` to
|
|
24
32
|
run participants without the additional analysis request. An explicit analysis
|
|
25
33
|
mapping using the default OpenAI API provider requires `maxCostUsd`. Admission
|
|
26
34
|
refuses the request when both its worst case and its expected cost plus a 10% margin are over
|
|
@@ -33,9 +41,10 @@ the cap admits no analysis at all, it says so instead. A study of more than 16 p
|
|
|
33
41
|
analysed in cohorts of at most 16, one request each, and one more request merges their reports;
|
|
34
42
|
the range covers every request and the analysis line names them (see
|
|
35
43
|
[runs with more than 16 participants](../contracts/study-analysis.md#runs-with-more-than-16-participants)).
|
|
36
|
-
The default $
|
|
37
|
-
|
|
38
|
-
|
|
44
|
+
The default $10 cap admits a single request at the evidence limits ($3.79 expected for one
|
|
45
|
+
participant, $4.54 for 16) and the analysis of 17 to 40 participants from $4.60 to $7.05 with no
|
|
46
|
+
evidence; a run that keeps more evidence can still be refused, and its skip names the
|
|
47
|
+
`--max-cost` that admits it. Use `analyze --dry-run --max-cost <usd>` on retained
|
|
39
48
|
evidence to inspect the expected cost, the worst case and the selected token
|
|
40
49
|
allowance before deliberately choosing a larger budget. Explicit
|
|
41
50
|
`maxOutputTokens` and `--max-output-tokens` limits are honored exactly. The output allowance
|
|
@@ -65,8 +74,8 @@ enforced output-token ceiling are unknown, so omit `maxCostUsd` and
|
|
|
65
74
|
allocated. There is no API fallback. Missing or unsupported account setup leaves
|
|
66
75
|
an explicit failed analysis state and the original recording intact. Use
|
|
67
76
|
`humanish doctor --study <study>` for setup checks; account allowance and model access
|
|
68
|
-
remain untested until a request. An omitted provider
|
|
69
|
-
|
|
77
|
+
remain untested until a request. An omitted provider means this analyst for a Codex
|
|
78
|
+
`local-agent` participant on any target, and OpenAI for every other participant. This setting
|
|
70
79
|
does not enable managed local desktops.
|
|
71
80
|
|
|
72
81
|
The same configuration works through `humanish run <study>`,
|
|
@@ -77,13 +86,34 @@ never enables analysis by default and rejects an explicit analysis mapping
|
|
|
77
86
|
before execution. `false` is accepted on every route. Dry runs show analysis
|
|
78
87
|
as skipped, without reading analysis credentials or making a provider request.
|
|
79
88
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
89
|
+
Automatic analysis starts when the run ends: every participant has stopped and the run has
|
|
90
|
+
published its bundle (`run.json` with an outcome, and `status.json` state `finished`). It runs in
|
|
91
|
+
the same process, so `humanish run` returns only after it. A participant who was blocked or
|
|
92
|
+
interrupted can still have useful retained evidence; analysis requires a verified live recording
|
|
93
|
+
whose participant streams all ended, not a successful participant outcome. An active, missing or
|
|
94
|
+
invalid recording is not analyzed. Default analysis also skips recordings containing only setup
|
|
95
|
+
or failure records with no retained participant activity. A desktop startup failure does not
|
|
96
|
+
start an analysis request. The original failure remains visible.
|
|
97
|
+
|
|
98
|
+
While analysis runs, the run reads as analyzing. `status.json` records `analysis.startedAt` and
|
|
99
|
+
its heartbeat keeps refreshing `updatedAt`; `humanish runs` lists the run as `passed, analyzing`
|
|
100
|
+
(or its other result); the TUI adds `analyzing` to the run; the served Observer shows **Writing
|
|
101
|
+
findings** in its status band and the Findings tab in a pending state with how long ago analysis
|
|
102
|
+
started; `humanish review` says the analysis is running and when it started. In a terminal, `run`
|
|
103
|
+
prints `Participants finished. Writing findings; this usually takes 1 to 4 minutes.` to stderr,
|
|
104
|
+
then `Still writing findings: <elapsed> elapsed.` every 20 seconds. With `--json` it prints only
|
|
105
|
+
the first line, on stderr.
|
|
106
|
+
|
|
107
|
+
A run whose process ended before it published (a crash, or a refusal after the run started) is
|
|
108
|
+
recorded `interrupted` with `reason: no-publish`, and one whose process was killed is marked
|
|
109
|
+
`interrupted` by `humanish reclaim`. `humanish analyze` accepts an interrupted run when every
|
|
110
|
+
participant stream ended; when one was still running it refuses with
|
|
111
|
+
`ANALYSIS_RUN_INCOMPLETE` and names `humanish run <study>`. It refuses a run that is still alive
|
|
112
|
+
with `ANALYSIS_RUN_ACTIVE`, and a run whose heartbeat stopped while its pid is still in use with
|
|
113
|
+
`ANALYSIS_RUN_UNCONFIRMED`, naming `humanish reclaim --run <id>`. When the process running an
|
|
114
|
+
automatic analysis stops before it finishes, `review` says so and names the `analyze` command
|
|
115
|
+
that runs it again. That command takes over the run's `.analysis-lock` when the process that
|
|
116
|
+
held it is gone, and says so in a warning.
|
|
87
117
|
|
|
88
118
|
CLI live starts disclose the selected analyst, its expected cost range and cap, or unknown account dollars before execution.
|
|
89
119
|
`humanish study check <study> --json` and the TUI study screen also expose the
|
|
@@ -95,7 +125,7 @@ runner continues after the TUI closes; reopening the TUI or Observer reads the
|
|
|
95
125
|
existing job and does not start another request. Concurrent or repeated automatic
|
|
96
126
|
invocations cannot silently retry a paid attempt. If a process disappears while
|
|
97
127
|
an attempt is in flight, its state can be unknown rather than falsely complete.
|
|
98
|
-
Use manual `humanish analyze --run <exact-run-id> --max-cost
|
|
128
|
+
Use manual `humanish analyze --run <exact-run-id> --max-cost 10` for an intentional
|
|
99
129
|
follow-up after inspecting the existing attempt and its accounting. For the account branch, use `humanish analyze --run <exact-run-id> --provider codex --rerun` without a dollar limit.
|
|
100
130
|
|
|
101
131
|
Stopping participant execution does not start a fresh automatic analysis. A
|
|
@@ -113,8 +143,12 @@ unknown analysis produces exit code 2 without discarding the recording. When the
|
|
|
113
143
|
run's own result is ok, the first line of the human output reads
|
|
114
144
|
`humanish run <study>: live run finished; the analysis did not complete`. A missing
|
|
115
145
|
`OPENAI_API_KEY` skips default analysis and preserves a successful run exit;
|
|
116
|
-
`automaticAnalysisTrigger: "default"` distinguishes that case in JSON.
|
|
117
|
-
|
|
146
|
+
`automaticAnalysisTrigger: "default"` distinguishes that case in JSON. The skip names the next
|
|
147
|
+
command: with Codex signed in with a ChatGPT account it is `humanish analyze --run <id>
|
|
148
|
+
--provider codex`, and the message names `review.analysis.provider: codex` as the setting that
|
|
149
|
+
makes that analyst the study's; otherwise it says to set the key and names `humanish analyze
|
|
150
|
+
--run <id> --max-cost 10`. So does a
|
|
151
|
+
default analysis whose expected cost, plus the margin, is over the default $10 cap:
|
|
118
152
|
it is skipped with `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED` and the run exits 0. For
|
|
119
153
|
any analysis refused for its cost, the job records the expected cost, the worst
|
|
120
154
|
case and the cap, and the CLI, `humanish review` and the Observer give them with
|
|
@@ -126,6 +160,28 @@ findings remain visibly partial; a valid partial result can succeed, while a
|
|
|
126
160
|
partial result with an analysis error still fails the command. Recorded task
|
|
127
161
|
outcomes and the deterministic review verdict are never rewritten by analysis.
|
|
128
162
|
|
|
163
|
+
After a live run the human output ends with the analysis line. When the analysis left findings,
|
|
164
|
+
the line gives its state and the top three findings follow, with the `humanish review --run <id>`
|
|
165
|
+
command that prints all of them. Otherwise the line says what happened and a `next:` line gives
|
|
166
|
+
the command that gets findings, when one can. A dry run keeps its one line,
|
|
167
|
+
`analysis: skipped for dry runs`.
|
|
168
|
+
|
|
169
|
+
`run --json` and `watch --json` carry the same on `automaticAnalysis`: `message`, the sentence the
|
|
170
|
+
human line is built from, and `nextCommand`, the command that gets findings or, when the analysis
|
|
171
|
+
left them, the `review` command (null when no command can). When the analysis left findings the
|
|
172
|
+
JSON also has `findings`:
|
|
173
|
+
|
|
174
|
+
| Field | Meaning |
|
|
175
|
+
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
176
|
+
| `analysisId` | The analysis the findings come from |
|
|
177
|
+
| `status` | `complete` or `partial` |
|
|
178
|
+
| `count` | How many findings the analysis made |
|
|
179
|
+
| `designFindingCount` | How many design findings it made; null for an analysis written before them |
|
|
180
|
+
| `top` | At most five findings, highest priority first: `id`, `title`, `headline`, `impact`, `confidence` and `participants` (the affected participant ids) |
|
|
181
|
+
| `review` | `humanish review --run <id>`, which prints every finding with its evidence |
|
|
182
|
+
| `observer` | The run's Observer page opened on the findings (`observer/index.html#/report`); null without an Observer |
|
|
183
|
+
| `path` | analysis.json relative to the project directory |
|
|
184
|
+
|
|
129
185
|
`automaticAnalysis.reason` is one of these codes, in `--json` and in the human line
|
|
130
186
|
`analysis: <state> (<reason>)`:
|
|
131
187
|
|
|
@@ -152,6 +208,7 @@ outcomes and the deterministic review verdict are never rewritten by analysis.
|
|
|
152
208
|
| `AUTOMATIC_ANALYSIS_PUBLICATION_FAILED` | `failed` | The analysis could not be published |
|
|
153
209
|
| `AUTOMATIC_ANALYSIS_CANCELLATION_UNAVAILABLE` | `unknown` | A cancellation could not be recorded |
|
|
154
210
|
| `AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN` | `unknown` | The outcome could not be determined |
|
|
211
|
+
| `AUTOMATIC_ANALYSIS_STOPPED` | `unknown` | Read only: the process running it stopped refreshing its job record |
|
|
155
212
|
|
|
156
213
|
A run refused before it was created (`run: not-created`) prints no `analysis:` line, since there
|
|
157
214
|
is no run to analyze and the refusal says why. Its JSON keeps the record, with
|
package/package.json
CHANGED
package/skills/humanish/SKILL.md
CHANGED
|
@@ -95,10 +95,13 @@ exact returned path, not a basename that could resolve to another manifest.
|
|
|
95
95
|
|
|
96
96
|
## Choosing a findings analyst
|
|
97
97
|
|
|
98
|
-
Analysis is separate from the participant.
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
98
|
+
Analysis is separate from the participant. A study that declares no analyst
|
|
99
|
+
gets one from its participant: a Codex `local-agent` participant, local or
|
|
100
|
+
hosted, defaults to a separate Codex account analyst; every other participant,
|
|
101
|
+
Claude Code included, defaults to the OpenAI API with a $10 admission budget.
|
|
102
|
+
Without `OPENAI_API_KEY` that OpenAI analysis is skipped, and the run output
|
|
103
|
+
names the next command: `analyze --provider codex` when Codex is signed in. To
|
|
104
|
+
select the restricted Codex ChatGPT account analyst on any supported study,
|
|
102
105
|
set `review.analysis.provider: codex` or pass `analyze --provider codex` on a
|
|
103
106
|
completed recording. This uses remote inference and the qualified CLI/login,
|
|
104
107
|
not local inference or the participant's existing conversation. See
|
|
@@ -133,7 +136,8 @@ the pinned runtime; a live local run also prepares it automatically. The normal
|
|
|
133
136
|
See [the complete example and limits](https://humanish.dev/docs/local-browser).
|
|
134
137
|
|
|
135
138
|
Local browsers currently require a loopback app URL with an explicit port above
|
|
136
|
-
1023 and use a 960×720 Chromium desktop
|
|
139
|
+
1023 and use a 960×720 Chromium desktop unless a device preset or
|
|
140
|
+
`execution.desktop.resolution` sets each participant's screen. For email-gated local apps, start
|
|
137
141
|
`humanish comms catch`, point the app's email sends at it, and declare
|
|
138
142
|
`comms.email.external.catchBaseUrl`. Use `doctor --study` to check its recipient
|
|
139
143
|
routes. This captures app sends without mailbox-provider credentials; it does
|
|
@@ -294,6 +298,63 @@ fails with `HUMANISH_STUDY_RETIRED_DIRECTORY`; move it to the matching `studies/
|
|
|
294
298
|
run looks like idle tiles — another reason to leave the cap out unless you
|
|
295
299
|
need it.
|
|
296
300
|
|
|
301
|
+
### Phone, tablet and desktop in one study
|
|
302
|
+
|
|
303
|
+
When the user asks to test several screen sizes, write one study with one
|
|
304
|
+
`participants` entry per size and set `device` on each entry. The participants run
|
|
305
|
+
at once, the Observer shows them side by side, and the findings compare them.
|
|
306
|
+
|
|
307
|
+
```yaml
|
|
308
|
+
schema: humanish.study.v3
|
|
309
|
+
id: devices
|
|
310
|
+
title: Phone, tablet and desktop
|
|
311
|
+
route: computer-use
|
|
312
|
+
mode: live
|
|
313
|
+
subject:
|
|
314
|
+
source: app-url
|
|
315
|
+
appUrl: "http://127.0.0.1:3000"
|
|
316
|
+
actor:
|
|
317
|
+
type: local-agent
|
|
318
|
+
localAgent: codex
|
|
319
|
+
persona: synthetic-new-user
|
|
320
|
+
mission: "Sign up and create your first project. Say what was hard to read or reach."
|
|
321
|
+
participants:
|
|
322
|
+
- id: phone
|
|
323
|
+
device: mobile
|
|
324
|
+
- id: tablet
|
|
325
|
+
device: tablet
|
|
326
|
+
- id: desktop
|
|
327
|
+
device: desktop
|
|
328
|
+
execution:
|
|
329
|
+
target: local
|
|
330
|
+
timeoutMs: 600000
|
|
331
|
+
desktop:
|
|
332
|
+
fidelity:
|
|
333
|
+
mobileEmulation: true
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
This runs on the local browser runtime with the user's Codex login. For a deployed
|
|
337
|
+
app, set `execution.target: e2b-desktop`, put the public URL in `subject.appUrl`,
|
|
338
|
+
add `policies.allowPublicTargets: true`, and give each entry a `target` (below).
|
|
339
|
+
|
|
340
|
+
- `device` takes a preset: `mobile` (414×896), `small-mobile`, `narrow-mobile`,
|
|
341
|
+
`tablet` (820×1180), `desktop` (1440×950) or `wide` (1920×1080). The preset
|
|
342
|
+
sizes the participant's screen and browser window.
|
|
343
|
+
- `fidelity.mobileEmulation: true` gives the phone participants a mobile viewport,
|
|
344
|
+
device pixel ratio, touch and a mobile user agent in Chromium. Tablet and
|
|
345
|
+
desktop participants are left as they are.
|
|
346
|
+
- Do not run one study per size, draw a device frame inside the page, or resize
|
|
347
|
+
the browser from the mission. Those produce desktop-ratio evidence labelled as
|
|
348
|
+
a phone, and three runs the Observer cannot compare.
|
|
349
|
+
- Participants on one public app (`policies.allowPublicTargets: true`) share its
|
|
350
|
+
data, so more than one is refused unless each entry sets `target`: a separate
|
|
351
|
+
test account, tenant or seeded URL per participant, or the same URL when the
|
|
352
|
+
task has each participant create its own account. A loopback or clone subject
|
|
353
|
+
needs no `target`.
|
|
354
|
+
- On a local study, `doctor --study` fails its runtime row when the installed
|
|
355
|
+
local runtime predates per-participant screens; `humanish runtime setup`
|
|
356
|
+
installs the pinned one. Do not move the study to E2B without asking the user.
|
|
357
|
+
|
|
297
358
|
Use committed `humanish/studies/*.yaml` for public-safe, reproducible studies. Use
|
|
298
359
|
ignored `.humanish/studies/*.yaml` or `.humanish/local/studies/*.yaml` for private repo
|
|
299
360
|
targets, local-only dogfood, or machine-specific settings. Never commit private
|
|
@@ -419,14 +480,15 @@ credits, or claim product behavior proof from a dry run.
|
|
|
419
480
|
When the target app can run locally, prove a known browser path with a
|
|
420
481
|
scripted-browser study after starting the app on loopback. The study replays a
|
|
421
482
|
`browser.steps` scenario (see above) on a desktop surface, plus a mobile one
|
|
422
|
-
with `count: 2`. The scripted steps make no model requests
|
|
423
|
-
|
|
424
|
-
`review.analysis: false` to skip it. Copy the shape of
|
|
483
|
+
with `count: 2`. The scripted steps make no model requests; the analysis after
|
|
484
|
+
the run does, and it is what produces findings. Copy the shape of
|
|
425
485
|
[`humanish/studies/scripted-demo.yaml`](https://github.com/danielgwilson/humanish/blob/main/humanish/studies/scripted-demo.yaml)
|
|
426
486
|
or the example in the
|
|
427
|
-
[study file reference](https://humanish.dev/docs/study-files#scripted-browser-scenarios)
|
|
428
|
-
|
|
429
|
-
|
|
487
|
+
[study file reference](https://humanish.dev/docs/study-files#scripted-browser-scenarios),
|
|
488
|
+
leaving out the demo's `review.analysis: false`, which keeps that zero-spend
|
|
489
|
+
demo from requesting analysis: set `subject.appUrl` to the loopback URL,
|
|
490
|
+
`scenario` to the scenario id, and `mode: live`. Set `review.analysis: false`
|
|
491
|
+
only when the user has chosen a run with no findings. Without `mode: live` the study is a dry run that opens
|
|
430
492
|
no browser.
|
|
431
493
|
|
|
432
494
|
```bash
|
|
@@ -441,6 +503,52 @@ Do not use `humanish watch --count ...` as a substitute for a scripted-browser
|
|
|
441
503
|
study. `watch` without a study starts a preview run for the Observer; a live scripted-browser
|
|
442
504
|
study captures desktop and mobile browser evidence against a running app.
|
|
443
505
|
|
|
506
|
+
## After a Live Run, Read and Report Its Findings
|
|
507
|
+
|
|
508
|
+
A live run ends with a separate analysis of its evidence, and `run --json`
|
|
509
|
+
carries what that analysis left. `findings.count` is how many findings it made;
|
|
510
|
+
`findings.top` holds at most five, each with its `title`, `headline`, `impact`,
|
|
511
|
+
`confidence` and the ids of the `participants` it affected; `findings.review` is
|
|
512
|
+
the command that prints every finding with its evidence; `findings.observer` is
|
|
513
|
+
the Observer page that opens on the findings. When the analysis left no
|
|
514
|
+
findings, `findings` is absent, `automaticAnalysis.message` says why, and
|
|
515
|
+
`automaticAnalysis.nextCommand` is the command that gets them (null when no
|
|
516
|
+
command can). Run that command, or tell the user to.
|
|
517
|
+
|
|
518
|
+
Read the full findings before reporting anything else:
|
|
519
|
+
|
|
520
|
+
```bash
|
|
521
|
+
npx humanish review --run <id> # the verdict, then each analysis finding
|
|
522
|
+
npx humanish review --run <id> --json # the same; findings are under `analysis`
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
Report each finding to the user: its title, impact, confidence and recovery, the participants it
|
|
526
|
+
affected, and the capture files it cites. These come from the post-run analysis of the evidence.
|
|
527
|
+
Label the participant's own report in the review summary as the participant's account. When
|
|
528
|
+
`analysis.state` is not `ready`, report `analysis.message` and the `analysis.next` command in
|
|
529
|
+
place of findings; `analysis.next` is null when no command can produce them. `latest` moves with
|
|
530
|
+
every run, including dry runs, so use the run id the run printed.
|
|
531
|
+
|
|
532
|
+
## Long Studies: `run` Blocks Until Its Findings
|
|
533
|
+
|
|
534
|
+
`humanish run <study>` returns only after every participant has ended and the automatic analysis
|
|
535
|
+
has finished. A participant with `endAfterMs` can keep a study running for up to 24 hours, and
|
|
536
|
+
analysis usually adds 1 to 4 minutes after the last participant ends. A shell timeout that sends
|
|
537
|
+
SIGTERM stops the run: humanish marks it interrupted and stops its sandboxes.
|
|
538
|
+
|
|
539
|
+
- Start a study that can outlive your shell call in tmux, for example
|
|
540
|
+
`tmux new -d -s study 'npx humanish run <study> --json --no-open > run-result.json 2> run.log'`.
|
|
541
|
+
`--detach` only skips the Observer server at the end; it does not move the run to the background.
|
|
542
|
+
- The first stderr line names the run: `Run <id> started. Follow it with humanish observe --run <id>`.
|
|
543
|
+
If you did not keep it, `npx humanish runs --json` lists runs newest first.
|
|
544
|
+
- In `runs --json`, a run whose participants are working has `display.state: running`; once they
|
|
545
|
+
end, its `display` is their result and `analysis.state` is `running` while findings are written.
|
|
546
|
+
- `npx humanish review --run <id> --json` says where the run is: participants running, analysis
|
|
547
|
+
running since a given time, or the findings.
|
|
548
|
+
- If the run's process died, `npx humanish reclaim --run <id>` stops its sandboxes and marks it
|
|
549
|
+
interrupted. `npx humanish analyze --run <id>` then analyzes it if every participant had ended,
|
|
550
|
+
and otherwise names the command that runs the study again.
|
|
551
|
+
|
|
444
552
|
## Optional Live E2B Study
|
|
445
553
|
|
|
446
554
|
Live headed E2B desktop participants are optional. Add the substrate dependency only
|
|
@@ -450,12 +558,18 @@ when the user explicitly wants live E2B execution:
|
|
|
450
558
|
npm i -D @e2b/desktop
|
|
451
559
|
```
|
|
452
560
|
|
|
453
|
-
Then confirm env var names are documented without values
|
|
561
|
+
Then confirm env var names are documented without values. Which keys a hosted
|
|
562
|
+
study needs depends on its participant and analyst:
|
|
454
563
|
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
OPENAI_API_KEY
|
|
458
|
-
|
|
564
|
+
| Participant | Keys |
|
|
565
|
+
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
566
|
+
| `openai-computer-use` | `E2B_API_KEY`, `OPENAI_API_KEY` |
|
|
567
|
+
| `local-agent` with `localAgent: codex` | `E2B_API_KEY` and `codex login`; the findings analysis uses the same Codex account |
|
|
568
|
+
| `local-agent` with `localAgent: claude` | `E2B_API_KEY` and `claude auth login`; `OPENAI_API_KEY` for findings; without it, a signed-in Codex can analyze the run (the run prints the command) |
|
|
569
|
+
|
|
570
|
+
A hosted Codex participant does not need `OPENAI_API_KEY`. Do not report a missing
|
|
571
|
+
`OPENAI_API_KEY` as the reason a Codex study on E2B fails; read the
|
|
572
|
+
`doctor --study` row instead.
|
|
459
573
|
|
|
460
574
|
Do not paste values into files, prompts, run bundles, issue drafts, or logs.
|
|
461
575
|
Load local values only at invocation time:
|
|
@@ -471,23 +585,31 @@ declared scenario is developer-experience testing. Private repos are allowed
|
|
|
471
585
|
only as explicit maintainer-authorized runs with repo redaction left on; never
|
|
472
586
|
publish their names, screenshots, logs, source snippets, or operational details.
|
|
473
587
|
|
|
474
|
-
##
|
|
475
|
-
|
|
476
|
-
|
|
588
|
+
## Setup errors and what they mean
|
|
589
|
+
|
|
590
|
+
Run `npx humanish doctor --study <id>` before the first live run and read every
|
|
591
|
+
`missing` row. The common ones:
|
|
592
|
+
|
|
593
|
+
- `codex_unsafe_configuration` on a hosted Codex participant: the row names the
|
|
594
|
+
setting humanish refused. Report it to the user; never edit, move or rename
|
|
595
|
+
their Codex, agent or skill config to make a run pass. An `AGENTS.md` in the
|
|
596
|
+
Codex home is admitted, and doctor notes that the participant also receives it.
|
|
597
|
+
- `subject.appUrl` must be loopback: a non-loopback URL needs
|
|
598
|
+
`policies.allowPublicTargets: true`, and with more than one participant, a
|
|
599
|
+
`target` on each entry.
|
|
600
|
+
- "Will be skipped: OPENAI_API_KEY is missing" on `post-run analysis` (an
|
|
601
|
+
`openai-computer-use` or Claude participant): the run works without findings,
|
|
602
|
+
and when Codex is signed in, its `next:` line gives the command that analyzes it with Codex.
|
|
603
|
+
`review.analysis.provider: codex` makes that the study's default.
|
|
604
|
+
- A local browser study refuses a non-loopback URL, a port below 1024, hosted
|
|
605
|
+
templates and `sandboxTimeoutMs`. The message names the setting; do not switch the
|
|
606
|
+
study to a hosted target unless the user agrees to E2B usage.
|
|
477
607
|
|
|
478
|
-
|
|
479
|
-
npx humanish review --run <id> # the verdict, then each analysis finding
|
|
480
|
-
npx humanish review --run <id> --json # the same; findings are under `analysis`
|
|
481
|
-
```
|
|
482
|
-
|
|
483
|
-
Report each finding to the user: its title, impact, confidence and recovery, the participants it
|
|
484
|
-
affected, and the capture files it cites. These come from the post-run analysis of the evidence.
|
|
485
|
-
Label the participant's own report in the review summary as the participant's account. When
|
|
486
|
-
`analysis.state` is not `ready`, report `analysis.message` and the `analysis.next` command in
|
|
487
|
-
place of findings; `analysis.next` is null when no command can produce them. `latest` moves with
|
|
488
|
-
every run, including dry runs, so use the run id the run printed.
|
|
608
|
+
## Reporting Back
|
|
489
609
|
|
|
490
|
-
|
|
610
|
+
Report a live run's findings first, as
|
|
611
|
+
[After a Live Run, Read and Report Its Findings](#after-a-live-run-read-and-report-its-findings)
|
|
612
|
+
describes. Then report:
|
|
491
613
|
|
|
492
614
|
- files changed in the target repo;
|
|
493
615
|
- exact proof commands run;
|