humanish 0.139.1 → 0.141.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (120) hide show
  1. package/CHANGELOG.md +23 -0
  2. package/README.md +5 -4
  3. package/dist/actors/codex/restricted-participant.js +1 -1
  4. package/dist/actors/computer-use/external-handoff.d.ts +22 -0
  5. package/dist/actors/computer-use/external-handoff.js +83 -0
  6. package/dist/actors/computer-use/loop/observation.d.ts +7 -0
  7. package/dist/actors/computer-use/loop/observation.js +17 -1
  8. package/dist/actors/computer-use/loop/types.d.ts +6 -0
  9. package/dist/analysis/analysis-limits.d.ts +6 -0
  10. package/dist/analysis/analysis-limits.js +6 -0
  11. package/dist/analysis/analysis-lock.d.ts +8 -0
  12. package/dist/analysis/analysis-lock.js +129 -0
  13. package/dist/analysis/automatic-completion.js +31 -0
  14. package/dist/analysis/automatic-config.d.ts +15 -6
  15. package/dist/analysis/automatic-config.js +25 -9
  16. package/dist/analysis/execute.js +1 -1
  17. package/dist/analysis/job.d.ts +1 -0
  18. package/dist/analysis/job.js +6 -2
  19. package/dist/analysis/recording-ended.d.ts +16 -0
  20. package/dist/analysis/recording-ended.js +29 -0
  21. package/dist/analysis/service.d.ts +1 -3
  22. package/dist/analysis/service.js +23 -50
  23. package/dist/analysis/source.d.ts +1 -1
  24. package/dist/browser-control/protocol.d.ts +8 -0
  25. package/dist/browser-control/protocol.js +21 -0
  26. package/dist/cli/commands/analysis-signals.d.ts +10 -4
  27. package/dist/cli/commands/analysis-signals.js +28 -6
  28. package/dist/cli/commands/runs.js +8 -2
  29. package/dist/cli/commands/study-route-computer-use.js +4 -5
  30. package/dist/cli/commands/study-route-scripted.js +5 -6
  31. package/dist/cli/commands/study-route-shared-world.js +4 -5
  32. package/dist/cli/commands/study-route-terminal.js +5 -6
  33. package/dist/cli/commands/study-run.js +1 -1
  34. package/dist/cli/doctor.d.ts +2 -0
  35. package/dist/cli/doctor.js +3 -0
  36. package/dist/cli/findings.d.ts +40 -14
  37. package/dist/cli/findings.js +117 -35
  38. package/dist/cli/io.d.ts +5 -1
  39. package/dist/cli/io.js +12 -2
  40. package/dist/cli/run-findings.d.ts +19 -0
  41. package/dist/cli/run-findings.js +106 -0
  42. package/dist/evidence/browser-hygiene.d.ts +12 -1
  43. package/dist/evidence/browser-hygiene.js +45 -2
  44. package/dist/feedback/export-bundle.js +1 -1
  45. package/dist/guest/bootstrap.d.ts +4 -2
  46. package/dist/guest/bootstrap.js +15 -7
  47. package/dist/guest/display-config.d.ts +31 -0
  48. package/dist/guest/display-config.js +35 -0
  49. package/dist/guest/external-links.d.ts +14 -0
  50. package/dist/guest/external-links.js +48 -0
  51. package/dist/guest/runtime-desktop.d.ts +9 -3
  52. package/dist/guest/runtime-desktop.js +64 -27
  53. package/dist/guest/runtime.d.ts +2 -1
  54. package/dist/guest/runtime.js +1 -1
  55. package/dist/guest-runtime-main.js +2 -1
  56. package/dist/observer/data.d.ts +4 -0
  57. package/dist/observer/run-routes.js +12 -3
  58. package/dist/observer-app.html +2 -2
  59. package/dist/routes/computer-use/e2b-desktop/desktop.js +27 -17
  60. package/dist/routes/computer-use/e2b-desktop/fidelity.js +10 -17
  61. package/dist/routes/computer-use/e2b-desktop/teardown.js +1 -1
  62. package/dist/routes/computer-use/in-process-desktop.js +2 -2
  63. package/dist/routes/computer-use/live-phase.js +1 -1
  64. package/dist/routes/computer-use/local-vm.js +34 -1
  65. package/dist/routes/computer-use/plan.js +1 -1
  66. package/dist/routes/computer-use/setup.js +2 -2
  67. package/dist/routes/scripted/plan.js +1 -1
  68. package/dist/routes/shared-world/plan.js +1 -1
  69. package/dist/routes/shared-world/route.js +1 -1
  70. package/dist/routes/terminal/plan.js +1 -1
  71. package/dist/run/display.d.ts +6 -1
  72. package/dist/run/display.js +11 -0
  73. package/dist/run/host-suspension.js +4 -1
  74. package/dist/run/preflight-receipts.d.ts +3 -12
  75. package/dist/run/preflight-receipts.js +7 -67
  76. package/dist/run/process-owner.d.ts +28 -0
  77. package/dist/run/process-owner.js +83 -0
  78. package/dist/run/projection.js +1 -1
  79. package/dist/run/reclaim.d.ts +6 -0
  80. package/dist/run/reclaim.js +32 -2
  81. package/dist/run/route-shell.js +2 -2
  82. package/dist/run/run-index.d.ts +3 -1
  83. package/dist/run/run-index.js +23 -11
  84. package/dist/run/run.d.ts +16 -5
  85. package/dist/run/run.js +25 -10
  86. package/dist/run/status.d.ts +57 -2
  87. package/dist/run/status.js +87 -1
  88. package/dist/run/stored-runs.d.ts +3 -0
  89. package/dist/run/stored-runs.js +9 -4
  90. package/dist/study/doctor.js +11 -6
  91. package/dist/study/init-templates.js +9 -3
  92. package/dist/study/mobile-emulation.d.ts +27 -0
  93. package/dist/study/mobile-emulation.js +60 -0
  94. package/dist/study/plan.js +1 -1
  95. package/dist/study/preflight-result.js +1 -1
  96. package/dist/study/run-study-events.d.ts +7 -1
  97. package/dist/study/summary.js +1 -1
  98. package/dist/study/warnings.js +5 -2
  99. package/dist/substrates/e2b/cdp-probe.d.ts +2 -6
  100. package/dist/substrates/e2b/desktop-browser.js +3 -1
  101. package/dist/substrates/e2b/desktop-cdp.d.ts +0 -2
  102. package/dist/substrates/e2b/desktop-cdp.js +0 -2
  103. package/dist/substrates/e2b/desktop-executor.d.ts +7 -0
  104. package/dist/substrates/e2b/desktop-executor.js +3 -0
  105. package/dist/substrates/e2b/desktop-handoff.d.ts +22 -0
  106. package/dist/substrates/e2b/desktop-handoff.js +86 -0
  107. package/dist/substrates/local/firecracker-desktop.d.ts +5 -0
  108. package/dist/substrates/local/firecracker-desktop.js +7 -1
  109. package/dist/substrates/local/runtime-config.d.ts +12 -0
  110. package/dist/substrates/local/runtime-config.js +55 -13
  111. package/dist/substrates/local/runtime-release.js +8 -8
  112. package/dist/substrates/local/runtime.js +1 -0
  113. package/dist/tui-app.js +115 -115
  114. package/docs/architecture/observer.md +7 -3
  115. package/docs/contracts/schemas.md +24 -3
  116. package/docs/contracts/study-analysis.md +15 -9
  117. package/docs/principles/invariants-and-defaults.md +1 -1
  118. package/docs/product/automatic-analysis.md +75 -18
  119. package/package.json +1 -1
  120. package/skills/humanish/SKILL.md +153 -31
@@ -159,9 +159,13 @@ allowlist, contained file reads, read-only routes and no-store security headers.
159
159
  Served data may also include `runtime` with `state`, `observedAt`, and
160
160
  `source: "local-run-status"`. This is a current observation of a contained,
161
161
  matching `status.json`; it never changes the run's recorded verdict or participant
162
- outcomes. A fresh heartbeat means running, an explicitly finalized record means
163
- finished, a record the CLI's signal handler wrote means interrupted, and stale or
164
- invalid timing means unknown. Missing, malformed, or mismatched records omit the
162
+ outcomes. A fresh heartbeat means running, a record its process wrote as `finished` means
163
+ finished, a record written as `interrupted` (by the CLI's signal handler, by a run that ended
164
+ without publishing, or by `humanish reclaim`) means interrupted, and stale or
165
+ invalid timing means unknown. While the run's own process analyzes it after the run ended,
166
+ `runtime.analysis` is `{ startedAt }`: the status band reads **Writing findings** and the
167
+ Findings view shows the analysis as pending, with how long ago it started, before the analysis
168
+ job exists. Missing, malformed, or mismatched records omit the
165
169
  observation. A stale heartbeat alone does not prove interruption, and stored PIDs
166
170
  are neither probed nor returned. Static rendering and export do not create this
167
171
  served-only observation.
@@ -1411,9 +1411,9 @@ proof claims zero on a `null` line, or when known spend exceeds the declared cap
1411
1411
 
1412
1412
  `humanish.run-status.v1` is `status.json`, written inside each run directory by
1413
1413
  every route at run start, refreshed on a fixed cadence while the run is
1414
- alive, and finalized when it ends: `{ schema, runId, state: running |
1414
+ alive, and rewritten when the run ends: `{ schema, runId, state: running |
1415
1415
  finished | interrupted, mode, study?, sandboxes?, pid, startedAt, updatedAt, completedAt?,
1416
- signal?, outcome? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
1416
+ signal?, reason?, outcome?, analysis? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
1417
1417
  `study`, and readers accept it. `sandboxes: none` is written at run start by a route that has no
1418
1418
  way to create a sandbox for the run: a scripted run against an `app-url` subject. A run that may
1419
1419
  create one has no `sandboxes` field. `humanish reclaim` reports such a run `clean` with
@@ -1453,6 +1453,24 @@ with it at publish: false means some participant or desktop usage has no price
1453
1453
  and the estimate is a lower bound. A record without it, as every record before
1454
1454
  0.109, says nothing about completeness, and readers treat it as unknown.
1455
1455
 
1456
+ `finished` means the run published its bundle. A process that leaves the run without publishing
1457
+ one, as a route that throws or returns a refusal after the run started does, writes
1458
+ `interrupted` with `reason: no-publish` and no `signal`; humanish 0.139.1 and earlier wrote
1459
+ `finished` with no `outcome` there. `humanish reclaim --run <id>` (not `--check`) rewrites a
1460
+ `running` record whose heartbeat is stale as `interrupted` with `reason: reclaimed` and
1461
+ `completedAt` set to its last `updatedAt`, and reports `markedInterrupted: true`. Either way the
1462
+ run is excluded from a study's duration and cost medians, which count only finished runs. Readers
1463
+ older than `reason` read such a record as interrupted.
1464
+
1465
+ `analysis` is added to a `finished` record when automatic analysis starts in the run's own
1466
+ process: `{ startedAt, endedAt?, state? }`. While it runs, the heartbeat refreshes `updatedAt`
1467
+ again; `completedAt` stays when the participants ended, so a run's duration excludes its
1468
+ analysis. When analysis ends, `endedAt` and `state` (the automatic analysis outcome's state:
1469
+ `complete`, `partial`, `failed`, `cancelled`, `skipped` or `unknown`) are written. An `analysis`
1470
+ without `endedAt` whose `updatedAt` is older than three touch intervals belonged to a process that
1471
+ stopped during analysis. `runAnalysisPhase` (`src/run/status.ts`) reads it as `running`,
1472
+ `stopped` or `ended`. Readers that predate the field ignore it.
1473
+
1456
1474
  `interrupted` with `signal` (`SIGINT`, `SIGTERM` or `SIGHUP`) is written by
1457
1475
  the CLI's run command when that signal stops a live run outside post-run
1458
1476
  analysis. It first prints one line saying it is stopping and refuses any further sandbox
@@ -1496,7 +1514,10 @@ cheapest-source-first: the `status.json` record, else the bundle, else the run
1496
1514
  directory alone. `{ runId, derivedFrom: status | bundle | directory, liveness,
1497
1515
  mode?, pid?, study?, startedAt?, updatedAt?, completedAt?, verdict?, ok?,
1498
1516
  failure?, participants?, estimatedCostUsd?, estimatedCostComplete?,
1499
- analysisCost?, durationMs? }`. `ok` and `failure` (the first execution failure)
1517
+ analysisCost?, durationMs?, analysis? }`. `analysis` is the status record's automatic analysis as
1518
+ `runAnalysisPhase` reads it, `{ state: running | stopped | ended, startedAt, endedAt?, outcome? }`;
1519
+ `humanish runs` and the TUI add `analyzing` or `analysis stopped` to the run's display from it, and
1520
+ `humanish runs --json` carries it on each run. `ok` and `failure` (the first execution failure)
1500
1521
  are the run's own outcome; `runDisplay` (`src/run/display.ts`) reads them with
1501
1522
  `liveness` and `verdict`, so the TUI and `humanish stats` show a run with
1502
1523
  `ok: false` as failed whatever its verdict. The point is cost: walking
@@ -3,15 +3,16 @@
3
3
  Study analysis is an independent interpretation of retained participant evidence.
4
4
  It is separate from the participant's account, recorded outcome, and the run's
5
5
  deterministic review verdict. Opening an Observer never starts a provider request.
6
- Supported live runs request analysis on completion by default, with a separate
7
- $3 cap on its expected cost. Set `review.analysis: false` to disable that request;
8
- see [automatic analysis](../product/automatic-analysis.md).
6
+ Supported live runs request analysis on completion by default. A study that declares no analyst
7
+ gets the Codex account analyst when its participant is a Codex `local-agent`, and the OpenAI
8
+ analyst with a separate $10 cap on its expected cost otherwise. Set `review.analysis: false` to
9
+ disable that request; see [automatic analysis](../product/automatic-analysis.md).
9
10
 
10
11
  ## Invocation
11
12
 
12
13
  ```bash
13
- humanish analyze --run latest --max-cost 3 --dry-run --json
14
- humanish analyze --run latest --max-cost 3 --json
14
+ humanish analyze --run latest --max-cost 10 --dry-run --json
15
+ humanish analyze --run latest --max-cost 10 --json
15
16
  humanish observe --run latest
16
17
  humanish analyze list --run latest --json
17
18
  humanish analyze show --run latest --json
@@ -116,9 +117,11 @@ the current merge instructions, billed $1.92, most of its input served from the
116
117
  provider's prompt cache, and each of its 466 citations resolved. Two runs do not
117
118
  calibrate the merge request's estimate.
118
119
 
119
- ## Explicit Codex account analysis
120
+ ## Codex account analysis
120
121
 
121
- An existing completed recording can use a separate restricted Codex analyst:
122
+ A study whose participant is a Codex `local-agent` uses this analyst unless it declares another,
123
+ and any study can select it with `review.analysis.provider: codex`. An existing completed
124
+ recording can use it too:
122
125
 
123
126
  ```bash
124
127
  humanish analyze --run latest --provider codex --dry-run --json
@@ -421,7 +424,7 @@ It contains only the attempt/run IDs, input/config/source digests, prompt versio
421
424
  and timestamp. The live caller retains a binding to that exact directory and
422
425
  publishes the final receipt once. A start without usable final accounting means
423
426
  dispatch and spend remain unresolved; it is not proof that a provider charged.
424
- A cancellation before transport can finalize with `usage.dispatched: false`.
427
+ A cancellation before transport can end with `usage.dispatched: false`.
425
428
  Admission refusals, missing credentials and rejected dispatch guards do not
426
429
  create a potentially paid attempt. Reuse does not create a new start.
427
430
 
@@ -486,7 +489,10 @@ The states without findings:
486
489
  - `skipped`: the automatic analysis was skipped or refused, for example without
487
490
  `OPENAI_API_KEY`, or the run's study file sets `review.analysis: false`
488
491
  (`AUTOMATIC_ANALYSIS_DISABLED`, read from the file when the view is built). `next` is
489
- `analyze --max-cost 3`, which runs an analysis anyway. For an analysis refused for its
492
+ `analyze --max-cost 10`, which runs an analysis anyway. For a skip without `OPENAI_API_KEY`
493
+ on a machine where Codex is signed in with a ChatGPT account (read when the view is built),
494
+ `next` is `analyze --provider codex` and the message names `review.analysis.provider: codex`
495
+ in the run's study file; otherwise the message says to set the key. For an analysis refused for its
490
496
  cost, the message gives the expected cost, the worst case and the cap, and `next` sets
491
497
  `--max-cost` to the worst case rounded up.
492
498
  - `failed`: the latest attempt failed or was cancelled and no usable version exists. `next`
@@ -90,7 +90,7 @@ silently drifting from one is not.
90
90
  | Default | Why it is the default | Legitimate override |
91
91
  | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
92
92
  | Dry-run | Spend safety (the live-spend rule sets the floor; dry-run keeps it far away) | `mode: live` |
93
- | Post-run analysis on supported live studies | Findings accompany the recording; a separate $3 cap on its expected cost is visible before execution | `review.analysis: false` disables it; an explicit mapping sets another analysis budget. Dry-run and unsupported routes never dispatch; missing default credentials produce a recorded skip |
93
+ | Post-run analysis on supported live studies | Findings accompany the recording; a Codex participant's run is analyzed by the same account, and otherwise a separate $10 cap on its expected cost is visible before execution | `review.analysis: false` disables it; an explicit mapping sets another analysis budget. Dry-run and unsupported routes never dispatch; missing default credentials produce a recorded skip |
94
94
  | Separate worlds (`per-lane-worlds`) | Isolation, attribution, reproducibility | `route: shared-world`: N participants against one mutable plane for scenarios that are about interaction between participants. Participants run at once, as many as `execution.concurrency` (two or more) and the E2B plan's limit allow, unless the study declares when each one starts (`participants[].startAfterMs`). The plane is one getHost-exposed subject sandbox with synthetic seeded data, or an operator-owned public deployment (external-public), driven by N actor sandboxes. The bundle declares the weaker `attributionClass: shared-world` and a verify-enforced `attributionLimits` ceiling (`best-effort-causal-attribution` and the rest of the concurrent set). Sequential turns at `execution.concurrency: 1` were removed in 0.106.0. |
95
95
  | External key placement | Smallest blast radius: when the keyed process (e.g. a computer-use provider loop) runs outside the sandbox, its key never enters | In-sandbox placement when the keyed process runs inside (an agent harness under test); declared per actor type, with a spend budget |
96
96
  | Loopback entry URLs | Public-safety: never drive third-party sites unbidden | `policies.allowPublicTargets` for an owner-declared deployment/preview (a Vercel preview of your own app). Multi-participant public/preview fan-out needs explicit `participants[].target` for every participant, so the adapter-owned topology is declared rather than inferred. Provisioned clone subjects always serve in-sandbox on loopback |
@@ -3,7 +3,14 @@
3
3
  Supported live studies automatically request analysis after each recording finishes. Findings remain
4
4
  separate from participant feedback and the recorded study verdict.
5
5
 
6
- The default is `gpt-6-astra` with high reasoning effort, a separate $3 cap on its
6
+ A study that declares no `review.analysis` gets its analyst from its participant (`actor`). A
7
+ Codex `local-agent` participant, on a local or hosted desktop, gets the Codex account analyst
8
+ described below, so a study run on a signed-in Codex needs no OpenAI API key for its findings.
9
+ Every other participant, including a Claude Code `local-agent` participant, gets the OpenAI
10
+ analyst. humanish never moves a study to the other analyst because a key is missing: the account
11
+ that pays for analysis is the one the study names or its participant implies.
12
+
13
+ The OpenAI default is `gpt-6-astra` with high reasoning effort, a separate $10 cap on its
7
14
  expected cost and a 600-second timeout. When no output limit is specified,
8
15
  humanish selects 32,768 tokens if the exact input's expected cost still fits that
9
16
  cap; otherwise it keeps the established 16,384-token allowance. This preserves
@@ -12,7 +19,7 @@ previously admitted studies without increasing their spending limit. To customiz
12
19
  ```yaml
13
20
  review:
14
21
  analysis:
15
- maxCostUsd: 3
22
+ maxCostUsd: 10
16
23
  # Optional overrides. Omit maxOutputTokens for budget-aware selection.
17
24
  model: gpt-6-astra
18
25
  timeoutMs: 600000
@@ -20,7 +27,8 @@ review:
20
27
  # question: Where did participants need to recover?
21
28
  ```
22
29
 
23
- Omitting `review.analysis` uses these defaults. Set `review.analysis: false` to
30
+ Omitting `review.analysis` uses these defaults for any participant other than a Codex
31
+ `local-agent`. Set `review.analysis: false` to
24
32
  run participants without the additional analysis request. An explicit analysis
25
33
  mapping using the default OpenAI API provider requires `maxCostUsd`. Admission
26
34
  refuses the request when both its worst case and its expected cost plus a 10% margin are over
@@ -33,9 +41,10 @@ the cap admits no analysis at all, it says so instead. A study of more than 16 p
33
41
  analysed in cohorts of at most 16, one request each, and one more request merges their reports;
34
42
  the range covers every request and the analysis line names them (see
35
43
  [runs with more than 16 participants](../contracts/study-analysis.md#runs-with-more-than-16-participants)).
36
- The default $3 cap refuses the analysis of 17 participants even for a run that keeps no evidence,
37
- so a default analysis of more than 16 participants is skipped with
38
- `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED` unless `review.analysis.maxCostUsd` admits it. Use `analyze --dry-run --max-cost <usd>` on retained
44
+ The default $10 cap admits a single request at the evidence limits ($3.79 expected for one
45
+ participant, $4.54 for 16) and the analysis of 17 to 40 participants from $4.60 to $7.05 with no
46
+ evidence; a run that keeps more evidence can still be refused, and its skip names the
47
+ `--max-cost` that admits it. Use `analyze --dry-run --max-cost <usd>` on retained
39
48
  evidence to inspect the expected cost, the worst case and the selected token
40
49
  allowance before deliberately choosing a larger budget. Explicit
41
50
  `maxOutputTokens` and `--max-output-tokens` limits are honored exactly. The output allowance
@@ -65,8 +74,8 @@ enforced output-token ceiling are unknown, so omit `maxCostUsd` and
65
74
  allocated. There is no API fallback. Missing or unsupported account setup leaves
66
75
  an explicit failed analysis state and the original recording intact. Use
67
76
  `humanish doctor --study <study>` for setup checks; account allowance and model access
68
- remain untested until a request. An omitted provider still means OpenAI, including
69
- hosted studies whose participant uses a local Codex or Claude login. This setting
77
+ remain untested until a request. An omitted provider means this analyst for a Codex
78
+ `local-agent` participant on any target, and OpenAI for every other participant. This setting
70
79
  does not enable managed local desktops.
71
80
 
72
81
  The same configuration works through `humanish run <study>`,
@@ -77,13 +86,34 @@ never enables analysis by default and rejects an explicit analysis mapping
77
86
  before execution. `false` is accepted on every route. Dry runs show analysis
78
87
  as skipped, without reading analysis credentials or making a provider request.
79
88
 
80
- Participant execution finishes and its recording is finalized before analysis is
81
- queued. A participant who was blocked or interrupted can still have useful
82
- retained evidence; analysis requires a verified live recording, not a successful
83
- participant outcome. An active, missing or invalid recording is not analyzed.
84
- Default analysis also skips recordings containing only setup or failure records
85
- with no retained participant activity. A desktop startup failure does not start
86
- an analysis request. The original failure remains visible.
89
+ Automatic analysis starts when the run ends: every participant has stopped and the run has
90
+ published its bundle (`run.json` with an outcome, and `status.json` state `finished`). It runs in
91
+ the same process, so `humanish run` returns only after it. A participant who was blocked or
92
+ interrupted can still have useful retained evidence; analysis requires a verified live recording
93
+ whose participant streams all ended, not a successful participant outcome. An active, missing or
94
+ invalid recording is not analyzed. Default analysis also skips recordings containing only setup
95
+ or failure records with no retained participant activity. A desktop startup failure does not
96
+ start an analysis request. The original failure remains visible.
97
+
98
+ While analysis runs, the run reads as analyzing. `status.json` records `analysis.startedAt` and
99
+ its heartbeat keeps refreshing `updatedAt`; `humanish runs` lists the run as `passed, analyzing`
100
+ (or its other result); the TUI adds `analyzing` to the run; the served Observer shows **Writing
101
+ findings** in its status band and the Findings tab in a pending state with how long ago analysis
102
+ started; `humanish review` says the analysis is running and when it started. In a terminal, `run`
103
+ prints `Participants finished. Writing findings; this usually takes 1 to 4 minutes.` to stderr,
104
+ then `Still writing findings: <elapsed> elapsed.` every 20 seconds. With `--json` it prints only
105
+ the first line, on stderr.
106
+
107
+ A run whose process ended before it published (a crash, or a refusal after the run started) is
108
+ recorded `interrupted` with `reason: no-publish`, and one whose process was killed is marked
109
+ `interrupted` by `humanish reclaim`. `humanish analyze` accepts an interrupted run when every
110
+ participant stream ended; when one was still running it refuses with
111
+ `ANALYSIS_RUN_INCOMPLETE` and names `humanish run <study>`. It refuses a run that is still alive
112
+ with `ANALYSIS_RUN_ACTIVE`, and a run whose heartbeat stopped while its pid is still in use with
113
+ `ANALYSIS_RUN_UNCONFIRMED`, naming `humanish reclaim --run <id>`. When the process running an
114
+ automatic analysis stops before it finishes, `review` says so and names the `analyze` command
115
+ that runs it again. That command takes over the run's `.analysis-lock` when the process that
116
+ held it is gone, and says so in a warning.
87
117
 
88
118
  CLI live starts disclose the selected analyst, its expected cost range and cap, or unknown account dollars before execution.
89
119
  `humanish study check <study> --json` and the TUI study screen also expose the
@@ -95,7 +125,7 @@ runner continues after the TUI closes; reopening the TUI or Observer reads the
95
125
  existing job and does not start another request. Concurrent or repeated automatic
96
126
  invocations cannot silently retry a paid attempt. If a process disappears while
97
127
  an attempt is in flight, its state can be unknown rather than falsely complete.
98
- Use manual `humanish analyze --run <exact-run-id> --max-cost 3` for an intentional
128
+ Use manual `humanish analyze --run <exact-run-id> --max-cost 10` for an intentional
99
129
  follow-up after inspecting the existing attempt and its accounting. For the account branch, use `humanish analyze --run <exact-run-id> --provider codex --rerun` without a dollar limit.
100
130
 
101
131
  Stopping participant execution does not start a fresh automatic analysis. A
@@ -113,8 +143,12 @@ unknown analysis produces exit code 2 without discarding the recording. When the
113
143
  run's own result is ok, the first line of the human output reads
114
144
  `humanish run <study>: live run finished; the analysis did not complete`. A missing
115
145
  `OPENAI_API_KEY` skips default analysis and preserves a successful run exit;
116
- `automaticAnalysisTrigger: "default"` distinguishes that case in JSON. So does a
117
- default analysis whose expected cost, plus the margin, is over the default $3 cap:
146
+ `automaticAnalysisTrigger: "default"` distinguishes that case in JSON. The skip names the next
147
+ command: with Codex signed in with a ChatGPT account it is `humanish analyze --run <id>
148
+ --provider codex`, and the message names `review.analysis.provider: codex` as the setting that
149
+ makes that analyst the study's; otherwise it says to set the key and names `humanish analyze
150
+ --run <id> --max-cost 10`. So does a
151
+ default analysis whose expected cost, plus the margin, is over the default $10 cap:
118
152
  it is skipped with `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED` and the run exits 0. For
119
153
  any analysis refused for its cost, the job records the expected cost, the worst
120
154
  case and the cap, and the CLI, `humanish review` and the Observer give them with
@@ -126,6 +160,28 @@ findings remain visibly partial; a valid partial result can succeed, while a
126
160
  partial result with an analysis error still fails the command. Recorded task
127
161
  outcomes and the deterministic review verdict are never rewritten by analysis.
128
162
 
163
+ After a live run the human output ends with the analysis line. When the analysis left findings,
164
+ the line gives its state and the top three findings follow, with the `humanish review --run <id>`
165
+ command that prints all of them. Otherwise the line says what happened and a `next:` line gives
166
+ the command that gets findings, when one can. A dry run keeps its one line,
167
+ `analysis: skipped for dry runs`.
168
+
169
+ `run --json` and `watch --json` carry the same on `automaticAnalysis`: `message`, the sentence the
170
+ human line is built from, and `nextCommand`, the command that gets findings or, when the analysis
171
+ left them, the `review` command (null when no command can). When the analysis left findings the
172
+ JSON also has `findings`:
173
+
174
+ | Field | Meaning |
175
+ | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
176
+ | `analysisId` | The analysis the findings come from |
177
+ | `status` | `complete` or `partial` |
178
+ | `count` | How many findings the analysis made |
179
+ | `designFindingCount` | How many design findings it made; null for an analysis written before them |
180
+ | `top` | At most five findings, highest priority first: `id`, `title`, `headline`, `impact`, `confidence` and `participants` (the affected participant ids) |
181
+ | `review` | `humanish review --run <id>`, which prints every finding with its evidence |
182
+ | `observer` | The run's Observer page opened on the findings (`observer/index.html#/report`); null without an Observer |
183
+ | `path` | analysis.json relative to the project directory |
184
+
129
185
  `automaticAnalysis.reason` is one of these codes, in `--json` and in the human line
130
186
  `analysis: <state> (<reason>)`:
131
187
 
@@ -152,6 +208,7 @@ outcomes and the deterministic review verdict are never rewritten by analysis.
152
208
  | `AUTOMATIC_ANALYSIS_PUBLICATION_FAILED` | `failed` | The analysis could not be published |
153
209
  | `AUTOMATIC_ANALYSIS_CANCELLATION_UNAVAILABLE` | `unknown` | A cancellation could not be recorded |
154
210
  | `AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN` | `unknown` | The outcome could not be determined |
211
+ | `AUTOMATIC_ANALYSIS_STOPPED` | `unknown` | Read only: the process running it stopped refreshing its job record |
155
212
 
156
213
  A run refused before it was created (`run: not-created`) prints no `analysis:` line, since there
157
214
  is no run to analyze and the refusal says why. Its JSON keeps the record, with
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "humanish",
3
- "version": "0.139.1",
3
+ "version": "0.141.0",
4
4
  "description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
5
5
  "keywords": [
6
6
  "cli",
@@ -95,10 +95,13 @@ exact returned path, not a basename that could resolve to another manifest.
95
95
 
96
96
  ## Choosing a findings analyst
97
97
 
98
- Analysis is separate from the participant. Hosted and manual defaults use the
99
- OpenAI API with its own admission budget. An explicitly local browser study with
100
- a Codex participant defaults to a separate Codex account analyst. To select that
101
- restricted Codex ChatGPT account analyst on other supported studies,
98
+ Analysis is separate from the participant. A study that declares no analyst
99
+ gets one from its participant: a Codex `local-agent` participant, local or
100
+ hosted, defaults to a separate Codex account analyst; every other participant,
101
+ Claude Code included, defaults to the OpenAI API with a $10 admission budget.
102
+ Without `OPENAI_API_KEY` that OpenAI analysis is skipped, and the run output
103
+ names the next command: `analyze --provider codex` when Codex is signed in. To
104
+ select the restricted Codex ChatGPT account analyst on any supported study,
102
105
  set `review.analysis.provider: codex` or pass `analyze --provider codex` on a
103
106
  completed recording. This uses remote inference and the qualified CLI/login,
104
107
  not local inference or the participant's existing conversation. See
@@ -133,7 +136,8 @@ the pinned runtime; a live local run also prepares it automatically. The normal
133
136
  See [the complete example and limits](https://humanish.dev/docs/local-browser).
134
137
 
135
138
  Local browsers currently require a loopback app URL with an explicit port above
136
- 1023 and use a 960×720 Chromium desktop. For email-gated local apps, start
139
+ 1023 and use a 960×720 Chromium desktop unless a device preset or
140
+ `execution.desktop.resolution` sets each participant's screen. For email-gated local apps, start
137
141
  `humanish comms catch`, point the app's email sends at it, and declare
138
142
  `comms.email.external.catchBaseUrl`. Use `doctor --study` to check its recipient
139
143
  routes. This captures app sends without mailbox-provider credentials; it does
@@ -294,6 +298,63 @@ fails with `HUMANISH_STUDY_RETIRED_DIRECTORY`; move it to the matching `studies/
294
298
  run looks like idle tiles — another reason to leave the cap out unless you
295
299
  need it.
296
300
 
301
+ ### Phone, tablet and desktop in one study
302
+
303
+ When the user asks to test several screen sizes, write one study with one
304
+ `participants` entry per size and set `device` on each entry. The participants run
305
+ at once, the Observer shows them side by side, and the findings compare them.
306
+
307
+ ```yaml
308
+ schema: humanish.study.v3
309
+ id: devices
310
+ title: Phone, tablet and desktop
311
+ route: computer-use
312
+ mode: live
313
+ subject:
314
+ source: app-url
315
+ appUrl: "http://127.0.0.1:3000"
316
+ actor:
317
+ type: local-agent
318
+ localAgent: codex
319
+ persona: synthetic-new-user
320
+ mission: "Sign up and create your first project. Say what was hard to read or reach."
321
+ participants:
322
+ - id: phone
323
+ device: mobile
324
+ - id: tablet
325
+ device: tablet
326
+ - id: desktop
327
+ device: desktop
328
+ execution:
329
+ target: local
330
+ timeoutMs: 600000
331
+ desktop:
332
+ fidelity:
333
+ mobileEmulation: true
334
+ ```
335
+
336
+ This runs on the local browser runtime with the user's Codex login. For a deployed
337
+ app, set `execution.target: e2b-desktop`, put the public URL in `subject.appUrl`,
338
+ add `policies.allowPublicTargets: true`, and give each entry a `target` (below).
339
+
340
+ - `device` takes a preset: `mobile` (414×896), `small-mobile`, `narrow-mobile`,
341
+ `tablet` (820×1180), `desktop` (1440×950) or `wide` (1920×1080). The preset
342
+ sizes the participant's screen and browser window.
343
+ - `fidelity.mobileEmulation: true` gives the phone participants a mobile viewport,
344
+ device pixel ratio, touch and a mobile user agent in Chromium. Tablet and
345
+ desktop participants are left as they are.
346
+ - Do not run one study per size, draw a device frame inside the page, or resize
347
+ the browser from the mission. Those produce desktop-ratio evidence labelled as
348
+ a phone, and three runs the Observer cannot compare.
349
+ - Participants on one public app (`policies.allowPublicTargets: true`) share its
350
+ data, so more than one is refused unless each entry sets `target`: a separate
351
+ test account, tenant or seeded URL per participant, or the same URL when the
352
+ task has each participant create its own account. A loopback or clone subject
353
+ needs no `target`.
354
+ - On a local study, `doctor --study` fails its runtime row when the installed
355
+ local runtime predates per-participant screens; `humanish runtime setup`
356
+ installs the pinned one. Do not move the study to E2B without asking the user.
357
+
297
358
  Use committed `humanish/studies/*.yaml` for public-safe, reproducible studies. Use
298
359
  ignored `.humanish/studies/*.yaml` or `.humanish/local/studies/*.yaml` for private repo
299
360
  targets, local-only dogfood, or machine-specific settings. Never commit private
@@ -419,14 +480,15 @@ credits, or claim product behavior proof from a dry run.
419
480
  When the target app can run locally, prove a known browser path with a
420
481
  scripted-browser study after starting the app on loopback. The study replays a
421
482
  `browser.steps` scenario (see above) on a desktop surface, plus a mobile one
422
- with `count: 2`. The scripted steps make no model requests. Post-run analysis
423
- still runs by default when `OPENAI_API_KEY` is set, with a $3 cap; keep
424
- `review.analysis: false` to skip it. Copy the shape of
483
+ with `count: 2`. The scripted steps make no model requests; the analysis after
484
+ the run does, and it is what produces findings. Copy the shape of
425
485
  [`humanish/studies/scripted-demo.yaml`](https://github.com/danielgwilson/humanish/blob/main/humanish/studies/scripted-demo.yaml)
426
486
  or the example in the
427
- [study file reference](https://humanish.dev/docs/study-files#scripted-browser-scenarios):
428
- set `subject.appUrl` to the loopback URL, `scenario` to the scenario id,
429
- and `mode: live`. Without `mode: live` the study is a dry run that opens
487
+ [study file reference](https://humanish.dev/docs/study-files#scripted-browser-scenarios),
488
+ leaving out the demo's `review.analysis: false`, which keeps that zero-spend
489
+ demo from requesting analysis: set `subject.appUrl` to the loopback URL,
490
+ `scenario` to the scenario id, and `mode: live`. Set `review.analysis: false`
491
+ only when the user has chosen a run with no findings. Without `mode: live` the study is a dry run that opens
430
492
  no browser.
431
493
 
432
494
  ```bash
@@ -441,6 +503,52 @@ Do not use `humanish watch --count ...` as a substitute for a scripted-browser
441
503
  study. `watch` without a study starts a preview run for the Observer; a live scripted-browser
442
504
  study captures desktop and mobile browser evidence against a running app.
443
505
 
506
+ ## After a Live Run, Read and Report Its Findings
507
+
508
+ A live run ends with a separate analysis of its evidence, and `run --json`
509
+ carries what that analysis left. `findings.count` is how many findings it made;
510
+ `findings.top` holds at most five, each with its `title`, `headline`, `impact`,
511
+ `confidence` and the ids of the `participants` it affected; `findings.review` is
512
+ the command that prints every finding with its evidence; `findings.observer` is
513
+ the Observer page that opens on the findings. When the analysis left no
514
+ findings, `findings` is absent, `automaticAnalysis.message` says why, and
515
+ `automaticAnalysis.nextCommand` is the command that gets them (null when no
516
+ command can). Run that command, or tell the user to.
517
+
518
+ Read the full findings before reporting anything else:
519
+
520
+ ```bash
521
+ npx humanish review --run <id> # the verdict, then each analysis finding
522
+ npx humanish review --run <id> --json # the same; findings are under `analysis`
523
+ ```
524
+
525
+ Report each finding to the user: its title, impact, confidence and recovery, the participants it
526
+ affected, and the capture files it cites. These come from the post-run analysis of the evidence.
527
+ Label the participant's own report in the review summary as the participant's account. When
528
+ `analysis.state` is not `ready`, report `analysis.message` and the `analysis.next` command in
529
+ place of findings; `analysis.next` is null when no command can produce them. `latest` moves with
530
+ every run, including dry runs, so use the run id the run printed.
531
+
532
+ ## Long Studies: `run` Blocks Until Its Findings
533
+
534
+ `humanish run <study>` returns only after every participant has ended and the automatic analysis
535
+ has finished. A participant with `endAfterMs` can keep a study running for up to 24 hours, and
536
+ analysis usually adds 1 to 4 minutes after the last participant ends. A shell timeout that sends
537
+ SIGTERM stops the run: humanish marks it interrupted and stops its sandboxes.
538
+
539
+ - Start a study that can outlive your shell call in tmux, for example
540
+ `tmux new -d -s study 'npx humanish run <study> --json --no-open > run-result.json 2> run.log'`.
541
+ `--detach` only skips the Observer server at the end; it does not move the run to the background.
542
+ - The first stderr line names the run: `Run <id> started. Follow it with humanish observe --run <id>`.
543
+ If you did not keep it, `npx humanish runs --json` lists runs newest first.
544
+ - In `runs --json`, a run whose participants are working has `display.state: running`; once they
545
+ end, its `display` is their result and `analysis.state` is `running` while findings are written.
546
+ - `npx humanish review --run <id> --json` says where the run is: participants running, analysis
547
+ running since a given time, or the findings.
548
+ - If the run's process died, `npx humanish reclaim --run <id>` stops its sandboxes and marks it
549
+ interrupted. `npx humanish analyze --run <id>` then analyzes it if every participant had ended,
550
+ and otherwise names the command that runs the study again.
551
+
444
552
  ## Optional Live E2B Study
445
553
 
446
554
  Live headed E2B desktop participants are optional. Add the substrate dependency only
@@ -450,12 +558,18 @@ when the user explicitly wants live E2B execution:
450
558
  npm i -D @e2b/desktop
451
559
  ```
452
560
 
453
- Then confirm env var names are documented without values:
561
+ Then confirm env var names are documented without values. Which keys a hosted
562
+ study needs depends on its participant and analyst:
454
563
 
455
- ```bash
456
- E2B_API_KEY
457
- OPENAI_API_KEY
458
- ```
564
+ | Participant | Keys |
565
+ | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
566
+ | `openai-computer-use` | `E2B_API_KEY`, `OPENAI_API_KEY` |
567
+ | `local-agent` with `localAgent: codex` | `E2B_API_KEY` and `codex login`; the findings analysis uses the same Codex account |
568
+ | `local-agent` with `localAgent: claude` | `E2B_API_KEY` and `claude auth login`; `OPENAI_API_KEY` for findings; without it, a signed-in Codex can analyze the run (the run prints the command) |
569
+
570
+ A hosted Codex participant does not need `OPENAI_API_KEY`. Do not report a missing
571
+ `OPENAI_API_KEY` as the reason a Codex study on E2B fails; read the
572
+ `doctor --study` row instead.
459
573
 
460
574
  Do not paste values into files, prompts, run bundles, issue drafts, or logs.
461
575
  Load local values only at invocation time:
@@ -471,23 +585,31 @@ declared scenario is developer-experience testing. Private repos are allowed
471
585
  only as explicit maintainer-authorized runs with repo redaction left on; never
472
586
  publish their names, screenshots, logs, source snippets, or operational details.
473
587
 
474
- ## Reporting Back
475
-
476
- After a live run, read its findings before anything else:
588
+ ## Setup errors and what they mean
589
+
590
+ Run `npx humanish doctor --study <id>` before the first live run and read every
591
+ `missing` row. The common ones:
592
+
593
+ - `codex_unsafe_configuration` on a hosted Codex participant: the row names the
594
+ setting humanish refused. Report it to the user; never edit, move or rename
595
+ their Codex, agent or skill config to make a run pass. An `AGENTS.md` in the
596
+ Codex home is admitted, and doctor notes that the participant also receives it.
597
+ - `subject.appUrl` must be loopback: a non-loopback URL needs
598
+ `policies.allowPublicTargets: true`, and with more than one participant, a
599
+ `target` on each entry.
600
+ - "Will be skipped: OPENAI_API_KEY is missing" on `post-run analysis` (an
601
+ `openai-computer-use` or Claude participant): the run works without findings,
602
+ and when Codex is signed in, its `next:` line gives the command that analyzes it with Codex.
603
+ `review.analysis.provider: codex` makes that the study's default.
604
+ - A local browser study refuses a non-loopback URL, a port below 1024, hosted
605
+ templates and `sandboxTimeoutMs`. The message names the setting; do not switch the
606
+ study to a hosted target unless the user agrees to E2B usage.
477
607
 
478
- ```bash
479
- npx humanish review --run <id> # the verdict, then each analysis finding
480
- npx humanish review --run <id> --json # the same; findings are under `analysis`
481
- ```
482
-
483
- Report each finding to the user: its title, impact, confidence and recovery, the participants it
484
- affected, and the capture files it cites. These come from the post-run analysis of the evidence.
485
- Label the participant's own report in the review summary as the participant's account. When
486
- `analysis.state` is not `ready`, report `analysis.message` and the `analysis.next` command in
487
- place of findings; `analysis.next` is null when no command can produce them. `latest` moves with
488
- every run, including dry runs, so use the run id the run printed.
608
+ ## Reporting Back
489
609
 
490
- Then report:
610
+ Report a live run's findings first, as
611
+ [After a Live Run, Read and Report Its Findings](#after-a-live-run-read-and-report-its-findings)
612
+ describes. Then report:
491
613
 
492
614
  - files changed in the target repo;
493
615
  - exact proof commands run;