humanish 0.140.0 → 0.141.0

package/docs/architecture/observer.md

@@ -159,9 +159,13 @@ allowlist, contained file reads, read-only routes and no-store security headers.
 Served data may also include `runtime` with `state`, `observedAt`, and
 `source: "local-run-status"`. This is a current observation of a contained,
 matching `status.json`; it never changes the run's recorded verdict or participant
-outcomes. A fresh heartbeat means running, an explicitly finalized record means
-finished, a record the CLI's signal handler wrote means interrupted, and stale or
-invalid timing means unknown. Missing, malformed, or mismatched records omit the
+outcomes. A fresh heartbeat means running, a record its process wrote as `finished` means
+finished, a record written as `interrupted` (by the CLI's signal handler, by a run that ended
+without publishing, or by `humanish reclaim`) means interrupted, and stale or
+invalid timing means unknown. While the run's own process analyzes it after the run ended,
+`runtime.analysis` is `{ startedAt }`: the status band reads **Writing findings** and the
+Findings view shows the analysis as pending, with how long ago it started, before the analysis
+job exists. Missing, malformed, or mismatched records omit the
 observation. A stale heartbeat alone does not prove interruption, and stored PIDs
 are neither probed nor returned. Static rendering and export do not create this
 served-only observation.

package/docs/contracts/schemas.md

@@ -1411,9 +1411,9 @@ proof claims zero on a `null` line, or when known spend exceeds the declared cap
 
 `humanish.run-status.v1` is `status.json`, written inside each run directory by
 every route at run start, refreshed on a fixed cadence while the run is
-alive, and finalized when it ends: `{ schema, runId, state: running |
+alive, and rewritten when the run ends: `{ schema, runId, state: running |
 finished | interrupted, mode, study?, sandboxes?, pid, startedAt, updatedAt, completedAt?,
-signal?, outcome? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
+signal?, reason?, outcome?, analysis? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
 `study`, and readers accept it. `sandboxes: none` is written at run start by a route that has no
 way to create a sandbox for the run: a scripted run against an `app-url` subject. A run that may
 create one has no `sandboxes` field. `humanish reclaim` reports such a run `clean` with
@@ -1453,6 +1453,24 @@ with it at publish: false means some participant or desktop usage has no price
 and the estimate is a lower bound. A record without it, as every record before
 0.109, says nothing about completeness, and readers treat it as unknown.
 
+`finished` means the run published its bundle. A process that leaves the run without publishing
+one, as a route that throws or returns a refusal after the run started does, writes
+`interrupted` with `reason: no-publish` and no `signal`; humanish 0.139.1 and earlier wrote
+`finished` with no `outcome` there. `humanish reclaim --run <id>` (not `--check`) rewrites a
+`running` record whose heartbeat is stale as `interrupted` with `reason: reclaimed` and
+`completedAt` set to its last `updatedAt`, and reports `markedInterrupted: true`. Either way the
+run is excluded from a study's duration and cost medians, which count only finished runs. Readers
+older than `reason` read such a record as interrupted.
+
+`analysis` is added to a `finished` record when automatic analysis starts in the run's own
+process: `{ startedAt, endedAt?, state? }`. While it runs, the heartbeat refreshes `updatedAt`
+again; `completedAt` stays when the participants ended, so a run's duration excludes its
+analysis. When analysis ends, `endedAt` and `state` (the automatic analysis outcome's state:
+`complete`, `partial`, `failed`, `cancelled`, `skipped` or `unknown`) are written. An `analysis`
+without `endedAt` whose `updatedAt` is older than three touch intervals belonged to a process that
+stopped during analysis. `runAnalysisPhase` (`src/run/status.ts`) reads it as `running`,
+`stopped` or `ended`. Readers that predate the field ignore it.
+
 `interrupted` with `signal` (`SIGINT`, `SIGTERM` or `SIGHUP`) is written by
 the CLI's run command when that signal stops a live run outside post-run
 analysis. It first prints one line saying it is stopping and refuses any further sandbox
@@ -1496,7 +1514,10 @@ cheapest-source-first: the `status.json` record, else the bundle, else the run
 directory alone. `{ runId, derivedFrom: status | bundle | directory, liveness,
 mode?, pid?, study?, startedAt?, updatedAt?, completedAt?, verdict?, ok?,
 failure?, participants?, estimatedCostUsd?, estimatedCostComplete?,
-analysisCost?, durationMs? }`. `ok` and `failure` (the first execution failure)
+analysisCost?, durationMs?, analysis? }`. `analysis` is the status record's automatic analysis as
+`runAnalysisPhase` reads it, `{ state: running | stopped | ended, startedAt, endedAt?, outcome? }`;
+`humanish runs` and the TUI add `analyzing` or `analysis stopped` to the run's display from it, and
+`humanish runs --json` carries it on each run. `ok` and `failure` (the first execution failure)
 are the run's own outcome; `runDisplay` (`src/run/display.ts`) reads them with
 `liveness` and `verdict`, so the TUI and `humanish stats` show a run with
 `ok: false` as failed whatever its verdict. The point is cost: walking

package/docs/contracts/study-analysis.md

@@ -424,7 +424,7 @@ It contains only the attempt/run IDs, input/config/source digests, prompt versio
 and timestamp. The live caller retains a binding to that exact directory and
 publishes the final receipt once. A start without usable final accounting means
 dispatch and spend remain unresolved; it is not proof that a provider charged.
-A cancellation before transport can finalize with `usage.dispatched: false`.
+A cancellation before transport can end with `usage.dispatched: false`.
 Admission refusals, missing credentials and rejected dispatch guards do not
 create a potentially paid attempt. Reuse does not create a new start.
 

package/docs/product/automatic-analysis.md

@@ -86,13 +86,34 @@ never enables analysis by default and rejects an explicit analysis mapping
 before execution. `false` is accepted on every route. Dry runs show analysis
 as skipped, without reading analysis credentials or making a provider request.
 
-Participant execution finishes and its recording is finalized before analysis is
-queued. A participant who was blocked or interrupted can still have useful
-retained evidence; analysis requires a verified live recording, not a successful
-participant outcome. An active, missing or invalid recording is not analyzed.
-Default analysis also skips recordings containing only setup or failure records
-with no retained participant activity. A desktop startup failure does not start
-an analysis request. The original failure remains visible.
+Automatic analysis starts when the run ends: every participant has stopped and the run has
+published its bundle (`run.json` with an outcome, and `status.json` state `finished`). It runs in
+the same process, so `humanish run` returns only after it. A participant who was blocked or
+interrupted can still have useful retained evidence; analysis requires a verified live recording
+whose participant streams all ended, not a successful participant outcome. An active, missing or
+invalid recording is not analyzed. Default analysis also skips recordings containing only setup
+or failure records with no retained participant activity. A desktop startup failure does not
+start an analysis request. The original failure remains visible.
+
+While analysis runs, the run reads as analyzing. `status.json` records `analysis.startedAt` and
+its heartbeat keeps refreshing `updatedAt`; `humanish runs` lists the run as `passed, analyzing`
+(or its other result); the TUI adds `analyzing` to the run; the served Observer shows **Writing
+findings** in its status band and the Findings tab in a pending state with how long ago analysis
+started; `humanish review` says the analysis is running and when it started. In a terminal, `run`
+prints `Participants finished. Writing findings; this usually takes 1 to 4 minutes.` to stderr,
+then `Still writing findings: <elapsed> elapsed.` every 20 seconds. With `--json` it prints only
+the first line, on stderr.
+
+A run whose process ended before it published (a crash, or a refusal after the run started) is
+recorded `interrupted` with `reason: no-publish`, and one whose process was killed is marked
+`interrupted` by `humanish reclaim`. `humanish analyze` accepts an interrupted run when every
+participant stream ended; when one was still running it refuses with
+`ANALYSIS_RUN_INCOMPLETE` and names `humanish run <study>`. It refuses a run that is still alive
+with `ANALYSIS_RUN_ACTIVE`, and a run whose heartbeat stopped while its pid is still in use with
+`ANALYSIS_RUN_UNCONFIRMED`, naming `humanish reclaim --run <id>`. When the process running an
+automatic analysis stops before it finishes, `review` says so and names the `analyze` command
+that runs it again. That command takes over the run's `.analysis-lock` when the process that
+held it is gone, and says so in a warning.
 
 CLI live starts disclose the selected analyst, its expected cost range and cap, or unknown account dollars before execution.
 `humanish study check <study> --json` and the TUI study screen also expose the
@@ -187,6 +208,7 @@ JSON also has `findings`:
 | `AUTOMATIC_ANALYSIS_PUBLICATION_FAILED`       | `failed`                  | The analysis could not be published                                   |
 | `AUTOMATIC_ANALYSIS_CANCELLATION_UNAVAILABLE` | `unknown`                 | A cancellation could not be recorded                                  |
 | `AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN`          | `unknown`                 | The outcome could not be determined                                   |
+| `AUTOMATIC_ANALYSIS_STOPPED`                  | `unknown`                 | Read only: the process running it stopped refreshing its job record   |
 
 A run refused before it was created (`run: not-created`) prints no `analysis:` line, since there
 is no run to analyze and the refusal says why. Its JSON keeps the record, with

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "humanish",
-  "version": "0.140.0",
+  "version": "0.141.0",
   "description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
   "keywords": [
     "cli",

package/skills/humanish/SKILL.md

@@ -298,6 +298,63 @@ fails with `HUMANISH_STUDY_RETIRED_DIRECTORY`; move it to the matching `studies/
   run looks like idle tiles — another reason to leave the cap out unless you
   need it.
 
+### Phone, tablet and desktop in one study
+
+When the user asks to test several screen sizes, write one study with one
+`participants` entry per size and set `device` on each entry. The participants run
+at once, the Observer shows them side by side, and the findings compare them.
+
+```yaml
+schema: humanish.study.v3
+id: devices
+title: Phone, tablet and desktop
+route: computer-use
+mode: live
+subject:
+  source: app-url
+  appUrl: "http://127.0.0.1:3000"
+actor:
+  type: local-agent
+  localAgent: codex
+  persona: synthetic-new-user
+  mission: "Sign up and create your first project. Say what was hard to read or reach."
+participants:
+  - id: phone
+    device: mobile
+  - id: tablet
+    device: tablet
+  - id: desktop
+    device: desktop
+execution:
+  target: local
+  timeoutMs: 600000
+  desktop:
+    fidelity:
+      mobileEmulation: true
+```
+
+This runs on the local browser runtime with the user's Codex login. For a deployed
+app, set `execution.target: e2b-desktop`, put the public URL in `subject.appUrl`,
+add `policies.allowPublicTargets: true`, and give each entry a `target` (below).
+
+- `device` takes a preset: `mobile` (414×896), `small-mobile`, `narrow-mobile`,
+  `tablet` (820×1180), `desktop` (1440×950) or `wide` (1920×1080). The preset
+  sizes the participant's screen and browser window.
+- `fidelity.mobileEmulation: true` gives the phone participants a mobile viewport,
+  device pixel ratio, touch and a mobile user agent in Chromium. Tablet and
+  desktop participants are left as they are.
+- Do not run one study per size, draw a device frame inside the page, or resize
+  the browser from the mission. Those produce desktop-ratio evidence labelled as
+  a phone, and three runs the Observer cannot compare.
+- Participants on one public app (`policies.allowPublicTargets: true`) share its
+  data, so more than one is refused unless each entry sets `target`: a separate
+  test account, tenant or seeded URL per participant, or the same URL when the
+  task has each participant create its own account. A loopback or clone subject
+  needs no `target`.
+- On a local study, `doctor --study` fails its runtime row when the installed
+  local runtime predates per-participant screens; `humanish runtime setup`
+  installs the pinned one. Do not move the study to E2B without asking the user.
+
 Use committed `humanish/studies/*.yaml` for public-safe, reproducible studies. Use
 ignored `.humanish/studies/*.yaml` or `.humanish/local/studies/*.yaml` for private repo
 targets, local-only dogfood, or machine-specific settings. Never commit private
@@ -472,6 +529,26 @@ Label the participant's own report in the review summary as the participant's ac
 place of findings; `analysis.next` is null when no command can produce them. `latest` moves with
 every run, including dry runs, so use the run id the run printed.
 
+## Long Studies: `run` Blocks Until Its Findings
+
+`humanish run <study>` returns only after every participant has ended and the automatic analysis
+has finished. A participant with `endAfterMs` can keep a study running for up to 24 hours, and
+analysis usually adds 1 to 4 minutes after the last participant ends. A shell timeout that sends
+SIGTERM stops the run: humanish marks it interrupted and stops its sandboxes.
+
+- Start a study that can outlive your shell call in tmux, for example
+  `tmux new -d -s study 'npx humanish run <study> --json --no-open > run-result.json 2> run.log'`.
+  `--detach` only skips the Observer server at the end; it does not move the run to the background.
+- The first stderr line names the run: `Run <id> started. Follow it with humanish observe --run <id>`.
+  If you did not keep it, `npx humanish runs --json` lists runs newest first.
+- In `runs --json`, a run whose participants are working has `display.state: running`; once they
+  end, its `display` is their result and `analysis.state` is `running` while findings are written.
+- `npx humanish review --run <id> --json` says where the run is: participants running, analysis
+  running since a given time, or the findings.
+- If the run's process died, `npx humanish reclaim --run <id>` stops its sandboxes and marks it
+  interrupted. `npx humanish analyze --run <id>` then analyzes it if every participant had ended,
+  and otherwise names the command that runs the study again.
+
 ## Optional Live E2B Study
 
 Live headed E2B desktop participants are optional. Add the substrate dependency only
@@ -481,12 +558,18 @@ when the user explicitly wants live E2B execution:
 npm i -D @e2b/desktop
 ```
 
-Then confirm env var names are documented without values:
+Then confirm env var names are documented without values. Which keys a hosted
+study needs depends on its participant and analyst:
 
-```bash
-E2B_API_KEY
-OPENAI_API_KEY
-```
+| Participant                             | Keys                                                                                                                                                 |
+| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `openai-computer-use`                   | `E2B_API_KEY`, `OPENAI_API_KEY`                                                                                                                      |
+| `local-agent` with `localAgent: codex`  | `E2B_API_KEY` and `codex login`; the findings analysis uses the same Codex account                                                                   |
+| `local-agent` with `localAgent: claude` | `E2B_API_KEY` and `claude auth login`; `OPENAI_API_KEY` for findings; without it, a signed-in Codex can analyze the run (the run prints the command) |
+
+A hosted Codex participant does not need `OPENAI_API_KEY`. Do not report a missing
+`OPENAI_API_KEY` as the reason a Codex study on E2B fails; read the
+`doctor --study` row instead.
 
 Do not paste values into files, prompts, run bundles, issue drafts, or logs.
 Load local values only at invocation time:
@@ -502,6 +585,26 @@ declared scenario is developer-experience testing. Private repos are allowed
 only as explicit maintainer-authorized runs with repo redaction left on; never
 publish their names, screenshots, logs, source snippets, or operational details.
 
+## Setup errors and what they mean
+
+Run `npx humanish doctor --study <id>` before the first live run and read every
+`missing` row. The common ones:
+
+- `codex_unsafe_configuration` on a hosted Codex participant: the row names the
+  setting humanish refused. Report it to the user; never edit, move or rename
+  their Codex, agent or skill config to make a run pass. An `AGENTS.md` in the
+  Codex home is admitted, and doctor notes that the participant also receives it.
+- `subject.appUrl` must be loopback: a non-loopback URL needs
+  `policies.allowPublicTargets: true`, and with more than one participant, a
+  `target` on each entry.
+- "Will be skipped: OPENAI_API_KEY is missing" on `post-run analysis` (an
+  `openai-computer-use` or Claude participant): the run works without findings,
+  and when Codex is signed in, its `next:` line gives the command that analyzes it with Codex.
+  `review.analysis.provider: codex` makes that the study's default.
+- A local browser study refuses a non-loopback URL, a port below 1024, hosted
+  templates and `sandboxTimeoutMs`. The message names the setting; do not switch the
+  study to a hosted target unless the user agrees to E2B usage.
+
 ## Reporting Back
 
 Report a live run's findings first, as