humanish 0.140.0 → 0.142.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +1 -0
- package/CHANGELOG.md +23 -0
- package/README.md +2 -0
- package/dist/actors/codex/restricted-participant.js +1 -1
- package/dist/actors/computer-use/external-handoff.d.ts +22 -0
- package/dist/actors/computer-use/external-handoff.js +83 -0
- package/dist/actors/computer-use/loop/observation.d.ts +7 -0
- package/dist/actors/computer-use/loop/observation.js +17 -1
- package/dist/actors/computer-use/loop/types.d.ts +6 -0
- package/dist/analysis/admission.d.ts +8 -0
- package/dist/analysis/admission.js +15 -4
- package/dist/analysis/analysis-limits.d.ts +6 -0
- package/dist/analysis/analysis-limits.js +6 -0
- package/dist/analysis/analysis-lock.d.ts +8 -0
- package/dist/analysis/analysis-lock.js +129 -0
- package/dist/analysis/automatic-completion.js +7 -0
- package/dist/analysis/automatic-config.d.ts +5 -2
- package/dist/analysis/automatic-config.js +23 -8
- package/dist/analysis/automatic.js +13 -7
- package/dist/analysis/execute.d.ts +17 -6
- package/dist/analysis/execute.js +37 -19
- package/dist/analysis/gemini-wire.d.ts +8 -0
- package/dist/analysis/gemini-wire.js +165 -0
- package/dist/analysis/image-tokens.d.ts +4 -2
- package/dist/analysis/image-tokens.js +24 -3
- package/dist/analysis/job.d.ts +2 -0
- package/dist/analysis/job.js +7 -2
- package/dist/analysis/openai-wire.d.ts +5 -0
- package/dist/analysis/openai-wire.js +101 -0
- package/dist/analysis/provider.d.ts +15 -7
- package/dist/analysis/provider.js +21 -101
- package/dist/analysis/recording-ended.d.ts +16 -0
- package/dist/analysis/recording-ended.js +29 -0
- package/dist/analysis/responses.d.ts +3 -0
- package/dist/analysis/responses.js +19 -0
- package/dist/analysis/service.d.ts +1 -3
- package/dist/analysis/service.js +34 -53
- package/dist/analysis/source.d.ts +1 -1
- package/dist/analysis/types.d.ts +8 -2
- package/dist/analysis/validation.d.ts +1 -0
- package/dist/analysis/validation.js +13 -12
- package/dist/browser-control/protocol.d.ts +8 -0
- package/dist/browser-control/protocol.js +21 -0
- package/dist/cli/commands/analysis-signals.d.ts +10 -4
- package/dist/cli/commands/analysis-signals.js +28 -6
- package/dist/cli/commands/analyze.js +9 -7
- package/dist/cli/commands/observe.js +26 -1
- package/dist/cli/commands/run-command.d.ts +1 -1
- package/dist/cli/commands/run-command.js +12 -0
- package/dist/cli/commands/runs.js +8 -2
- package/dist/cli/commands/study-remote.d.ts +13 -0
- package/dist/cli/commands/study-remote.js +125 -0
- package/dist/cli/commands/study-route-computer-use.js +28 -4
- package/dist/cli/commands/study-route-scripted.js +3 -2
- package/dist/cli/commands/study-route-shared-world.js +3 -2
- package/dist/cli/commands/study-route-terminal.js +3 -2
- package/dist/cli/commands/study-run.js +34 -0
- package/dist/cli/commands/watch.js +5 -0
- package/dist/cli/doctor.js +7 -2
- package/dist/cli/findings.d.ts +30 -0
- package/dist/cli/findings.js +114 -15
- package/dist/cli/io.d.ts +8 -0
- package/dist/cli/io.js +3 -1
- package/dist/evidence/browser-hygiene.d.ts +12 -1
- package/dist/evidence/browser-hygiene.js +45 -2
- package/dist/feedback/export-bundle.js +1 -1
- package/dist/guest/external-links.d.ts +14 -0
- package/dist/guest/external-links.js +48 -0
- package/dist/guest/runtime-desktop.js +5 -2
- package/dist/keys/key-resolution.d.ts +10 -0
- package/dist/keys/key-resolution.js +33 -5
- package/dist/observer/data.d.ts +4 -0
- package/dist/observer/listen.d.ts +7 -2
- package/dist/observer/listen.js +10 -4
- package/dist/observer/render.d.ts +2 -0
- package/dist/observer/render.js +1 -1
- package/dist/observer/run-routes.js +12 -3
- package/dist/observer-app.html +2 -2
- package/dist/remote/launch.d.ts +53 -0
- package/dist/remote/launch.js +320 -0
- package/dist/remote/observer-proxy.d.ts +11 -0
- package/dist/remote/observer-proxy.js +66 -0
- package/dist/remote/plan.d.ts +35 -0
- package/dist/remote/plan.js +114 -0
- package/dist/remote/reattach.d.ts +17 -0
- package/dist/remote/reattach.js +81 -0
- package/dist/remote/runner.d.ts +68 -0
- package/dist/remote/runner.js +216 -0
- package/dist/routes/computer-use/e2b-desktop/desktop.js +27 -17
- package/dist/routes/computer-use/e2b-desktop/start.js +1 -1
- package/dist/routes/computer-use/e2b-desktop/teardown.js +1 -1
- package/dist/routes/computer-use/in-process-desktop.js +2 -2
- package/dist/routes/computer-use/live-phase.js +1 -1
- package/dist/routes/computer-use/setup.js +2 -2
- package/dist/routes/computer-use/types.d.ts +2 -0
- package/dist/routes/scripted/types.d.ts +2 -0
- package/dist/routes/shared-world/route.js +1 -1
- package/dist/routes/shared-world/types.d.ts +2 -0
- package/dist/routes/terminal/types.d.ts +2 -0
- package/dist/run/costs.d.ts +2 -1
- package/dist/run/display.d.ts +6 -1
- package/dist/run/display.js +11 -0
- package/dist/run/host-suspension.js +4 -1
- package/dist/run/preflight-receipts.d.ts +3 -12
- package/dist/run/preflight-receipts.js +7 -67
- package/dist/run/pricing.d.ts +13 -1
- package/dist/run/pricing.js +50 -4
- package/dist/run/process-owner.d.ts +28 -0
- package/dist/run/process-owner.js +83 -0
- package/dist/run/projection.js +1 -1
- package/dist/run/reclaim.d.ts +6 -0
- package/dist/run/reclaim.js +32 -2
- package/dist/run/results.d.ts +1 -1
- package/dist/run/route-shell.js +1 -1
- package/dist/run/run-cost.d.ts +2 -1
- package/dist/run/run-cost.js +6 -1
- package/dist/run/run-index.d.ts +3 -1
- package/dist/run/run-index.js +23 -11
- package/dist/run/run.d.ts +29 -5
- package/dist/run/run.js +43 -17
- package/dist/run/sandbox-creates.d.ts +10 -3
- package/dist/run/sandbox-creates.js +48 -6
- package/dist/run/status.d.ts +57 -2
- package/dist/run/status.js +87 -1
- package/dist/run/stored-runs.d.ts +3 -0
- package/dist/run/stored-runs.js +9 -4
- package/dist/run-study.d.ts +5 -0
- package/dist/study/doctor.js +14 -1
- package/dist/study/requirements.d.ts +1 -1
- package/dist/study/requirements.js +8 -6
- package/dist/study/route-inputs.js +4 -0
- package/dist/study/run-study-events.d.ts +8 -2
- package/dist/study/run-study-events.js +9 -2
- package/dist/substrates/e2b/base-module.d.ts +3 -0
- package/dist/substrates/e2b/base-module.js +40 -0
- package/dist/substrates/e2b/desktop-browser.d.ts +1 -1
- package/dist/substrates/e2b/desktop-browser.js +27 -7
- package/dist/substrates/e2b/desktop-executor.d.ts +7 -0
- package/dist/substrates/e2b/desktop-executor.js +3 -0
- package/dist/substrates/e2b/desktop-handoff.d.ts +22 -0
- package/dist/substrates/e2b/desktop-handoff.js +86 -0
- package/dist/substrates/e2b/sdk.d.ts +24 -0
- package/dist/substrates/e2b/sdk.js +5 -1
- package/dist/substrates/e2b/shell-only.d.ts +5 -0
- package/dist/substrates/e2b/shell-only.js +27 -0
- package/dist/substrates/local/runtime-release.js +8 -8
- package/dist/substrates/modal/sandbox.js +27 -28
- package/dist/substrates/modal/sdk.d.ts +7 -0
- package/dist/tui-app.js +115 -115
- package/docs/architecture/observer.md +7 -3
- package/docs/contracts/schemas.md +24 -3
- package/docs/contracts/study-analysis.md +52 -4
- package/docs/product/automatic-analysis.md +55 -9
- package/package.json +2 -2
- package/skills/humanish/SKILL.md +118 -5
|
@@ -159,9 +159,13 @@ allowlist, contained file reads, read-only routes and no-store security headers.
|
|
|
159
159
|
Served data may also include `runtime` with `state`, `observedAt`, and
|
|
160
160
|
`source: "local-run-status"`. This is a current observation of a contained,
|
|
161
161
|
matching `status.json`; it never changes the run's recorded verdict or participant
|
|
162
|
-
outcomes. A fresh heartbeat means running,
|
|
163
|
-
finished, a record the CLI's signal handler
|
|
164
|
-
|
|
162
|
+
outcomes. A fresh heartbeat means running, a record its process wrote as `finished` means
|
|
163
|
+
finished, a record written as `interrupted` (by the CLI's signal handler, by a run that ended
|
|
164
|
+
without publishing, or by `humanish reclaim`) means interrupted, and stale or
|
|
165
|
+
invalid timing means unknown. While the run's own process analyzes it after the run ended,
|
|
166
|
+
`runtime.analysis` is `{ startedAt }`: the status band reads **Writing findings** and the
|
|
167
|
+
Findings view shows the analysis as pending, with how long ago it started, before the analysis
|
|
168
|
+
job exists. Missing, malformed, or mismatched records omit the
|
|
165
169
|
observation. A stale heartbeat alone does not prove interruption, and stored PIDs
|
|
166
170
|
are neither probed nor returned. Static rendering and export do not create this
|
|
167
171
|
served-only observation.
|
|
@@ -1411,9 +1411,9 @@ proof claims zero on a `null` line, or when known spend exceeds the declared cap
|
|
|
1411
1411
|
|
|
1412
1412
|
`humanish.run-status.v1` is `status.json`, written inside each run directory by
|
|
1413
1413
|
every route at run start, refreshed on a fixed cadence while the run is
|
|
1414
|
-
alive, and
|
|
1414
|
+
alive, and rewritten when the run ends: `{ schema, runId, state: running |
|
|
1415
1415
|
finished | interrupted, mode, study?, sandboxes?, pid, startedAt, updatedAt, completedAt?,
|
|
1416
|
-
signal?, outcome? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
|
|
1416
|
+
signal?, reason?, outcome?, analysis? }`. A record written by 0.108 or earlier also has `lab`, with the same value as
|
|
1417
1417
|
`study`, and readers accept it. `sandboxes: none` is written at run start by a route that has no
|
|
1418
1418
|
way to create a sandbox for the run: a scripted run against an `app-url` subject. A run that may
|
|
1419
1419
|
create one has no `sandboxes` field. `humanish reclaim` reports such a run `clean` with
|
|
@@ -1453,6 +1453,24 @@ with it at publish: false means some participant or desktop usage has no price
|
|
|
1453
1453
|
and the estimate is a lower bound. A record without it, as every record before
|
|
1454
1454
|
0.109, says nothing about completeness, and readers treat it as unknown.
|
|
1455
1455
|
|
|
1456
|
+
`finished` means the run published its bundle. A process that leaves the run without publishing
|
|
1457
|
+
one, as a route that throws or returns a refusal after the run started does, writes
|
|
1458
|
+
`interrupted` with `reason: no-publish` and no `signal`; humanish 0.139.1 and earlier wrote
|
|
1459
|
+
`finished` with no `outcome` there. `humanish reclaim --run <id>` (not `--check`) rewrites a
|
|
1460
|
+
`running` record whose heartbeat is stale as `interrupted` with `reason: reclaimed` and
|
|
1461
|
+
`completedAt` set to its last `updatedAt`, and reports `markedInterrupted: true`. Either way the
|
|
1462
|
+
run is excluded from a study's duration and cost medians, which count only finished runs. Readers
|
|
1463
|
+
older than `reason` read such a record as interrupted.
|
|
1464
|
+
|
|
1465
|
+
`analysis` is added to a `finished` record when automatic analysis starts in the run's own
|
|
1466
|
+
process: `{ startedAt, endedAt?, state? }`. While it runs, the heartbeat refreshes `updatedAt`
|
|
1467
|
+
again; `completedAt` stays when the participants ended, so a run's duration excludes its
|
|
1468
|
+
analysis. When analysis ends, `endedAt` and `state` (the automatic analysis outcome's state:
|
|
1469
|
+
`complete`, `partial`, `failed`, `cancelled`, `skipped` or `unknown`) are written. An `analysis`
|
|
1470
|
+
without `endedAt` whose `updatedAt` is older than three touch intervals belonged to a process that
|
|
1471
|
+
stopped during analysis. `runAnalysisPhase` (`src/run/status.ts`) reads it as `running`,
|
|
1472
|
+
`stopped` or `ended`. Readers that predate the field ignore it.
|
|
1473
|
+
|
|
1456
1474
|
`interrupted` with `signal` (`SIGINT`, `SIGTERM` or `SIGHUP`) is written by
|
|
1457
1475
|
the CLI's run command when that signal stops a live run outside post-run
|
|
1458
1476
|
analysis. It first prints one line saying it is stopping and refuses any further sandbox
|
|
@@ -1496,7 +1514,10 @@ cheapest-source-first: the `status.json` record, else the bundle, else the run
|
|
|
1496
1514
|
directory alone. `{ runId, derivedFrom: status | bundle | directory, liveness,
|
|
1497
1515
|
mode?, pid?, study?, startedAt?, updatedAt?, completedAt?, verdict?, ok?,
|
|
1498
1516
|
failure?, participants?, estimatedCostUsd?, estimatedCostComplete?,
|
|
1499
|
-
analysisCost?, durationMs? }`. `
|
|
1517
|
+
analysisCost?, durationMs?, analysis? }`. `analysis` is the status record's automatic analysis as
|
|
1518
|
+
`runAnalysisPhase` reads it, `{ state: running | stopped | ended, startedAt, endedAt?, outcome? }`;
|
|
1519
|
+
`humanish runs` and the TUI add `analyzing` or `analysis stopped` to the run's display from it, and
|
|
1520
|
+
`humanish runs --json` carries it on each run. `ok` and `failure` (the first execution failure)
|
|
1500
1521
|
are the run's own outcome; `runDisplay` (`src/run/display.ts`) reads them with
|
|
1501
1522
|
`liveness` and `verdict`, so the TUI and `humanish stats` show a run with
|
|
1502
1523
|
`ok: false` as failed whatever its verdict. The point is cost: walking
|
|
@@ -5,7 +5,8 @@ It is separate from the participant's account, recorded outcome, and the run's
|
|
|
5
5
|
deterministic review verdict. Opening an Observer never starts a provider request.
|
|
6
6
|
Supported live runs request analysis on completion by default. A study that declares no analyst
|
|
7
7
|
gets the Codex account analyst when its participant is a Codex `local-agent`, and the OpenAI
|
|
8
|
-
analyst with a separate $10 cap on its expected cost otherwise.
|
|
8
|
+
analyst with a separate $10 cap on its expected cost otherwise. A study can also select the
|
|
9
|
+
[Gemini API analyst](#gemini-api-analysis). Set `review.analysis: false` to
|
|
9
10
|
disable that request; see [automatic analysis](../product/automatic-analysis.md).
|
|
10
11
|
|
|
11
12
|
## Invocation
|
|
@@ -117,6 +118,49 @@ the current merge instructions, billed $1.92, most of its input served from the
|
|
|
117
118
|
provider's prompt cache, and each of its 466 citations resolved. Two runs do not
|
|
118
119
|
calibrate the merge request's estimate.
|
|
119
120
|
|
|
121
|
+
## Gemini API analysis
|
|
122
|
+
|
|
123
|
+
`review.analysis.provider: gemini`, or `analyze --provider gemini`, sends the same request to
|
|
124
|
+
Google's Gemini API:
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
humanish keys set gemini
|
|
128
|
+
humanish analyze --run latest --provider gemini --max-cost 2 --dry-run --json
|
|
129
|
+
humanish analyze --run latest --provider gemini --model gemini-3.8-flash --max-cost 2 --json
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
The request goes to `generativelanguage.googleapis.com/v1beta/models/<model>:generateContent`
|
|
133
|
+
with the key in the `x-goog-api-key` header. It carries the same instructions, packet and
|
|
134
|
+
captures, with each capture at `MEDIA_RESOLUTION_HIGH`, thinking level `HIGH`, one candidate,
|
|
135
|
+
the standard service tier, no tools and `store: false`. The key is `GEMINI_API_KEY` when set,
|
|
136
|
+
else `GOOGLE_API_KEY`. Google's client libraries prefer `GOOGLE_API_KEY`; humanish does not, so a
|
|
137
|
+
`GOOGLE_API_KEY` exported for another Google API never replaces the stored Gemini key. Key
|
|
138
|
+
discovery fills neither name while the other is already in the environment. Without either, the attempt ends with
|
|
139
|
+
`ANALYSIS_API_KEY_MISSING` and nothing is sent; the OpenAI key is never used.
|
|
140
|
+
|
|
141
|
+
The supported model is `gemini-3.8-flash`, the one with a completed live analysis. The other
|
|
142
|
+
Gemini models whose pages list image input and structured output join once a live analysis on
|
|
143
|
+
each completes. `--max-cost` is required, and admission, the cost cap, `--max-output-tokens`
|
|
144
|
+
(at most 32,768) and the partial result for a bill above the worst case work as on OpenAI.
|
|
145
|
+
Admission prices each capture at 1,120 tokens, the media-resolution guide's figure for a
|
|
146
|
+
high-resolution image, and expects twice the output it expects from OpenAI: one
|
|
147
|
+
`gemini-3.8-flash` analysis of one participant wrote 24,246 output tokens. Usage maps
|
|
148
|
+
`promptTokenCount` to input, `candidatesTokenCount` plus `thoughtsTokenCount` to output, and
|
|
149
|
+
`cachedContentTokenCount` to cached input; a usage record whose `totalTokenCount` disagrees
|
|
150
|
+
with those parts is kept as unknown. Rates are Google's paid standard tier. `gemini-3.8-flash`
|
|
151
|
+
has a published price change on 2027-01-01, and each request is priced at the rates of the day
|
|
152
|
+
it is sent. A free-tier key bills nothing, and Google's terms
|
|
153
|
+
let it use free-tier content to improve its products.
|
|
154
|
+
|
|
155
|
+
Gemini refuses the analysis schemas while they carry `maxItems`, so the request leaves those
|
|
156
|
+
array limits out of the schema and states them, by field path, at the end of the system
|
|
157
|
+
instruction. The response is still checked against the full schema. A `STOP` candidate
|
|
158
|
+
whose only non-thought part is the JSON answer completes. `MAX_TOKENS` and `CONTINUATION` end
|
|
159
|
+
as `output_incomplete`. A blocked prompt and the `SAFETY`, `RECITATION`, `LANGUAGE`,
|
|
160
|
+
`BLOCKLIST`, `PROHIBITED_CONTENT`, `SPII`, `ESCALATION` and image-safety finish reasons end as
|
|
161
|
+
`refusal`. Any other finish reason or answer shape ends as `invalid_response`. An HTTP error
|
|
162
|
+
keeps only its status code, which the attempt's warnings name.
|
|
163
|
+
|
|
120
164
|
## Codex account analysis
|
|
121
165
|
|
|
122
166
|
A study whose participant is a Codex `local-agent` uses this analyst unless it declares another,
|
|
@@ -424,7 +468,7 @@ It contains only the attempt/run IDs, input/config/source digests, prompt versio
|
|
|
424
468
|
and timestamp. The live caller retains a binding to that exact directory and
|
|
425
469
|
publishes the final receipt once. A start without usable final accounting means
|
|
426
470
|
dispatch and spend remain unresolved; it is not proof that a provider charged.
|
|
427
|
-
A cancellation before transport can
|
|
471
|
+
A cancellation before transport can end with `usage.dispatched: false`.
|
|
428
472
|
Admission refusals, missing credentials and rejected dispatch guards do not
|
|
429
473
|
create a potentially paid attempt. Reuse does not create a new start.
|
|
430
474
|
|
|
@@ -492,11 +536,15 @@ The states without findings:
|
|
|
492
536
|
`analyze --max-cost 10`, which runs an analysis anyway. For a skip without `OPENAI_API_KEY`
|
|
493
537
|
on a machine where Codex is signed in with a ChatGPT account (read when the view is built),
|
|
494
538
|
`next` is `analyze --provider codex` and the message names `review.analysis.provider: codex`
|
|
495
|
-
in the run's study file; otherwise the message says to set the key.
|
|
539
|
+
in the run's study file; otherwise the message says to set the key. A study analyzed on Gemini
|
|
540
|
+
is skipped with `AUTOMATIC_ANALYSIS_GEMINI_KEY_MISSING`; its message names `keys set gemini`,
|
|
541
|
+
and `next` is `analyze --provider gemini --model <model> --max-cost 10`.
|
|
542
|
+
For an analysis refused for its
|
|
496
543
|
cost, the message gives the expected cost, the worst case and the cap, and `next` sets
|
|
497
544
|
`--max-cost` to the worst case rounded up.
|
|
498
545
|
- `failed`: the latest attempt failed or was cancelled and no usable version exists. `next`
|
|
499
|
-
reruns it with the same analyst
|
|
546
|
+
reruns it with the same analyst: `--provider codex` for the Codex account analyst, and
|
|
547
|
+
`--provider gemini --model <model>` for a Gemini one.
|
|
500
548
|
- `none`: no analysis ran for this live run and nothing records why, for example a run that
|
|
501
549
|
started from a library call, or one whose study file has moved or changed id.
|
|
502
550
|
- `dry_run`: a dry run has no participant evidence to analyze. `next` is `run` with the run's own
|
|
@@ -50,7 +50,8 @@ allowance before deliberately choosing a larger budget. Explicit
|
|
|
50
50
|
`maxOutputTokens` and `--max-output-tokens` limits are honored exactly. The output allowance
|
|
51
51
|
includes reasoning as well as the report; exhausting it does not produce a usable
|
|
52
52
|
report and never starts an automatic retry. Analysis
|
|
53
|
-
sends selected retained text and captures to OpenAI using `OPENAI_API_KEY
|
|
53
|
+
sends selected retained text and captures to OpenAI using `OPENAI_API_KEY`, or to Google with
|
|
54
|
+
`provider: gemini`.
|
|
54
55
|
Analysis runs in the humanish runner using its credentials. This setting adds no
|
|
55
56
|
credential channel to the target application; each participant's route keeps
|
|
56
57
|
its existing authentication boundary. Review the separate analysis budget before running a manifest live; an actor's
|
|
@@ -78,6 +79,27 @@ remain untested until a request. An omitted provider means this analyst for a Co
|
|
|
78
79
|
`local-agent` participant on any target, and OpenAI for every other participant. This setting
|
|
79
80
|
does not enable managed local desktops.
|
|
80
81
|
|
|
82
|
+
To use Google's Gemini API for the separate analyst:
|
|
83
|
+
|
|
84
|
+
```yaml
|
|
85
|
+
review:
|
|
86
|
+
analysis:
|
|
87
|
+
provider: gemini
|
|
88
|
+
model: gemini-3.8-flash
|
|
89
|
+
maxCostUsd: 2
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Gemini analysis sends the same evidence packet and captures to Google, at high thinking level,
|
|
93
|
+
with each capture at high media resolution. It reads `GEMINI_API_KEY`, which `humanish keys set
|
|
94
|
+
gemini` stores, and `GOOGLE_API_KEY` only when `GEMINI_API_KEY` is unset. Key discovery treats
|
|
95
|
+
the two names as one key: a project's `.humanish/local/provider.env` or the user store never adds
|
|
96
|
+
one while the other is already in the environment. A missing Gemini key skips the analysis
|
|
97
|
+
(`AUTOMATIC_ANALYSIS_GEMINI_KEY_MISSING`); the OpenAI key is never used in its place. Admission,
|
|
98
|
+
`maxCostUsd`, `maxOutputTokens` and priced usage work as they do on OpenAI, at the paid-tier
|
|
99
|
+
rates in `src/run/pricing.ts`; requests made with a free-tier key bill nothing, and Google's
|
|
100
|
+
terms let it use free-tier content to improve its products. The supported model is
|
|
101
|
+
`gemini-3.8-flash`, the one with a completed live analysis.
|
|
102
|
+
|
|
81
103
|
The same configuration works through `humanish run <study>`,
|
|
82
104
|
`watch <study>`, and TUI live starts. Direct library calls to the five recording
|
|
83
105
|
producers honor it too. Supported routes are computer-use, scripted-browser,
|
|
@@ -86,13 +108,34 @@ never enables analysis by default and rejects an explicit analysis mapping
|
|
|
86
108
|
before execution. `false` is accepted on every route. Dry runs show analysis
|
|
87
109
|
as skipped, without reading analysis credentials or making a provider request.
|
|
88
110
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
111
|
+
Automatic analysis starts when the run ends: every participant has stopped and the run has
|
|
112
|
+
published its bundle (`run.json` with an outcome, and `status.json` state `finished`). It runs in
|
|
113
|
+
the same process, so `humanish run` returns only after it. A participant who was blocked or
|
|
114
|
+
interrupted can still have useful retained evidence; analysis requires a verified live recording
|
|
115
|
+
whose participant streams all ended, not a successful participant outcome. An active, missing or
|
|
116
|
+
invalid recording is not analyzed. Default analysis also skips recordings containing only setup
|
|
117
|
+
or failure records with no retained participant activity. A desktop startup failure does not
|
|
118
|
+
start an analysis request. The original failure remains visible.
|
|
119
|
+
|
|
120
|
+
While analysis runs, the run reads as analyzing. `status.json` records `analysis.startedAt` and
|
|
121
|
+
its heartbeat keeps refreshing `updatedAt`; `humanish runs` lists the run as `passed, analyzing`
|
|
122
|
+
(or its other result); the TUI adds `analyzing` to the run; the served Observer shows **Writing
|
|
123
|
+
findings** in its status band and the Findings tab in a pending state with how long ago analysis
|
|
124
|
+
started; `humanish review` says the analysis is running and when it started. In a terminal, `run`
|
|
125
|
+
prints `Participants finished. Writing findings; this usually takes 1 to 4 minutes.` to stderr,
|
|
126
|
+
then `Still writing findings: <elapsed> elapsed.` every 20 seconds. With `--json` it prints only
|
|
127
|
+
the first line, on stderr.
|
|
128
|
+
|
|
129
|
+
A run whose process ended before it published (a crash, or a refusal after the run started) is
|
|
130
|
+
recorded `interrupted` with `reason: no-publish`, and one whose process was killed is marked
|
|
131
|
+
`interrupted` by `humanish reclaim`. `humanish analyze` accepts an interrupted run when every
|
|
132
|
+
participant stream ended; when one was still running it refuses with
|
|
133
|
+
`ANALYSIS_RUN_INCOMPLETE` and names `humanish run <study>`. It refuses a run that is still alive
|
|
134
|
+
with `ANALYSIS_RUN_ACTIVE`, and a run whose heartbeat stopped while its pid is still in use with
|
|
135
|
+
`ANALYSIS_RUN_UNCONFIRMED`, naming `humanish reclaim --run <id>`. When the process running an
|
|
136
|
+
automatic analysis stops before it finishes, `review` says so and names the `analyze` command
|
|
137
|
+
that runs it again. That command takes over the run's `.analysis-lock` when the process that
|
|
138
|
+
held it is gone, and says so in a warning.
|
|
96
139
|
|
|
97
140
|
CLI live starts disclose the selected analyst, its expected cost range and cap, or unknown account dollars before execution.
|
|
98
141
|
`humanish study check <study> --json` and the TUI study screen also expose the
|
|
@@ -126,7 +169,8 @@ run's own result is ok, the first line of the human output reads
|
|
|
126
169
|
command: with Codex signed in with a ChatGPT account it is `humanish analyze --run <id>
|
|
127
170
|
--provider codex`, and the message names `review.analysis.provider: codex` as the setting that
|
|
128
171
|
makes that analyst the study's; otherwise it says to set the key and names `humanish analyze
|
|
129
|
-
--run <id> --max-cost 10`.
|
|
172
|
+
--run <id> --max-cost 10`. A study analyzed on Gemini names `humanish keys set gemini` and
|
|
173
|
+
`humanish analyze --run <id> --provider gemini --model <model> --max-cost 10`. So does a
|
|
130
174
|
default analysis whose expected cost, plus the margin, is over the default $10 cap:
|
|
131
175
|
it is skipped with `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED` and the run exits 0. For
|
|
132
176
|
any analysis refused for its cost, the job records the expected cost, the worst
|
|
@@ -176,6 +220,7 @@ JSON also has `findings`:
|
|
|
176
220
|
| `AUTOMATIC_ANALYSIS_SOURCE_UNAVAILABLE` | `skipped` | This invocation published no run to analyze, or the run id is invalid |
|
|
177
221
|
| `AUTOMATIC_ANALYSIS_SOURCE_CHANGED` | `failed` | The run directory changed after the run published its bundle |
|
|
178
222
|
| `AUTOMATIC_ANALYSIS_KEY_MISSING` | `skipped` | No OpenAI API key |
|
|
223
|
+
| `AUTOMATIC_ANALYSIS_GEMINI_KEY_MISSING` | `skipped` | No Gemini API key for a study analyzed on Gemini |
|
|
179
224
|
| `AUTOMATIC_ANALYSIS_NO_PARTICIPANT_EVIDENCE` | `skipped` | A default analysis found no participant evidence |
|
|
180
225
|
| `AUTOMATIC_ANALYSIS_ACTOR_CANCELLED` | `skipped` | The harness cancelled the participants, or a signal stopped the run |
|
|
181
226
|
| `AUTOMATIC_ANALYSIS_CLEANUP_UNCONFIRMED` | `skipped` | A local VM study could not confirm its cleanup; the command exits 2 |
|
|
@@ -187,6 +232,7 @@ JSON also has `findings`:
|
|
|
187
232
|
| `AUTOMATIC_ANALYSIS_PUBLICATION_FAILED` | `failed` | The analysis could not be published |
|
|
188
233
|
| `AUTOMATIC_ANALYSIS_CANCELLATION_UNAVAILABLE` | `unknown` | A cancellation could not be recorded |
|
|
189
234
|
| `AUTOMATIC_ANALYSIS_OUTCOME_UNKNOWN` | `unknown` | The outcome could not be determined |
|
|
235
|
+
| `AUTOMATIC_ANALYSIS_STOPPED` | `unknown` | Read only: the process running it stopped refreshing its job record |
|
|
190
236
|
|
|
191
237
|
A run refused before it was created (`run: not-created`) prints no `analysis:` line, since there
|
|
192
238
|
is no run to analyze and the refusal says why. Its JSON keeps the record, with
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "humanish",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.142.0",
|
|
4
4
|
"description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"cli",
|
|
@@ -96,7 +96,7 @@
|
|
|
96
96
|
"comms:agentmail:contract": "node --import tsx scripts/agentmail-contract-check.mjs",
|
|
97
97
|
"format": "oxfmt",
|
|
98
98
|
"format:check": "oxfmt --check",
|
|
99
|
-
"lint": "node scripts/check-lint-cap.mjs --max-warnings=
|
|
99
|
+
"lint": "node scripts/check-lint-cap.mjs --max-warnings=410",
|
|
100
100
|
"knip": "knip",
|
|
101
101
|
"prose:check": "node scripts/check-code-prose.mjs",
|
|
102
102
|
"vocabulary:check": "tsx scripts/check-retired-words.ts",
|
package/skills/humanish/SKILL.md
CHANGED
|
@@ -108,6 +108,16 @@ not local inference or the participant's existing conversation. See
|
|
|
108
108
|
[the analysis contract](../../docs/contracts/study-analysis.md) for the current
|
|
109
109
|
CLI/model qualification and setup limits.
|
|
110
110
|
|
|
111
|
+
To analyze on Gemini, the user stores a key with `npx humanish keys set gemini`
|
|
112
|
+
(`GEMINI_API_KEY`; `GOOGLE_API_KEY` is read only when `GEMINI_API_KEY` is
|
|
113
|
+
unset), then sets `review.analysis: {provider: gemini, maxCostUsd: <n>}` or
|
|
114
|
+
passes `analyze --provider gemini --max-cost <n>`. The supported model is
|
|
115
|
+
`gemini-3.8-flash`. This sends the evidence packet and
|
|
116
|
+
captures to Google. Admission, the cost cap and priced usage work as on OpenAI,
|
|
117
|
+
and a missing Gemini key never falls back to OpenAI. On a free-tier key, Google
|
|
118
|
+
may use the content to improve its products; tell the user before a study with
|
|
119
|
+
private evidence runs on one.
|
|
120
|
+
|
|
111
121
|
When post-run analysis is refused for its cost, the run output and `humanish
|
|
112
122
|
review` give its expected cost, its worst case and the command that runs it,
|
|
113
123
|
`humanish analyze --run <id> --max-cost <n>`. Show the user those costs and run
|
|
@@ -298,6 +308,63 @@ fails with `HUMANISH_STUDY_RETIRED_DIRECTORY`; move it to the matching `studies/
|
|
|
298
308
|
run looks like idle tiles — another reason to leave the cap out unless you
|
|
299
309
|
need it.
|
|
300
310
|
|
|
311
|
+
### Phone, tablet and desktop in one study
|
|
312
|
+
|
|
313
|
+
When the user asks to test several screen sizes, write one study with one
|
|
314
|
+
`participants` entry per size and set `device` on each entry. The participants run
|
|
315
|
+
at once, the Observer shows them side by side, and the findings compare them.
|
|
316
|
+
|
|
317
|
+
```yaml
|
|
318
|
+
schema: humanish.study.v3
|
|
319
|
+
id: devices
|
|
320
|
+
title: Phone, tablet and desktop
|
|
321
|
+
route: computer-use
|
|
322
|
+
mode: live
|
|
323
|
+
subject:
|
|
324
|
+
source: app-url
|
|
325
|
+
appUrl: "http://127.0.0.1:3000"
|
|
326
|
+
actor:
|
|
327
|
+
type: local-agent
|
|
328
|
+
localAgent: codex
|
|
329
|
+
persona: synthetic-new-user
|
|
330
|
+
mission: "Sign up and create your first project. Say what was hard to read or reach."
|
|
331
|
+
participants:
|
|
332
|
+
- id: phone
|
|
333
|
+
device: mobile
|
|
334
|
+
- id: tablet
|
|
335
|
+
device: tablet
|
|
336
|
+
- id: desktop
|
|
337
|
+
device: desktop
|
|
338
|
+
execution:
|
|
339
|
+
target: local
|
|
340
|
+
timeoutMs: 600000
|
|
341
|
+
desktop:
|
|
342
|
+
fidelity:
|
|
343
|
+
mobileEmulation: true
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
This runs on the local browser runtime with the user's Codex login. For a deployed
|
|
347
|
+
app, set `execution.target: e2b-desktop`, put the public URL in `subject.appUrl`,
|
|
348
|
+
add `policies.allowPublicTargets: true`, and give each entry a `target` (below).
|
|
349
|
+
|
|
350
|
+
- `device` takes a preset: `mobile` (414×896), `small-mobile`, `narrow-mobile`,
|
|
351
|
+
`tablet` (820×1180), `desktop` (1440×950) or `wide` (1920×1080). The preset
|
|
352
|
+
sizes the participant's screen and browser window.
|
|
353
|
+
- `fidelity.mobileEmulation: true` gives the phone participants a mobile viewport,
|
|
354
|
+
device pixel ratio, touch and a mobile user agent in Chromium. Tablet and
|
|
355
|
+
desktop participants are left as they are.
|
|
356
|
+
- Do not run one study per size, draw a device frame inside the page, or resize
|
|
357
|
+
the browser from the mission. Those produce desktop-ratio evidence labelled as
|
|
358
|
+
a phone, and three runs the Observer cannot compare.
|
|
359
|
+
- Participants on one public app (`policies.allowPublicTargets: true`) share its
|
|
360
|
+
data, so more than one is refused unless each entry sets `target`: a separate
|
|
361
|
+
test account, tenant or seeded URL per participant, or the same URL when the
|
|
362
|
+
task has each participant create its own account. A loopback or clone subject
|
|
363
|
+
needs no `target`.
|
|
364
|
+
- On a local study, `doctor --study` fails its runtime row when the installed
|
|
365
|
+
local runtime predates per-participant screens; `humanish runtime setup`
|
|
366
|
+
installs the pinned one. Do not move the study to E2B without asking the user.
|
|
367
|
+
|
|
301
368
|
Use committed `humanish/studies/*.yaml` for public-safe, reproducible studies. Use
|
|
302
369
|
ignored `.humanish/studies/*.yaml` or `.humanish/local/studies/*.yaml` for private repo
|
|
303
370
|
targets, local-only dogfood, or machine-specific settings. Never commit private
|
|
@@ -472,6 +539,26 @@ Label the participant's own report in the review summary as the participant's ac
|
|
|
472
539
|
place of findings; `analysis.next` is null when no command can produce them. `latest` moves with
|
|
473
540
|
every run, including dry runs, so use the run id the run printed.
|
|
474
541
|
|
|
542
|
+
## Long Studies: `run` Blocks Until Its Findings
|
|
543
|
+
|
|
544
|
+
`humanish run <study>` returns only after every participant has ended and the automatic analysis
|
|
545
|
+
has finished. A participant with `endAfterMs` can keep a study running for up to 24 hours, and
|
|
546
|
+
analysis usually adds 1 to 4 minutes after the last participant ends. A shell timeout that sends
|
|
547
|
+
SIGTERM stops the run: humanish marks it interrupted and stops its sandboxes.
|
|
548
|
+
|
|
549
|
+
- Start a study that can outlive your shell call in tmux, for example
|
|
550
|
+
`tmux new -d -s study 'npx humanish run <study> --json --no-open > run-result.json 2> run.log'`.
|
|
551
|
+
`--detach` only skips the Observer server at the end; it does not move the run to the background.
|
|
552
|
+
- The first stderr line names the run: `Run <id> started. Follow it with humanish observe --run <id>`.
|
|
553
|
+
If you did not keep it, `npx humanish runs --json` lists runs newest first.
|
|
554
|
+
- In `runs --json`, a run whose participants are working has `display.state: running`; once they
|
|
555
|
+
end, its `display` is their result and `analysis.state` is `running` while findings are written.
|
|
556
|
+
- `npx humanish review --run <id> --json` says where the run is: participants running, analysis
|
|
557
|
+
running since a given time, or the findings.
|
|
558
|
+
- If the run's process died, `npx humanish reclaim --run <id>` stops its sandboxes and marks it
|
|
559
|
+
interrupted. `npx humanish analyze --run <id>` then analyzes it if every participant had ended,
|
|
560
|
+
and otherwise names the command that runs the study again.
|
|
561
|
+
|
|
475
562
|
## Optional Live E2B Study
|
|
476
563
|
|
|
477
564
|
Live headed E2B desktop participants are optional. Add the substrate dependency only
|
|
@@ -481,12 +568,18 @@ when the user explicitly wants live E2B execution:
|
|
|
481
568
|
npm i -D @e2b/desktop
|
|
482
569
|
```
|
|
483
570
|
|
|
484
|
-
Then confirm env var names are documented without values
|
|
571
|
+
Then confirm env var names are documented without values. Which keys a hosted
|
|
572
|
+
study needs depends on its participant and analyst:
|
|
485
573
|
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
OPENAI_API_KEY
|
|
489
|
-
|
|
574
|
+
| Participant | Keys |
|
|
575
|
+
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
576
|
+
| `openai-computer-use` | `E2B_API_KEY`, `OPENAI_API_KEY` |
|
|
577
|
+
| `local-agent` with `localAgent: codex` | `E2B_API_KEY` and `codex login`; the findings analysis uses the same Codex account |
|
|
578
|
+
| `local-agent` with `localAgent: claude` | `E2B_API_KEY` and `claude auth login`; `OPENAI_API_KEY` for findings; without it, a signed-in Codex can analyze the run (the run prints the command) |
|
|
579
|
+
|
|
580
|
+
A hosted Codex participant does not need `OPENAI_API_KEY`. Do not report a missing
|
|
581
|
+
`OPENAI_API_KEY` as the reason a Codex study on E2B fails; read the
|
|
582
|
+
`doctor --study` row instead.
|
|
490
583
|
|
|
491
584
|
Do not paste values into files, prompts, run bundles, issue drafts, or logs.
|
|
492
585
|
Load local values only at invocation time:
|
|
@@ -502,6 +595,26 @@ declared scenario is developer-experience testing. Private repos are allowed
|
|
|
502
595
|
only as explicit maintainer-authorized runs with repo redaction left on; never
|
|
503
596
|
publish their names, screenshots, logs, source snippets, or operational details.
|
|
504
597
|
|
|
598
|
+
## Setup errors and what they mean
|
|
599
|
+
|
|
600
|
+
Run `npx humanish doctor --study <id>` before the first live run and read every
|
|
601
|
+
`missing` row. The common ones:
|
|
602
|
+
|
|
603
|
+
- `codex_unsafe_configuration` on a hosted Codex participant: the row names the
|
|
604
|
+
setting humanish refused. Report it to the user; never edit, move or rename
|
|
605
|
+
their Codex, agent or skill config to make a run pass. An `AGENTS.md` in the
|
|
606
|
+
Codex home is admitted, and doctor notes that the participant also receives it.
|
|
607
|
+
- `subject.appUrl` must be loopback: a non-loopback URL needs
|
|
608
|
+
`policies.allowPublicTargets: true`, and with more than one participant, a
|
|
609
|
+
`target` on each entry.
|
|
610
|
+
- "Will be skipped: OPENAI_API_KEY is missing" on `post-run analysis` (an
|
|
611
|
+
`openai-computer-use` or Claude participant): the run works without findings,
|
|
612
|
+
and when Codex is signed in, its `next:` line gives the command that analyzes it with Codex.
|
|
613
|
+
`review.analysis.provider: codex` makes that the study's default.
|
|
614
|
+
- A local browser study refuses a non-loopback URL, a port below 1024, hosted
|
|
615
|
+
templates and `sandboxTimeoutMs`. The message names the setting; do not switch the
|
|
616
|
+
study to a hosted target unless the user agrees to E2B usage.
|
|
617
|
+
|
|
505
618
|
## Reporting Back
|
|
506
619
|
|
|
507
620
|
Report a live run's findings first, as
|