humanish 0.145.0 → 0.147.0

package/docs/contracts/schemas.md

@@ -76,17 +76,17 @@ map each one to its v3 key.
 A study declares its route and composes code primitives; it is not a hardcoded kind. The
 top-level keys:
 
-| Key                                                               | Meaning                                                                                                                                                                       |
-| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `schema`, `id`, `title`, `description`                            | the document's identity                                                                                                                                                       |
-| `route`                                                           | required: `preview`, `computer-use`, `shared-world`, `terminal` or `scripted`                                                                                                 |
-| `mode`                                                            | `dry-run` (the default) or `live`                                                                                                                                             |
-| `actor`                                                           | who drives it, one object: `type`, `model`, `localAgent`, `persona`, `mission`, `tasks`, `maxOutputTokens`, `maxWaitMs`, `idleWaitMs`, `reasoningEffort`, `stopWhen`, `dwell` |
-| `participants`                                                    | a count, a homogeneous `{ count, instruction }`, or a list of entries (below)                                                                                                 |
-| `surfaces`                                                        | scripted only: `[desktop]` or `[desktop, mobile]`                                                                                                                             |
-| `caps`                                                            | the route's budget (below)                                                                                                                                                    |
-| `scenario`                                                        | scripted only: a committed scenario id or path                                                                                                                                |
-| `subject`, `execution`, `policies`, `review`, `defaults`, `comms` | as below                                                                                                                                                                      |
+| Key                                                               | Meaning                                                                                                                                                                                      |
+| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `schema`, `id`, `title`, `description`                            | the document's identity                                                                                                                                                                      |
+| `route`                                                           | required: `preview`, `computer-use`, `shared-world`, `terminal` or `scripted`                                                                                                                |
+| `mode`                                                            | `dry-run` (the default) or `live`                                                                                                                                                            |
+| `actor`                                                           | who drives it, one object: `type`, `model`, `localAgent`, `persona`, `mission`, `tasks`, `maxOutputTokens`, `maxWaitMs`, `idleWaitMs`, `reasoningEffort`, `serviceTier`, `stopWhen`, `dwell` |
+| `participants`                                                    | a count, a homogeneous `{ count, instruction }`, or a list of entries (below)                                                                                                                |
+| `surfaces`                                                        | scripted only: `[desktop]` or `[desktop, mobile]`                                                                                                                                            |
+| `caps`                                                            | the route's budget (below)                                                                                                                                                                   |
+| `scenario`                                                        | scripted only: a committed scenario id or path                                                                                                                                               |
+| `subject`, `execution`, `policies`, `review`, `defaults`, `comms` | as below                                                                                                                                                                                     |
 
 The parser reads only the declared route's keys. A key that route never reads is an error that
 names the route, such as "route: computer-use does not read scenario. Remove it.", and an
@@ -212,7 +212,7 @@ The rest of this section names keys by their v3 spelling:
   browser target; `entry` is a shared-world participant's same-origin path;
 - `participants[].files` (computer-use on `execution.target: e2b-desktop` only):
   1 to 20 `{ from, as }` entries, each a file copied onto that participant's desktop
-  before its browser opens. `from` is relative to the folder that holds the study file
+  before its session starts. `from` is relative to the folder that holds the study file
   (`cwd` for a library config); `as` is relative to the desktop user's home folder.
   The parser refuses an absolute path, a `.`, `..` or empty part, a backslash, an `as`
   that starts with `~`, lands in `subject/` or on a name starting with `.humanish`, looks
@@ -1148,6 +1148,14 @@ Core-owned fields:
   is part of who the participant was rather than of how the instrument was tuned
   (docs/principles/actor-fidelity.md), so a trace without it is a result missing
   half its sample description
+- optional `modelSettings.serviceTier`: `fast` when the study declared
+  `actor.serviceTier: fast`, so every first-party OpenAI CUA request carried
+  `service_tier: "fast"` (OpenAI Fast mode). Absent when undeclared. Each
+  `tokenUsage.turns[]` record of such a trace has `serviceTier`, the tier the
+  response says served it: `priority` or `fast` bill at the model's Fast mode rate
+  in `src/run/pricing.ts`, and `default`, which OpenAI serves when Fast mode
+  traffic ramps too fast, bills at the standard rate. `estimatedCost.breakdown.fastTurns`
+  counts the requests priced at the fast rate.
 - optional `modelSettings.maxOutputTokens`: the declared positive integer
   `actor.maxOutputTokens`, passed to every first-party OpenAI CUA response
   request, including reasoning output. Absent when undeclared. The first request

package/docs/contracts/study-analysis.md

@@ -27,8 +27,10 @@ cap; a refused dry run says why, and names the `--max-cost` that admits it.
 
 The default provider is `openai`; its default model is `gpt-6.1-sol`, at medium reasoning effort
 ([reasoning effort](#reasoning-effort)). A request sends selected retained text and
-captures to OpenAI, without tools, redirects, provider-side response storage, or
-automatic retries. `--question` adds a reviewer question; it never changes the
+captures to OpenAI, without tools, redirects or provider-side response storage. A request that
+runs out of output is sent once more while the cap admits it
+([a request that runs out of output](#a-request-that-runs-out-of-output)); no other failure is
+sent again. `--question` adds a reviewer question; it never changes the
 participant assignment. For OpenAI API analysis, `--max-cost` is required, including for dry-run
 admission. Admission refuses the request when both its worst case and its expected cost plus a
 10% margin are over it; the provider bill is not limited by it.
@@ -60,12 +62,39 @@ times for those billed $1 or more.
 A refusal's message gives the expected cost, the worst case, the cap, and
 `humanish analyze --run <id> --max-cost <n>` with `n` the worst case rounded up.
 
-The default deadline is ten minutes for each request. With no explicit output-token limit, analysis
-uses 32,768 tokens if admission fits the declared budget, or retains the prior
-16,384-token allowance otherwise. Explicit limits are never adjusted. Selection
-happens before dispatch; `admission.outputTokenAllowance` exposes it in dry-run,
-and the saved configuration, digest and automatic job bind the selected allowance.
-This selection never increases the spending limit or starts a retry.
+The default deadline is ten minutes for each request. With no explicit output-token limit,
+`src/analysis/output-limit.ts` sizes the allowance to the packet. An OpenAI analysis at none, low
+or medium effort of up to 16 participants gets 12,288 tokens plus 2,048 per participant, at most
+32,768: 14,336 for one participant and 18,432 for three. A high-effort analysis, a Gemini
+analysis and a run analysed in cohorts get 32,768. If admission refuses an allowance above
+16,384 under the declared budget, the analysis keeps 16,384. Explicit limits are never adjusted.
+Selection happens before dispatch; `admission.outputTokenAllowance` exposes it in dry-run, and
+the saved configuration, digest and automatic job bind the selected allowance. This selection
+never increases the spending limit.
+
+The sizes come from the finished reports on the 2026-10-22 and 2026-10-23 benchmark recordings:
+the Taskly analyses listed in `docs/evidence/benchmark/2026-10-2*.json`, one participant with 8 to
+14 captures, and the movie-game analyses of
+[analysis models on fixed recordings](https://github.com/danielgwilson/humanish/blob/main/docs/evidence/benchmark/2026-10-22-analysis-models.md), three
+participants with 40 to 49 captures. Each analysis's output tokens are read from its retained
+artifact.
+
+| Analyst and effort   | One participant: passed, most output | Three participants: passed, most output |
+| -------------------- | ------------------------------------ | --------------------------------------- |
+| `gpt-6.1-sol` low    | 24, 4,308                            | 6, 6,391                                |
+| `gpt-6.1-sol` medium | 142, 5,659                           | 36, 9,000                               |
+| `gpt-6-luna` medium  | 27, 8,354                            | 2, 11,313                               |
+| `gpt-6.1-sol` high   | 22, 10,086                           | 6, 15,797                               |
+| `gpt-6-luna` high    | 78, 19,880                           | 9, 26,439                               |
+| `gemini-3.8-flash`   | 23, 29,739                           | 1, 26,154                               |
+
+At low and medium effort the sized limit is 1.7 times the most output of a finished report of
+one participant and 1.6 times that of three; for `gpt-6.1-sol` at medium it is 2.5 and 2.0 times.
+At high effort a finished `gpt-6-luna` report of three participants used 26,439 tokens, so 32,768
+has no room to lower. A `gpt-6.1-sol` analysis at medium that ran out of output on a
+one-participant run had used all 32,768 tokens in 374 seconds and cost $0.39; at 14,336 tokens the
+same request stops at 44% of that output.
+
 The configured wall-clock deadline covers the entire request, including waiting
 for response headers and reading the body. A request-scoped dispatcher prevents
 Node's separate fetch timeout from cutting a longer configured deadline short;
@@ -77,6 +106,29 @@ success. Findings keep reported concerns and observed recovery distinct across
 participants. Other supported models can be selected explicitly, but evidence
 reference validation does not certify their interpretation of small visual details.
 
+## A request that runs out of output
+
+When an OpenAI or Gemini request ends with `output_incomplete`, humanish sends the same request
+once more, with a fresh deadline. It does so only when the spend so far, plus what admission
+holds for the attempt's requests still running and admission's figure for one more request (its
+expected cost plus 10%, or its worst case when that is lower), is within `maxCostUsd`. The spend
+so far counts every response of the attempt, the one that ran out included. When a response did
+not report its usage, the spend is unknown and the request is not sent again. A request is sent
+again at most once; when the second one also runs out, the attempt fails with
+`analysis_output_incomplete`. A cohort request and the merge request are each sent again the same
+way. Codex account analysis has no dollar cap to check and is never sent again. A timeout, an HTTP
+error and a rejected report are not sent again: a timed-out request has already used the
+ten-minute deadline.
+
+The attempt keeps one execution start, one receipt and one `analysis.json`. Their `usage` is the
+sum of every response, both requests of a retry included, and `estimatedCostUsd` prices each
+response as its own request. The schemas are unchanged. A warning in `warnings` (and in the human
+output of `analyze` and of a live run, under its `analysis:` line) names the request, the input
+and output tokens and the cost of the response that ran out, and says whether humanish sent it
+again; a refused retry gives the spend, the held amount and the cap. The result is partial with
+`analysis_admission_estimate_exceeded` only when the bill passes admission's worst case plus the
+worst case of each request sent again, or the cap.
+
 ## Runs with more than 16 participants
 
 One analysis covers up to 128 participants, the most streams it reads from a run
@@ -104,7 +156,8 @@ whole packet as a single request's report does.
 The attempt has one execution start, one receipt and one `analysis.json`, with the
 run's participants, coverage and evidence and the summed usage of every request
 sent. The artifact schema is unchanged. Its stored limits grew to hold eight
-cohorts: 6,400 evidence items, 320 captures and 16 MiB. If a cohort request fails
+cohorts: 6,400 evidence items, 320 captures and 16 MiB. A cohort request or the merge request that
+runs out of output is sent once more under the rule above. If a cohort request fails
 or is rejected, no further request starts, requests already sent finish, no merge
 request is sent and the attempt fails with that request's code, with no findings;
 a warning names the cohort's size. A failed merge request fails the attempt the same way. The usage
@@ -261,8 +314,8 @@ stopped, an operator can remove the empty lock directory and retry.
 
 ## Evidence and findings
 
-Each request's packet admits up to 16 participants, 800 evidence items, 40 PNG
-captures, 160 KiB of text and 20 MiB of images. Individual source files, image
+Each request's packet admits up to 16 participants, 800 evidence items, 40
+captures (PNG or lossless WebP), 160 KiB of text and 20 MiB of images. Individual source files, image
 dimensions and result sizes have separate limits. Analysis reads run.json up to
 16 MiB and each participant's trace up to 32 MiB, as verify and the Observer do.
 It reads the traces one at a time and keeps of each only what selection can

package/docs/product/automatic-analysis.md

@@ -16,15 +16,17 @@ expected cost and a 600-second timeout. A study or command that names a model bu
 that model at high; only the default analyst, with no model named, runs at medium. On the same
 recordings it matched `gpt-6-astra` at high, the earlier default, at about one seventh of the cost
 ([analysis models on fixed recordings](https://github.com/danielgwilson/humanish/blob/main/docs/evidence/benchmark/2026-10-22-analysis-models.md)). When no output limit is specified,
-humanish selects 32,768 tokens if the exact input's expected cost still fits that
-cap; otherwise it keeps the established 16,384-token allowance. This preserves
-previously admitted studies without increasing their spending limit. To customize it:
+humanish sizes it to the analysis: at low or medium effort, 12,288 tokens plus 2,048 per
+participant (14,336 for one), and 32,768 at high effort, on Gemini and past 16 participants. An
+allowance above 16,384 that the cap would refuse falls back to 16,384, so a study its cap admitted
+before stays admitted ([output limit](../contracts/study-analysis.md#invocation)). A
+request that runs out of output is sent once more while the cap admits it. To customize it:
 
 ```yaml
 review:
   analysis:
     maxCostUsd: 10
-    # Optional overrides. Omit maxOutputTokens for budget-aware selection.
+    # Optional overrides. Omit maxOutputTokens for a limit sized to the analysis.
     # A named model without reasoningEffort runs at high.
     model: gpt-6.1-sol
     reasoningEffort: medium
@@ -57,8 +59,9 @@ effort; on 24 Taskly analyses `gpt-6.1-sol` at medium wrote a mean 3,747 and cos
 evidence to inspect the expected cost, the worst case and the selected token
 allowance before deliberately choosing a larger budget. Explicit
 `maxOutputTokens` and `--max-output-tokens` limits are honored exactly. The output allowance
-includes reasoning as well as the report; exhausting it does not produce a usable
-report and never starts an automatic retry. Analysis
+includes reasoning as well as the report; a request that exhausts it has no usable report. It
+is sent once more when the spend so far and admission's figure for one more request fit the cap,
+and the CLI says so under the analysis line; the attempt's usage counts both requests. Analysis
 sends selected retained text and captures to OpenAI using `OPENAI_API_KEY`, or to Google with
 `provider: gemini`.
 Analysis runs in the humanish runner using its credentials. This setting adds no

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "humanish",
-  "version": "0.145.0",
+  "version": "0.147.0",
   "description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
   "keywords": [
     "cli",
@@ -105,6 +105,7 @@
     "site:hero:proof": "node scripts/site-hero-wrap-proof.mjs"
   },
   "dependencies": {
+    "@jsquash/webp": "1.5.0",
     "commander": "^15.0.0",
     "parse5": "^8.0.1",
     "playwright-core": "^1.63.0",
@@ -120,13 +121,13 @@
     "@types/pngjs": "^6.0.5",
     "axe-core": "4.14.0",
     "fast-check": "catalog:",
-    "knip": "6.41.0",
+    "knip": "6.42.1",
     "modal": "^0.11.1",
     "oxc-parser": "catalog:",
-    "oxfmt": "0.73.0",
-    "oxlint": "1.88.0",
+    "oxfmt": "0.74.0",
+    "oxlint": "1.89.0",
     "oxlint-tsgolint": "7.0.2004",
-    "skills": "1.7.2",
+    "skills": "1.7.4",
     "tsx": "^4.23.15",
     "typescript": "catalog:",
     "vitest": "catalog:"