humanish 0.113.0 → 0.114.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (68) hide show
  1. package/CHANGELOG.md +22 -0
  2. package/CONTEXT.md +1 -0
  3. package/README.md +1 -1
  4. package/dist/actors/computer-use/openai-conversation.js +9 -7
  5. package/dist/actors/contract.d.ts +4 -2
  6. package/dist/analysis/admission.d.ts +37 -0
  7. package/dist/analysis/admission.js +60 -0
  8. package/dist/analysis/automatic-completion.d.ts +0 -6
  9. package/dist/analysis/automatic-completion.js +3 -3
  10. package/dist/analysis/automatic-config.d.ts +10 -2
  11. package/dist/analysis/automatic-config.js +29 -14
  12. package/dist/analysis/automatic.js +7 -0
  13. package/dist/analysis/execute.d.ts +19 -5
  14. package/dist/analysis/execute.js +58 -27
  15. package/dist/analysis/image-tokens.d.ts +2 -0
  16. package/dist/analysis/image-tokens.js +1 -1
  17. package/dist/analysis/job.d.ts +9 -1
  18. package/dist/analysis/job.js +11 -2
  19. package/dist/analysis/service.js +6 -3
  20. package/dist/cli/commands/analyze.js +7 -2
  21. package/dist/cli/commands/notes.d.ts +3 -0
  22. package/dist/cli/commands/notes.js +144 -0
  23. package/dist/cli/commands/observe.js +1 -1
  24. package/dist/cli/commands/study-run.js +2 -1
  25. package/dist/cli/findings.js +4 -1
  26. package/dist/cli/io.d.ts +1 -0
  27. package/dist/cli/io.js +11 -12
  28. package/dist/cli/program.js +2 -0
  29. package/dist/evidence/encoded-text.d.ts +15 -2
  30. package/dist/evidence/encoded-text.js +68 -4
  31. package/dist/evidence/secret-scrub.d.ts +1 -1
  32. package/dist/evidence/secret-scrub.js +23 -2
  33. package/dist/feedback/draft.d.ts +19 -0
  34. package/dist/feedback/draft.js +51 -1
  35. package/dist/feedback/export-bundle.js +2 -2
  36. package/dist/feedback/export.js +47 -3
  37. package/dist/feedback/feedback.js +42 -2
  38. package/dist/observer/artifact.d.ts +5 -0
  39. package/dist/observer/artifact.js +18 -2
  40. package/dist/observer/notes-route.d.ts +21 -0
  41. package/dist/observer/notes-route.js +140 -0
  42. package/dist/observer/pinned-files.d.ts +8 -1
  43. package/dist/observer/pinned-files.js +32 -3
  44. package/dist/observer/render.js +38 -14
  45. package/dist/observer/run-routes.d.ts +8 -2
  46. package/dist/observer/run-routes.js +47 -6
  47. package/dist/observer/serve.d.ts +3 -0
  48. package/dist/observer/serve.js +28 -3
  49. package/dist/observer-app.html +7 -6
  50. package/dist/run/contained-output.d.ts +7 -0
  51. package/dist/run/contained-output.js +35 -1
  52. package/dist/run/note-files.d.ts +73 -0
  53. package/dist/run/note-files.js +184 -0
  54. package/dist/run/notes.d.ts +37 -0
  55. package/dist/run/notes.js +198 -0
  56. package/dist/run/participant-caption.d.ts +9 -0
  57. package/dist/run/participant-caption.js +19 -0
  58. package/dist/study/doctor.js +1 -1
  59. package/dist/study/preflight-result.js +2 -1
  60. package/dist/study/summary.js +2 -2
  61. package/docs/contracts/errors.md +2 -2
  62. package/docs/contracts/run-bundle.md +57 -0
  63. package/docs/contracts/schemas.md +5 -2
  64. package/docs/contracts/study-analysis.md +26 -8
  65. package/docs/principles/invariants-and-defaults.md +2 -2
  66. package/docs/product/automatic-analysis.md +22 -16
  67. package/package.json +2 -2
  68. package/skills/humanish/SKILL.md +5 -0
@@ -576,6 +576,63 @@ updatedAt: "<ISO timestamp>"
576
576
 
577
577
  The latest pointer may move. Run bundle directories should not.
578
578
 
579
+ ## Reviewer notes
580
+
581
+ Each note a person adds while reviewing a run is its own file in the run directory,
582
+ `notes/<id>.json`, written by `humanish notes --add` and by an Observer served on loopback.
583
+ `notes/` is absent until the first note.
584
+
585
+ ```json
586
+ {
587
+ "schema": "humanish.run-note.v1",
588
+ "runId": "example-2026-06-02t10-00-00-000z-proof",
589
+ "id": "note-20260602t103000000z-000000000001",
590
+ "atMs": 151000,
591
+ "participant": "stream-001",
592
+ "nearest": { "participant": "stream-001", "itemId": "turn-07-call-01" },
593
+ "text": "They looked for the save button here.",
594
+ "author": "you",
595
+ "createdAt": "2026-06-02T10:30:00.000Z",
596
+ "editedAt": null
597
+ }
598
+ ```
599
+
600
+ - `id`: `note-`, the creation time as `yyyymmddthhmmssmmmz`, `-` and 12 hex digits of
601
+ random bits. The file is named `<id>.json`, so the files sort in the order they were
602
+ added.
603
+ - `runId`: the run directory's name. A file whose `runId` or `id` does not match its run or
604
+ its file name is not read.
605
+ - `atMs`: milliseconds from the run clock's start, the earliest timed capture or
606
+ desktop video of any participant. The Observer's study timeline shows that moment as
607
+ 00:00. A participant's captures count only when every one is stamped and the stamps
608
+ never go back.
609
+ - `participant`: the stream id the note belongs to, or `null` for the whole run.
610
+ `humanish notes --add --participant` and the Observer also take the participant's id from
611
+ the study (the stream's `laneId`) and record its stream id. A name that is one stream's id
612
+ and another stream's study id is refused.
613
+ - `nearest`: the latest stamped trace item, a capture or an event, at or before the
614
+ moment, among the note's participant or every participant; `null` when there is none.
615
+ - `text`: 1 to 2000 characters, after the run's known values, secret-shaped values and
616
+ local paths are replaced.
617
+ - `author`: `"you"` in this version.
618
+ - `createdAt`, `editedAt`: ISO-8601. Notes cannot be edited yet, so `editedAt` is `null`.
619
+
620
+ A note file is written once and never rewritten: its bytes go to a temporary file inside
621
+ `notes/`, which `link` gives the note's name only when no file has that name, and a taken name
622
+ gets a new id. Adding a note reads no other note, so notes added at the same time from any
623
+ process are all kept, and there is no lock. Readers list at most 1000 entries of `notes/` and
624
+ read at most 500 notes, each within 16 KiB and refused before reading past that; any other
625
+ entry, or a file that is not a note of the run, is skipped and named. The Observer servers serve
626
+ any file under `notes/` only within 16 KiB. A run takes no note past 500.
627
+ `verify` scans the note files with every other run file, so a secret-shaped value in a note
628
+ keeps the run from `share_ready`. A bundle export copies them; an HTML export carries the notes
629
+ read-only after checking the set it read.
630
+
631
+ Threat model: forged web requests are refused, whether from a page on another site or through a
632
+ name rebound to 127.0.0.1, by the token, Origin and Host checks on `POST /api/notes`. A local
633
+ process with write access to the run directory is out of scope: it already has the user's
634
+ privileges and can change any file in the run.
635
+
579
636
  ## Verify Result Share Safety
580
637
 
581
638
  `humanish.verify-result.v1` includes a machine-readable `shareSafety` block in
@@ -1008,9 +1008,12 @@ Core-owned fields:
1008
1008
  `cacheWriteInput`. `summarizedTurns` counts the earlier exchanges the latest
1009
1009
  `explicit_context` request no longer carried whole: lines of the note, or
1010
1010
  only counted once the note reached its cap.
1011
- `requests[]` has one entry per participant request: its `mode` and, in
1011
+ `requests[]` has one entry per participant turn or impressions request: its `mode` and, in
1012
1012
  `explicit_context`, `carriedExchanges`, `carriedScreenshots` and
1013
- `estimatedInputTokens`. Billed input per request is in `tokenUsage.turns`. A
1013
+ `estimatedInputTokens`. The impressions-only request has `kind: impressions`;
1014
+ the optional `kind` field is absent on participant turns and in older bundles.
1015
+ Impressions carry the context the next turn would carry, with the same budget
1016
+ and summarization of older turns. Billed input per request is in `tokenUsage.turns`. A
1014
1017
  participant that ran in `explicit_context` gets a run warning. Absent on other
1015
1018
  providers and on earlier bundles
1016
1019
  - optional `affordanceUse` (`humanish.affordance-use.v1`): which kind of route this
@@ -4,7 +4,7 @@ Study analysis is an independent interpretation of retained participant evidence
4
4
  It is separate from the participant's account, recorded outcome, and the run's
5
5
  deterministic review verdict. Opening an Observer never starts a provider request.
6
6
  Supported live runs request analysis on completion by default, with a separate
7
- $3 admission estimate limit. Set `review.analysis: false` to disable that request;
7
+ $3 cap on its expected cost. Set `review.analysis: false` to disable that request;
8
8
  see [automatic analysis](../product/automatic-analysis.md).
9
9
 
10
10
  ## Invocation
@@ -20,16 +20,30 @@ humanish analyze show --run latest --json
20
20
  The source must be a verified, completed live run. A dry-run contract bundle is
21
21
  not a participant study. Here, `analyze --dry-run` means checking an existing
22
22
  study's input and admission estimate without credentials, a provider request,
23
- or a new analysis artifact.
23
+ or a new analysis artifact. It prints the expected cost, the worst case and the
24
+ cap; a refused dry run says why, and names the `--max-cost` that admits it.
24
25
 
25
26
  The default provider is `openai`; its default model is `gpt-6-astra`, with high reasoning effort. A request sends selected retained text and
26
27
  captures to OpenAI, without tools, redirects, provider-side response storage, or
27
28
  automatic retries. `--question` adds a reviewer question; it never changes the
28
29
  participant assignment. For OpenAI API analysis, `--max-cost` is required, including for dry-run
29
- admission. It bounds a conservative estimate, not an exact provider bill.
30
- `--timeout-ms` and `--max-output-tokens` bound the request. An exceeded admission
31
- estimate retains valid findings and usage but returns a partial result and a
32
- nonzero command exit, including when that version is reused.
30
+ admission. Admission compares the expected cost plus a 10% margin with it; the
31
+ provider bill is not limited by it.
32
+ `--timeout-ms` and `--max-output-tokens` bound the request. A bill above the
33
+ worst case or the cap retains valid findings and usage but returns a partial
34
+ result and a nonzero command exit, including when that version is reused.
35
+
36
+ The estimate counts the instructions, evidence packet and result schema at 3
37
+ UTF-8 bytes per input token, adds 2,048 framing tokens and each capture's
38
+ high-detail image tokens, and prices input at the model's highest input rate.
39
+ The expected output is 12,000 tokens plus 1,000 per participant, at most the
40
+ output allowance; the worst case spends the whole allowance. `admission` in
41
+ `--json` output has `estimatedCostUsd` (the expected cost), `worstCaseCostUsd`,
42
+ `maxCostUsd`, `inputTokenAllowance` (the input tokens priced) and
43
+ `outputTokenAllowance`. On 148 billed gpt-6-astra analyses the expected cost was
44
+ 1.08 to 7.3 times the bill, and 1.08 to 1.32 times for those billed $1 or more.
45
+ A refusal's message gives the expected cost, the worst case, the cap, and
46
+ `humanish analyze --run <id> --max-cost <n>` with `n` the worst case rounded up.
33
47
 
34
48
  The default deadline is ten minutes. With no explicit output-token limit, analysis
35
49
  uses 32,768 tokens if admission fits the declared budget, or retains the prior
@@ -304,7 +318,9 @@ describes:
304
318
  The optional automatic job is separate from the immutable analysis. Its view
305
319
  binds terminal state to the exact execution receipt and report. A stale or
306
320
  unverifiable job remains unknown; reading or exporting it never dispatches.
307
- Automatic job metadata is omitted from shared derivatives. See
321
+ Automatic job metadata is omitted from shared derivatives. A job refused for its
322
+ cost also records `admission` (`expectedCostUsd`, `worstCaseCostUsd`,
323
+ `maxCostUsd`), which `review` and the Observer read; other jobs omit it. See
308
324
  [automatic analysis](../product/automatic-analysis.md).
309
325
 
310
326
  Version and correction directories are claimed exclusively; publication is
@@ -384,7 +400,9 @@ The states without findings:
384
400
  - `skipped`: the automatic analysis was skipped or refused, for example without
385
401
  `OPENAI_API_KEY`, or the run's study file sets `review.analysis: false`
386
402
  (`AUTOMATIC_ANALYSIS_DISABLED`, read from the file when the view is built). `next` is
387
- `analyze --max-cost 3`, which runs an analysis anyway.
403
+ `analyze --max-cost 3`, which runs an analysis anyway. For an analysis refused for its
404
+ cost, the message gives the expected cost, the worst case and the cap, and `next` sets
405
+ `--max-cost` to the worst case rounded up.
388
406
  - `failed`: the latest attempt failed or was cancelled and no usable version exists. `next`
389
407
  reruns it with the same analyst, `--provider codex` for the Codex account analyst.
390
408
  - `none`: no analysis ran for this live run and nothing records why, for example a run that
@@ -39,7 +39,7 @@ certify (see the conformance suite).
39
39
  turn a dry run, preview, reader or unsupported route into provider or sandbox
40
40
  spend. An explicitly live supported study includes the disclosed default
41
41
  post-run analysis budget unless `review.analysis: false` disables it. Analysis
42
- has a separate admission estimate limit, not a provider billing cap or part of
42
+ has a separate cap on its expected cost, not a provider billing cap or part of
43
43
  the actor budget. Spend-bearing tests retain their explicit env gates.
44
44
  4. **Evidence verifies fail-closed.** A run bundle that cannot pass verification (schema,
45
45
  redaction status, artifact presence, public-safety scan) is a failed run, even when the
@@ -90,7 +90,7 @@ silently drifting from one is not.
90
90
  | Default | Why it is the default | Legitimate override |
91
91
  | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
92
92
  | Dry-run | Spend safety (the live-spend rule sets the floor; dry-run keeps it far away) | `mode: live` |
93
- | Post-run analysis on supported live studies | Findings accompany the recording; a separate $3 admission estimate limit is visible before execution | `review.analysis: false` disables it; an explicit mapping sets another analysis budget. Dry-run and unsupported routes never dispatch; missing default credentials produce a recorded skip |
93
+ | Post-run analysis on supported live studies | Findings accompany the recording; a separate $3 cap on its expected cost is visible before execution | `review.analysis: false` disables it; an explicit mapping sets another analysis budget. Dry-run and unsupported routes never dispatch; missing default credentials produce a recorded skip |
94
94
  | Separate worlds (`per-lane-worlds`) | Isolation, attribution, reproducibility | `route: shared-world`: N participants against one mutable plane for scenarios that are about interaction between participants. Every participant runs at once: `execution.concurrency` must be two or more, and an omitted value fills to the number of participants. The plane is one getHost-exposed subject sandbox with synthetic seeded data, or an operator-owned public deployment (external-public), driven by N actor sandboxes. The bundle declares the weaker `attributionClass: shared-world` and a verify-enforced `attributionLimits` ceiling (`best-effort-causal-attribution` and the rest of the concurrent set). Sequential turns at `execution.concurrency: 1` were removed in 0.106.0. |
95
95
  | External key placement | Smallest blast radius: when the keyed process (e.g. a computer-use provider loop) runs outside the sandbox, its key never enters | In-sandbox placement when the keyed process runs inside (an agent harness under test); declared per actor type, with a spend budget |
96
96
  | Loopback entry URLs | Public-safety: never drive third-party sites unbidden | `policies.allowPublicTargets` for an owner-declared deployment/preview (a Vercel preview of your own app). Multi-participant public/preview fan-out needs explicit `participants[].target` for every participant, so the adapter-owned topology is declared rather than inferred. Provisioned clone subjects always serve in-sandbox on loopback |
@@ -3,10 +3,10 @@
3
3
  Supported live studies automatically request analysis after each recording finishes. Findings remain
4
4
  separate from participant feedback and the recorded study verdict.
5
5
 
6
- The default is `gpt-6-astra` with high reasoning effort, a separate $3 admission
7
- estimate limit and a 600-second timeout. When no output limit is specified,
8
- humanish selects 32,768 tokens if the exact input's admission estimate fits that
9
- budget; otherwise it keeps the established 16,384-token allowance. This preserves
6
+ The default is `gpt-6-astra` with high reasoning effort, a separate $3 cap on its
7
+ expected cost and a 600-second timeout. When no output limit is specified,
8
+ humanish selects 32,768 tokens if the exact input's expected cost still fits that
9
+ cap; otherwise it keeps the established 16,384-token allowance. This preserves
10
10
  previously admitted studies without increasing their spending limit. To customize it:
11
11
 
12
12
  ```yaml
@@ -22,11 +22,15 @@ review:
22
22
 
23
23
  Omitting `review.analysis` uses these defaults. Set `review.analysis: false` to
24
24
  run participants without the additional analysis request. An explicit analysis
25
- mapping using the default OpenAI API provider requires `maxCostUsd`. This limits an admission estimate, not the
26
- provider's final bill, and is separate from participant spending limits. Analysis
27
- can decline a large study before dispatch when its conservative estimate exceeds
28
- that limit. Use `analyze --dry-run --max-cost <usd>` on retained evidence to inspect
29
- the estimate and selected token allowance before deliberately choosing a larger budget. Explicit
25
+ mapping using the default OpenAI API provider requires `maxCostUsd`. Admission
26
+ compares the expected cost plus a 10% margin with it; it does not limit the
27
+ provider's final bill, and it is separate from participant spending limits. The
28
+ estimate gives two numbers: the expected cost, and the worst case if the analyst
29
+ spends its whole output allowance. `humanish study check` gives the expected cost
30
+ range for the study's participant count, from a run that keeps no evidence to one
31
+ at the evidence limits. Use `analyze --dry-run --max-cost <usd>` on retained
32
+ evidence to inspect the expected cost, the worst case and the selected token
33
+ allowance before deliberately choosing a larger budget. Explicit
30
34
  `maxOutputTokens` and `--max-output-tokens` limits are honored exactly. The output allowance
31
35
  includes reasoning as well as the report; exhausting it does not produce a usable
32
36
  report and never starts an automatic retry. Analysis
@@ -74,7 +78,7 @@ Default analysis also skips recordings containing only setup or failure records
74
78
  with no retained participant activity. A desktop startup failure does not start
75
79
  an analysis request. The original failure remains visible.
76
80
 
77
- CLI live starts disclose the selected analyst and its separate admission estimate limit or unknown account dollars before execution.
81
+ CLI live starts disclose the selected analyst, its expected cost range and cap, or unknown account dollars before execution.
78
82
  `humanish study check <study> --json` and the TUI study screen also expose the
79
83
  resolved budget without dispatching analysis. Library callers can inspect
80
84
  `resolveAutomaticAnalysis` or `automaticAnalysisBudget` before running.
@@ -101,11 +105,13 @@ for post-run analysis, and `ok` for the overall request. Failed, cancelled or
101
105
  unknown analysis produces exit code 2 without discarding the recording. A missing
102
106
  `OPENAI_API_KEY` skips default analysis and preserves a successful run exit;
103
107
  `automaticAnalysisTrigger: "default"` distinguishes that case in JSON. So does a
104
- default analysis whose conservative estimate is over the default $3 limit: it is
105
- skipped with `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED`, the run exits 0, and the CLI
106
- prints the `humanish analyze --run <id> --max-cost <usd>` command that runs it.
107
- A missing key or an over-limit estimate for an explicitly configured analysis
108
- remains a failed overall request. Every skip is retained for review and does not
108
+ default analysis whose expected cost, plus the margin, is over the default $3 cap:
109
+ it is skipped with `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED` and the run exits 0. For
110
+ any analysis refused for its cost, the job records the expected cost, the worst
111
+ case and the cap, and the CLI, `humanish review` and the Observer give them with
112
+ the `humanish analyze --run <id> --max-cost <n>` command that runs it, `n` being
113
+ the worst case rounded up. A missing key or an over-cap estimate for an explicitly
114
+ configured analysis remains a failed overall request. Every skip is retained for review and does not
109
115
  retry automatically. Partial
110
116
  findings remain visibly partial; a valid partial result can succeed, while a
111
117
  partial result with an analysis error still fails the command. Recorded task
@@ -119,7 +125,7 @@ outcomes and the deterministic review verdict are never rewritten by analysis.
119
125
  | none (`null`) | `complete` | Analysis completed |
120
126
  | `AUTOMATIC_ANALYSIS_REUSED` | that of the reused result | An earlier analysis of this run was reused |
121
127
  | `AUTOMATIC_ANALYSIS_LIMITATIONS` | `partial` | Analysis completed with recorded limitations |
122
- | `AUTOMATIC_ANALYSIS_ADMISSION_EXCEEDED` | `partial` | Usage passed the admission estimate |
128
+ | `AUTOMATIC_ANALYSIS_ADMISSION_EXCEEDED` | `partial` | Usage passed the worst case or the cap |
123
129
  | `AUTOMATIC_ANALYSIS_FAILED` | `failed` or `partial` | Analysis failed, or an unexpected error stopped it |
124
130
  | `AUTOMATIC_ANALYSIS_CANCELLED` | `cancelled` | Ctrl-C or the TUI's **Cancel analysis** stopped it |
125
131
  | `AUTOMATIC_ANALYSIS_DRY_RUN` | `skipped` | A dry run records no participant evidence |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "humanish",
3
- "version": "0.113.0",
3
+ "version": "0.114.0",
4
4
  "description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
5
5
  "keywords": [
6
6
  "cli",
@@ -91,7 +91,7 @@
91
91
  "comms:agentmail:contract": "node --import tsx scripts/agentmail-contract-check.mjs",
92
92
  "format": "oxfmt",
93
93
  "format:check": "oxfmt --check",
94
- "lint": "node scripts/check-lint-cap.mjs --max-warnings=440",
94
+ "lint": "node scripts/check-lint-cap.mjs --max-warnings=439",
95
95
  "knip": "knip",
96
96
  "prose:check": "node scripts/check-code-prose.mjs",
97
97
  "vocabulary:check": "tsx scripts/check-retired-words.ts",
@@ -105,6 +105,11 @@ not local inference or the participant's existing conversation. See
105
105
  [the analysis contract](../../docs/contracts/study-analysis.md) for the current
106
106
  CLI/model qualification and setup limits.
107
107
 
108
+ When post-run analysis is refused for its cost, the run output and `humanish
109
+ review` give its expected cost, its worst case and the command that runs it,
110
+ `humanish analyze --run <id> --max-cost <n>`. Show the user those costs and run
111
+ the command only once they agree to the spend.
112
+
108
113
  Do not pass numeric `maxCostUsd`/`maxOutputTokens` or their CLI flags to the
109
114
  account branch. It cannot enforce those ceilings and rejects them. Account dollar
110
115
  cost remains unknown even when token usage is reported. There is no fallback to