humanish 0.111.4 → 0.113.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (158) hide show
  1. package/CHANGELOG.md +37 -0
  2. package/README.md +11 -0
  3. package/TELEMETRY.md +21 -0
  4. package/dist/actors/codex/restricted-participant-policy.d.ts +21 -1
  5. package/dist/actors/codex/restricted-participant-policy.js +20 -2
  6. package/dist/actors/codex/restricted-participant.d.ts +1 -1
  7. package/dist/actors/codex/restricted-participant.js +2 -1
  8. package/dist/actors/computer-use/loop/debrief.d.ts +12 -4
  9. package/dist/actors/computer-use/loop/debrief.js +57 -20
  10. package/dist/actors/computer-use/loop/ending.d.ts +5 -2
  11. package/dist/actors/computer-use/loop/impressions.d.ts +14 -0
  12. package/dist/actors/computer-use/loop/impressions.js +40 -0
  13. package/dist/actors/computer-use/loop/session.d.ts +3 -1
  14. package/dist/actors/computer-use/loop/session.js +2 -0
  15. package/dist/actors/computer-use/loop/trace.js +2 -0
  16. package/dist/actors/computer-use/loop/types.d.ts +8 -1
  17. package/dist/actors/computer-use/loop.d.ts +2 -1
  18. package/dist/actors/computer-use/loop.js +15 -3
  19. package/dist/actors/computer-use/openai-conversation.d.ts +96 -0
  20. package/dist/actors/computer-use/openai-conversation.js +238 -0
  21. package/dist/actors/computer-use/openai-provider.js +38 -48
  22. package/dist/actors/contract.d.ts +45 -3
  23. package/dist/actors/contract.js +13 -0
  24. package/dist/analysis/diagnostics.js +7 -12
  25. package/dist/analysis/evidence-sources.d.ts +3 -1
  26. package/dist/analysis/evidence-sources.js +10 -1
  27. package/dist/analysis/evidence.js +2 -2
  28. package/dist/analysis/execute.d.ts +1 -1
  29. package/dist/analysis/execute.js +37 -5
  30. package/dist/analysis/service.js +7 -1
  31. package/dist/analysis/types.d.ts +23 -0
  32. package/dist/analysis/validation.d.ts +80 -35
  33. package/dist/analysis/validation.js +58 -11
  34. package/dist/cli/commands/runs.js +37 -20
  35. package/dist/cli/commands/setup.js +95 -6
  36. package/dist/cli/commands/study-format.js +7 -4
  37. package/dist/cli/commands/study.js +1 -1
  38. package/dist/cli/commands/tui.js +1 -1
  39. package/dist/cli/doctor.d.ts +2 -0
  40. package/dist/cli/doctor.js +18 -0
  41. package/dist/cli/findings.d.ts +34 -3
  42. package/dist/cli/findings.js +136 -18
  43. package/dist/cli/io.d.ts +4 -1
  44. package/dist/cli/io.js +3 -2
  45. package/dist/cli/program.js +35 -1
  46. package/dist/cli/secret-prompt.d.ts +5 -1
  47. package/dist/cli/secret-prompt.js +6 -2
  48. package/dist/cli/update-check-worker.d.ts +1 -0
  49. package/dist/cli/update-check-worker.js +5 -0
  50. package/dist/cli/update-check.d.ts +46 -0
  51. package/dist/cli/update-check.js +203 -0
  52. package/dist/feedback/draft.js +32 -4
  53. package/dist/keys/key-resolution.js +2 -0
  54. package/dist/keys/key-status.d.ts +37 -0
  55. package/dist/keys/key-status.js +68 -0
  56. package/dist/observer/run-routes.js +8 -4
  57. package/dist/observer-app.html +4 -4
  58. package/dist/routes/computer-use/bundle.d.ts +1 -1
  59. package/dist/routes/computer-use/bundle.js +6 -16
  60. package/dist/routes/computer-use/fanout-bundle.js +40 -37
  61. package/dist/routes/computer-use/fanout-records.d.ts +2 -0
  62. package/dist/routes/computer-use/fanout-records.js +18 -26
  63. package/dist/routes/computer-use/local-vm.js +9 -2
  64. package/dist/routes/computer-use/participant-desktop.d.ts +6 -0
  65. package/dist/routes/computer-use/participant-execution.js +13 -12
  66. package/dist/routes/computer-use/participant-facts.d.ts +2 -2
  67. package/dist/routes/computer-use/participant-facts.js +10 -1
  68. package/dist/routes/computer-use/rerun-selection.js +23 -21
  69. package/dist/routes/computer-use/result.js +10 -17
  70. package/dist/routes/computer-use/self-report.js +7 -1
  71. package/dist/routes/computer-use/setup.js +14 -0
  72. package/dist/routes/computer-use/single-bundle.d.ts +2 -0
  73. package/dist/routes/computer-use/single-bundle.js +81 -37
  74. package/dist/routes/computer-use/types.d.ts +4 -1
  75. package/dist/routes/scripted/bundle.d.ts +1 -1
  76. package/dist/routes/scripted/bundle.js +9 -19
  77. package/dist/routes/scripted/surface-records.d.ts +0 -1
  78. package/dist/routes/scripted/surface-records.js +27 -13
  79. package/dist/routes/shared-world/bundle-records.d.ts +2 -2
  80. package/dist/routes/shared-world/bundle-records.js +24 -30
  81. package/dist/routes/shared-world/bundle.d.ts +1 -1
  82. package/dist/routes/shared-world/bundle.js +71 -79
  83. package/dist/routes/shared-world/participant-specs.d.ts +1 -2
  84. package/dist/routes/shared-world/participant-specs.js +0 -8
  85. package/dist/routes/shared-world/result.js +8 -12
  86. package/dist/routes/terminal/bundle.d.ts +1 -1
  87. package/dist/routes/terminal/bundle.js +10 -19
  88. package/dist/routes/terminal/dry-run.js +1 -0
  89. package/dist/routes/terminal/live-sandbox.js +9 -2
  90. package/dist/routes/terminal/plan.js +1 -0
  91. package/dist/routes/terminal/runtime.d.ts +1 -0
  92. package/dist/routes/terminal/runtime.js +1 -0
  93. package/dist/routes/terminal/session.js +1 -0
  94. package/dist/run/costs.js +9 -20
  95. package/dist/run/detail.d.ts +1 -1
  96. package/dist/run/detail.js +13 -4
  97. package/dist/run/display.d.ts +2 -2
  98. package/dist/run/display.js +15 -18
  99. package/dist/run/dry-run.js +10 -25
  100. package/dist/run/judge.d.ts +24 -0
  101. package/dist/run/judge.js +54 -0
  102. package/dist/run/liveness.d.ts +18 -0
  103. package/dist/run/liveness.js +24 -0
  104. package/dist/run/outcomes.d.ts +8 -1
  105. package/dist/run/outcomes.js +58 -12
  106. package/dist/run/participant-caption.d.ts +31 -0
  107. package/dist/run/participant-caption.js +60 -0
  108. package/dist/run/participant-records.d.ts +4 -2
  109. package/dist/run/participant-records.js +6 -0
  110. package/dist/run/review-markdown.d.ts +11 -0
  111. package/dist/run/review-markdown.js +22 -0
  112. package/dist/run/run-cost.d.ts +7 -0
  113. package/dist/run/run-cost.js +25 -5
  114. package/dist/run/run-index.d.ts +4 -50
  115. package/dist/run/run-index.js +57 -87
  116. package/dist/run/run.d.ts +2 -2
  117. package/dist/run/run.js +5 -3
  118. package/dist/run/stats.js +2 -12
  119. package/dist/run/transient-comms-secrets.d.ts +8 -3
  120. package/dist/run/transient-comms-secrets.js +21 -1
  121. package/dist/run-study.js +17 -1
  122. package/dist/study/discover.js +6 -1
  123. package/dist/study/doctor.d.ts +2 -0
  124. package/dist/study/doctor.js +45 -7
  125. package/dist/study/init-templates.js +10 -5
  126. package/dist/study/keys.d.ts +1 -0
  127. package/dist/study/keys.js +1 -1
  128. package/dist/study/parse/execution.js +7 -1
  129. package/dist/study/plan-types.d.ts +1 -0
  130. package/dist/study/plan.js +3 -2
  131. package/dist/study/preflight-result.d.ts +1 -0
  132. package/dist/study/preflight-result.js +1 -1
  133. package/dist/study/preflight.d.ts +2 -0
  134. package/dist/study/preflight.js +54 -14
  135. package/dist/study/types.d.ts +4 -1
  136. package/dist/study/warnings.d.ts +7 -1
  137. package/dist/study/warnings.js +82 -4
  138. package/dist/substrates/local/capacity.d.ts +75 -0
  139. package/dist/substrates/local/capacity.js +143 -0
  140. package/dist/substrates/local/firecracker-desktop.d.ts +8 -0
  141. package/dist/substrates/local/firecracker-desktop.js +34 -3
  142. package/dist/substrates/local/runtime-host.d.ts +16 -3
  143. package/dist/substrates/local/runtime-host.js +49 -4
  144. package/dist/substrates/local/runtime.d.ts +10 -0
  145. package/dist/substrates/local/runtime.js +44 -5
  146. package/dist/tui-app.js +75 -75
  147. package/dist/verify/liveness.d.ts +2 -2
  148. package/dist/verify/liveness.js +7 -4
  149. package/docs/architecture/actor-contract.md +20 -6
  150. package/docs/architecture/terminal-product-route.md +14 -0
  151. package/docs/contracts/feedback.md +8 -1
  152. package/docs/contracts/run-bundle.md +10 -0
  153. package/docs/contracts/schemas.md +65 -34
  154. package/docs/contracts/study-analysis.md +86 -22
  155. package/package.json +2 -1
  156. package/skills/humanish/SKILL.md +21 -10
  157. package/dist/actors/computer-use/openai-requests.d.ts +0 -62
  158. package/dist/actors/computer-use/openai-requests.js +0 -144
@@ -14,8 +14,8 @@ export interface UnfinishedRun {
14
14
  }
15
15
  /**
16
16
  * One warning, and the unfinished-run facts, when the run is not finished, by the run index's
17
- * rule: its status record when it has a usable one, else a participant still `running` in its
18
- * bundle's `simulations[]`. Nothing for a finished run.
17
+ * rule: its bundle outcome first, then usable status, then the bundle's simulations. Nothing
18
+ * for a finished run.
19
19
  */
20
20
  export declare function runNotFinished(runPaths: PreparedRunArtifactPaths, bundle: RunBundle, nowMs?: number): Promise<{
21
21
  warnings: string[];
@@ -7,7 +7,7 @@ import path from "node:path";
7
7
  import { readRunJsonIfExists, readRunTextIfExists } from "../run/locate.js";
8
8
  import { isRecord } from "../run/type-guards.js";
9
9
  import { RECLAIM_RECEIPT_ARTIFACT } from "../run/reclaim.js";
10
- import { runLiveness } from "../run/run-index.js";
10
+ import { runLiveness } from "../run/liveness.js";
11
11
  import { parseSandboxReceipts, SANDBOX_RECEIPTS_ARTIFACT } from "../run/sandbox-receipts.js";
12
12
  import { RUN_STATUS_FILE } from "../run/status.js";
13
13
  import { plural } from "../run/text.js";
@@ -17,8 +17,8 @@ const RUN_NOT_FINISHED = "RUN_NOT_FINISHED";
17
17
  const GONE_STATES = new Set(["killed", "already-gone"]);
18
18
  /**
19
19
  * One warning, and the unfinished-run facts, when the run is not finished, by the run index's
20
- * rule: its status record when it has a usable one, else a participant still `running` in its
21
- * bundle's `simulations[]`. Nothing for a finished run.
20
+ * rule: its bundle outcome first, then usable status, then the bundle's simulations. Nothing
21
+ * for a finished run.
22
22
  */
23
23
  export async function runNotFinished(runPaths, bundle, nowMs = Date.now()) {
24
24
  const runId = path.basename(runPaths.absoluteRunRoot);
@@ -27,7 +27,10 @@ export async function runNotFinished(runPaths, bundle, nowMs = Date.now()) {
27
27
  if (liveness === "finished")
28
28
  return { warnings: [] };
29
29
  const seen = [];
30
- if (record === undefined) {
30
+ if (bundle.outcome?.state === "interrupted") {
31
+ seen.push(`run.json records interruption by ${bundle.outcome.signal} at ${bundle.outcome.at}`);
32
+ }
33
+ else if (record === undefined) {
31
34
  const running = bundle.simulations.filter((simulation) => simulation.status === "running");
32
35
  seen.push(`the run has no usable ${RUN_STATUS_FILE} and ${running.length} of ${bundle.simulations.length} simulations are still running`);
33
36
  }
@@ -350,22 +350,36 @@ this paragraph is the record of that decision.
350
350
 
351
351
  `ComputerUseProvider.debrief` is an optional read-only request after a structured
352
352
  `stopWhen` or dwell stop. It returns a `ComputerUseTurn` with `closingReport` containing
353
- `summary` and `frictionReports`; an empty friction list is valid. The loop rejects
353
+ `summary`, `frictionReports` and `impressions`; an empty friction or impressions list is valid,
354
+ and a report without `impressions` (an older provider) is still accepted. The loop rejects
354
355
  actions, pending safety checks, and invalid report shapes. It redacts accepted
355
356
  reports, records them in `ActorTrace.debrief`, and projects one readable message
356
- without invoking action or communication callbacks.
357
+ without invoking action or communication callbacks. Each impression becomes its own message
358
+ and goes to `ActorTrace.impressions`; a session without them records why in the same field.
359
+ A turn that ends the session itself may carry `closingReport` too, and the loop keeps its
360
+ impressions the same way without a second request.
361
+
362
+ `ComputerUseProvider.requestImpressions` is the optional request after the participant ended
363
+ the session itself without impressions. It asks for impressions only, returns them as
364
+ `impressions` on the turn, and follows the debrief's rules and guards. The loop records it as
365
+ `ActorTrace.debrief` with `trigger: "participant_end"` and no report, so the participant's
366
+ reason, outcome and friction come from its final message as before. A failed, timed-out or
367
+ cut-off request leaves `ActorTrace.impressions` as `not_collected` with the reason.
357
368
 
358
369
  This request uses the final observation and retained provider history. It does
359
370
  not expose the hidden stop criterion or change the original task outcome. It is
360
- skipped before any participant turn, after natural completion, without provider
361
- support, or without remaining time and known budget. An attempted request's
371
+ skipped before any participant turn, after natural completion (apart from the impressions-only
372
+ request), without provider support, or without remaining time and known budget. An attempted request's
362
373
  unreported usage remains an unknown cost line. `counts.debriefCalls` is separate
363
374
  from interaction turns, and reported usage contributes to aggregate cost.
364
375
 
365
376
  The OpenAI implementation makes one request with tools disabled and structured
366
- output; no HTTP or policy retries. Stateless/ZDR mode does not offer retrospective
377
+ output; no HTTP or policy retries. Its output limit is 3072 tokens, or the declared
378
+ `maxOutputTokens` when lower. Stateless/ZDR mode does not offer retrospective
367
379
  reporting: its client-carried conversation summarizes its oldest turns past a token
368
- budget, so it may not hold the whole session a retrospective report needs. The
380
+ budget, so it may not hold the whole session a retrospective report needs. It does offer
381
+ the impressions-only request, which reads the carried conversation: its newest
382
+ exchanges whole and earlier turns as a text note. The
369
383
  [paired live receipt](https://github.com/danielgwilson/humanish/blob/main/docs/evidence/computer-use/structured-closing-report-2026-09-05.md)
370
384
  records both report recovery and control failures in the separate legacy parser.
371
385
 
@@ -84,6 +84,20 @@ at the base tier, with cached input at the cached rate, and says so with
84
84
  checked against `caps.maxUsd` keeps the provider line unmeasured, and the studied
85
85
  product's no-spend boundary still needs separate interpretation.
86
86
 
87
+ ## Participant telemetry
88
+
89
+ Terminal participant commands and product setup receive `DO_NOT_TRACK=1` by default.
90
+ Their child processes inherit it, so studied CLIs that honor the convention can exclude
91
+ synthetic usage. To study the product's telemetry behavior, set
92
+ `execution.terminal.doNotTrack: false`; humanish then leaves the variable unset.
93
+ Only booleans are accepted, and `true` or an omitted field enables the default.
94
+
95
+ The resolved `doNotTrack` boolean is recorded with `humanish.actor-runtime.v1` in
96
+ `actor.json`, `streams[].actor.runtime` in `run.json`, and `runtime` in
97
+ `terminal-ledgers.json`. A dry run records it in the JSON message of its
98
+ `terminal-lab.runtime.declared` event. Older bundles omit it. This records the
99
+ command environment policy; it does not prove that the studied CLI honors the variable.
100
+
87
101
  ## Runtime auth: raw-key placement and remaining provider access
88
102
 
89
103
  `execution.runtimeAuth: openai-egress` is the default: a study that declares no
@@ -66,7 +66,9 @@ Listing does not create a draft or check public-sharing eligibility.
66
66
 
67
67
  For completed computer-use and shared-world sessions, participant-reported
68
68
  friction comes from the redacted closing report, a completed debrief's
69
- `frictionReports`, and observed-report clauses in earlier messages.
69
+ `frictionReports`, and observed-report clauses in earlier messages. The
70
+ participant's closing impressions (`ActorTrace.impressions`) are opinions about
71
+ the product and never count as friction.
70
72
  Matching `stopWhen` or ending a dwell window does not discard an earlier
71
73
  report or change the successful completion verdict. Exact repeated reports
72
74
  appear once in the participant's candidate. Harness notices, reasoning, and observed
@@ -84,6 +86,11 @@ was not repeated in the message trace.
84
86
  Builds structured feedback from the first usable run candidate, or the candidate
85
87
  selected with `--candidate`. With no candidate, a live run produces a summary of
86
88
  its recorded review; that fallback does not add a finding to the candidate list.
89
+ With `--analysis` and `--finding`, the draft is built from that analysis finding
90
+ instead. When the finding's observations cite a participant's closing impression,
91
+ `actual` quotes it after the observations as
92
+ `Participant <id> said at the end (<kind>): <text>`. An impression is quoted only
93
+ when the cited evidence entry carries its text whole; uncited impressions stay out.
87
94
  A dry run tests no product behavior, so `draft`, `verify`, `issue` and
88
95
  `issue-url` refuse it with `HUMANISH_FEEDBACK_REQUIRES_LIVE_RUN`. All drafts
89
96
  require a verified `share_ready` live run and are written under the run bundle,
@@ -6,6 +6,13 @@
6
6
  source of truth. This page explains the stable public fields and the extension rules; it does not
7
7
  version the runtime separately.
8
8
 
9
+ Terminal runs record the resolved telemetry setting as `doNotTrack` in
10
+ `streams[].actor.runtime` (`humanish.actor-runtime.v1`), also retained in `actor.json`
11
+ and `terminal-ledgers.json`. `true` means participant and product setup commands receive
12
+ `DO_NOT_TRACK=1`; `false` means humanish leaves it unset. Dry runs record the same boolean
13
+ in the JSON message of the `terminal-lab.runtime.declared` event. Older bundles omit it.
14
+ The setting does not attest that the studied CLI honors the variable.
15
+
9
16
  ## Purpose
10
17
 
11
18
  A run bundle is the durable evidence packet for one harness run. It should be
@@ -154,6 +161,9 @@ writing it, and on runs recorded before the field existed. For those, readers
154
161
  take liveness from `status.json` and, for an older finished run, its `ok` and
155
162
  `execution` from `status.json`'s `outcome`. That record is the only copy of an
156
163
  older run's `ok`; for a run with `outcome` in `run.json`, `status.json` copies it.
164
+ The bundle's outcome wins even if the process stopped before updating `status.json`.
165
+ Without an outcome or a usable status record naming the run, a bundle with a simulation still
166
+ `running` reads as interrupted. A fresh status record keeps an in-progress bundle running.
157
167
 
158
168
  Every surface that shows whether a run passed calls `runDisplay`
159
169
  (`src/run/display.ts`) with the run's liveness, verdict and `ok`: the Observer
@@ -24,38 +24,38 @@ workflow without leaking private upstream truth into core.
24
24
 
25
25
  ## Contract Index
26
26
 
27
- | Contract | Schema | Public-safe fixture |
28
- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- |
29
- | Run bundle | `humanish.run-bundle.v1` | `synthetic-run-bundle` |
30
- | Adapter | `humanish.adapter.v1` | `synthetic-cli-adapter` |
31
- | Study file | `humanish.study.v3` | `first-run` |
32
- | Persona | `humanish.persona.v1` | `synthetic-maintainer` |
33
- | Scenario | `humanish.scenario.v1` | `first-run-smoke` |
34
- | Actor trace | `humanish.actor-trace.v1` | `synthetic-actor-trace` |
35
- | Substrate | reserved (never shipped) | none |
36
- | Evidence stream | reserved (streams live inside the run bundle) | see [`run-bundle.md`](run-bundle.md) |
37
- | Review | `humanish.review.v1` | `contract-proof-review` |
38
- | Study analysis | `humanish.study-analysis.v1` | see [study analysis](study-analysis.md) and synthetic analysis fixtures in `tests/` |
39
- | Study analysis correction | `humanish.study-analysis-correction.v1` | see [study analysis](study-analysis.md#human-review-and-sharing) |
40
- | Analysis findings view | `humanish.analysis-findings.v1` (`review --json` field `analysis`; read-only projection of the selected analysis) | see [study analysis](study-analysis.md#reading-findings) |
41
- | Analysis execution receipt | `humanish.analysis-execution.v1` | see [study analysis](study-analysis.md#durable-records) |
42
- | Analysis execution start | `humanish.analysis-execution-start.v1` | see [study analysis](study-analysis.md#durable-records) |
43
- | Verification | `humanish.verify-result.v1` | `five-check-verify` |
44
- | Policy | `humanish.policy.v1` (fixture-only; not engine-validated) | `public-safety-policy` |
45
- | Feedback | `humanish.feedback.v1` | `public-safe-feedback` |
46
- | Terminal cost ledger | `humanish.terminal-cost-ledger.v1` | see Terminal Cost Ledger below |
47
- | Terminal no-spend proof | `humanish.terminal-no-spend-proof.v1` | see Terminal Cost Ledger below |
48
- | Pricing (operator-editable rates) | `humanish.pricing.v1` (`src/run/pricing.ts`; dated per-model + E2B desktop rates) | see Run Cost Summary And Estimated Actor Cost below |
49
- | Run cost summary | `humanish.run-cost-summary.v1` (additive `RunBundle.cost`; estimate, never a charge) | see Run Cost Summary And Estimated Actor Cost below |
50
- | Estimated actor cost | `humanish.actor-estimated-cost.v1` (additive `ActorTrace.estimatedCost`) | see Run Cost Summary And Estimated Actor Cost below |
51
- | Model settings | `humanish.model-settings.v1` (additive `ActorTrace.modelSettings`; the reasoning effort the request carried) | see Actor Trace below |
52
- | Affordance use | `humanish.affordance-use.v1` (additive `ActorTrace.affordanceUse`; per-class counts of the routes an actor took) | see Affordance Use below |
53
- | Adapter score | `humanish.adapter-score.v1` (`RunBundle.adapterScore`; namespaced; route-specific acceptance semantics) | see Product-Adapter Extension Seam below |
54
- | Adapter artifact | `humanish.adapter-artifact.v1` (`RunBundle.adapterArtifacts[]`; namespaced; local relative proof references) | see Product-Adapter Extension Seam below |
55
- | Shared-world evidence | `humanish.shared-world.v1` (additive `RunBundle.sharedWorld` + `RunBundle.attributionClass`; `topologyMode: sequential \| concurrent`) | see Shared-World Evidence below |
56
- | Comms thread | `humanish.comms-thread.v1` (off-app email/SMS the app sent, captured; a `kind: log` run-dir artifact of digests only: from/to/subject/link digests + one OTP count, never raw) | see `comms` under Study File |
57
- | Serve result | `humanish.serve-result.v1` (`src/observer/serve.ts` is authoritative) | none (command result envelope; see Serve Result below) |
58
- | Serve control plane | reserved (`/_humanish/api/*` answers `501` `HUMANISH_SERVE_CONTROL_PLANE_DISABLED` in v1) | none |
27
+ | Contract | Schema | Public-safe fixture |
28
+ | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
29
+ | Run bundle | `humanish.run-bundle.v1` | `synthetic-run-bundle` |
30
+ | Adapter | `humanish.adapter.v1` | `synthetic-cli-adapter` |
31
+ | Study file | `humanish.study.v3` | `first-run` |
32
+ | Persona | `humanish.persona.v1` | `synthetic-maintainer` |
33
+ | Scenario | `humanish.scenario.v1` | `first-run-smoke` |
34
+ | Actor trace | `humanish.actor-trace.v1` | `synthetic-actor-trace` |
35
+ | Substrate | reserved (never shipped) | none |
36
+ | Evidence stream | reserved (streams live inside the run bundle) | see [`run-bundle.md`](run-bundle.md) |
37
+ | Review | `humanish.review.v1` | `contract-proof-review` |
38
+ | Study analysis | `humanish.study-analysis.v1` (prompt `study-evidence-7` adds `findings[].headline`, `findings[].experience` and `designFindings[]`; older artifacts omit them; `study-evidence-8` adds participant impressions to the instructions, with no field change) | see [study analysis](study-analysis.md) and synthetic analysis fixtures in `tests/` |
39
+ | Study analysis correction | `humanish.study-analysis-correction.v1` | see [study analysis](study-analysis.md#human-review-and-sharing) |
40
+ | Analysis findings view | `humanish.analysis-findings.v1` (`review --json` field `analysis`; read-only projection of the selected analysis) | see [study analysis](study-analysis.md#reading-findings) |
41
+ | Analysis execution receipt | `humanish.analysis-execution.v1` | see [study analysis](study-analysis.md#durable-records) |
42
+ | Analysis execution start | `humanish.analysis-execution-start.v1` | see [study analysis](study-analysis.md#durable-records) |
43
+ | Verification | `humanish.verify-result.v1` | `five-check-verify` |
44
+ | Policy | `humanish.policy.v1` (fixture-only; not engine-validated) | `public-safety-policy` |
45
+ | Feedback | `humanish.feedback.v1` | `public-safe-feedback` |
46
+ | Terminal cost ledger | `humanish.terminal-cost-ledger.v1` | see Terminal Cost Ledger below |
47
+ | Terminal no-spend proof | `humanish.terminal-no-spend-proof.v1` | see Terminal Cost Ledger below |
48
+ | Pricing (operator-editable rates) | `humanish.pricing.v1` (`src/run/pricing.ts`; dated per-model + E2B desktop rates) | see Run Cost Summary And Estimated Actor Cost below |
49
+ | Run cost summary | `humanish.run-cost-summary.v1` (additive `RunBundle.cost`; estimate, never a charge) | see Run Cost Summary And Estimated Actor Cost below |
50
+ | Estimated actor cost | `humanish.actor-estimated-cost.v1` (additive `ActorTrace.estimatedCost`) | see Run Cost Summary And Estimated Actor Cost below |
51
+ | Model settings | `humanish.model-settings.v1` (additive `ActorTrace.modelSettings`; the reasoning effort the request carried) | see Actor Trace below |
52
+ | Affordance use | `humanish.affordance-use.v1` (additive `ActorTrace.affordanceUse`; per-class counts of the routes an actor took) | see Affordance Use below |
53
+ | Adapter score | `humanish.adapter-score.v1` (`RunBundle.adapterScore`; namespaced; route-specific acceptance semantics) | see Product-Adapter Extension Seam below |
54
+ | Adapter artifact | `humanish.adapter-artifact.v1` (`RunBundle.adapterArtifacts[]`; namespaced; local relative proof references) | see Product-Adapter Extension Seam below |
55
+ | Shared-world evidence | `humanish.shared-world.v1` (additive `RunBundle.sharedWorld` + `RunBundle.attributionClass`; `topologyMode: sequential \| concurrent`) | see Shared-World Evidence below |
56
+ | Comms thread | `humanish.comms-thread.v1` (off-app email/SMS the app sent, captured; a `kind: log` run-dir artifact of digests only: from/to/subject/link digests + one OTP count, never raw) | see `comms` under Study File |
57
+ | Serve result | `humanish.serve-result.v1` (`src/observer/serve.ts` is authoritative) | none (command result envelope; see Serve Result below) |
58
+ | Serve control plane | reserved (`/_humanish/api/*` answers `501` `HUMANISH_SERVE_CONTROL_PLANE_DISABLED` in v1) | none |
59
59
 
60
60
  ## Study File
61
61
 
@@ -296,6 +296,14 @@ instruction?, target?, entry?, host?, reasoningEffort?, stopWhen?, dwell? }` bec
296
296
  metadata; with `openai-egress` it stays in the host-side E2B header transform
297
297
  set at sandbox creation, and the agent gets a placeholder. A dry-run neither
298
298
  reads nor injects it;
299
+ - `execution.terminal.doNotTrack`: boolean, default `true`. Terminal participant and product
300
+ setup commands receive `DO_NOT_TRACK=1`. Set `false` to leave that variable unset when
301
+ studying the product's telemetry behavior. Non-boolean values are refused at parse.
302
+ The resolved boolean is `doNotTrack` in `humanish.actor-runtime.v1`: `actor.json`,
303
+ `streams[].actor.runtime` in `run.json`, and `runtime` in `terminal-ledgers.json` for live
304
+ runs; the JSON message of the `terminal-lab.runtime.declared` event for dry runs.
305
+ Older bundles omit this field. It records the command environment policy, not whether
306
+ the studied CLI honored it;
299
307
  - `mode`: `dry-run` (a synthetic bundle, no spend; the default) or `live`. On the
300
308
  scripted route `live` gates real browser actuation against the declared app.
301
309
  Provider spend stays $0 there by mechanism (no model runs);
@@ -962,8 +970,9 @@ Core-owned fields:
962
970
  - optional `modelSettings.maxOutputTokens`: the declared positive integer
963
971
  `actor.maxOutputTokens`, passed to every first-party OpenAI CUA response
964
972
  request, including reasoning output. Absent when undeclared. The first request
965
- of a session and closing reports use `min(maxOutputTokens, 1024)`, or 1024 when
966
- undeclared; the first request also carries the opening screenshot as an
973
+ of a session uses `min(maxOutputTokens, 1024)`, or 1024 when undeclared, and
974
+ closing requests use `min(maxOutputTokens, 3072)`, or 3072 when undeclared; the
975
+ first request also carries the opening screenshot as an
967
976
  `input_image` after the instructions. Unsupported routes,
968
977
  custom provider/session hooks, and per-participant overrides fail before allocation.
969
978
  This is an output-token limit, not an input-token, request-count or billing cap.
@@ -1006,6 +1015,28 @@ Core-owned fields:
1006
1015
  providers and on earlier bundles
1007
1016
  - optional `affordanceUse` (`humanish.affordance-use.v1`): which kind of route this
1008
1017
  actor took (see Affordance Use below)
1018
+ - optional `impressions`: what a computer-use participant said about the product
1019
+ at the end of its session. `status: collected` carries `items[]`, each a
1020
+ `kind` (`unclear`, `unfinished`, `untrustworthy`, `liked`, `missing`,
1021
+ `unlike_my_work`), a redacted first-person `text` of at most 500 characters
1022
+ and the `messageId` of the trace `message` that quotes it as
1023
+ `Impression (<kind>): <text>`. They are not counted in `counts.messages`. Up
1024
+ to six come from the closing account: a completed debrief's report, the
1025
+ participant's own final account when its provider writes a structured one
1026
+ (the Codex participant), or, when the participant ended the session itself
1027
+ without them, one impressions-only request (OpenAI computer-use, threaded or
1028
+ `explicit_context`). That request is recorded as `debrief` with `trigger:
1029
+ participant_end` and never has a `report` or `messageId`; it leaves the
1030
+ participant's `reason`, outcome and friction as they were. An empty `items`
1031
+ list means the participant named none. `status: not_collected` carries a
1032
+ `reason`: the closing report or impressions request was skipped or failed
1033
+ (including a reply cut off by the output limit), the participant ended the
1034
+ session without a structured closing account and its provider cannot ask for
1035
+ impressions (the Claude local agent), the closing account had no
1036
+ impressions, or the session stopped before a closing account. Every
1037
+ computer-use loop trace records one; absent on other routes and on earlier
1038
+ bundles. `debrief.report` keeps only `summary` and `frictionReports`.
1039
+ Impression messages do not count as participant-reported friction
1009
1040
  - optional `estimatedCost` (`humanish.actor-estimated-cost.v1`): a token-derived
1010
1041
  cost estimate for this participant (see Run Cost Summary And Estimated Actor Cost).
1011
1042
  It is deliberately a different field from `tokenUsage.costUsd`: a bare
@@ -161,6 +161,42 @@ thought. Older reports may omit this field and remain readable. The disclosure
161
161
  is model-generated assessment, not an independent completeness audit or a human
162
162
  reviewer annotation.
163
163
 
164
+ ## Headlines and design findings
165
+
166
+ The `study-evidence-7` prompt writes each finding twice. `headline` is one plain sentence about what
167
+ happened, for a designer, product manager or developer who did not watch the session.
168
+ `experience` is one to three plain sentences: what the person tried, what got in their way, and
169
+ how it seemed to feel. Neither names participant or evidence IDs. A concern known only from what
170
+ participants said still reads as something they said. `title`, `summary` and `observations` keep
171
+ their evidence-level wording, and validation applies to them as before.
172
+
173
+ The same prompt adds `designFindings`: problems a product designer would notice in the supplied
174
+ captures, whether or not a participant mentioned them. Each has an `id` (`D1`, `D2`, ...), a
175
+ `headline`, the `screen` in plain words, what a designer would `notice`, `whyItMatters` to a
176
+ person using the product, one `suggestion`, a `severity` (`major` when it misleads or blocks,
177
+ `moderate` when it slows or confuses, `minor` when it is polish), a `confidence`,
178
+ `seenByStreamIds` and `evidenceIds`. Validation rejects a design finding that cites no retained
179
+ capture, cites evidence outside the packet, repeats an ID, or lists a participant in
180
+ `seenByStreamIds` without a cited capture of that participant. An empty list means the review found no design
181
+ problem in the captures. The model judges legibility and size from images alone; it does not
182
+ measure text or control sizes.
183
+
184
+ New provider responses must include `headline` and `experience` on every finding and a
185
+ `designFindings` list. A stored artifact with prompt `study-evidence-7` or later must carry them
186
+ too. Artifacts from earlier prompts load without them and render by title and summary. The
187
+ Observer applies the same prompt-version rules and the same field limits when it reads a saved
188
+ analysis, and treats an analysis that breaks them as unavailable.
189
+
190
+ The narrative scrub removes a run's known transient values, such as a received one-time code or
191
+ link, from every generated text field. It finds each value as written and in its encoded forms:
192
+ percent-encoded, JSON-escaped, base64, base64url and hex, and where escapes split it. A field
193
+ with a value is stored decoded with the value replaced; a field without one keeps its spelling. A
194
+ known value in any of these forms in an ID, enum or reference refuses the response.
195
+
196
+ When a reviewer amends a finding, its headline and experience describe the replaced claim.
197
+ Text output, the Observer and feedback drafts show the reviewer's claim in their place and say the
198
+ finding was corrected; the stored analysis keeps the original.
199
+
164
200
  Every observation cites packet-local evidence IDs. The model cannot choose a
165
201
  filesystem path or fetch another resource. Validation checks participant
166
202
  membership, unique counts, quote fidelity, evidence type and reference
@@ -203,7 +239,7 @@ feedback quotes cite evidence IDs that the run's own `analysis.json` resolves.
203
239
  Participant reviews cite distinct packet-local evidence IDs belonging only to
204
240
  that participant's stream; source event IDs are not citation IDs. Shared
205
241
  interactions can cite multiple included participants in findings and concern
206
- reviews under the existing exposure rules. The `study-evidence-6` prompt makes
242
+ reviews under the existing exposure rules. The `study-evidence-6` prompt made
207
243
  this distinction explicit without changing the validator or historical reports.
208
244
 
209
245
  Elapsed replay time starts at the first retained capture. It is not a video
@@ -219,11 +255,32 @@ guarantee that the selected capture is the most relevant. Context-only and
219
255
  nonvisual evidence retain their basis and original event. All cited moments stay
220
256
  available, with exact recording links.
221
257
 
258
+ A finding with a headline leads its Observer row with the headline and its open panel with the
259
+ experience. The title, summary, assessment and observations sit in a closed **Evidence**
260
+ disclosure beneath. A **Design findings** section follows the ranked findings, grouped by
261
+ severity, with each cited capture as a thumbnail that opens that frame. Findings without a
262
+ headline render as described below.
263
+
222
264
  Confidence, recovery and the first full evidence limitation remain visible when
223
265
  a finding opens. Exposure and remaining unique limits are one disclosure away;
224
266
  observation details retain each original claim and its specific limitation.
225
267
  This presentation does not rewrite the saved analysis or reviewer corrections.
226
268
 
269
+ ## Participant impressions
270
+
271
+ A computer-use participant's closing impressions (`ActorTrace.impressions`) reach the packet as
272
+ ordinary `message` entries of that participant's stream, one per impression, with text
273
+ `Impression (<kind>): <text>`. They are quote-eligible and attributed by `streamId`, so feedback,
274
+ participant_statement observations and design findings can cite them under the existing rules.
275
+ Selection admits them right after the participant's last entry, ahead of the rest of its
276
+ session. The evidence mapping and `captureVersion` are unchanged.
277
+
278
+ The `study-evidence-8` prompt adds one paragraph: impressions are the participant's own opinions,
279
+ used as participant statements, and an `unlike my work` impression together with a capture of
280
+ that screen is a strong design finding. Validation is unchanged. A design finding still needs at
281
+ least one cited capture, so an impression plus the capture of its screen makes a design finding,
282
+ and an impression alone supports only a finding about what the participant said.
283
+
227
284
  ## Durable records
228
285
 
229
286
  The frozen `humanish.observer-data.v1` schema is unchanged. A companion
@@ -291,31 +348,35 @@ is disabled. Existing adapter captures and logs retain their contained routes.
291
348
  ## Reading findings
292
349
 
293
350
  `humanish review` prints a run's analysis findings after its participant review, and
294
- `humanish analyze show` prints them alone. Each finding shows its title, impact, confidence and
295
- recovery, the participants it affected out of those exposed, the frames its evidence cites with
296
- their time since the first retained capture, the capture files, its next step and the latest
297
- human review note. A live run's human output ends with up to three findings, one line each, and
298
- the `review` command that prints all of them. `analyze show --json` prints the validated
351
+ `humanish analyze show` prints them alone. Each finding leads with its headline and experience,
352
+ then shows its title as `evidence:`, its impact, confidence and recovery, the participants it
353
+ affected out of those exposed, the frames its evidence cites with their time since the first
354
+ retained capture, the capture files, its next step and the latest human review note. A finding
355
+ without a headline starts at its title. Design findings follow, most severe first, each with its
356
+ screen, notice, why it matters, suggestion, who saw it and its capture files. A live run's human
357
+ output ends with up to three findings and up to three design findings, one line each, and the
358
+ `review` command that prints all of them. `analyze show --json` prints the validated
299
359
  analysis record and its corrections, as before. Reading findings never starts an analysis.
300
360
 
301
361
  `review --json` carries the same view as its `analysis` field, schema
302
362
  `humanish.analysis-findings.v1`, built in `src/cli/findings.ts`:
303
363
 
304
- | Field | Meaning |
305
- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
306
- | `schema` | `humanish.analysis-findings.v1`. |
307
- | `runId` | The run the view reads. |
308
- | `state` | `ready`, `stale`, `running`, `none`, `skipped`, `failed`, `dry_run` or `unavailable`. Only `ready` and `stale` carry findings. |
309
- | `reason` | The stable code behind a state other than `ready`, or null: an automatic analysis reason such as `AUTOMATIC_ANALYSIS_KEY_MISSING`, a failed attempt's error code, or a load warning such as `ANALYSIS_SOURCE_CHANGED`. |
310
- | `message` | What happened, in words. |
311
- | `next` | The command that gets findings for this run, or for a dry run the command that starts a live one. Null when no command can, as for a run whose participants left no evidence. |
312
- | `analysisId`, `status`, `provider`, `model`, `completedAt`, `estimatedCostUsd` | The analysis version `analyze show` selects: the newest ready one, else the newest stale one. Null without one. |
313
- | `runPath`, `path` | The run directory and the version's `analysis.json`, relative to the project directory. |
314
- | `summary`, `limitations` | The analysis's own summary and limitations. |
315
- | `findings[]` | Highest priority first: `id`, `title`, `summary`, `impact` (`blocked_task`, `friction`, `recovery`, `uncertain`), `confidence` (`low`, `medium`, `high`), `recovery` (`recovered`, `not_observed`, `unknown`), `affected[]` (`streamId`, `label`), `exposedCount`, `evidence[]`, `nextStep` and `correction`. |
316
- | `findings[].evidence[]` | Each cited item once: `id`, `streamId`, `kind`, `bases` (how the observations used it), `frame` (the latest retained capture's index from 0, null before any capture), `elapsedMs` (time since the first retained capture, not a video offset), `at` and `capture` (the file relative to the project directory, null for nonvisual evidence). |
317
- | `findings[].correction` | The latest `analyze correct` note on that finding version (`status`, `reason`, `replacementClaim`), or null. |
318
- | `warnings` | Load warnings for the selected version. |
364
+ | Field | Meaning |
365
+ | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
366
+ | `schema` | `humanish.analysis-findings.v1`. |
367
+ | `runId` | The run the view reads. |
368
+ | `state` | `ready`, `stale`, `running`, `none`, `skipped`, `failed`, `dry_run` or `unavailable`. Only `ready` and `stale` carry findings. |
369
+ | `reason` | The stable code behind a state other than `ready`, or null: an automatic analysis reason such as `AUTOMATIC_ANALYSIS_KEY_MISSING`, a failed attempt's error code, or a load warning such as `ANALYSIS_SOURCE_CHANGED`. |
370
+ | `message` | What happened, in words. |
371
+ | `next` | The command that gets findings for this run, or for a dry run the command that starts a live one. Null when no command can, as for a run whose participants left no evidence. |
372
+ | `analysisId`, `status`, `provider`, `model`, `completedAt`, `estimatedCostUsd` | The analysis version `analyze show` selects: the newest ready one, else the newest stale one. Null without one. |
373
+ | `runPath`, `path` | The run directory and the version's `analysis.json`, relative to the project directory. |
374
+ | `summary`, `limitations` | The analysis's own summary and limitations. |
375
+ | `findings[]` | Highest priority first: `id`, `headline` and `experience` (null in analyses before `study-evidence-7`), `title`, `summary`, `impact` (`blocked_task`, `friction`, `recovery`, `uncertain`), `confidence` (`low`, `medium`, `high`), `recovery` (`recovered`, `not_observed`, `unknown`), `affected[]` (`streamId`, `label`), `exposedCount`, `evidence[]`, `nextStep` and `correction`. |
376
+ | `findings[].evidence[]` | Each cited item once: `id`, `streamId`, `kind`, `bases` (how the observations used it), `frame` (the latest retained capture's index from 0, null before any capture), `elapsedMs` (time since the first retained capture, not a video offset), `at` and `capture` (the file relative to the project directory, null for nonvisual evidence). |
377
+ | `findings[].correction` | The latest `analyze correct` note on that finding version (`status`, `reason`, `replacementClaim`), or null. When `status` is `amended`, `replacementClaim` replaces the finding's headline and experience. |
378
+ | `designFindings` | Most severe first: `id`, `headline`, `screen`, `notice`, `whyItMatters`, `suggestion`, `severity` (`major`, `moderate`, `minor`), `confidence`, `seenBy[]` (`streamId`, `label`) and `evidence[]` (as `findings[].evidence[]`, without `bases`). Null without an analysis result, or for an analysis written before design findings. |
379
+ | `warnings` | Load warnings for the selected version. |
319
380
 
320
381
  The states without findings:
321
382
 
@@ -352,7 +413,10 @@ humanish feedback issue --run latest --analysis <id> --finding F1 --repo owner/r
352
413
 
353
414
  Corrections append against exact analysis and finding hashes. Confirming one
354
415
  version does not approve a later claim. Dismissed findings cannot become feedback
355
- drafts; amendments preserve the original and record the replacement. Feedback
416
+ drafts; amendments preserve the original and record the replacement. A reason or replacement
417
+ claim that matches a sensitive pattern is refused. When the correction is made inside the run's
418
+ own invocation, one that holds a known transient value, as written or encoded, is refused too; a
419
+ separate `analyze correct` command has no transient values to check against. Feedback
356
420
  drafts include source/version/evidence references and remain explicitly
357
421
  independent of participant-authored candidates. No command above posts to GitHub.
358
422
  Each analysis has room for 256 correction inventory entries, including interrupted
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "humanish",
3
- "version": "0.111.4",
3
+ "version": "0.113.0",
4
4
  "description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
5
5
  "keywords": [
6
6
  "cli",
@@ -76,6 +76,7 @@
76
76
  "tui:test": "pnpm --filter humanish-tui test",
77
77
  "release:dogfood": "node scripts/release-dogfood.mjs",
78
78
  "bench": "tsx bench/run.ts",
79
+ "arch:bench": "tsx scripts/arch-bench.ts",
79
80
  "browser-control:proof": "node scripts/browser-control-proof.mjs",
80
81
  "docs:generate": "tsx scripts/generate-cli-docs.ts",
81
82
  "docs:check": "tsx scripts/generate-cli-docs.ts --check && tsx scripts/check-doc-paths.ts && tsx scripts/check-doc-symbols.ts && tsx scripts/check-doc-comments.ts && tsx scripts/check-doc-study-fields.ts",
@@ -190,16 +190,6 @@ Legacy profiles without it retain medium defaults for missing patience and
190
190
  technical confidence. Avoid contradictory prose and traits; inspect the compiled
191
191
  brief with `humanish study show <study> --json` before running.
192
192
 
193
- For an autonomous participant study, use a computer-use/local-agent study and write
194
- its `mission` as a believable situation and desired outcome. Supply fixture facts
195
- needed to act, but do not supply selectors, click sequences, hidden success rules,
196
- expected defects or recovery tricks. For example: “Saturday's event needs two
197
- setup volunteers and one cleanup volunteer. See whether this app helps you
198
- organize the event and keep people informed.” Do not instruct a participant to
199
- find a specific UI control merely to make the study succeed. Allow discovery,
200
- recovery and stopping appropriate to the participant. Keep researcher criteria in
201
- `tasks[].success`, separate from participant-facing `tasks[].goal`.
202
-
203
193
  Scripted regression checks are a different use of scenarios. When the requested
204
194
  work is deterministic verification of a known path, a `browser.steps` manifest
205
195
  can drive it with explicit selectors and assertions. It does not establish
@@ -232,6 +222,27 @@ tokens into scenario files.
232
222
 
233
223
  ## Authoring Studies
234
224
 
225
+ For an autonomous participant study, write `mission`, participant `instruction`
226
+ and `tasks[].goal` as a believable situation and desired outcome. Supply fixture
227
+ facts needed to act. Leave discovery, recovery and stopping to the participant.
228
+ Keep researcher criteria in `tasks[].success`.
229
+
230
+ | Bad: scripts the controls | Good: gives a situation |
231
+ | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
232
+ | Click "New event", type a title into the field, then press "Save" and open "Settings". | Saturday's event needs two setup volunteers and one cleanup volunteer. See whether this app helps you organize the event and keep people informed. |
233
+
234
+ `humanish study check` and `humanish run` warn about step lists, chains of UI
235
+ actions, quoted control labels, CSS selectors and URLs beyond the subject's entry
236
+ point. They quote up to three matched lines and suggest a situation-style mission.
237
+ The checks apply to computer-use and shared-world studies; scripted-browser and
238
+ terminal studies and `tasks[].success` are excluded. These warnings never refuse
239
+ a run, and the run records them in its bundle.
240
+
241
+ Give each participant a persona with a short, fictional `background` describing
242
+ their relevant experience and situation. A missing persona or background also
243
+ warns. Use `humanish study show <study> --json` to see what the participant receives.
244
+ Do not supply hidden success rules, expected defects or recovery tricks.
245
+
235
246
  Write reusable studies as `.yaml` study files (`humanish.study.v3`):
236
247
 
237
248
  ```yaml