humanish 0.112.0 → 0.114.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +41 -0
- package/CONTEXT.md +1 -0
- package/README.md +12 -1
- package/TELEMETRY.md +21 -0
- package/dist/actors/codex/restricted-participant-policy.d.ts +21 -1
- package/dist/actors/codex/restricted-participant-policy.js +20 -2
- package/dist/actors/codex/restricted-participant.d.ts +1 -1
- package/dist/actors/codex/restricted-participant.js +2 -1
- package/dist/actors/computer-use/loop/debrief.d.ts +12 -4
- package/dist/actors/computer-use/loop/debrief.js +57 -20
- package/dist/actors/computer-use/loop/ending.d.ts +5 -2
- package/dist/actors/computer-use/loop/impressions.d.ts +14 -0
- package/dist/actors/computer-use/loop/impressions.js +40 -0
- package/dist/actors/computer-use/loop/session.d.ts +3 -1
- package/dist/actors/computer-use/loop/session.js +2 -0
- package/dist/actors/computer-use/loop/trace.js +2 -0
- package/dist/actors/computer-use/loop/types.d.ts +8 -1
- package/dist/actors/computer-use/loop.d.ts +2 -1
- package/dist/actors/computer-use/loop.js +15 -3
- package/dist/actors/computer-use/openai-conversation.d.ts +9 -4
- package/dist/actors/computer-use/openai-conversation.js +60 -29
- package/dist/actors/computer-use/openai-provider.js +18 -12
- package/dist/actors/contract.d.ts +47 -5
- package/dist/actors/contract.js +13 -0
- package/dist/analysis/admission.d.ts +37 -0
- package/dist/analysis/admission.js +60 -0
- package/dist/analysis/automatic-completion.d.ts +0 -6
- package/dist/analysis/automatic-completion.js +3 -3
- package/dist/analysis/automatic-config.d.ts +10 -2
- package/dist/analysis/automatic-config.js +29 -14
- package/dist/analysis/automatic.js +7 -0
- package/dist/analysis/evidence-sources.d.ts +3 -1
- package/dist/analysis/evidence-sources.js +10 -1
- package/dist/analysis/evidence.js +2 -2
- package/dist/analysis/execute.d.ts +20 -6
- package/dist/analysis/execute.js +61 -28
- package/dist/analysis/image-tokens.d.ts +2 -0
- package/dist/analysis/image-tokens.js +1 -1
- package/dist/analysis/job.d.ts +9 -1
- package/dist/analysis/job.js +11 -2
- package/dist/analysis/service.js +6 -3
- package/dist/cli/commands/analyze.js +7 -2
- package/dist/cli/commands/notes.d.ts +3 -0
- package/dist/cli/commands/notes.js +144 -0
- package/dist/cli/commands/observe.js +1 -1
- package/dist/cli/commands/runs.js +33 -10
- package/dist/cli/commands/setup.js +95 -6
- package/dist/cli/commands/study-format.js +7 -4
- package/dist/cli/commands/study-run.js +2 -1
- package/dist/cli/commands/study.js +1 -1
- package/dist/cli/commands/tui.js +1 -1
- package/dist/cli/doctor.d.ts +2 -0
- package/dist/cli/doctor.js +18 -0
- package/dist/cli/findings.js +4 -1
- package/dist/cli/io.d.ts +3 -0
- package/dist/cli/io.js +11 -12
- package/dist/cli/program.js +37 -1
- package/dist/cli/secret-prompt.d.ts +5 -1
- package/dist/cli/secret-prompt.js +6 -2
- package/dist/cli/update-check-worker.d.ts +1 -0
- package/dist/cli/update-check-worker.js +5 -0
- package/dist/cli/update-check.d.ts +46 -0
- package/dist/cli/update-check.js +203 -0
- package/dist/evidence/encoded-text.d.ts +15 -2
- package/dist/evidence/encoded-text.js +68 -4
- package/dist/evidence/secret-scrub.d.ts +1 -1
- package/dist/evidence/secret-scrub.js +23 -2
- package/dist/feedback/draft.d.ts +19 -0
- package/dist/feedback/draft.js +69 -1
- package/dist/feedback/export-bundle.js +2 -2
- package/dist/feedback/export.js +47 -3
- package/dist/feedback/feedback.js +42 -2
- package/dist/keys/key-resolution.js +2 -0
- package/dist/keys/key-status.d.ts +37 -0
- package/dist/keys/key-status.js +68 -0
- package/dist/observer/artifact.d.ts +5 -0
- package/dist/observer/artifact.js +18 -2
- package/dist/observer/notes-route.d.ts +21 -0
- package/dist/observer/notes-route.js +140 -0
- package/dist/observer/pinned-files.d.ts +8 -1
- package/dist/observer/pinned-files.js +32 -3
- package/dist/observer/render.js +38 -14
- package/dist/observer/run-routes.d.ts +8 -2
- package/dist/observer/run-routes.js +47 -6
- package/dist/observer/serve.d.ts +3 -0
- package/dist/observer/serve.js +28 -3
- package/dist/observer-app.html +7 -6
- package/dist/routes/computer-use/local-vm.js +9 -2
- package/dist/routes/computer-use/participant-desktop.d.ts +6 -0
- package/dist/routes/computer-use/participant-execution.js +9 -1
- package/dist/routes/computer-use/rerun-selection.js +23 -21
- package/dist/routes/computer-use/result.js +3 -2
- package/dist/routes/computer-use/self-report.js +7 -1
- package/dist/routes/computer-use/setup.js +14 -0
- package/dist/routes/computer-use/types.d.ts +4 -1
- package/dist/routes/shared-world/bundle-records.js +8 -7
- package/dist/routes/shared-world/bundle.js +1 -1
- package/dist/run/contained-output.d.ts +7 -0
- package/dist/run/contained-output.js +35 -1
- package/dist/run/detail.js +4 -4
- package/dist/run/note-files.d.ts +73 -0
- package/dist/run/note-files.js +184 -0
- package/dist/run/notes.d.ts +37 -0
- package/dist/run/notes.js +198 -0
- package/dist/run/participant-caption.d.ts +11 -1
- package/dist/run/participant-caption.js +30 -1
- package/dist/run/participant-records.d.ts +3 -1
- package/dist/run/participant-records.js +6 -0
- package/dist/run-study.js +17 -1
- package/dist/study/discover.js +6 -1
- package/dist/study/doctor.d.ts +2 -0
- package/dist/study/doctor.js +46 -8
- package/dist/study/init-templates.js +10 -5
- package/dist/study/plan.js +3 -2
- package/dist/study/preflight-result.d.ts +1 -0
- package/dist/study/preflight-result.js +3 -2
- package/dist/study/preflight.d.ts +2 -0
- package/dist/study/preflight.js +54 -14
- package/dist/study/summary.js +2 -2
- package/dist/study/warnings.d.ts +5 -0
- package/dist/study/warnings.js +79 -2
- package/dist/substrates/local/capacity.d.ts +75 -0
- package/dist/substrates/local/capacity.js +143 -0
- package/dist/substrates/local/firecracker-desktop.d.ts +8 -0
- package/dist/substrates/local/firecracker-desktop.js +34 -3
- package/dist/substrates/local/runtime-host.d.ts +16 -3
- package/dist/substrates/local/runtime-host.js +49 -4
- package/dist/substrates/local/runtime.d.ts +10 -0
- package/dist/substrates/local/runtime.js +44 -5
- package/dist/tui-app.js +58 -58
- package/docs/architecture/actor-contract.md +20 -6
- package/docs/contracts/errors.md +2 -2
- package/docs/contracts/feedback.md +8 -1
- package/docs/contracts/run-bundle.md +57 -0
- package/docs/contracts/schemas.md +62 -36
- package/docs/contracts/study-analysis.md +41 -8
- package/docs/principles/invariants-and-defaults.md +2 -2
- package/docs/product/automatic-analysis.md +22 -16
- package/package.json +2 -2
- package/skills/humanish/SKILL.md +26 -10
|
@@ -350,22 +350,36 @@ this paragraph is the record of that decision.
|
|
|
350
350
|
|
|
351
351
|
`ComputerUseProvider.debrief` is an optional read-only request after a structured
|
|
352
352
|
`stopWhen` or dwell stop. It returns a `ComputerUseTurn` with `closingReport` containing
|
|
353
|
-
`summary` and `
|
|
353
|
+
`summary`, `frictionReports` and `impressions`; an empty friction or impressions list is valid,
|
|
354
|
+
and a report without `impressions` (an older provider) is still accepted. The loop rejects
|
|
354
355
|
actions, pending safety checks, and invalid report shapes. It redacts accepted
|
|
355
356
|
reports, records them in `ActorTrace.debrief`, and projects one readable message
|
|
356
|
-
without invoking action or communication callbacks.
|
|
357
|
+
without invoking action or communication callbacks. Each impression becomes its own message
|
|
358
|
+
and goes to `ActorTrace.impressions`; a session without them records why in the same field.
|
|
359
|
+
A turn that ends the session itself may carry `closingReport` too, and the loop keeps its
|
|
360
|
+
impressions the same way without a second request.
|
|
361
|
+
|
|
362
|
+
`ComputerUseProvider.requestImpressions` is the optional request after the participant ended
|
|
363
|
+
the session itself without impressions. It asks for impressions only, returns them as
|
|
364
|
+
`impressions` on the turn, and follows the debrief's rules and guards. The loop records it as
|
|
365
|
+
`ActorTrace.debrief` with `trigger: "participant_end"` and no report, so the participant's
|
|
366
|
+
reason, outcome and friction come from its final message as before. A failed, timed-out or
|
|
367
|
+
cut-off request leaves `ActorTrace.impressions` as `not_collected` with the reason.
|
|
357
368
|
|
|
358
369
|
This request uses the final observation and retained provider history. It does
|
|
359
370
|
not expose the hidden stop criterion or change the original task outcome. It is
|
|
360
|
-
skipped before any participant turn, after natural completion
|
|
361
|
-
support, or without remaining time and known budget. An attempted request's
|
|
371
|
+
skipped before any participant turn, after natural completion (apart from the impressions-only
|
|
372
|
+
request), without provider support, or without remaining time and known budget. An attempted request's
|
|
362
373
|
unreported usage remains an unknown cost line. `counts.debriefCalls` is separate
|
|
363
374
|
from interaction turns, and reported usage contributes to aggregate cost.
|
|
364
375
|
|
|
365
376
|
The OpenAI implementation makes one request with tools disabled and structured
|
|
366
|
-
output; no HTTP or policy retries.
|
|
377
|
+
output; no HTTP or policy retries. Its output limit is 3072 tokens, or the declared
|
|
378
|
+
`maxOutputTokens` when lower. Stateless/ZDR mode does not offer retrospective
|
|
367
379
|
reporting: its client-carried conversation summarizes its oldest turns past a token
|
|
368
|
-
budget, so it may not hold the whole session a retrospective report needs.
|
|
380
|
+
budget, so it may not hold the whole session a retrospective report needs. It does offer
|
|
381
|
+
the impressions-only request, which reads the carried conversation: its newest
|
|
382
|
+
exchanges whole and earlier turns as a text note. The
|
|
369
383
|
[paired live receipt](https://github.com/danielgwilson/humanish/blob/main/docs/evidence/computer-use/structured-closing-report-2026-09-05.md)
|
|
370
384
|
records both report recovery and control failures in the separate legacy parser.
|
|
371
385
|
|
package/docs/contracts/errors.md
CHANGED
|
@@ -46,8 +46,8 @@ Counts are the distinct quoted codes in `src/` on 2026-10-03.
|
|
|
46
46
|
The lowercase families are values recorded inside artifacts and receipts, and an artifact keeps
|
|
47
47
|
the code it was written with. Automatic analysis reads two of them back:
|
|
48
48
|
`analysis_admission_estimate_exceeded` becomes the reason `AUTOMATIC_ANALYSIS_ADMISSION_EXCEEDED`,
|
|
49
|
-
and `analysis_budget_exceeded`
|
|
50
|
-
`humanish analyze --max-cost
|
|
49
|
+
and `analysis_budget_exceeded` records the expected cost, the worst case and the cap on the job,
|
|
50
|
+
and prints them with the command that reruns it, `humanish analyze --run <id> --max-cost <n>`. Validation also accepts a partial artifact whose error is
|
|
51
51
|
`analysis_admission_estimate_exceeded`.
|
|
52
52
|
|
|
53
53
|
Share-safety reasons in `humanish verify` (`VERIFY_FAILED`, `RAW_SCREENSHOTS` and the rest) are
|
|
@@ -66,7 +66,9 @@ Listing does not create a draft or check public-sharing eligibility.
|
|
|
66
66
|
|
|
67
67
|
For completed computer-use and shared-world sessions, participant-reported
|
|
68
68
|
friction comes from the redacted closing report, a completed debrief's
|
|
69
|
-
`frictionReports`, and observed-report clauses in earlier messages.
|
|
69
|
+
`frictionReports`, and observed-report clauses in earlier messages. The
|
|
70
|
+
participant's closing impressions (`ActorTrace.impressions`) are opinions about
|
|
71
|
+
the product and never count as friction.
|
|
70
72
|
Matching `stopWhen` or ending a dwell window does not discard an earlier
|
|
71
73
|
report or change the successful completion verdict. Exact repeated reports
|
|
72
74
|
appear once in the participant's candidate. Harness notices, reasoning, and observed
|
|
@@ -84,6 +86,11 @@ was not repeated in the message trace.
|
|
|
84
86
|
Builds structured feedback from the first usable run candidate, or the candidate
|
|
85
87
|
selected with `--candidate`. With no candidate, a live run produces a summary of
|
|
86
88
|
its recorded review; that fallback does not add a finding to the candidate list.
|
|
89
|
+
With `--analysis` and `--finding`, the draft is built from that analysis finding
|
|
90
|
+
instead. When the finding's observations cite a participant's closing impression,
|
|
91
|
+
`actual` quotes it after the observations as
|
|
92
|
+
`Participant <id> said at the end (<kind>): <text>`. An impression is quoted only
|
|
93
|
+
when the cited evidence entry carries its text whole; uncited impressions stay out.
|
|
87
94
|
A dry run tests no product behavior, so `draft`, `verify`, `issue` and
|
|
88
95
|
`issue-url` refuse it with `HUMANISH_FEEDBACK_REQUIRES_LIVE_RUN`. All drafts
|
|
89
96
|
require a verified `share_ready` live run and are written under the run bundle,
|
|
@@ -576,6 +576,63 @@ updatedAt: "<ISO timestamp>"
|
|
|
576
576
|
|
|
577
577
|
The latest pointer may move. Run bundle directories should not.
|
|
578
578
|
|
|
579
|
+
## Reviewer notes
|
|
580
|
+
|
|
581
|
+
Each note a person adds while reviewing a run is its own file in the run directory,
|
|
582
|
+
`notes/<id>.json`, written by `humanish notes --add` and by an Observer served on loopback.
|
|
583
|
+
`notes/` is absent until the first note.
|
|
584
|
+
|
|
585
|
+
```json
|
|
586
|
+
{
|
|
587
|
+
"schema": "humanish.run-note.v1",
|
|
588
|
+
"runId": "example-2026-06-02t10-00-00-000z-proof",
|
|
589
|
+
"id": "note-20260602t103000000z-000000000001",
|
|
590
|
+
"atMs": 151000,
|
|
591
|
+
"participant": "stream-001",
|
|
592
|
+
"nearest": { "participant": "stream-001", "itemId": "turn-07-call-01" },
|
|
593
|
+
"text": "They looked for the save button here.",
|
|
594
|
+
"author": "you",
|
|
595
|
+
"createdAt": "2026-06-02T10:30:00.000Z",
|
|
596
|
+
"editedAt": null
|
|
597
|
+
}
|
|
598
|
+
```
|
|
599
|
+
|
|
600
|
+
- `id`: `note-`, the creation time as `yyyymmddthhmmssmmmz`, `-` and 12 hex digits of
|
|
601
|
+
random bits. The file is named `<id>.json`, so the files sort in the order they were
|
|
602
|
+
added.
|
|
603
|
+
- `runId`: the run directory's name. A file whose `runId` or `id` does not match its run or
|
|
604
|
+
its file name is not read.
|
|
605
|
+
- `atMs`: milliseconds from the run clock's start, the earliest timed capture or
|
|
606
|
+
desktop video of any participant. The Observer's study timeline shows that moment as
|
|
607
|
+
00:00. A participant's captures count only when every one is stamped and the stamps
|
|
608
|
+
never go back.
|
|
609
|
+
- `participant`: the stream id the note belongs to, or `null` for the whole run.
|
|
610
|
+
`humanish notes --add --participant` and the Observer also take the participant's id from
|
|
611
|
+
the study (the stream's `laneId`) and record its stream id. A name that is one stream's id
|
|
612
|
+
and another stream's study id is refused.
|
|
613
|
+
- `nearest`: the latest stamped trace item, a capture or an event, at or before the
|
|
614
|
+
moment, among the note's participant or every participant; `null` when there is none.
|
|
615
|
+
- `text`: 1 to 2000 characters, after the run's known values, secret-shaped values and
|
|
616
|
+
local paths are replaced.
|
|
617
|
+
- `author`: `"you"` in this version.
|
|
618
|
+
- `createdAt`, `editedAt`: ISO-8601. Notes cannot be edited yet, so `editedAt` is `null`.
|
|
619
|
+
|
|
620
|
+
A note file is written once and never rewritten: its bytes go to a temporary file inside
|
|
621
|
+
`notes/`, which `link` gives the note's name only when no file has that name, and a taken name
|
|
622
|
+
gets a new id. Adding a note reads no other note, so notes added at the same time from any
|
|
623
|
+
process are all kept, and there is no lock. Readers list at most 1000 entries of `notes/` and
|
|
624
|
+
read at most 500 notes, each within 16 KiB and refused before reading past that; any other
|
|
625
|
+
entry, or a file that is not a note of the run, is skipped and named. The Observer servers serve
|
|
626
|
+
any file under `notes/` only within 16 KiB. A run takes no note past 500.
|
|
627
|
+
`verify` scans the note files with every other run file, so a secret-shaped value in a note
|
|
628
|
+
keeps the run from `share_ready`. A bundle export copies them; an HTML export carries the notes
|
|
629
|
+
read-only after checking the set it read.
|
|
630
|
+
|
|
631
|
+
Threat model: forged web requests are refused, whether from a page on another site or through a
|
|
632
|
+
name rebound to 127.0.0.1, by the token, Origin and Host checks on `POST /api/notes`. A local
|
|
633
|
+
process with write access to the run directory is out of scope: it already has the user's
|
|
634
|
+
privileges and can change any file in the run.
|
|
635
|
+
|
|
579
636
|
## Verify Result Share Safety
|
|
580
637
|
|
|
581
638
|
`humanish.verify-result.v1` includes a machine-readable `shareSafety` block in
|
|
@@ -24,38 +24,38 @@ workflow without leaking private upstream truth into core.
|
|
|
24
24
|
|
|
25
25
|
## Contract Index
|
|
26
26
|
|
|
27
|
-
| Contract | Schema
|
|
28
|
-
| --------------------------------- |
|
|
29
|
-
| Run bundle | `humanish.run-bundle.v1`
|
|
30
|
-
| Adapter | `humanish.adapter.v1`
|
|
31
|
-
| Study file | `humanish.study.v3`
|
|
32
|
-
| Persona | `humanish.persona.v1`
|
|
33
|
-
| Scenario | `humanish.scenario.v1`
|
|
34
|
-
| Actor trace | `humanish.actor-trace.v1`
|
|
35
|
-
| Substrate | reserved (never shipped)
|
|
36
|
-
| Evidence stream | reserved (streams live inside the run bundle)
|
|
37
|
-
| Review | `humanish.review.v1`
|
|
38
|
-
| Study analysis | `humanish.study-analysis.v1` (prompt `study-evidence-7` adds `findings[].headline`, `findings[].experience` and `designFindings[]`; older artifacts omit them)
|
|
39
|
-
| Study analysis correction | `humanish.study-analysis-correction.v1`
|
|
40
|
-
| Analysis findings view | `humanish.analysis-findings.v1` (`review --json` field `analysis`; read-only projection of the selected analysis)
|
|
41
|
-
| Analysis execution receipt | `humanish.analysis-execution.v1`
|
|
42
|
-
| Analysis execution start | `humanish.analysis-execution-start.v1`
|
|
43
|
-
| Verification | `humanish.verify-result.v1`
|
|
44
|
-
| Policy | `humanish.policy.v1` (fixture-only; not engine-validated)
|
|
45
|
-
| Feedback | `humanish.feedback.v1`
|
|
46
|
-
| Terminal cost ledger | `humanish.terminal-cost-ledger.v1`
|
|
47
|
-
| Terminal no-spend proof | `humanish.terminal-no-spend-proof.v1`
|
|
48
|
-
| Pricing (operator-editable rates) | `humanish.pricing.v1` (`src/run/pricing.ts`; dated per-model + E2B desktop rates)
|
|
49
|
-
| Run cost summary | `humanish.run-cost-summary.v1` (additive `RunBundle.cost`; estimate, never a charge)
|
|
50
|
-
| Estimated actor cost | `humanish.actor-estimated-cost.v1` (additive `ActorTrace.estimatedCost`)
|
|
51
|
-
| Model settings | `humanish.model-settings.v1` (additive `ActorTrace.modelSettings`; the reasoning effort the request carried)
|
|
52
|
-
| Affordance use | `humanish.affordance-use.v1` (additive `ActorTrace.affordanceUse`; per-class counts of the routes an actor took)
|
|
53
|
-
| Adapter score | `humanish.adapter-score.v1` (`RunBundle.adapterScore`; namespaced; route-specific acceptance semantics)
|
|
54
|
-
| Adapter artifact | `humanish.adapter-artifact.v1` (`RunBundle.adapterArtifacts[]`; namespaced; local relative proof references)
|
|
55
|
-
| Shared-world evidence | `humanish.shared-world.v1` (additive `RunBundle.sharedWorld` + `RunBundle.attributionClass`; `topologyMode: sequential \| concurrent`)
|
|
56
|
-
| Comms thread | `humanish.comms-thread.v1` (off-app email/SMS the app sent, captured; a `kind: log` run-dir artifact of digests only: from/to/subject/link digests + one OTP count, never raw)
|
|
57
|
-
| Serve result | `humanish.serve-result.v1` (`src/observer/serve.ts` is authoritative)
|
|
58
|
-
| Serve control plane | reserved (`/_humanish/api/*` answers `501` `HUMANISH_SERVE_CONTROL_PLANE_DISABLED` in v1)
|
|
27
|
+
| Contract | Schema | Public-safe fixture |
|
|
28
|
+
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
|
|
29
|
+
| Run bundle | `humanish.run-bundle.v1` | `synthetic-run-bundle` |
|
|
30
|
+
| Adapter | `humanish.adapter.v1` | `synthetic-cli-adapter` |
|
|
31
|
+
| Study file | `humanish.study.v3` | `first-run` |
|
|
32
|
+
| Persona | `humanish.persona.v1` | `synthetic-maintainer` |
|
|
33
|
+
| Scenario | `humanish.scenario.v1` | `first-run-smoke` |
|
|
34
|
+
| Actor trace | `humanish.actor-trace.v1` | `synthetic-actor-trace` |
|
|
35
|
+
| Substrate | reserved (never shipped) | none |
|
|
36
|
+
| Evidence stream | reserved (streams live inside the run bundle) | see [`run-bundle.md`](run-bundle.md) |
|
|
37
|
+
| Review | `humanish.review.v1` | `contract-proof-review` |
|
|
38
|
+
| Study analysis | `humanish.study-analysis.v1` (prompt `study-evidence-7` adds `findings[].headline`, `findings[].experience` and `designFindings[]`; older artifacts omit them; `study-evidence-8` adds participant impressions to the instructions, with no field change) | see [study analysis](study-analysis.md) and synthetic analysis fixtures in `tests/` |
|
|
39
|
+
| Study analysis correction | `humanish.study-analysis-correction.v1` | see [study analysis](study-analysis.md#human-review-and-sharing) |
|
|
40
|
+
| Analysis findings view | `humanish.analysis-findings.v1` (`review --json` field `analysis`; read-only projection of the selected analysis) | see [study analysis](study-analysis.md#reading-findings) |
|
|
41
|
+
| Analysis execution receipt | `humanish.analysis-execution.v1` | see [study analysis](study-analysis.md#durable-records) |
|
|
42
|
+
| Analysis execution start | `humanish.analysis-execution-start.v1` | see [study analysis](study-analysis.md#durable-records) |
|
|
43
|
+
| Verification | `humanish.verify-result.v1` | `five-check-verify` |
|
|
44
|
+
| Policy | `humanish.policy.v1` (fixture-only; not engine-validated) | `public-safety-policy` |
|
|
45
|
+
| Feedback | `humanish.feedback.v1` | `public-safe-feedback` |
|
|
46
|
+
| Terminal cost ledger | `humanish.terminal-cost-ledger.v1` | see Terminal Cost Ledger below |
|
|
47
|
+
| Terminal no-spend proof | `humanish.terminal-no-spend-proof.v1` | see Terminal Cost Ledger below |
|
|
48
|
+
| Pricing (operator-editable rates) | `humanish.pricing.v1` (`src/run/pricing.ts`; dated per-model + E2B desktop rates) | see Run Cost Summary And Estimated Actor Cost below |
|
|
49
|
+
| Run cost summary | `humanish.run-cost-summary.v1` (additive `RunBundle.cost`; estimate, never a charge) | see Run Cost Summary And Estimated Actor Cost below |
|
|
50
|
+
| Estimated actor cost | `humanish.actor-estimated-cost.v1` (additive `ActorTrace.estimatedCost`) | see Run Cost Summary And Estimated Actor Cost below |
|
|
51
|
+
| Model settings | `humanish.model-settings.v1` (additive `ActorTrace.modelSettings`; the reasoning effort the request carried) | see Actor Trace below |
|
|
52
|
+
| Affordance use | `humanish.affordance-use.v1` (additive `ActorTrace.affordanceUse`; per-class counts of the routes an actor took) | see Affordance Use below |
|
|
53
|
+
| Adapter score | `humanish.adapter-score.v1` (`RunBundle.adapterScore`; namespaced; route-specific acceptance semantics) | see Product-Adapter Extension Seam below |
|
|
54
|
+
| Adapter artifact | `humanish.adapter-artifact.v1` (`RunBundle.adapterArtifacts[]`; namespaced; local relative proof references) | see Product-Adapter Extension Seam below |
|
|
55
|
+
| Shared-world evidence | `humanish.shared-world.v1` (additive `RunBundle.sharedWorld` + `RunBundle.attributionClass`; `topologyMode: sequential \| concurrent`) | see Shared-World Evidence below |
|
|
56
|
+
| Comms thread | `humanish.comms-thread.v1` (off-app email/SMS the app sent, captured; a `kind: log` run-dir artifact of digests only: from/to/subject/link digests + one OTP count, never raw) | see `comms` under Study File |
|
|
57
|
+
| Serve result | `humanish.serve-result.v1` (`src/observer/serve.ts` is authoritative) | none (command result envelope; see Serve Result below) |
|
|
58
|
+
| Serve control plane | reserved (`/_humanish/api/*` answers `501` `HUMANISH_SERVE_CONTROL_PLANE_DISABLED` in v1) | none |
|
|
59
59
|
|
|
60
60
|
## Study File
|
|
61
61
|
|
|
@@ -970,8 +970,9 @@ Core-owned fields:
|
|
|
970
970
|
- optional `modelSettings.maxOutputTokens`: the declared positive integer
|
|
971
971
|
`actor.maxOutputTokens`, passed to every first-party OpenAI CUA response
|
|
972
972
|
request, including reasoning output. Absent when undeclared. The first request
|
|
973
|
-
of a session
|
|
974
|
-
|
|
973
|
+
of a session uses `min(maxOutputTokens, 1024)`, or 1024 when undeclared, and
|
|
974
|
+
closing requests use `min(maxOutputTokens, 3072)`, or 3072 when undeclared; the
|
|
975
|
+
first request also carries the opening screenshot as an
|
|
975
976
|
`input_image` after the instructions. Unsupported routes,
|
|
976
977
|
custom provider/session hooks, and per-participant overrides fail before allocation.
|
|
977
978
|
This is an output-token limit, not an input-token, request-count or billing cap.
|
|
@@ -1007,13 +1008,38 @@ Core-owned fields:
|
|
|
1007
1008
|
`cacheWriteInput`. `summarizedTurns` counts the earlier exchanges the latest
|
|
1008
1009
|
`explicit_context` request no longer carried whole: lines of the note, or
|
|
1009
1010
|
only counted once the note reached its cap.
|
|
1010
|
-
`requests[]` has one entry per participant request: its `mode` and, in
|
|
1011
|
+
`requests[]` has one entry per participant turn or impressions request: its `mode` and, in
|
|
1011
1012
|
`explicit_context`, `carriedExchanges`, `carriedScreenshots` and
|
|
1012
|
-
`estimatedInputTokens`.
|
|
1013
|
+
`estimatedInputTokens`. The impressions-only request has `kind: impressions`;
|
|
1014
|
+
the optional `kind` field is absent on participant turns and in older bundles.
|
|
1015
|
+
Impressions carry the context the next turn would carry, with the same budget
|
|
1016
|
+
and summarization of older turns. Billed input per request is in `tokenUsage.turns`. A
|
|
1013
1017
|
participant that ran in `explicit_context` gets a run warning. Absent on other
|
|
1014
1018
|
providers and on earlier bundles
|
|
1015
1019
|
- optional `affordanceUse` (`humanish.affordance-use.v1`): which kind of route this
|
|
1016
1020
|
actor took (see Affordance Use below)
|
|
1021
|
+
- optional `impressions`: what a computer-use participant said about the product
|
|
1022
|
+
at the end of its session. `status: collected` carries `items[]`, each a
|
|
1023
|
+
`kind` (`unclear`, `unfinished`, `untrustworthy`, `liked`, `missing`,
|
|
1024
|
+
`unlike_my_work`), a redacted first-person `text` of at most 500 characters
|
|
1025
|
+
and the `messageId` of the trace `message` that quotes it as
|
|
1026
|
+
`Impression (<kind>): <text>`. They are not counted in `counts.messages`. Up
|
|
1027
|
+
to six come from the closing account: a completed debrief's report, the
|
|
1028
|
+
participant's own final account when its provider writes a structured one
|
|
1029
|
+
(the Codex participant), or, when the participant ended the session itself
|
|
1030
|
+
without them, one impressions-only request (OpenAI computer-use, threaded or
|
|
1031
|
+
`explicit_context`). That request is recorded as `debrief` with `trigger:
|
|
1032
|
+
participant_end` and never has a `report` or `messageId`; it leaves the
|
|
1033
|
+
participant's `reason`, outcome and friction as they were. An empty `items`
|
|
1034
|
+
list means the participant named none. `status: not_collected` carries a
|
|
1035
|
+
`reason`: the closing report or impressions request was skipped or failed
|
|
1036
|
+
(including a reply cut off by the output limit), the participant ended the
|
|
1037
|
+
session without a structured closing account and its provider cannot ask for
|
|
1038
|
+
impressions (the Claude local agent), the closing account had no
|
|
1039
|
+
impressions, or the session stopped before a closing account. Every
|
|
1040
|
+
computer-use loop trace records one; absent on other routes and on earlier
|
|
1041
|
+
bundles. `debrief.report` keeps only `summary` and `frictionReports`.
|
|
1042
|
+
Impression messages do not count as participant-reported friction
|
|
1017
1043
|
- optional `estimatedCost` (`humanish.actor-estimated-cost.v1`): a token-derived
|
|
1018
1044
|
cost estimate for this participant (see Run Cost Summary And Estimated Actor Cost).
|
|
1019
1045
|
It is deliberately a different field from `tokenUsage.costUsd`: a bare
|
|
@@ -4,7 +4,7 @@ Study analysis is an independent interpretation of retained participant evidence
|
|
|
4
4
|
It is separate from the participant's account, recorded outcome, and the run's
|
|
5
5
|
deterministic review verdict. Opening an Observer never starts a provider request.
|
|
6
6
|
Supported live runs request analysis on completion by default, with a separate
|
|
7
|
-
$3
|
|
7
|
+
$3 cap on its expected cost. Set `review.analysis: false` to disable that request;
|
|
8
8
|
see [automatic analysis](../product/automatic-analysis.md).
|
|
9
9
|
|
|
10
10
|
## Invocation
|
|
@@ -20,16 +20,30 @@ humanish analyze show --run latest --json
|
|
|
20
20
|
The source must be a verified, completed live run. A dry-run contract bundle is
|
|
21
21
|
not a participant study. Here, `analyze --dry-run` means checking an existing
|
|
22
22
|
study's input and admission estimate without credentials, a provider request,
|
|
23
|
-
or a new analysis artifact.
|
|
23
|
+
or a new analysis artifact. It prints the expected cost, the worst case and the
|
|
24
|
+
cap; a refused dry run says why, and names the `--max-cost` that admits it.
|
|
24
25
|
|
|
25
26
|
The default provider is `openai`; its default model is `gpt-6-astra`, with high reasoning effort. A request sends selected retained text and
|
|
26
27
|
captures to OpenAI, without tools, redirects, provider-side response storage, or
|
|
27
28
|
automatic retries. `--question` adds a reviewer question; it never changes the
|
|
28
29
|
participant assignment. For OpenAI API analysis, `--max-cost` is required, including for dry-run
|
|
29
|
-
admission.
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
30
|
+
admission. Admission compares the expected cost plus a 10% margin with it; the
|
|
31
|
+
provider bill is not limited by it.
|
|
32
|
+
`--timeout-ms` and `--max-output-tokens` bound the request. A bill above the
|
|
33
|
+
worst case or the cap retains valid findings and usage but returns a partial
|
|
34
|
+
result and a nonzero command exit, including when that version is reused.
|
|
35
|
+
|
|
36
|
+
The estimate counts the instructions, evidence packet and result schema at 3
|
|
37
|
+
UTF-8 bytes per input token, adds 2,048 framing tokens and each capture's
|
|
38
|
+
high-detail image tokens, and prices input at the model's highest input rate.
|
|
39
|
+
The expected output is 12,000 tokens plus 1,000 per participant, at most the
|
|
40
|
+
output allowance; the worst case spends the whole allowance. `admission` in
|
|
41
|
+
`--json` output has `estimatedCostUsd` (the expected cost), `worstCaseCostUsd`,
|
|
42
|
+
`maxCostUsd`, `inputTokenAllowance` (the input tokens priced) and
|
|
43
|
+
`outputTokenAllowance`. On 148 billed gpt-6-astra analyses the expected cost was
|
|
44
|
+
1.08 to 7.3 times the bill, and 1.08 to 1.32 times for those billed $1 or more.
|
|
45
|
+
A refusal's message gives the expected cost, the worst case, the cap, and
|
|
46
|
+
`humanish analyze --run <id> --max-cost <n>` with `n` the worst case rounded up.
|
|
33
47
|
|
|
34
48
|
The default deadline is ten minutes. With no explicit output-token limit, analysis
|
|
35
49
|
uses 32,768 tokens if admission fits the declared budget, or retains the prior
|
|
@@ -266,6 +280,21 @@ a finding opens. Exposure and remaining unique limits are one disclosure away;
|
|
|
266
280
|
observation details retain each original claim and its specific limitation.
|
|
267
281
|
This presentation does not rewrite the saved analysis or reviewer corrections.
|
|
268
282
|
|
|
283
|
+
## Participant impressions
|
|
284
|
+
|
|
285
|
+
A computer-use participant's closing impressions (`ActorTrace.impressions`) reach the packet as
|
|
286
|
+
ordinary `message` entries of that participant's stream, one per impression, with text
|
|
287
|
+
`Impression (<kind>): <text>`. They are quote-eligible and attributed by `streamId`, so feedback,
|
|
288
|
+
participant_statement observations and design findings can cite them under the existing rules.
|
|
289
|
+
Selection admits them right after the participant's last entry, ahead of the rest of its
|
|
290
|
+
session. The evidence mapping and `captureVersion` are unchanged.
|
|
291
|
+
|
|
292
|
+
The `study-evidence-8` prompt adds one paragraph: impressions are the participant's own opinions,
|
|
293
|
+
used as participant statements, and an `unlike my work` impression together with a capture of
|
|
294
|
+
that screen is a strong design finding. Validation is unchanged. A design finding still needs at
|
|
295
|
+
least one cited capture, so an impression plus the capture of its screen makes a design finding,
|
|
296
|
+
and an impression alone supports only a finding about what the participant said.
|
|
297
|
+
|
|
269
298
|
## Durable records
|
|
270
299
|
|
|
271
300
|
The frozen `humanish.observer-data.v1` schema is unchanged. A companion
|
|
@@ -289,7 +318,9 @@ describes:
|
|
|
289
318
|
The optional automatic job is separate from the immutable analysis. Its view
|
|
290
319
|
binds terminal state to the exact execution receipt and report. A stale or
|
|
291
320
|
unverifiable job remains unknown; reading or exporting it never dispatches.
|
|
292
|
-
Automatic job metadata is omitted from shared derivatives.
|
|
321
|
+
Automatic job metadata is omitted from shared derivatives. A job refused for its
|
|
322
|
+
cost also records `admission` (`expectedCostUsd`, `worstCaseCostUsd`,
|
|
323
|
+
`maxCostUsd`), which `review` and the Observer read; other jobs omit it. See
|
|
293
324
|
[automatic analysis](../product/automatic-analysis.md).
|
|
294
325
|
|
|
295
326
|
Version and correction directories are claimed exclusively; publication is
|
|
@@ -369,7 +400,9 @@ The states without findings:
|
|
|
369
400
|
- `skipped`: the automatic analysis was skipped or refused, for example without
|
|
370
401
|
`OPENAI_API_KEY`, or the run's study file sets `review.analysis: false`
|
|
371
402
|
(`AUTOMATIC_ANALYSIS_DISABLED`, read from the file when the view is built). `next` is
|
|
372
|
-
`analyze --max-cost 3`, which runs an analysis anyway.
|
|
403
|
+
`analyze --max-cost 3`, which runs an analysis anyway. For an analysis refused for its
|
|
404
|
+
cost, the message gives the expected cost, the worst case and the cap, and `next` sets
|
|
405
|
+
`--max-cost` to the worst case rounded up.
|
|
373
406
|
- `failed`: the latest attempt failed or was cancelled and no usable version exists. `next`
|
|
374
407
|
reruns it with the same analyst, `--provider codex` for the Codex account analyst.
|
|
375
408
|
- `none`: no analysis ran for this live run and nothing records why, for example a run that
|
|
@@ -39,7 +39,7 @@ certify (see the conformance suite).
|
|
|
39
39
|
turn a dry run, preview, reader or unsupported route into provider or sandbox
|
|
40
40
|
spend. An explicitly live supported study includes the disclosed default
|
|
41
41
|
post-run analysis budget unless `review.analysis: false` disables it. Analysis
|
|
42
|
-
has a separate
|
|
42
|
+
has a separate cap on its expected cost, not a provider billing cap or part of
|
|
43
43
|
the actor budget. Spend-bearing tests retain their explicit env gates.
|
|
44
44
|
4. **Evidence verifies fail-closed.** A run bundle that cannot pass verification (schema,
|
|
45
45
|
redaction status, artifact presence, public-safety scan) is a failed run, even when the
|
|
@@ -90,7 +90,7 @@ silently drifting from one is not.
|
|
|
90
90
|
| Default | Why it is the default | Legitimate override |
|
|
91
91
|
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
92
92
|
| Dry-run | Spend safety (the live-spend rule sets the floor; dry-run keeps it far away) | `mode: live` |
|
|
93
|
-
| Post-run analysis on supported live studies | Findings accompany the recording; a separate $3
|
|
93
|
+
| Post-run analysis on supported live studies | Findings accompany the recording; a separate $3 cap on its expected cost is visible before execution | `review.analysis: false` disables it; an explicit mapping sets another analysis budget. Dry-run and unsupported routes never dispatch; missing default credentials produce a recorded skip |
|
|
94
94
|
| Separate worlds (`per-lane-worlds`) | Isolation, attribution, reproducibility | `route: shared-world`: N participants against one mutable plane for scenarios that are about interaction between participants. Every participant runs at once: `execution.concurrency` must be two or more, and an omitted value fills to the number of participants. The plane is one getHost-exposed subject sandbox with synthetic seeded data, or an operator-owned public deployment (external-public), driven by N actor sandboxes. The bundle declares the weaker `attributionClass: shared-world` and a verify-enforced `attributionLimits` ceiling (`best-effort-causal-attribution` and the rest of the concurrent set). Sequential turns at `execution.concurrency: 1` were removed in 0.106.0. |
|
|
95
95
|
| External key placement | Smallest blast radius: when the keyed process (e.g. a computer-use provider loop) runs outside the sandbox, its key never enters | In-sandbox placement when the keyed process runs inside (an agent harness under test); declared per actor type, with a spend budget |
|
|
96
96
|
| Loopback entry URLs | Public-safety: never drive third-party sites unbidden | `policies.allowPublicTargets` for an owner-declared deployment/preview (a Vercel preview of your own app). Multi-participant public/preview fan-out needs explicit `participants[].target` for every participant, so the adapter-owned topology is declared rather than inferred. Provisioned clone subjects always serve in-sandbox on loopback |
|
|
@@ -3,10 +3,10 @@
|
|
|
3
3
|
Supported live studies automatically request analysis after each recording finishes. Findings remain
|
|
4
4
|
separate from participant feedback and the recorded study verdict.
|
|
5
5
|
|
|
6
|
-
The default is `gpt-6-astra` with high reasoning effort, a separate $3
|
|
7
|
-
|
|
8
|
-
humanish selects 32,768 tokens if the exact input's
|
|
9
|
-
|
|
6
|
+
The default is `gpt-6-astra` with high reasoning effort, a separate $3 cap on its
|
|
7
|
+
expected cost and a 600-second timeout. When no output limit is specified,
|
|
8
|
+
humanish selects 32,768 tokens if the exact input's expected cost still fits that
|
|
9
|
+
cap; otherwise it keeps the established 16,384-token allowance. This preserves
|
|
10
10
|
previously admitted studies without increasing their spending limit. To customize it:
|
|
11
11
|
|
|
12
12
|
```yaml
|
|
@@ -22,11 +22,15 @@ review:
|
|
|
22
22
|
|
|
23
23
|
Omitting `review.analysis` uses these defaults. Set `review.analysis: false` to
|
|
24
24
|
run participants without the additional analysis request. An explicit analysis
|
|
25
|
-
mapping using the default OpenAI API provider requires `maxCostUsd`.
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
25
|
+
mapping using the default OpenAI API provider requires `maxCostUsd`. Admission
|
|
26
|
+
compares the expected cost plus a 10% margin with it; it does not limit the
|
|
27
|
+
provider's final bill, and it is separate from participant spending limits. The
|
|
28
|
+
estimate gives two numbers: the expected cost, and the worst case if the analyst
|
|
29
|
+
spends its whole output allowance. `humanish study check` gives the expected cost
|
|
30
|
+
range for the study's participant count, from a run that keeps no evidence to one
|
|
31
|
+
at the evidence limits. Use `analyze --dry-run --max-cost <usd>` on retained
|
|
32
|
+
evidence to inspect the expected cost, the worst case and the selected token
|
|
33
|
+
allowance before deliberately choosing a larger budget. Explicit
|
|
30
34
|
`maxOutputTokens` and `--max-output-tokens` limits are honored exactly. The output allowance
|
|
31
35
|
includes reasoning as well as the report; exhausting it does not produce a usable
|
|
32
36
|
report and never starts an automatic retry. Analysis
|
|
@@ -74,7 +78,7 @@ Default analysis also skips recordings containing only setup or failure records
|
|
|
74
78
|
with no retained participant activity. A desktop startup failure does not start
|
|
75
79
|
an analysis request. The original failure remains visible.
|
|
76
80
|
|
|
77
|
-
CLI live starts disclose the selected analyst
|
|
81
|
+
CLI live starts disclose the selected analyst, its expected cost range and cap, or unknown account dollars before execution.
|
|
78
82
|
`humanish study check <study> --json` and the TUI study screen also expose the
|
|
79
83
|
resolved budget without dispatching analysis. Library callers can inspect
|
|
80
84
|
`resolveAutomaticAnalysis` or `automaticAnalysisBudget` before running.
|
|
@@ -101,11 +105,13 @@ for post-run analysis, and `ok` for the overall request. Failed, cancelled or
|
|
|
101
105
|
unknown analysis produces exit code 2 without discarding the recording. A missing
|
|
102
106
|
`OPENAI_API_KEY` skips default analysis and preserves a successful run exit;
|
|
103
107
|
`automaticAnalysisTrigger: "default"` distinguishes that case in JSON. So does a
|
|
104
|
-
default analysis whose
|
|
105
|
-
skipped with `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
108
|
+
default analysis whose expected cost, plus the margin, is over the default $3 cap:
|
|
109
|
+
it is skipped with `AUTOMATIC_ANALYSIS_ADMISSION_REFUSED` and the run exits 0. For
|
|
110
|
+
any analysis refused for its cost, the job records the expected cost, the worst
|
|
111
|
+
case and the cap, and the CLI, `humanish review` and the Observer give them with
|
|
112
|
+
the `humanish analyze --run <id> --max-cost <n>` command that runs it, `n` being
|
|
113
|
+
the worst case rounded up. A missing key or an over-cap estimate for an explicitly
|
|
114
|
+
configured analysis remains a failed overall request. Every skip is retained for review and does not
|
|
109
115
|
retry automatically. Partial
|
|
110
116
|
findings remain visibly partial; a valid partial result can succeed, while a
|
|
111
117
|
partial result with an analysis error still fails the command. Recorded task
|
|
@@ -119,7 +125,7 @@ outcomes and the deterministic review verdict are never rewritten by analysis.
|
|
|
119
125
|
| none (`null`) | `complete` | Analysis completed |
|
|
120
126
|
| `AUTOMATIC_ANALYSIS_REUSED` | that of the reused result | An earlier analysis of this run was reused |
|
|
121
127
|
| `AUTOMATIC_ANALYSIS_LIMITATIONS` | `partial` | Analysis completed with recorded limitations |
|
|
122
|
-
| `AUTOMATIC_ANALYSIS_ADMISSION_EXCEEDED` | `partial` | Usage passed the
|
|
128
|
+
| `AUTOMATIC_ANALYSIS_ADMISSION_EXCEEDED` | `partial` | Usage passed the worst case or the cap |
|
|
123
129
|
| `AUTOMATIC_ANALYSIS_FAILED` | `failed` or `partial` | Analysis failed, or an unexpected error stopped it |
|
|
124
130
|
| `AUTOMATIC_ANALYSIS_CANCELLED` | `cancelled` | Ctrl-C or the TUI's **Cancel analysis** stopped it |
|
|
125
131
|
| `AUTOMATIC_ANALYSIS_DRY_RUN` | `skipped` | A dry run records no participant evidence |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "humanish",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.114.0",
|
|
4
4
|
"description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"cli",
|
|
@@ -91,7 +91,7 @@
|
|
|
91
91
|
"comms:agentmail:contract": "node --import tsx scripts/agentmail-contract-check.mjs",
|
|
92
92
|
"format": "oxfmt",
|
|
93
93
|
"format:check": "oxfmt --check",
|
|
94
|
-
"lint": "node scripts/check-lint-cap.mjs --max-warnings=
|
|
94
|
+
"lint": "node scripts/check-lint-cap.mjs --max-warnings=439",
|
|
95
95
|
"knip": "knip",
|
|
96
96
|
"prose:check": "node scripts/check-code-prose.mjs",
|
|
97
97
|
"vocabulary:check": "tsx scripts/check-retired-words.ts",
|
package/skills/humanish/SKILL.md
CHANGED
|
@@ -105,6 +105,11 @@ not local inference or the participant's existing conversation. See
|
|
|
105
105
|
[the analysis contract](../../docs/contracts/study-analysis.md) for the current
|
|
106
106
|
CLI/model qualification and setup limits.
|
|
107
107
|
|
|
108
|
+
When post-run analysis is refused for its cost, the run output and `humanish
|
|
109
|
+
review` give its expected cost, its worst case and the command that runs it,
|
|
110
|
+
`humanish analyze --run <id> --max-cost <n>`. Show the user those costs and run
|
|
111
|
+
the command only once they agree to the spend.
|
|
112
|
+
|
|
108
113
|
Do not pass numeric `maxCostUsd`/`maxOutputTokens` or their CLI flags to the
|
|
109
114
|
account branch. It cannot enforce those ceilings and rejects them. Account dollar
|
|
110
115
|
cost remains unknown even when token usage is reported. There is no fallback to
|
|
@@ -190,16 +195,6 @@ Legacy profiles without it retain medium defaults for missing patience and
|
|
|
190
195
|
technical confidence. Avoid contradictory prose and traits; inspect the compiled
|
|
191
196
|
brief with `humanish study show <study> --json` before running.
|
|
192
197
|
|
|
193
|
-
For an autonomous participant study, use a computer-use/local-agent study and write
|
|
194
|
-
its `mission` as a believable situation and desired outcome. Supply fixture facts
|
|
195
|
-
needed to act, but do not supply selectors, click sequences, hidden success rules,
|
|
196
|
-
expected defects or recovery tricks. For example: “Saturday's event needs two
|
|
197
|
-
setup volunteers and one cleanup volunteer. See whether this app helps you
|
|
198
|
-
organize the event and keep people informed.” Do not instruct a participant to
|
|
199
|
-
find a specific UI control merely to make the study succeed. Allow discovery,
|
|
200
|
-
recovery and stopping appropriate to the participant. Keep researcher criteria in
|
|
201
|
-
`tasks[].success`, separate from participant-facing `tasks[].goal`.
|
|
202
|
-
|
|
203
198
|
Scripted regression checks are a different use of scenarios. When the requested
|
|
204
199
|
work is deterministic verification of a known path, a `browser.steps` manifest
|
|
205
200
|
can drive it with explicit selectors and assertions. It does not establish
|
|
@@ -232,6 +227,27 @@ tokens into scenario files.
|
|
|
232
227
|
|
|
233
228
|
## Authoring Studies
|
|
234
229
|
|
|
230
|
+
For an autonomous participant study, write `mission`, participant `instruction`
|
|
231
|
+
and `tasks[].goal` as a believable situation and desired outcome. Supply fixture
|
|
232
|
+
facts needed to act. Leave discovery, recovery and stopping to the participant.
|
|
233
|
+
Keep researcher criteria in `tasks[].success`.
|
|
234
|
+
|
|
235
|
+
| Bad: scripts the controls | Good: gives a situation |
|
|
236
|
+
| -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
237
|
+
| Click "New event", type a title into the field, then press "Save" and open "Settings". | Saturday's event needs two setup volunteers and one cleanup volunteer. See whether this app helps you organize the event and keep people informed. |
|
|
238
|
+
|
|
239
|
+
`humanish study check` and `humanish run` warn about step lists, chains of UI
|
|
240
|
+
actions, quoted control labels, CSS selectors and URLs beyond the subject's entry
|
|
241
|
+
point. They quote up to three matched lines and suggest a situation-style mission.
|
|
242
|
+
The checks apply to computer-use and shared-world studies; scripted-browser and
|
|
243
|
+
terminal studies and `tasks[].success` are excluded. These warnings never refuse
|
|
244
|
+
a run, and the run records them in its bundle.
|
|
245
|
+
|
|
246
|
+
Give each participant a persona with a short, fictional `background` describing
|
|
247
|
+
their relevant experience and situation. A missing persona or background also
|
|
248
|
+
warns. Use `humanish study show <study> --json` to see what the participant receives.
|
|
249
|
+
Do not supply hidden success rules, expected defects or recovery tricks.
|
|
250
|
+
|
|
235
251
|
Write reusable studies as `.yaml` study files (`humanish.study.v3`):
|
|
236
252
|
|
|
237
253
|
```yaml
|