humanish 0.111.4 → 0.113.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +37 -0
- package/README.md +11 -0
- package/TELEMETRY.md +21 -0
- package/dist/actors/codex/restricted-participant-policy.d.ts +21 -1
- package/dist/actors/codex/restricted-participant-policy.js +20 -2
- package/dist/actors/codex/restricted-participant.d.ts +1 -1
- package/dist/actors/codex/restricted-participant.js +2 -1
- package/dist/actors/computer-use/loop/debrief.d.ts +12 -4
- package/dist/actors/computer-use/loop/debrief.js +57 -20
- package/dist/actors/computer-use/loop/ending.d.ts +5 -2
- package/dist/actors/computer-use/loop/impressions.d.ts +14 -0
- package/dist/actors/computer-use/loop/impressions.js +40 -0
- package/dist/actors/computer-use/loop/session.d.ts +3 -1
- package/dist/actors/computer-use/loop/session.js +2 -0
- package/dist/actors/computer-use/loop/trace.js +2 -0
- package/dist/actors/computer-use/loop/types.d.ts +8 -1
- package/dist/actors/computer-use/loop.d.ts +2 -1
- package/dist/actors/computer-use/loop.js +15 -3
- package/dist/actors/computer-use/openai-conversation.d.ts +96 -0
- package/dist/actors/computer-use/openai-conversation.js +238 -0
- package/dist/actors/computer-use/openai-provider.js +38 -48
- package/dist/actors/contract.d.ts +45 -3
- package/dist/actors/contract.js +13 -0
- package/dist/analysis/diagnostics.js +7 -12
- package/dist/analysis/evidence-sources.d.ts +3 -1
- package/dist/analysis/evidence-sources.js +10 -1
- package/dist/analysis/evidence.js +2 -2
- package/dist/analysis/execute.d.ts +1 -1
- package/dist/analysis/execute.js +37 -5
- package/dist/analysis/service.js +7 -1
- package/dist/analysis/types.d.ts +23 -0
- package/dist/analysis/validation.d.ts +80 -35
- package/dist/analysis/validation.js +58 -11
- package/dist/cli/commands/runs.js +37 -20
- package/dist/cli/commands/setup.js +95 -6
- package/dist/cli/commands/study-format.js +7 -4
- package/dist/cli/commands/study.js +1 -1
- package/dist/cli/commands/tui.js +1 -1
- package/dist/cli/doctor.d.ts +2 -0
- package/dist/cli/doctor.js +18 -0
- package/dist/cli/findings.d.ts +34 -3
- package/dist/cli/findings.js +136 -18
- package/dist/cli/io.d.ts +4 -1
- package/dist/cli/io.js +3 -2
- package/dist/cli/program.js +35 -1
- package/dist/cli/secret-prompt.d.ts +5 -1
- package/dist/cli/secret-prompt.js +6 -2
- package/dist/cli/update-check-worker.d.ts +1 -0
- package/dist/cli/update-check-worker.js +5 -0
- package/dist/cli/update-check.d.ts +46 -0
- package/dist/cli/update-check.js +203 -0
- package/dist/feedback/draft.js +32 -4
- package/dist/keys/key-resolution.js +2 -0
- package/dist/keys/key-status.d.ts +37 -0
- package/dist/keys/key-status.js +68 -0
- package/dist/observer/run-routes.js +8 -4
- package/dist/observer-app.html +4 -4
- package/dist/routes/computer-use/bundle.d.ts +1 -1
- package/dist/routes/computer-use/bundle.js +6 -16
- package/dist/routes/computer-use/fanout-bundle.js +40 -37
- package/dist/routes/computer-use/fanout-records.d.ts +2 -0
- package/dist/routes/computer-use/fanout-records.js +18 -26
- package/dist/routes/computer-use/local-vm.js +9 -2
- package/dist/routes/computer-use/participant-desktop.d.ts +6 -0
- package/dist/routes/computer-use/participant-execution.js +13 -12
- package/dist/routes/computer-use/participant-facts.d.ts +2 -2
- package/dist/routes/computer-use/participant-facts.js +10 -1
- package/dist/routes/computer-use/rerun-selection.js +23 -21
- package/dist/routes/computer-use/result.js +10 -17
- package/dist/routes/computer-use/self-report.js +7 -1
- package/dist/routes/computer-use/setup.js +14 -0
- package/dist/routes/computer-use/single-bundle.d.ts +2 -0
- package/dist/routes/computer-use/single-bundle.js +81 -37
- package/dist/routes/computer-use/types.d.ts +4 -1
- package/dist/routes/scripted/bundle.d.ts +1 -1
- package/dist/routes/scripted/bundle.js +9 -19
- package/dist/routes/scripted/surface-records.d.ts +0 -1
- package/dist/routes/scripted/surface-records.js +27 -13
- package/dist/routes/shared-world/bundle-records.d.ts +2 -2
- package/dist/routes/shared-world/bundle-records.js +24 -30
- package/dist/routes/shared-world/bundle.d.ts +1 -1
- package/dist/routes/shared-world/bundle.js +71 -79
- package/dist/routes/shared-world/participant-specs.d.ts +1 -2
- package/dist/routes/shared-world/participant-specs.js +0 -8
- package/dist/routes/shared-world/result.js +8 -12
- package/dist/routes/terminal/bundle.d.ts +1 -1
- package/dist/routes/terminal/bundle.js +10 -19
- package/dist/routes/terminal/dry-run.js +1 -0
- package/dist/routes/terminal/live-sandbox.js +9 -2
- package/dist/routes/terminal/plan.js +1 -0
- package/dist/routes/terminal/runtime.d.ts +1 -0
- package/dist/routes/terminal/runtime.js +1 -0
- package/dist/routes/terminal/session.js +1 -0
- package/dist/run/costs.js +9 -20
- package/dist/run/detail.d.ts +1 -1
- package/dist/run/detail.js +13 -4
- package/dist/run/display.d.ts +2 -2
- package/dist/run/display.js +15 -18
- package/dist/run/dry-run.js +10 -25
- package/dist/run/judge.d.ts +24 -0
- package/dist/run/judge.js +54 -0
- package/dist/run/liveness.d.ts +18 -0
- package/dist/run/liveness.js +24 -0
- package/dist/run/outcomes.d.ts +8 -1
- package/dist/run/outcomes.js +58 -12
- package/dist/run/participant-caption.d.ts +31 -0
- package/dist/run/participant-caption.js +60 -0
- package/dist/run/participant-records.d.ts +4 -2
- package/dist/run/participant-records.js +6 -0
- package/dist/run/review-markdown.d.ts +11 -0
- package/dist/run/review-markdown.js +22 -0
- package/dist/run/run-cost.d.ts +7 -0
- package/dist/run/run-cost.js +25 -5
- package/dist/run/run-index.d.ts +4 -50
- package/dist/run/run-index.js +57 -87
- package/dist/run/run.d.ts +2 -2
- package/dist/run/run.js +5 -3
- package/dist/run/stats.js +2 -12
- package/dist/run/transient-comms-secrets.d.ts +8 -3
- package/dist/run/transient-comms-secrets.js +21 -1
- package/dist/run-study.js +17 -1
- package/dist/study/discover.js +6 -1
- package/dist/study/doctor.d.ts +2 -0
- package/dist/study/doctor.js +45 -7
- package/dist/study/init-templates.js +10 -5
- package/dist/study/keys.d.ts +1 -0
- package/dist/study/keys.js +1 -1
- package/dist/study/parse/execution.js +7 -1
- package/dist/study/plan-types.d.ts +1 -0
- package/dist/study/plan.js +3 -2
- package/dist/study/preflight-result.d.ts +1 -0
- package/dist/study/preflight-result.js +1 -1
- package/dist/study/preflight.d.ts +2 -0
- package/dist/study/preflight.js +54 -14
- package/dist/study/types.d.ts +4 -1
- package/dist/study/warnings.d.ts +7 -1
- package/dist/study/warnings.js +82 -4
- package/dist/substrates/local/capacity.d.ts +75 -0
- package/dist/substrates/local/capacity.js +143 -0
- package/dist/substrates/local/firecracker-desktop.d.ts +8 -0
- package/dist/substrates/local/firecracker-desktop.js +34 -3
- package/dist/substrates/local/runtime-host.d.ts +16 -3
- package/dist/substrates/local/runtime-host.js +49 -4
- package/dist/substrates/local/runtime.d.ts +10 -0
- package/dist/substrates/local/runtime.js +44 -5
- package/dist/tui-app.js +75 -75
- package/dist/verify/liveness.d.ts +2 -2
- package/dist/verify/liveness.js +7 -4
- package/docs/architecture/actor-contract.md +20 -6
- package/docs/architecture/terminal-product-route.md +14 -0
- package/docs/contracts/feedback.md +8 -1
- package/docs/contracts/run-bundle.md +10 -0
- package/docs/contracts/schemas.md +65 -34
- package/docs/contracts/study-analysis.md +86 -22
- package/package.json +2 -1
- package/skills/humanish/SKILL.md +21 -10
- package/dist/actors/computer-use/openai-requests.d.ts +0 -62
- package/dist/actors/computer-use/openai-requests.js +0 -144
|
@@ -14,8 +14,8 @@ export interface UnfinishedRun {
|
|
|
14
14
|
}
|
|
15
15
|
/**
|
|
16
16
|
* One warning, and the unfinished-run facts, when the run is not finished, by the run index's
|
|
17
|
-
* rule: its
|
|
18
|
-
*
|
|
17
|
+
* rule: its bundle outcome first, then usable status, then the bundle's simulations. Nothing
|
|
18
|
+
* for a finished run.
|
|
19
19
|
*/
|
|
20
20
|
export declare function runNotFinished(runPaths: PreparedRunArtifactPaths, bundle: RunBundle, nowMs?: number): Promise<{
|
|
21
21
|
warnings: string[];
|
package/dist/verify/liveness.js
CHANGED
|
@@ -7,7 +7,7 @@ import path from "node:path";
|
|
|
7
7
|
import { readRunJsonIfExists, readRunTextIfExists } from "../run/locate.js";
|
|
8
8
|
import { isRecord } from "../run/type-guards.js";
|
|
9
9
|
import { RECLAIM_RECEIPT_ARTIFACT } from "../run/reclaim.js";
|
|
10
|
-
import { runLiveness } from "../run/
|
|
10
|
+
import { runLiveness } from "../run/liveness.js";
|
|
11
11
|
import { parseSandboxReceipts, SANDBOX_RECEIPTS_ARTIFACT } from "../run/sandbox-receipts.js";
|
|
12
12
|
import { RUN_STATUS_FILE } from "../run/status.js";
|
|
13
13
|
import { plural } from "../run/text.js";
|
|
@@ -17,8 +17,8 @@ const RUN_NOT_FINISHED = "RUN_NOT_FINISHED";
|
|
|
17
17
|
const GONE_STATES = new Set(["killed", "already-gone"]);
|
|
18
18
|
/**
|
|
19
19
|
* One warning, and the unfinished-run facts, when the run is not finished, by the run index's
|
|
20
|
-
* rule: its
|
|
21
|
-
*
|
|
20
|
+
* rule: its bundle outcome first, then usable status, then the bundle's simulations. Nothing
|
|
21
|
+
* for a finished run.
|
|
22
22
|
*/
|
|
23
23
|
export async function runNotFinished(runPaths, bundle, nowMs = Date.now()) {
|
|
24
24
|
const runId = path.basename(runPaths.absoluteRunRoot);
|
|
@@ -27,7 +27,10 @@ export async function runNotFinished(runPaths, bundle, nowMs = Date.now()) {
|
|
|
27
27
|
if (liveness === "finished")
|
|
28
28
|
return { warnings: [] };
|
|
29
29
|
const seen = [];
|
|
30
|
-
if (
|
|
30
|
+
if (bundle.outcome?.state === "interrupted") {
|
|
31
|
+
seen.push(`run.json records interruption by ${bundle.outcome.signal} at ${bundle.outcome.at}`);
|
|
32
|
+
}
|
|
33
|
+
else if (record === undefined) {
|
|
31
34
|
const running = bundle.simulations.filter((simulation) => simulation.status === "running");
|
|
32
35
|
seen.push(`the run has no usable ${RUN_STATUS_FILE} and ${running.length} of ${bundle.simulations.length} simulations are still running`);
|
|
33
36
|
}
|
|
@@ -350,22 +350,36 @@ this paragraph is the record of that decision.
|
|
|
350
350
|
|
|
351
351
|
`ComputerUseProvider.debrief` is an optional read-only request after a structured
|
|
352
352
|
`stopWhen` or dwell stop. It returns a `ComputerUseTurn` with `closingReport` containing
|
|
353
|
-
`summary` and `
|
|
353
|
+
`summary`, `frictionReports` and `impressions`; an empty friction or impressions list is valid,
|
|
354
|
+
and a report without `impressions` (an older provider) is still accepted. The loop rejects
|
|
354
355
|
actions, pending safety checks, and invalid report shapes. It redacts accepted
|
|
355
356
|
reports, records them in `ActorTrace.debrief`, and projects one readable message
|
|
356
|
-
without invoking action or communication callbacks.
|
|
357
|
+
without invoking action or communication callbacks. Each impression becomes its own message
|
|
358
|
+
and goes to `ActorTrace.impressions`; a session without them records why in the same field.
|
|
359
|
+
A turn that ends the session itself may carry `closingReport` too, and the loop keeps its
|
|
360
|
+
impressions the same way without a second request.
|
|
361
|
+
|
|
362
|
+
`ComputerUseProvider.requestImpressions` is the optional request after the participant ended
|
|
363
|
+
the session itself without impressions. It asks for impressions only, returns them as
|
|
364
|
+
`impressions` on the turn, and follows the debrief's rules and guards. The loop records it as
|
|
365
|
+
`ActorTrace.debrief` with `trigger: "participant_end"` and no report, so the participant's
|
|
366
|
+
reason, outcome and friction come from its final message as before. A failed, timed-out or
|
|
367
|
+
cut-off request leaves `ActorTrace.impressions` as `not_collected` with the reason.
|
|
357
368
|
|
|
358
369
|
This request uses the final observation and retained provider history. It does
|
|
359
370
|
not expose the hidden stop criterion or change the original task outcome. It is
|
|
360
|
-
skipped before any participant turn, after natural completion
|
|
361
|
-
support, or without remaining time and known budget. An attempted request's
|
|
371
|
+
skipped before any participant turn, after natural completion (apart from the impressions-only
|
|
372
|
+
request), without provider support, or without remaining time and known budget. An attempted request's
|
|
362
373
|
unreported usage remains an unknown cost line. `counts.debriefCalls` is separate
|
|
363
374
|
from interaction turns, and reported usage contributes to aggregate cost.
|
|
364
375
|
|
|
365
376
|
The OpenAI implementation makes one request with tools disabled and structured
|
|
366
|
-
output; no HTTP or policy retries.
|
|
377
|
+
output; no HTTP or policy retries. Its output limit is 3072 tokens, or the declared
|
|
378
|
+
`maxOutputTokens` when lower. Stateless/ZDR mode does not offer retrospective
|
|
367
379
|
reporting: its client-carried conversation summarizes its oldest turns past a token
|
|
368
|
-
budget, so it may not hold the whole session a retrospective report needs.
|
|
380
|
+
budget, so it may not hold the whole session a retrospective report needs. It does offer
|
|
381
|
+
the impressions-only request, which reads the carried conversation: its newest
|
|
382
|
+
exchanges whole and earlier turns as a text note. The
|
|
369
383
|
[paired live receipt](https://github.com/danielgwilson/humanish/blob/main/docs/evidence/computer-use/structured-closing-report-2026-09-05.md)
|
|
370
384
|
records both report recovery and control failures in the separate legacy parser.
|
|
371
385
|
|
|
@@ -84,6 +84,20 @@ at the base tier, with cached input at the cached rate, and says so with
|
|
|
84
84
|
checked against `caps.maxUsd` keeps the provider line unmeasured, and the studied
|
|
85
85
|
product's no-spend boundary still needs separate interpretation.
|
|
86
86
|
|
|
87
|
+
## Participant telemetry
|
|
88
|
+
|
|
89
|
+
Terminal participant commands and product setup receive `DO_NOT_TRACK=1` by default.
|
|
90
|
+
Their child processes inherit it, so studied CLIs that honor the convention can exclude
|
|
91
|
+
synthetic usage. To study the product's telemetry behavior, set
|
|
92
|
+
`execution.terminal.doNotTrack: false`; humanish then leaves the variable unset.
|
|
93
|
+
Only booleans are accepted, and `true` or an omitted field enables the default.
|
|
94
|
+
|
|
95
|
+
The resolved `doNotTrack` boolean is recorded with `humanish.actor-runtime.v1` in
|
|
96
|
+
`actor.json`, `streams[].actor.runtime` in `run.json`, and `runtime` in
|
|
97
|
+
`terminal-ledgers.json`. A dry run records it in the JSON message of its
|
|
98
|
+
`terminal-lab.runtime.declared` event. Older bundles omit it. This records the
|
|
99
|
+
command environment policy; it does not prove that the studied CLI honors the variable.
|
|
100
|
+
|
|
87
101
|
## Runtime auth: raw-key placement and remaining provider access
|
|
88
102
|
|
|
89
103
|
`execution.runtimeAuth: openai-egress` is the default: a study that declares no
|
|
@@ -66,7 +66,9 @@ Listing does not create a draft or check public-sharing eligibility.
|
|
|
66
66
|
|
|
67
67
|
For completed computer-use and shared-world sessions, participant-reported
|
|
68
68
|
friction comes from the redacted closing report, a completed debrief's
|
|
69
|
-
`frictionReports`, and observed-report clauses in earlier messages.
|
|
69
|
+
`frictionReports`, and observed-report clauses in earlier messages. The
|
|
70
|
+
participant's closing impressions (`ActorTrace.impressions`) are opinions about
|
|
71
|
+
the product and never count as friction.
|
|
70
72
|
Matching `stopWhen` or ending a dwell window does not discard an earlier
|
|
71
73
|
report or change the successful completion verdict. Exact repeated reports
|
|
72
74
|
appear once in the participant's candidate. Harness notices, reasoning, and observed
|
|
@@ -84,6 +86,11 @@ was not repeated in the message trace.
|
|
|
84
86
|
Builds structured feedback from the first usable run candidate, or the candidate
|
|
85
87
|
selected with `--candidate`. With no candidate, a live run produces a summary of
|
|
86
88
|
its recorded review; that fallback does not add a finding to the candidate list.
|
|
89
|
+
With `--analysis` and `--finding`, the draft is built from that analysis finding
|
|
90
|
+
instead. When the finding's observations cite a participant's closing impression,
|
|
91
|
+
`actual` quotes it after the observations as
|
|
92
|
+
`Participant <id> said at the end (<kind>): <text>`. An impression is quoted only
|
|
93
|
+
when the cited evidence entry carries its text whole; uncited impressions stay out.
|
|
87
94
|
A dry run tests no product behavior, so `draft`, `verify`, `issue` and
|
|
88
95
|
`issue-url` refuse it with `HUMANISH_FEEDBACK_REQUIRES_LIVE_RUN`. All drafts
|
|
89
96
|
require a verified `share_ready` live run and are written under the run bundle,
|
|
@@ -6,6 +6,13 @@
|
|
|
6
6
|
source of truth. This page explains the stable public fields and the extension rules; it does not
|
|
7
7
|
version the runtime separately.
|
|
8
8
|
|
|
9
|
+
Terminal runs record the resolved telemetry setting as `doNotTrack` in
|
|
10
|
+
`streams[].actor.runtime` (`humanish.actor-runtime.v1`), also retained in `actor.json`
|
|
11
|
+
and `terminal-ledgers.json`. `true` means participant and product setup commands receive
|
|
12
|
+
`DO_NOT_TRACK=1`; `false` means humanish leaves it unset. Dry runs record the same boolean
|
|
13
|
+
in the JSON message of the `terminal-lab.runtime.declared` event. Older bundles omit it.
|
|
14
|
+
The setting does not attest that the studied CLI honors the variable.
|
|
15
|
+
|
|
9
16
|
## Purpose
|
|
10
17
|
|
|
11
18
|
A run bundle is the durable evidence packet for one harness run. It should be
|
|
@@ -154,6 +161,9 @@ writing it, and on runs recorded before the field existed. For those, readers
|
|
|
154
161
|
take liveness from `status.json` and, for an older finished run, its `ok` and
|
|
155
162
|
`execution` from `status.json`'s `outcome`. That record is the only copy of an
|
|
156
163
|
older run's `ok`; for a run with `outcome` in `run.json`, `status.json` copies it.
|
|
164
|
+
The bundle's outcome wins even if the process stopped before updating `status.json`.
|
|
165
|
+
Without an outcome or a usable status record naming the run, a bundle with a simulation still
|
|
166
|
+
`running` reads as interrupted. A fresh status record keeps an in-progress bundle running.
|
|
157
167
|
|
|
158
168
|
Every surface that shows whether a run passed calls `runDisplay`
|
|
159
169
|
(`src/run/display.ts`) with the run's liveness, verdict and `ok`: the Observer
|
|
@@ -24,38 +24,38 @@ workflow without leaking private upstream truth into core.
|
|
|
24
24
|
|
|
25
25
|
## Contract Index
|
|
26
26
|
|
|
27
|
-
| Contract | Schema
|
|
28
|
-
| --------------------------------- |
|
|
29
|
-
| Run bundle | `humanish.run-bundle.v1`
|
|
30
|
-
| Adapter | `humanish.adapter.v1`
|
|
31
|
-
| Study file | `humanish.study.v3`
|
|
32
|
-
| Persona | `humanish.persona.v1`
|
|
33
|
-
| Scenario | `humanish.scenario.v1`
|
|
34
|
-
| Actor trace | `humanish.actor-trace.v1`
|
|
35
|
-
| Substrate | reserved (never shipped)
|
|
36
|
-
| Evidence stream | reserved (streams live inside the run bundle)
|
|
37
|
-
| Review | `humanish.review.v1`
|
|
38
|
-
| Study analysis | `humanish.study-analysis.v1`
|
|
39
|
-
| Study analysis correction | `humanish.study-analysis-correction.v1`
|
|
40
|
-
| Analysis findings view | `humanish.analysis-findings.v1` (`review --json` field `analysis`; read-only projection of the selected analysis)
|
|
41
|
-
| Analysis execution receipt | `humanish.analysis-execution.v1`
|
|
42
|
-
| Analysis execution start | `humanish.analysis-execution-start.v1`
|
|
43
|
-
| Verification | `humanish.verify-result.v1`
|
|
44
|
-
| Policy | `humanish.policy.v1` (fixture-only; not engine-validated)
|
|
45
|
-
| Feedback | `humanish.feedback.v1`
|
|
46
|
-
| Terminal cost ledger | `humanish.terminal-cost-ledger.v1`
|
|
47
|
-
| Terminal no-spend proof | `humanish.terminal-no-spend-proof.v1`
|
|
48
|
-
| Pricing (operator-editable rates) | `humanish.pricing.v1` (`src/run/pricing.ts`; dated per-model + E2B desktop rates)
|
|
49
|
-
| Run cost summary | `humanish.run-cost-summary.v1` (additive `RunBundle.cost`; estimate, never a charge)
|
|
50
|
-
| Estimated actor cost | `humanish.actor-estimated-cost.v1` (additive `ActorTrace.estimatedCost`)
|
|
51
|
-
| Model settings | `humanish.model-settings.v1` (additive `ActorTrace.modelSettings`; the reasoning effort the request carried)
|
|
52
|
-
| Affordance use | `humanish.affordance-use.v1` (additive `ActorTrace.affordanceUse`; per-class counts of the routes an actor took)
|
|
53
|
-
| Adapter score | `humanish.adapter-score.v1` (`RunBundle.adapterScore`; namespaced; route-specific acceptance semantics)
|
|
54
|
-
| Adapter artifact | `humanish.adapter-artifact.v1` (`RunBundle.adapterArtifacts[]`; namespaced; local relative proof references)
|
|
55
|
-
| Shared-world evidence | `humanish.shared-world.v1` (additive `RunBundle.sharedWorld` + `RunBundle.attributionClass`; `topologyMode: sequential \| concurrent`)
|
|
56
|
-
| Comms thread | `humanish.comms-thread.v1` (off-app email/SMS the app sent, captured; a `kind: log` run-dir artifact of digests only: from/to/subject/link digests + one OTP count, never raw)
|
|
57
|
-
| Serve result | `humanish.serve-result.v1` (`src/observer/serve.ts` is authoritative)
|
|
58
|
-
| Serve control plane | reserved (`/_humanish/api/*` answers `501` `HUMANISH_SERVE_CONTROL_PLANE_DISABLED` in v1)
|
|
27
|
+
| Contract | Schema | Public-safe fixture |
|
|
28
|
+
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
|
|
29
|
+
| Run bundle | `humanish.run-bundle.v1` | `synthetic-run-bundle` |
|
|
30
|
+
| Adapter | `humanish.adapter.v1` | `synthetic-cli-adapter` |
|
|
31
|
+
| Study file | `humanish.study.v3` | `first-run` |
|
|
32
|
+
| Persona | `humanish.persona.v1` | `synthetic-maintainer` |
|
|
33
|
+
| Scenario | `humanish.scenario.v1` | `first-run-smoke` |
|
|
34
|
+
| Actor trace | `humanish.actor-trace.v1` | `synthetic-actor-trace` |
|
|
35
|
+
| Substrate | reserved (never shipped) | none |
|
|
36
|
+
| Evidence stream | reserved (streams live inside the run bundle) | see [`run-bundle.md`](run-bundle.md) |
|
|
37
|
+
| Review | `humanish.review.v1` | `contract-proof-review` |
|
|
38
|
+
| Study analysis | `humanish.study-analysis.v1` (prompt `study-evidence-7` adds `findings[].headline`, `findings[].experience` and `designFindings[]`; older artifacts omit them; `study-evidence-8` adds participant impressions to the instructions, with no field change) | see [study analysis](study-analysis.md) and synthetic analysis fixtures in `tests/` |
|
|
39
|
+
| Study analysis correction | `humanish.study-analysis-correction.v1` | see [study analysis](study-analysis.md#human-review-and-sharing) |
|
|
40
|
+
| Analysis findings view | `humanish.analysis-findings.v1` (`review --json` field `analysis`; read-only projection of the selected analysis) | see [study analysis](study-analysis.md#reading-findings) |
|
|
41
|
+
| Analysis execution receipt | `humanish.analysis-execution.v1` | see [study analysis](study-analysis.md#durable-records) |
|
|
42
|
+
| Analysis execution start | `humanish.analysis-execution-start.v1` | see [study analysis](study-analysis.md#durable-records) |
|
|
43
|
+
| Verification | `humanish.verify-result.v1` | `five-check-verify` |
|
|
44
|
+
| Policy | `humanish.policy.v1` (fixture-only; not engine-validated) | `public-safety-policy` |
|
|
45
|
+
| Feedback | `humanish.feedback.v1` | `public-safe-feedback` |
|
|
46
|
+
| Terminal cost ledger | `humanish.terminal-cost-ledger.v1` | see Terminal Cost Ledger below |
|
|
47
|
+
| Terminal no-spend proof | `humanish.terminal-no-spend-proof.v1` | see Terminal Cost Ledger below |
|
|
48
|
+
| Pricing (operator-editable rates) | `humanish.pricing.v1` (`src/run/pricing.ts`; dated per-model + E2B desktop rates) | see Run Cost Summary And Estimated Actor Cost below |
|
|
49
|
+
| Run cost summary | `humanish.run-cost-summary.v1` (additive `RunBundle.cost`; estimate, never a charge) | see Run Cost Summary And Estimated Actor Cost below |
|
|
50
|
+
| Estimated actor cost | `humanish.actor-estimated-cost.v1` (additive `ActorTrace.estimatedCost`) | see Run Cost Summary And Estimated Actor Cost below |
|
|
51
|
+
| Model settings | `humanish.model-settings.v1` (additive `ActorTrace.modelSettings`; the reasoning effort the request carried) | see Actor Trace below |
|
|
52
|
+
| Affordance use | `humanish.affordance-use.v1` (additive `ActorTrace.affordanceUse`; per-class counts of the routes an actor took) | see Affordance Use below |
|
|
53
|
+
| Adapter score | `humanish.adapter-score.v1` (`RunBundle.adapterScore`; namespaced; route-specific acceptance semantics) | see Product-Adapter Extension Seam below |
|
|
54
|
+
| Adapter artifact | `humanish.adapter-artifact.v1` (`RunBundle.adapterArtifacts[]`; namespaced; local relative proof references) | see Product-Adapter Extension Seam below |
|
|
55
|
+
| Shared-world evidence | `humanish.shared-world.v1` (additive `RunBundle.sharedWorld` + `RunBundle.attributionClass`; `topologyMode: sequential \| concurrent`) | see Shared-World Evidence below |
|
|
56
|
+
| Comms thread | `humanish.comms-thread.v1` (off-app email/SMS the app sent, captured; a `kind: log` run-dir artifact of digests only: from/to/subject/link digests + one OTP count, never raw) | see `comms` under Study File |
|
|
57
|
+
| Serve result | `humanish.serve-result.v1` (`src/observer/serve.ts` is authoritative) | none (command result envelope; see Serve Result below) |
|
|
58
|
+
| Serve control plane | reserved (`/_humanish/api/*` answers `501` `HUMANISH_SERVE_CONTROL_PLANE_DISABLED` in v1) | none |
|
|
59
59
|
|
|
60
60
|
## Study File
|
|
61
61
|
|
|
@@ -296,6 +296,14 @@ instruction?, target?, entry?, host?, reasoningEffort?, stopWhen?, dwell? }` bec
|
|
|
296
296
|
metadata; with `openai-egress` it stays in the host-side E2B header transform
|
|
297
297
|
set at sandbox creation, and the agent gets a placeholder. A dry-run neither
|
|
298
298
|
reads nor injects it;
|
|
299
|
+
- `execution.terminal.doNotTrack`: boolean, default `true`. Terminal participant and product
|
|
300
|
+
setup commands receive `DO_NOT_TRACK=1`. Set `false` to leave that variable unset when
|
|
301
|
+
studying the product's telemetry behavior. Non-boolean values are refused at parse.
|
|
302
|
+
The resolved boolean is `doNotTrack` in `humanish.actor-runtime.v1`: `actor.json`,
|
|
303
|
+
`streams[].actor.runtime` in `run.json`, and `runtime` in `terminal-ledgers.json` for live
|
|
304
|
+
runs; the JSON message of the `terminal-lab.runtime.declared` event for dry runs.
|
|
305
|
+
Older bundles omit this field. It records the command environment policy, not whether
|
|
306
|
+
the studied CLI honored it;
|
|
299
307
|
- `mode`: `dry-run` (a synthetic bundle, no spend; the default) or `live`. On the
|
|
300
308
|
scripted route `live` gates real browser actuation against the declared app.
|
|
301
309
|
Provider spend stays $0 there by mechanism (no model runs);
|
|
@@ -962,8 +970,9 @@ Core-owned fields:
|
|
|
962
970
|
- optional `modelSettings.maxOutputTokens`: the declared positive integer
|
|
963
971
|
`actor.maxOutputTokens`, passed to every first-party OpenAI CUA response
|
|
964
972
|
request, including reasoning output. Absent when undeclared. The first request
|
|
965
|
-
of a session
|
|
966
|
-
|
|
973
|
+
of a session uses `min(maxOutputTokens, 1024)`, or 1024 when undeclared, and
|
|
974
|
+
closing requests use `min(maxOutputTokens, 3072)`, or 3072 when undeclared; the
|
|
975
|
+
first request also carries the opening screenshot as an
|
|
967
976
|
`input_image` after the instructions. Unsupported routes,
|
|
968
977
|
custom provider/session hooks, and per-participant overrides fail before allocation.
|
|
969
978
|
This is an output-token limit, not an input-token, request-count or billing cap.
|
|
@@ -1006,6 +1015,28 @@ Core-owned fields:
|
|
|
1006
1015
|
providers and on earlier bundles
|
|
1007
1016
|
- optional `affordanceUse` (`humanish.affordance-use.v1`): which kind of route this
|
|
1008
1017
|
actor took (see Affordance Use below)
|
|
1018
|
+
- optional `impressions`: what a computer-use participant said about the product
|
|
1019
|
+
at the end of its session. `status: collected` carries `items[]`, each a
|
|
1020
|
+
`kind` (`unclear`, `unfinished`, `untrustworthy`, `liked`, `missing`,
|
|
1021
|
+
`unlike_my_work`), a redacted first-person `text` of at most 500 characters
|
|
1022
|
+
and the `messageId` of the trace `message` that quotes it as
|
|
1023
|
+
`Impression (<kind>): <text>`. They are not counted in `counts.messages`. Up
|
|
1024
|
+
to six come from the closing account: a completed debrief's report, the
|
|
1025
|
+
participant's own final account when its provider writes a structured one
|
|
1026
|
+
(the Codex participant), or, when the participant ended the session itself
|
|
1027
|
+
without them, one impressions-only request (OpenAI computer-use, threaded or
|
|
1028
|
+
`explicit_context`). That request is recorded as `debrief` with `trigger:
|
|
1029
|
+
participant_end` and never has a `report` or `messageId`; it leaves the
|
|
1030
|
+
participant's `reason`, outcome and friction as they were. An empty `items`
|
|
1031
|
+
list means the participant named none. `status: not_collected` carries a
|
|
1032
|
+
`reason`: the closing report or impressions request was skipped or failed
|
|
1033
|
+
(including a reply cut off by the output limit), the participant ended the
|
|
1034
|
+
session without a structured closing account and its provider cannot ask for
|
|
1035
|
+
impressions (the Claude local agent), the closing account had no
|
|
1036
|
+
impressions, or the session stopped before a closing account. Every
|
|
1037
|
+
computer-use loop trace records one; absent on other routes and on earlier
|
|
1038
|
+
bundles. `debrief.report` keeps only `summary` and `frictionReports`.
|
|
1039
|
+
Impression messages do not count as participant-reported friction
|
|
1009
1040
|
- optional `estimatedCost` (`humanish.actor-estimated-cost.v1`): a token-derived
|
|
1010
1041
|
cost estimate for this participant (see Run Cost Summary And Estimated Actor Cost).
|
|
1011
1042
|
It is deliberately a different field from `tokenUsage.costUsd`: a bare
|
|
@@ -161,6 +161,42 @@ thought. Older reports may omit this field and remain readable. The disclosure
|
|
|
161
161
|
is model-generated assessment, not an independent completeness audit or a human
|
|
162
162
|
reviewer annotation.
|
|
163
163
|
|
|
164
|
+
## Headlines and design findings
|
|
165
|
+
|
|
166
|
+
The `study-evidence-7` prompt writes each finding twice. `headline` is one plain sentence about what
|
|
167
|
+
happened, for a designer, product manager or developer who did not watch the session.
|
|
168
|
+
`experience` is one to three plain sentences: what the person tried, what got in their way, and
|
|
169
|
+
how it seemed to feel. Neither names participant or evidence IDs. A concern known only from what
|
|
170
|
+
participants said still reads as something they said. `title`, `summary` and `observations` keep
|
|
171
|
+
their evidence-level wording, and validation applies to them as before.
|
|
172
|
+
|
|
173
|
+
The same prompt adds `designFindings`: problems a product designer would notice in the supplied
|
|
174
|
+
captures, whether or not a participant mentioned them. Each has an `id` (`D1`, `D2`, ...), a
|
|
175
|
+
`headline`, the `screen` in plain words, what a designer would `notice`, `whyItMatters` to a
|
|
176
|
+
person using the product, one `suggestion`, a `severity` (`major` when it misleads or blocks,
|
|
177
|
+
`moderate` when it slows or confuses, `minor` when it is polish), a `confidence`,
|
|
178
|
+
`seenByStreamIds` and `evidenceIds`. Validation rejects a design finding that cites no retained
|
|
179
|
+
capture, cites evidence outside the packet, repeats an ID, or lists a participant in
|
|
180
|
+
`seenByStreamIds` without a cited capture of that participant. An empty list means the review found no design
|
|
181
|
+
problem in the captures. The model judges legibility and size from images alone; it does not
|
|
182
|
+
measure text or control sizes.
|
|
183
|
+
|
|
184
|
+
New provider responses must include `headline` and `experience` on every finding and a
|
|
185
|
+
`designFindings` list. A stored artifact with prompt `study-evidence-7` or later must carry them
|
|
186
|
+
too. Artifacts from earlier prompts load without them and render by title and summary. The
|
|
187
|
+
Observer applies the same prompt-version rules and the same field limits when it reads a saved
|
|
188
|
+
analysis, and treats an analysis that breaks them as unavailable.
|
|
189
|
+
|
|
190
|
+
The narrative scrub removes a run's known transient values, such as a received one-time code or
|
|
191
|
+
link, from every generated text field. It finds each value as written and in its encoded forms:
|
|
192
|
+
percent-encoded, JSON-escaped, base64, base64url and hex, and where escapes split it. A field
|
|
193
|
+
with a value is stored decoded with the value replaced; a field without one keeps its spelling. A
|
|
194
|
+
known value in any of these forms in an ID, enum or reference refuses the response.
|
|
195
|
+
|
|
196
|
+
When a reviewer amends a finding, its headline and experience describe the replaced claim.
|
|
197
|
+
Text output, the Observer and feedback drafts show the reviewer's claim in their place and say the
|
|
198
|
+
finding was corrected; the stored analysis keeps the original.
|
|
199
|
+
|
|
164
200
|
Every observation cites packet-local evidence IDs. The model cannot choose a
|
|
165
201
|
filesystem path or fetch another resource. Validation checks participant
|
|
166
202
|
membership, unique counts, quote fidelity, evidence type and reference
|
|
@@ -203,7 +239,7 @@ feedback quotes cite evidence IDs that the run's own `analysis.json` resolves.
|
|
|
203
239
|
Participant reviews cite distinct packet-local evidence IDs belonging only to
|
|
204
240
|
that participant's stream; source event IDs are not citation IDs. Shared
|
|
205
241
|
interactions can cite multiple included participants in findings and concern
|
|
206
|
-
reviews under the existing exposure rules. The `study-evidence-6` prompt
|
|
242
|
+
reviews under the existing exposure rules. The `study-evidence-6` prompt made
|
|
207
243
|
this distinction explicit without changing the validator or historical reports.
|
|
208
244
|
|
|
209
245
|
Elapsed replay time starts at the first retained capture. It is not a video
|
|
@@ -219,11 +255,32 @@ guarantee that the selected capture is the most relevant. Context-only and
|
|
|
219
255
|
nonvisual evidence retain their basis and original event. All cited moments stay
|
|
220
256
|
available, with exact recording links.
|
|
221
257
|
|
|
258
|
+
A finding with a headline leads its Observer row with the headline and its open panel with the
|
|
259
|
+
experience. The title, summary, assessment and observations sit in a closed **Evidence**
|
|
260
|
+
disclosure beneath. A **Design findings** section follows the ranked findings, grouped by
|
|
261
|
+
severity, with each cited capture as a thumbnail that opens that frame. Findings without a
|
|
262
|
+
headline render as described below.
|
|
263
|
+
|
|
222
264
|
Confidence, recovery and the first full evidence limitation remain visible when
|
|
223
265
|
a finding opens. Exposure and remaining unique limits are one disclosure away;
|
|
224
266
|
observation details retain each original claim and its specific limitation.
|
|
225
267
|
This presentation does not rewrite the saved analysis or reviewer corrections.
|
|
226
268
|
|
|
269
|
+
## Participant impressions
|
|
270
|
+
|
|
271
|
+
A computer-use participant's closing impressions (`ActorTrace.impressions`) reach the packet as
|
|
272
|
+
ordinary `message` entries of that participant's stream, one per impression, with text
|
|
273
|
+
`Impression (<kind>): <text>`. They are quote-eligible and attributed by `streamId`, so feedback,
|
|
274
|
+
participant_statement observations and design findings can cite them under the existing rules.
|
|
275
|
+
Selection admits them right after the participant's last entry, ahead of the rest of its
|
|
276
|
+
session. The evidence mapping and `captureVersion` are unchanged.
|
|
277
|
+
|
|
278
|
+
The `study-evidence-8` prompt adds one paragraph: impressions are the participant's own opinions,
|
|
279
|
+
used as participant statements, and an `unlike my work` impression together with a capture of
|
|
280
|
+
that screen is a strong design finding. Validation is unchanged. A design finding still needs at
|
|
281
|
+
least one cited capture, so an impression plus the capture of its screen makes a design finding,
|
|
282
|
+
and an impression alone supports only a finding about what the participant said.
|
|
283
|
+
|
|
227
284
|
## Durable records
|
|
228
285
|
|
|
229
286
|
The frozen `humanish.observer-data.v1` schema is unchanged. A companion
|
|
@@ -291,31 +348,35 @@ is disabled. Existing adapter captures and logs retain their contained routes.
|
|
|
291
348
|
## Reading findings
|
|
292
349
|
|
|
293
350
|
`humanish review` prints a run's analysis findings after its participant review, and
|
|
294
|
-
`humanish analyze show` prints them alone. Each finding
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
351
|
+
`humanish analyze show` prints them alone. Each finding leads with its headline and experience,
|
|
352
|
+
then shows its title as `evidence:`, its impact, confidence and recovery, the participants it
|
|
353
|
+
affected out of those exposed, the frames its evidence cites with their time since the first
|
|
354
|
+
retained capture, the capture files, its next step and the latest human review note. A finding
|
|
355
|
+
without a headline starts at its title. Design findings follow, most severe first, each with its
|
|
356
|
+
screen, notice, why it matters, suggestion, who saw it and its capture files. A live run's human
|
|
357
|
+
output ends with up to three findings and up to three design findings, one line each, and the
|
|
358
|
+
`review` command that prints all of them. `analyze show --json` prints the validated
|
|
299
359
|
analysis record and its corrections, as before. Reading findings never starts an analysis.
|
|
300
360
|
|
|
301
361
|
`review --json` carries the same view as its `analysis` field, schema
|
|
302
362
|
`humanish.analysis-findings.v1`, built in `src/cli/findings.ts`:
|
|
303
363
|
|
|
304
|
-
| Field | Meaning
|
|
305
|
-
| ------------------------------------------------------------------------------ |
|
|
306
|
-
| `schema` | `humanish.analysis-findings.v1`.
|
|
307
|
-
| `runId` | The run the view reads.
|
|
308
|
-
| `state` | `ready`, `stale`, `running`, `none`, `skipped`, `failed`, `dry_run` or `unavailable`. Only `ready` and `stale` carry findings.
|
|
309
|
-
| `reason` | The stable code behind a state other than `ready`, or null: an automatic analysis reason such as `AUTOMATIC_ANALYSIS_KEY_MISSING`, a failed attempt's error code, or a load warning such as `ANALYSIS_SOURCE_CHANGED`.
|
|
310
|
-
| `message` | What happened, in words.
|
|
311
|
-
| `next` | The command that gets findings for this run, or for a dry run the command that starts a live one. Null when no command can, as for a run whose participants left no evidence.
|
|
312
|
-
| `analysisId`, `status`, `provider`, `model`, `completedAt`, `estimatedCostUsd` | The analysis version `analyze show` selects: the newest ready one, else the newest stale one. Null without one.
|
|
313
|
-
| `runPath`, `path` | The run directory and the version's `analysis.json`, relative to the project directory.
|
|
314
|
-
| `summary`, `limitations` | The analysis's own summary and limitations.
|
|
315
|
-
| `findings[]` | Highest priority first: `id`, `title`, `summary`, `impact` (`blocked_task`, `friction`, `recovery`, `uncertain`), `confidence` (`low`, `medium`, `high`), `recovery` (`recovered`, `not_observed`, `unknown`), `affected[]` (`streamId`, `label`), `exposedCount`, `evidence[]`, `nextStep` and `correction`.
|
|
316
|
-
| `findings[].evidence[]` | Each cited item once: `id`, `streamId`, `kind`, `bases` (how the observations used it), `frame` (the latest retained capture's index from 0, null before any capture), `elapsedMs` (time since the first retained capture, not a video offset), `at` and `capture` (the file relative to the project directory, null for nonvisual evidence).
|
|
317
|
-
| `findings[].correction` | The latest `analyze correct` note on that finding version (`status`, `reason`, `replacementClaim`), or null.
|
|
318
|
-
| `
|
|
364
|
+
| Field | Meaning |
|
|
365
|
+
| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
366
|
+
| `schema` | `humanish.analysis-findings.v1`. |
|
|
367
|
+
| `runId` | The run the view reads. |
|
|
368
|
+
| `state` | `ready`, `stale`, `running`, `none`, `skipped`, `failed`, `dry_run` or `unavailable`. Only `ready` and `stale` carry findings. |
|
|
369
|
+
| `reason` | The stable code behind a state other than `ready`, or null: an automatic analysis reason such as `AUTOMATIC_ANALYSIS_KEY_MISSING`, a failed attempt's error code, or a load warning such as `ANALYSIS_SOURCE_CHANGED`. |
|
|
370
|
+
| `message` | What happened, in words. |
|
|
371
|
+
| `next` | The command that gets findings for this run, or for a dry run the command that starts a live one. Null when no command can, as for a run whose participants left no evidence. |
|
|
372
|
+
| `analysisId`, `status`, `provider`, `model`, `completedAt`, `estimatedCostUsd` | The analysis version `analyze show` selects: the newest ready one, else the newest stale one. Null without one. |
|
|
373
|
+
| `runPath`, `path` | The run directory and the version's `analysis.json`, relative to the project directory. |
|
|
374
|
+
| `summary`, `limitations` | The analysis's own summary and limitations. |
|
|
375
|
+
| `findings[]` | Highest priority first: `id`, `headline` and `experience` (null in analyses before `study-evidence-7`), `title`, `summary`, `impact` (`blocked_task`, `friction`, `recovery`, `uncertain`), `confidence` (`low`, `medium`, `high`), `recovery` (`recovered`, `not_observed`, `unknown`), `affected[]` (`streamId`, `label`), `exposedCount`, `evidence[]`, `nextStep` and `correction`. |
|
|
376
|
+
| `findings[].evidence[]` | Each cited item once: `id`, `streamId`, `kind`, `bases` (how the observations used it), `frame` (the latest retained capture's index from 0, null before any capture), `elapsedMs` (time since the first retained capture, not a video offset), `at` and `capture` (the file relative to the project directory, null for nonvisual evidence). |
|
|
377
|
+
| `findings[].correction` | The latest `analyze correct` note on that finding version (`status`, `reason`, `replacementClaim`), or null. When `status` is `amended`, `replacementClaim` replaces the finding's headline and experience. |
|
|
378
|
+
| `designFindings` | Most severe first: `id`, `headline`, `screen`, `notice`, `whyItMatters`, `suggestion`, `severity` (`major`, `moderate`, `minor`), `confidence`, `seenBy[]` (`streamId`, `label`) and `evidence[]` (as `findings[].evidence[]`, without `bases`). Null without an analysis result, or for an analysis written before design findings. |
|
|
379
|
+
| `warnings` | Load warnings for the selected version. |
|
|
319
380
|
|
|
320
381
|
The states without findings:
|
|
321
382
|
|
|
@@ -352,7 +413,10 @@ humanish feedback issue --run latest --analysis <id> --finding F1 --repo owner/r
|
|
|
352
413
|
|
|
353
414
|
Corrections append against exact analysis and finding hashes. Confirming one
|
|
354
415
|
version does not approve a later claim. Dismissed findings cannot become feedback
|
|
355
|
-
drafts; amendments preserve the original and record the replacement.
|
|
416
|
+
drafts; amendments preserve the original and record the replacement. A reason or replacement
|
|
417
|
+
claim that matches a sensitive pattern is refused. When the correction is made inside the run's
|
|
418
|
+
own invocation, one that holds a known transient value, as written or encoded, is refused too; a
|
|
419
|
+
separate `analyze correct` command has no transient values to check against. Feedback
|
|
356
420
|
drafts include source/version/evidence references and remain explicitly
|
|
357
421
|
independent of participant-authored candidates. No command above posts to GitHub.
|
|
358
422
|
Each analysis has room for 256 correction inventory entries, including interrupted
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "humanish",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.113.0",
|
|
4
4
|
"description": "Synthetic user research for apps, CLIs, and agent-facing product flows.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"cli",
|
|
@@ -76,6 +76,7 @@
|
|
|
76
76
|
"tui:test": "pnpm --filter humanish-tui test",
|
|
77
77
|
"release:dogfood": "node scripts/release-dogfood.mjs",
|
|
78
78
|
"bench": "tsx bench/run.ts",
|
|
79
|
+
"arch:bench": "tsx scripts/arch-bench.ts",
|
|
79
80
|
"browser-control:proof": "node scripts/browser-control-proof.mjs",
|
|
80
81
|
"docs:generate": "tsx scripts/generate-cli-docs.ts",
|
|
81
82
|
"docs:check": "tsx scripts/generate-cli-docs.ts --check && tsx scripts/check-doc-paths.ts && tsx scripts/check-doc-symbols.ts && tsx scripts/check-doc-comments.ts && tsx scripts/check-doc-study-fields.ts",
|
package/skills/humanish/SKILL.md
CHANGED
|
@@ -190,16 +190,6 @@ Legacy profiles without it retain medium defaults for missing patience and
|
|
|
190
190
|
technical confidence. Avoid contradictory prose and traits; inspect the compiled
|
|
191
191
|
brief with `humanish study show <study> --json` before running.
|
|
192
192
|
|
|
193
|
-
For an autonomous participant study, use a computer-use/local-agent study and write
|
|
194
|
-
its `mission` as a believable situation and desired outcome. Supply fixture facts
|
|
195
|
-
needed to act, but do not supply selectors, click sequences, hidden success rules,
|
|
196
|
-
expected defects or recovery tricks. For example: “Saturday's event needs two
|
|
197
|
-
setup volunteers and one cleanup volunteer. See whether this app helps you
|
|
198
|
-
organize the event and keep people informed.” Do not instruct a participant to
|
|
199
|
-
find a specific UI control merely to make the study succeed. Allow discovery,
|
|
200
|
-
recovery and stopping appropriate to the participant. Keep researcher criteria in
|
|
201
|
-
`tasks[].success`, separate from participant-facing `tasks[].goal`.
|
|
202
|
-
|
|
203
193
|
Scripted regression checks are a different use of scenarios. When the requested
|
|
204
194
|
work is deterministic verification of a known path, a `browser.steps` manifest
|
|
205
195
|
can drive it with explicit selectors and assertions. It does not establish
|
|
@@ -232,6 +222,27 @@ tokens into scenario files.
|
|
|
232
222
|
|
|
233
223
|
## Authoring Studies
|
|
234
224
|
|
|
225
|
+
For an autonomous participant study, write `mission`, participant `instruction`
|
|
226
|
+
and `tasks[].goal` as a believable situation and desired outcome. Supply fixture
|
|
227
|
+
facts needed to act. Leave discovery, recovery and stopping to the participant.
|
|
228
|
+
Keep researcher criteria in `tasks[].success`.
|
|
229
|
+
|
|
230
|
+
| Bad: scripts the controls | Good: gives a situation |
|
|
231
|
+
| -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
232
|
+
| Click "New event", type a title into the field, then press "Save" and open "Settings". | Saturday's event needs two setup volunteers and one cleanup volunteer. See whether this app helps you organize the event and keep people informed. |
|
|
233
|
+
|
|
234
|
+
`humanish study check` and `humanish run` warn about step lists, chains of UI
|
|
235
|
+
actions, quoted control labels, CSS selectors and URLs beyond the subject's entry
|
|
236
|
+
point. They quote up to three matched lines and suggest a situation-style mission.
|
|
237
|
+
The checks apply to computer-use and shared-world studies; scripted-browser and
|
|
238
|
+
terminal studies and `tasks[].success` are excluded. These warnings never refuse
|
|
239
|
+
a run, and the run records them in its bundle.
|
|
240
|
+
|
|
241
|
+
Give each participant a persona with a short, fictional `background` describing
|
|
242
|
+
their relevant experience and situation. A missing persona or background also
|
|
243
|
+
warns. Use `humanish study show <study> --json` to see what the participant receives.
|
|
244
|
+
Do not supply hidden success rules, expected defects or recovery tricks.
|
|
245
|
+
|
|
235
246
|
Write reusable studies as `.yaml` study files (`humanish.study.v3`):
|
|
236
247
|
|
|
237
248
|
```yaml
|