humanish 0.62.0 → 0.64.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +106 -109
- package/dist/cua-actor-lab.js +13 -2
- package/dist/cua-actor-lab.js.map +1 -1
- package/dist/e2b-terminal-lab.d.ts +5 -1
- package/dist/e2b-terminal-lab.js +40 -8
- package/dist/e2b-terminal-lab.js.map +1 -1
- package/dist/program.js +4 -1
- package/dist/program.js.map +1 -1
- package/dist/run.js +5 -1
- package/dist/run.js.map +1 -1
- package/dist/telemetry.d.ts +10 -0
- package/dist/telemetry.js +44 -1
- package/dist/telemetry.js.map +1 -1
- package/dist/terminal-token-usage.d.ts +14 -0
- package/dist/terminal-token-usage.js +93 -0
- package/dist/terminal-token-usage.js.map +1 -0
- package/docs/contracts/schemas.md +1 -1
- package/docs/goals/current.md +1 -1
- package/docs/ramp/README.md +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# humanish
|
|
2
2
|
|
|
3
|
-
Synthetic user research for apps, CLIs, and agent-facing product flows
|
|
4
|
-
|
|
3
|
+
Synthetic user research for apps, CLIs, and agent-facing product flows.
|
|
4
|
+
Open-source and public-safe.
|
|
5
5
|
|
|
6
|
-
Humanish runs studies. Realistic synthetic participants
|
|
7
|
-
goals, patience, and skill
|
|
6
|
+
Humanish runs studies. Realistic synthetic participants, each with its own
|
|
7
|
+
goals, patience, and skill, actually use your product on hosted desktops while
|
|
8
8
|
you watch. A study leaves verifiable evidence: screenshots, action traces,
|
|
9
9
|
per-task completion funnels, participant outcomes with the denominator
|
|
10
10
|
attached, and estimated cost lines. A fail-closed share-safety gate stands
|
|
@@ -26,7 +26,7 @@ studied; it is not a Humanish adopter or endorser.
|
|
|
26
26
|
Every design decision in humanish is checked against the three people a study
|
|
27
27
|
actually involves ([docs/principles/three-roles.md](docs/principles/three-roles.md)):
|
|
28
28
|
|
|
29
|
-
- **The researcher**
|
|
29
|
+
- **The researcher** (usually a coding agent driving the CLI) declares the
|
|
30
30
|
protocol: personas, discrete tasks with success criteria the participant
|
|
31
31
|
never sees, and a budget set once at the study level, the way recruiting
|
|
32
32
|
decisions are made.
|
|
@@ -37,12 +37,11 @@ actually involves ([docs/principles/three-roles.md](docs/principles/three-roles.
|
|
|
37
37
|
instrument. A participant abandoning a task is a finding about the product,
|
|
38
38
|
not a harness failure.
|
|
39
39
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
are committed under
|
|
40
|
+
In practice: a two-participant study of an email-gated signup on a public
|
|
41
|
+
open-source document app completed end to end (signup, verification mail read
|
|
42
|
+
in a captured inbox, signed-in dashboard), and the keyboard-first participant
|
|
43
|
+
reported, unprompted, that the signature step could not be completed without a
|
|
44
|
+
mouse. The panel cost about $1.50 and the receipts are committed under
|
|
46
45
|
[docs/goals/email-gated-signup/receipts/](docs/goals/email-gated-signup/receipts/).
|
|
47
46
|
|
|
48
47
|
## Install
|
|
@@ -56,11 +55,11 @@ npx humanish run try-live # a REAL study on a hosted desktop, capped at $2
|
|
|
56
55
|
|
|
57
56
|
`try-live` studies a real public app so that your first live run works without
|
|
58
57
|
configuring anything; point its `subject` at your own app once you have seen it
|
|
59
|
-
run. It needs `E2B_API_KEY`, and a model
|
|
58
|
+
run. It needs `E2B_API_KEY`, and a model: either a provider key or a coding
|
|
60
59
|
agent you are already signed in to (see below).
|
|
61
60
|
|
|
62
61
|
**Install it, do not one-shot it.** A live run needs the optional peer
|
|
63
|
-
`@e2b/desktop`, and Node resolves that relative to humanish itself
|
|
62
|
+
`@e2b/desktop`, and Node resolves that relative to humanish itself, so a
|
|
64
63
|
one-shot `npx humanish@latest` can never find it, no matter what your project
|
|
65
64
|
has installed. `npm i -D humanish @e2b/desktop` once, then `npx humanish …`
|
|
66
65
|
resolves the local copy and works. The dry-run path (`humanish run first-run`)
|
|
@@ -83,7 +82,7 @@ for skills.sh discovery.
|
|
|
83
82
|
## A First Live Run Without a Provider API Key
|
|
84
83
|
|
|
85
84
|
A live study normally needs a provider API key. If you already have a coding
|
|
86
|
-
agent signed in
|
|
85
|
+
agent signed in (Codex on a ChatGPT plan, Claude Code on a Max plan), humanish
|
|
87
86
|
can use it as the participant's brain instead, and then the only credential it
|
|
88
87
|
needs is `E2B_API_KEY`.
|
|
89
88
|
|
|
@@ -110,8 +109,8 @@ Three things to know before you rely on it:
|
|
|
110
109
|
- **It is not free.** Subscription usage consumes your own plan. Runs driven this
|
|
111
110
|
way record `estimatedCostUsd: null` with `reason: "no_token_usage"` rather than
|
|
112
111
|
`$0`, because `$0` would be untrue. Rate limits on those plans are built for
|
|
113
|
-
interactive coding; humanish fails closed with the CLI's own message
|
|
114
|
-
|
|
112
|
+
interactive coding; humanish fails closed with the CLI's own message and does
|
|
113
|
+
not retry into them.
|
|
115
114
|
- **It is slower.** Roughly 9 seconds per turn against about 3 for a direct API
|
|
116
115
|
call, so give the lane a longer `execution.timeoutMs` than you would otherwise.
|
|
117
116
|
- **The evidence says which brain ran it.** The trace records
|
|
@@ -120,8 +119,8 @@ Three things to know before you rely on it:
|
|
|
120
119
|
|
|
121
120
|
## Telemetry
|
|
122
121
|
|
|
123
|
-
humanish collects anonymous usage data by default
|
|
124
|
-
worked, roughly how long it took
|
|
122
|
+
humanish collects anonymous usage data by default (which command ran, whether it
|
|
123
|
+
worked, roughly how long it took) so the maintainers can tell whether anyone
|
|
125
124
|
reaches a working first run. It never sends your labs, subjects, personas, paths,
|
|
126
125
|
or evidence, and there is no field in the payload that could.
|
|
127
126
|
|
|
@@ -141,7 +140,7 @@ boundary is three planks, each enforced where it actually holds:
|
|
|
141
140
|
**1. This repo and the published package are kept public-safe by CI.** Every
|
|
142
141
|
push runs a public-surface scan (secret/key/path shapes, a sha256 binary-asset
|
|
143
142
|
allowlist, over both tracked files and the packed npm payload) plus a
|
|
144
|
-
full-history gitleaks scan. That protects what we ship
|
|
143
|
+
full-history gitleaks scan. That protects what we ship; it does not scan your
|
|
145
144
|
repo.
|
|
146
145
|
|
|
147
146
|
**2. The harness never persists secret values into run artifacts.** On every
|
|
@@ -152,8 +151,8 @@ evidence; values never are. Pixels are the exception: a raw screenshot shows
|
|
|
152
151
|
whatever was on screen, which is why plank 3 exists.
|
|
153
152
|
|
|
154
153
|
**3. Run bundles are local by default.** Evidence lands under gitignored
|
|
155
|
-
`.humanish/`, and no command publishes it for you. Sharing evidence
|
|
156
|
-
screenshots, pasting transcripts, attaching bundles to issues
|
|
154
|
+
`.humanish/`, and no command publishes it for you. Sharing evidence (committing
|
|
155
|
+
screenshots, pasting transcripts, attaching bundles to issues) is a deliberate
|
|
157
156
|
act, and reviewing what you share is on you. Use synthetic personas and
|
|
158
157
|
synthetic data so there is nothing sensitive to capture in the first place.
|
|
159
158
|
|
|
@@ -212,33 +211,33 @@ npx humanish watch --json --no-open
|
|
|
212
211
|
## The Terminal Surface
|
|
213
212
|
|
|
214
213
|
Every other humanish command is built so an agent can drive it. `humanish tui`
|
|
215
|
-
|
|
214
|
+
takes the screen and waits for a person.
|
|
216
215
|
|
|
217
216
|
```bash
|
|
218
217
|
npx humanish tui
|
|
219
218
|
```
|
|
220
219
|
|
|
221
|
-
Arrow keys move, `enter` opens, `esc` goes back, `q` quits.
|
|
222
|
-
you
|
|
223
|
-
lab, one run
|
|
224
|
-
watching changes from running to its verdict
|
|
225
|
-
you.
|
|
226
|
-
|
|
227
|
-
- **labs
|
|
228
|
-
something running now sort first. Each row carries what to expect from a
|
|
229
|
-
run of it; a lab with no live history says `no live runs yet`
|
|
230
|
-
|
|
231
|
-
- **lab
|
|
220
|
+
Arrow keys move, `enter` opens, `esc` goes back, `q` quits. There are three
|
|
221
|
+
screens, and you move between objects rather than between states: the set of
|
|
222
|
+
labs, one lab, one run. A run's lifecycle
|
|
223
|
+
renders in place, so a run you are watching changes from running to its verdict
|
|
224
|
+
without the screen moving under you.
|
|
225
|
+
|
|
226
|
+
- **labs**: every lab in the project, whether or not it has ever run. Labs with
|
|
227
|
+
something running now sort first. Each row carries what to expect from a live
|
|
228
|
+
run of it; a lab with no live history says `no live runs yet` and does not
|
|
229
|
+
quote a median from dry runs, which spend nothing and take no time.
|
|
230
|
+
- **lab**: that lab's history, and two ways to start it. A dry run starts on one
|
|
232
231
|
keypress because it cannot cost anything; a live run is armed by the first
|
|
233
232
|
`enter` and committed by the second, restating the cost in between.
|
|
234
|
-
- **run
|
|
233
|
+
- **run**: who is in the run, what they are currently thinking, and how far they
|
|
235
234
|
have got, with time and money underneath. A terminal cannot show screenshots,
|
|
236
235
|
so the run's self-contained Observer artifact is named for you to open.
|
|
237
236
|
|
|
238
|
-
A run you start from the surface is
|
|
237
|
+
A run you start from the surface is detached: it keeps going if you quit the
|
|
239
238
|
TUI, and it survives losing the SSH session you started it over. The surface
|
|
240
239
|
follows it by reading `.humanish/runs/<id>/status.json`, holding no handle on
|
|
241
|
-
it
|
|
240
|
+
it, so you can quit mid-run, reopen, and find it still there.
|
|
242
241
|
|
|
243
242
|
Requires an interactive terminal and Node 22 or newer. It refuses anything else
|
|
244
243
|
with a structured error rather than rendering escape codes into a pipe:
|
|
@@ -260,7 +259,7 @@ Every other command still works on Node 20; only this surface needs 22.
|
|
|
260
259
|
## Serve the Library
|
|
261
260
|
|
|
262
261
|
`humanish watch` follows one attached run; `humanish serve` serves the whole
|
|
263
|
-
local run library under `.humanish/runs
|
|
262
|
+
local run library under `.humanish/runs/`, a library index plus every run's
|
|
264
263
|
Observer page:
|
|
265
264
|
|
|
266
265
|
```bash
|
|
@@ -273,34 +272,34 @@ npx humanish serve --expose --public-url https://observer.example.com
|
|
|
273
272
|
The first serves the library on loopback only. The second is the phone path:
|
|
274
273
|
ngrok's edge authenticates viewers with Google OAuth (restricted to your
|
|
275
274
|
`--allow-email`/`--allow-domain` allow rules) before any request reaches the
|
|
276
|
-
loopback server
|
|
275
|
+
loopback server; humanish carries no in-process auth. The third is a secretless
|
|
277
276
|
safe observer: no login, but only runs whose `humanish verify` shareSafety is
|
|
278
|
-
`share_ready` exist at all
|
|
277
|
+
`share_ready` exist at all; everything else is absent and 404s. The fourth
|
|
279
278
|
trusts an edge you already secure (Cloudflare Access, Tailscale, a proxy you own)
|
|
280
279
|
and just binds loopback behind it.
|
|
281
280
|
|
|
282
281
|
In every mode the server binds `127.0.0.1`; exposure only ever happens through
|
|
283
282
|
an authenticated edge forwarding to the loopback port. Exposure is fail-closed:
|
|
284
283
|
`--expose` always needs a reachable public origin (a `--tunnel` or a `--public-url`,
|
|
285
|
-
even under `--safe`), and then requires
|
|
286
|
-
or a `--public-url` you secure)
|
|
284
|
+
even under `--safe`), and then requires either edge auth (`--oauth` on the tunnel,
|
|
285
|
+
or a `--public-url` you secure) or `--safe`. `--oauth google` with no allow rule
|
|
287
286
|
lets any Google account in and warns loudly.
|
|
288
287
|
|
|
289
288
|
### Watch a live run from your phone
|
|
290
289
|
|
|
291
290
|
`humanish watch <cua-lab> --expose --tunnel ngrok --oauth google --allow-email
|
|
292
|
-
you@example.com` streams the
|
|
291
|
+
you@example.com` streams the live desktop of a computer-use run to an
|
|
293
292
|
edge-authenticated remote viewer while it plays. The attached server comes up
|
|
294
293
|
during the run and survives a timed-out/failed run, so you can inspect a failed
|
|
295
294
|
run's evidence too. A live run is never `share_ready`, so `watch --expose` always
|
|
296
295
|
requires edge auth; `--safe` is a `serve` library filter and is rejected on watch
|
|
297
|
-
(`HUMANISH_WATCH_SAFE_NOT_APPLICABLE`). An exposed watch serves
|
|
298
|
-
run
|
|
296
|
+
(`HUMANISH_WATCH_SAFE_NOT_APPLICABLE`). An exposed watch serves only the attached
|
|
297
|
+
run (its history lists just that run and every other run id 404s), so a remote
|
|
299
298
|
viewer can never reach your other runs' raw evidence.
|
|
300
299
|
|
|
301
|
-
Live E2B desktop stream URLs are served
|
|
302
|
-
edge auth; `serve` never injects them (remote viewers of the library see
|
|
303
|
-
persisted evidence
|
|
300
|
+
Live E2B desktop stream URLs are served only on `watch --expose`, and only behind
|
|
301
|
+
edge auth; `serve` never injects them (remote viewers of the library see only
|
|
302
|
+
persisted evidence: screenshots, events, terminal tails). See
|
|
304
303
|
[Serve architecture](docs/architecture/serve.md).
|
|
305
304
|
|
|
306
305
|
## Commands
|
|
@@ -309,7 +308,7 @@ persisted evidence — screenshots, events, terminal tails — only). See
|
|
|
309
308
|
| --- | --- |
|
|
310
309
|
| `humanish init` | Scaffold committed `humanish/` source and ignored `.humanish/` runtime state. |
|
|
311
310
|
| `humanish doctor` | Explain readiness and missing setup. |
|
|
312
|
-
| `humanish tui` | Interactive terminal surface for browsing labs and runs and starting a run. Humans only
|
|
311
|
+
| `humanish tui` | Interactive terminal surface for browsing labs and runs and starting a run. Humans only; it refuses a non-interactive stdin or stdout. |
|
|
313
312
|
| `humanish run --dry-run` | Generate a synthetic run bundle without browser, keys, or provider spend. |
|
|
314
313
|
| `humanish run --app-url http://127.0.0.1:<port>` | Capture live desktop/mobile browser evidence against a running local app. |
|
|
315
314
|
| `humanish watch [lab]` | Run sims or a named lab, open Observer, and keep watching. |
|
|
@@ -361,16 +360,16 @@ or Observer data.
|
|
|
361
360
|
### Computer-Use Labs
|
|
362
361
|
|
|
363
362
|
A computer-use lab dispatches a **registered computer-use actor** (`actors[0].type`,
|
|
364
|
-
resolved against the actor registry
|
|
363
|
+
resolved against the actor registry, e.g. `openai-computer-use`) to drive an app in
|
|
365
364
|
a hosted E2B desktop browser and emit an evidence bundle under gitignored
|
|
366
|
-
`.humanish/` (full-fidelity screenshots by default
|
|
365
|
+
`.humanish/` (full-fidelity screenshots by default, see below; length-only typed
|
|
367
366
|
text; provider-neutral `humanish.actor-trace.v1` on the stream). Two subjects route
|
|
368
367
|
here:
|
|
369
368
|
|
|
370
369
|
- **`subject.source: clone`** (+ `execution.target: e2b-desktop` + a computer-use
|
|
371
|
-
actor): the lab clones your repo
|
|
370
|
+
actor): the lab clones your repo into the sandbox, runs your declared
|
|
372
371
|
`serve.install`/`serve.build`/`serve.start` commands (detached, with readiness
|
|
373
|
-
probing), and drives the served app at `serve.url`. Subject env var
|
|
372
|
+
probing), and drives the served app at `serve.url`. Subject env var names declared
|
|
374
373
|
in `subject.env` are provisioned from `--env-file` (names land in evidence; values
|
|
375
374
|
never do). The bundle records provenance: repo, cloned commit, env names.
|
|
376
375
|
- **`subject.source: app-url`**: you (a library caller) provision the app yourself
|
|
@@ -405,55 +404,53 @@ you own (a Vercel preview, staging), use an `app-url` subject with
|
|
|
405
404
|
only declared subject env names do. `humanish init` scaffolds an example at
|
|
406
405
|
`humanish/labs/cua-browser.yaml`.
|
|
407
406
|
|
|
408
|
-
**Off-app email/SMS verification (`comms`).** When a flow is gated behind an email
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
407
|
+
**Off-app email/SMS verification (`comms`).** When a flow is gated behind an email or
|
|
408
|
+
SMS the app itself sends (a signup verification link, a one-time code, a magic link),
|
|
409
|
+
add a `comms:` block to the lab. Humanish redirects the app's email-API sends (via
|
|
410
|
+
one adopter-named env var: with the official Resend SDK it is `RESEND_BASE_URL`,
|
|
412
411
|
which the SDK reads on its own, so nothing in the app has to change; other providers
|
|
413
412
|
need the app to actually pass the env through, and a run whose catch captures zero
|
|
414
|
-
sends warns at teardown) into a catch **inside**
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
the full `comms:` shape.
|
|
413
|
+
sends warns at teardown) into a catch **inside** the sandbox, so nothing leaves the
|
|
414
|
+
machine. Every lane gets a deterministic inbox address automatically, and each
|
|
415
|
+
persona's prompt carries the full handoff: the address to sign up with, the inbox URL
|
|
416
|
+
to open, and that waiting for an email is a next step, not a blocker. The run bundle
|
|
417
|
+
gets a digest-only `humanish.comms-thread.v1` artifact (from/to/subject/link
|
|
418
|
+
digests + an OTP count; no raw address, link, or code persists); the readable proof a
|
|
419
|
+
persona saw the email is its inbox-page screenshots. Hosted on the clone/local-tree
|
|
420
|
+
computer-use lanes and the concurrent shared-world route (warned inert elsewhere),
|
|
421
|
+
vendor-neutral (Resend/SendGrid shaped, or a custom profile).
|
|
422
|
+
See `docs/contracts/schemas.md` for the full `comms:` shape.
|
|
425
423
|
|
|
426
424
|
**Screenshots are full-fidelity by default.** Run bundles live in gitignored
|
|
427
|
-
`.humanish/`, so the Observer shows exactly what the persona saw
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
**Device presets.** `execution.desktop.device` picks the hosted desktop screen size —
|
|
425
|
+
`.humanish/`, so the Observer shows exactly what the persona saw. Set
|
|
426
|
+
`policies.redactScreenshots: true` to persist blurred thumbnails at capture instead
|
|
427
|
+
(for unowned subjects, or bundles you intend to share as-is). Raw bundles stay local
|
|
428
|
+
in gitignored `.humanish/`; nothing scans the pixels, so review them before sharing
|
|
429
|
+
anywhere. A redact-on-export step is planned. The frame sent to the model is always
|
|
430
|
+
full-resolution regardless. (Doctrine: `docs/principles/invariants-and-defaults.md`.
|
|
431
|
+
Redaction binds the publish boundary, not capture.) `humanish verify` reports
|
|
432
|
+
raw-screenshot bundles as `shareSafety.status: local_only`; `humanish feedback issue`
|
|
433
|
+
refuses them until the run is share-ready.
|
|
434
|
+
|
|
435
|
+
**Device presets.** `execution.desktop.device` picks the hosted desktop screen size:
|
|
439
436
|
`mobile` (414×896), `small-mobile` (360×740), `narrow-mobile` (320×700), `tablet`
|
|
440
437
|
(820×1180), `desktop` (1440×950, default), or `wide` (1920×1080). The values are copied
|
|
441
|
-
from the mature in-house sims
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
measure it through CDP and record it separately from requested/verified screen geometry.
|
|
445
|
-
without that measurement seam omit the viewport rather than guessing. A
|
|
446
|
-
responsive CSS still fires, and the model is *told* its device in the
|
|
447
|
-
how those sims run organic mobile lanes. There is no touch input, the
|
|
448
|
-
isn't rendered, and the user-agent stays desktop on this route; true
|
|
449
|
-
arrives with the deterministic CDP actor. Device is run-wide
|
|
450
|
-
(N personas × devices) lands with fan-out.
|
|
451
|
-
hatch that overrides the preset.
|
|
438
|
+
from the mature in-house sims. **Honest fidelity:** on the computer-use / E2B-desktop
|
|
439
|
+
route width/height size the virtual display and browser outer window. The actual page
|
|
440
|
+
viewport is smaller because browser chrome occupies space; Chromium-family live bundles
|
|
441
|
+
measure it through CDP and record it separately from requested/verified screen geometry.
|
|
442
|
+
Browsers without that measurement seam omit the viewport rather than guessing. A
|
|
443
|
+
site's width-based responsive CSS still fires, and the model is *told* its device in the
|
|
444
|
+
prompt, matching how those sims run organic mobile lanes. There is no touch input, the
|
|
445
|
+
device-pixel-ratio isn't rendered, and the user-agent stays desktop on this route; true
|
|
446
|
+
touch/DPR/UA emulation arrives with the deterministic CDP actor. Device is run-wide
|
|
447
|
+
today; per-*persona* device (N personas × devices) lands with fan-out.
|
|
448
|
+
`execution.desktop.resolution` is a raw escape hatch that overrides the preset.
|
|
452
449
|
|
|
453
450
|
**Desktop browser choice.** Hosted computer-use lanes and shared-world actor seats use the
|
|
454
451
|
route's historical opener unless you set `execution.desktop.browser` to `chrome`, `chromium`,
|
|
455
|
-
or `firefox`. A concrete value means "launch this browser or fail"
|
|
456
|
-
|
|
452
|
+
or `firefox`. A concrete value means "launch this browser or fail"; it never silently
|
|
453
|
+
falls back to whatever the image prefers. When configured, run bundles record the requested
|
|
457
454
|
browser and the resolved in-sandbox command as `desktopBrowser`.
|
|
458
455
|
|
|
459
456
|
**Deterministic stop conditions.** Freeform computer-use actors can keep acting after the
|
|
@@ -490,34 +487,34 @@ browser-observed stops, set `execution.desktop.browser: chrome` or `chromium`.
|
|
|
490
487
|
|
|
491
488
|
**Cost tracking (estimated).** Computer-use run bundles carry an advisory `cost` block: a
|
|
492
489
|
per-lane token-derived model estimate plus one aggregate E2B desktop-minute estimate. Every
|
|
493
|
-
dollar figure is an
|
|
490
|
+
dollar figure is an estimate, never a provider charge. It is a rate-table multiply, always
|
|
494
491
|
surfaced as "~$X estimated (rates as of `<date>`)" in the Observer and the run library, and it
|
|
495
492
|
carries the pricing date + source so a token-derived number is never mistaken for an
|
|
496
493
|
authoritative bill. Unknown model/rate is declared absent (`null` + a reason), never guessed or
|
|
497
494
|
silently zeroed; dry-runs invent no spend. The rates live in
|
|
498
|
-
[`src/pricing.ts`](src/pricing.ts) as **operator-editable, dated estimates
|
|
499
|
-
rate is still a `placeholder` stand-in; update the numbers
|
|
495
|
+
[`src/pricing.ts`](src/pricing.ts) as **operator-editable, dated estimates**. The E2B desktop
|
|
496
|
+
rate is still a `placeholder` stand-in; update the numbers and the `asOf` date when providers
|
|
500
497
|
change pricing. On models that bill prompt-cache writes and long-context requests at their own
|
|
501
498
|
rates (OpenAI's 5.6 family), the estimate prices both exactly from the trace's per-request
|
|
502
499
|
usage ledger.
|
|
503
500
|
|
|
504
501
|
**Choosing the model.** Computer-use lanes default to `gpt-5.6-sol` (the 5.6-generation
|
|
505
502
|
flagship; `gpt-5.6` is OpenAI's alias for the same model). Configure it per lab with
|
|
506
|
-
`actors[0].model
|
|
503
|
+
`actors[0].model`. Any id in the rate table prices cleanly (`gpt-5.6-terra` and
|
|
507
504
|
`gpt-5.6-luna` are the cheaper tiers; `gpt-5.5` stays priced for pinned labs). A run with a
|
|
508
|
-
spend cap (`execution.caps`) refuses an unpriced model at preflight
|
|
509
|
-
|
|
510
|
-
know.
|
|
505
|
+
spend cap (`execution.caps`) refuses an unpriced model at preflight, so add a dated rate to
|
|
506
|
+
`src/pricing.ts` before capping a model the table does not know.
|
|
511
507
|
|
|
512
508
|
**Fail-closed spend cap.** Set `execution.caps.maxUsd` on a computer-use lab to abort a session
|
|
513
|
-
the moment its running estimated spend crosses the cap
|
|
514
|
-
terminal lane's `scenario.caps.maxUsd
|
|
509
|
+
the moment its running estimated spend crosses the cap, a runaway-retry guard that mirrors the
|
|
510
|
+
terminal lane's `scenario.caps.maxUsd`. It is a **per-lane** cap: enforced inside each lane's loop,
|
|
515
511
|
so an N-lane fan-out can spend up to N × `maxUsd` before any lane aborts (the run bundle warns with
|
|
516
512
|
the true ~N × cap ceiling; a shared run-level budget is future work). A lane that did real work then
|
|
517
513
|
hits its cap passes (`budget_reached`); a zero-action runaway that crosses it fails (`gave_up`).
|
|
518
514
|
Absent = uncapped (the historical CUA behavior); `maxUsd: 0` = no-spend. A cap on a model
|
|
519
|
-
`src/pricing.ts` cannot price is refused at preflight (`HUMANISH_CUA_LAB_UNPRICED_CAP`)
|
|
520
|
-
run uncapped
|
|
515
|
+
`src/pricing.ts` cannot price is refused at preflight (`HUMANISH_CUA_LAB_UNPRICED_CAP`)
|
|
516
|
+
rather than run uncapped: an unenforceable cap is more dangerous than none, so add a rate
|
|
517
|
+
or drop the cap.
|
|
521
518
|
|
|
522
519
|
**Failed-lane reruns.** Multi-lane CUA fan-out can be rerun surgically without mutating
|
|
523
520
|
the source run:
|
|
@@ -531,10 +528,10 @@ This creates a new linked run containing only the failed/blocked/timed-out/hollo
|
|
|
531
528
|
(or the explicit `--lanes` selection). The new `run.json` records `rerun.sourceRunId`,
|
|
532
529
|
selected lane ids, and previous lane statuses; the source run's verdict is left unchanged.
|
|
533
530
|
This is intentionally not automatic retry; a passing rerun is evidence of a
|
|
534
|
-
nondeterminism candidate
|
|
531
|
+
nondeterminism candidate and does not license erasing the original red lane.
|
|
535
532
|
|
|
536
533
|
**Run-owned cleanup.** Live providers can record resource evidence in `run.json`.
|
|
537
|
-
Stored bundle IDs are mutable evidence
|
|
534
|
+
Stored bundle IDs are mutable evidence and do not authorize provider mutation. The
|
|
538
535
|
cleanup command writes a durable inspection receipt until Humanish has a
|
|
539
536
|
verified resource-lease contract. Resources already recorded as killed become
|
|
540
537
|
`already_clean`; recorded live or unknown resources become `failed`, which
|
|
@@ -551,7 +548,7 @@ orphan sweep is maintainer-only, opt-in, and verifies provider metadata before
|
|
|
551
548
|
calling provider cleanup.
|
|
552
549
|
|
|
553
550
|
Trust note: `serve` commands run inside the disposable sandbox with the declared
|
|
554
|
-
subject env provisioned
|
|
551
|
+
subject env provisioned, the same trust class as a repo's package.json scripts.
|
|
555
552
|
Only run lab configs you trust, and declare only the env names that the subject
|
|
556
553
|
genuinely needs. (Since 0.5.0, a clone × e2b-desktop lab whose actor is a
|
|
557
554
|
registered computer-use actor routes here and requires `serve`; on earlier
|
|
@@ -563,9 +560,9 @@ The computer-use loop is provider- and substrate-agnostic. You can point a lab a
|
|
|
563
560
|
**already-running local dev server** (`subject.source: local-app`) and drive it
|
|
564
561
|
through its in-process JS contract (`window.app.getState()` etc.) with a custom
|
|
565
562
|
`CuaExecutor` (screenshot optional, `appState` as the progress signal) paired with a
|
|
566
|
-
**non-vision** `CuaProvider` (`requiresFrame` falsey)
|
|
567
|
-
Observer, the evidence bundle, redaction, and the friction loop, with **
|
|
568
|
-
desktop and
|
|
563
|
+
**non-vision** `CuaProvider` (`requiresFrame` falsey), keeping personas, the
|
|
564
|
+
Observer, the evidence bundle, redaction, and the friction loop, with **no E2B
|
|
565
|
+
desktop and no clone**. Supply `cuaHooks.buildExecutor` + `buildProvider` to
|
|
569
566
|
`runLab` (a config-only run with no hooks fails closed with a structured error). See
|
|
570
567
|
[State-driven executor](docs/architecture/state-driven-executor.md).
|
|
571
568
|
|
package/dist/cua-actor-lab.js
CHANGED
|
@@ -289,7 +289,12 @@ function laneSpecsAndPlan(config, opts = {}) {
|
|
|
289
289
|
const simId = `sim-${String(i + 1).padStart(3, "0")}`;
|
|
290
290
|
const streamId = `stream-${String(i + 1).padStart(3, "0")}`;
|
|
291
291
|
const device = resolveLaneDevice(config, lane);
|
|
292
|
-
|
|
292
|
+
// A lane's persona FALLS BACK to actors[0].persona, matching this field's own doc comment
|
|
293
|
+
// in src/lab-config.ts and its sibling resolutions (stopWhen, reasoningEffort) two lines
|
|
294
|
+
// below. Reading only lane.persona when a roster was present meant every fan-out lane of
|
|
295
|
+
// every lab that declared actors[0].persona ran with no persona at all: no personaLine in
|
|
296
|
+
// the prompt, traitsApplied [], and nothing warned (#512).
|
|
297
|
+
const personaId = (lane?.persona ?? actor?.persona);
|
|
293
298
|
const resolvedPersona = personaId === undefined ? undefined : opts.personas?.get(personaId);
|
|
294
299
|
const composed = composeLaneInstructions({
|
|
295
300
|
mission,
|
|
@@ -1218,7 +1223,13 @@ function stripNegatedNonBlockerPhrases(text) {
|
|
|
1218
1223
|
return text
|
|
1219
1224
|
.replace(/\bno\s+(?:real\s+|remaining\s+|actual\s+)?(?:blocker|blockers|blocking issue|blocking issues|error|errors|failure|failures)\s+(?:was\s+|were\s+)?(?:encountered|observed|found|hit|seen|reported|detected)\b/g, "")
|
|
1220
1225
|
.replace(/\bwithout\s+(?:a\s+|any\s+)?(?:real\s+|remaining\s+|actual\s+)?(?:blocker|blockers|blocking issue|blocking issues|error|errors|failure|failures)\b/g, "")
|
|
1221
|
-
.replace(/\bnot\s+(?:blocked|a blocker|an error|failed)\b/g, "")
|
|
1226
|
+
.replace(/\bnot\s+(?:blocked|a blocker|an error|failed)\b/g, "")
|
|
1227
|
+
// "No functional failures blocked me" downgraded a clean passing run to a lab failure on
|
|
1228
|
+
// 2026-09-01. The adjective list above is closed (real|remaining|actual), so an ordinary
|
|
1229
|
+
// qualifier like "functional" slipped through and the trailing verb "blocked" tripped the
|
|
1230
|
+
// scan. Allow up to two intervening words, and cover the verb form directly.
|
|
1231
|
+
.replace(/\bno\s+(?:\w+\s+){0,2}(?:blocker|blockers|blocking issues?|errors?|failures?|problems?|issues?)\b(?:\s+(?:blocked|stopped|prevented)\s+(?:me|us|it))?/g, " ")
|
|
1232
|
+
.replace(/\bnothing\s+(?:\w+\s+){0,2}(?:blocked|stopped|prevented)\s+(?:me|us|it)\b/g, " ");
|
|
1222
1233
|
}
|
|
1223
1234
|
/**
|
|
1224
1235
|
* Remove double-quoted spans and markdown blockquote lines before the blocker scan, so a persona
|