humanish 0.36.0 → 0.37.0

package/docs/principles/actor-fidelity.md

@@ -0,0 +1,165 @@
+# Actor Fidelity: whose behavior is this evidence about?
+
+Status: research digest + design position. Sources are public papers and open-source
+harnesses; every claim below carries its citation so it can be argued with rather
+than inherited.
+
+## The question
+
+A computer-use actor in a study of a consumer web app typed a `javascript:` URL into
+the browser address bar to get past a step. The run finished green.
+
+Whether that is a defect depends entirely on a question the harness never asked:
+**who is this study's user?**
+
+- If the declared users are **people**, it is a defect — and a costly one. The actor
+  routed around the friction the study existed to measure, so a passing run proves
+  nothing about the human experience. Worse, it is silent: nothing in the bundle
+  distinguishes that run from one where a person clicked through.
+- If the declared users are **agents** — as they are for an agent-facing CLI, API, or
+  MCP surface — the same act is faithful. It is the user population behaving normally,
+  and the fact that the agent had to reach for that affordance is itself a product
+  finding about how legible the surface is to its actual users.
+
+The proof roadmap already anticipated this: its strongest evidence class,
+`user-census`, is defined as "the users of the product ARE agents, and the lab runs
+real production harnesses." This page is the layer that makes the distinction
+operable.
+
+## What the evidence says
+
+### Direct URL navigation is a human affordance
+
+WebLINX built its action space from 2,337 real human demonstrations; `load(url)`
+appears in 2,324 of them — 99.4%, roughly 1.6 times per session
+([arXiv 2402.05930](https://arxiv.org/abs/2402.05930)). Treating address-bar
+navigation as non-human would make a human-declared lane *less* faithful, not more.
+The anomalous class is script execution and developer tooling, not URL entry.
+BrowserGym already factors these apart: its `nav` subset is exactly
+`{goto, go_back, go_forward}`, separate from everything else
+([arXiv 2412.05467](https://arxiv.org/abs/2412.05467)).
+
+### Prose instructions are a weak control for action modality
+
+- Tool-restriction constraints in realistic agentic prompts are honored 19.9–27.2% of
+  the time (AgentIF).
+- When a constraint conflicts with the task goal, obedience falls to 9.6–45.8%, and
+  models usually do not register the conflict ("Control Illusion").
+- Rule-file effects are largely content-independent: random rules tie with curated
+  ones, and shuffling changes little — measured specifically on agent skills and
+  persona definitions ([arXiv 2604.11088](https://arxiv.org/abs/2604.11088)). A
+  modality rule can therefore appear to work while doing nothing.
+- Persona adherence decays over a long trajectory rather than holding
+  ([arXiv 2512.12775](https://arxiv.org/abs/2512.12775)).
+
+The one large-scale prompt-level modality constraint in the literature —
+Online-Mind2Web instructing agents not to use search — was not trusted by its own
+authors, who published the measured constrained-vs-unconstrained delta (26% vs 31%)
+instead of asserting the instruction took effect
+([arXiv 2504.01382](https://arxiv.org/abs/2504.01382)). That is the discipline this
+project copies: where a constraint cannot be mechanically enforced, measure what it
+was worth.
+
+### Broad motivational reframes have backfired; narrow countable ones have worked
+
+Making realism the actor's stated objective is an appealing fix, and the nearest
+measured attempts went the wrong way. A findings-informed realism persona prompt
+lowered overall simulator fidelity (User-Sim Index 70.9 → 64.6) and worsened outcome
+calibration (0.18 → 0.29): "moving some behaviors closer to humans can move others
+further away" ([arXiv 2603.11245](https://arxiv.org/abs/2603.11245)). Explicitly
+licensing persona-faithful failure fixed the *marginal* rate five-fold but left the
+*conditional* structure nearly unchanged — it taught the simulator to disengage
+uniformly rather than to disengage when a real user would
+([arXiv 2606.20708](https://arxiv.org/abs/2606.20708)). By contrast, a narrow,
+countable instruction did produce a measured improvement
+([arXiv 2601.17087](https://arxiv.org/abs/2601.17087)).
+
+Two further cautions for prompt design: telling an actor it is being observed and
+scored is a documented behavior-changing cue whose effect grows with model scale
+([arXiv 2505.17815](https://arxiv.org/abs/2505.17815)), and stating the grading
+criterion to the actor invites optimization of the stated metric. Pre-action
+self-checks are worse than neutral: deliberation before acting degrades constraint
+adherence in most models measured ([arXiv 2505.11423](https://arxiv.org/abs/2505.11423)),
+and chain-of-thought is an unreliable report of what actually drove an action
+([arXiv 2505.05410](https://arxiv.org/abs/2505.05410)) — an actor that takes a
+shortcut and then narrates a plausible justification for it is the expected output,
+not an edge case.
+
+What does hold, in the same literature: periodic re-injection of a short persona
+contract (large, cheap, replicated), and grounding a persona in specifics rather
+than exhorting it to be faithful (interview-grounded personas measurably beat prose
+descriptions).
+
+### Outcome-only scoring cannot see modality
+
+WebArena's URL evaluator scores on the final page URL, so a single `goto` to the
+reference URL earns full credit ([arXiv 2307.13854](https://arxiv.org/abs/2307.13854)).
+τ-bench states the general form: a state-based reward is necessary but not
+sufficient, since it cannot see whether policy was followed
+([arXiv 2406.12045](https://arxiv.org/abs/2406.12045)). And the gradient is real —
+calling site APIs instead of driving the UI roughly doubles WebArena scores
+([arXiv 2410.16464](https://arxiv.org/abs/2410.16464)). Any harness that scores only
+outcomes should expect actors to find that gradient, and should not be surprised when
+a green run proves nothing.
+
+## The position this project takes
+
+**Declare, record, and let the adopter judge.**
+
+1. **Declared population is a property of a persona, not of a lab.** One study program
+   can legitimately run agents and people against the same surface; the comparison
+   between them is a finding, not a contradiction. Prior art runs both populations
+   through one instrument and reports the difference
+   ([SusBench](https://arxiv.org/abs/2510.11035); MAS-Bench's GUI/shortcut split,
+   [arXiv 2509.06477](https://arxiv.org/abs/2509.06477)).
+
+2. **Record which affordance class each action used.** Recording is the layer that
+   makes every other claim checkable, including whether the persona prompt worked at
+   all. It is also the only honest answer to a defect whose verdict depends on a
+   declaration the harness cannot verify.
+
+3. **Do not bake a verdict.** Whether an affordance invalidates a study is product
+   semantics, and product semantics belong to the adopter's scorer
+   (`review.scorer.ref`). The harness emits facts; the adopter decides what they mean.
+
+4. **Prompt guidance ships as a nudge whose take-rate is reported, never as the
+   mechanism.** The evidence above is what that sentence is standing on.
+
+5. **Fail closed only on harness integrity, never on product semantics** — a lane that
+   cannot report what it did is a broken instrument, which is a different thing from
+   an actor that behaved unexpectedly.
+
+## What this does not claim
+
+Fidelity here is scoped to **affordance and decision** level: which routes an actor
+took, and whether they belong to the declared population. It is not a claim of
+behavioral realism. At the kinematic level — pointer paths, timing, motor noise — the
+gap between agents and people is total and trivially detectable
+([arXiv 2604.09574](https://arxiv.org/abs/2604.09574)). Any claim beyond the
+affordance/decision level would be dishonest, and this project does not make one.
+
+Two further limits worth stating plainly. Restricting script execution and developer
+tooling for *fidelity* reasons has no precedent — where such restrictions exist
+elsewhere, the stated motive is code-execution safety — so this is new ground rather
+than an inherited convention. And nobody has yet measured whether affordance-class
+recording changes what adopters decide; that is the experiment this layer makes
+possible, not a result it can assume.
+
+## Vocabulary
+
+Terms are borrowed rather than coined, so results stay comparable with the
+surrounding literature:
+
+| Term | Source | Meaning here |
+| --- | --- | --- |
+| action space / action set | WebArena, BrowserGym, OSWorld | the set of actions an actor can express |
+| `nav` subset | BrowserGym | `goto`, `go_back`, `go_forward` — direct navigation, a human affordance |
+| naturalistic actions | AndroidWorld | the human-modality subset, contrasted with exposed function-calling APIs |
+| shortcut action | MAS-Bench | a non-UI route to the same outcome (API, deep link, script) |
+| algorithmic fidelity | Argyle et al., *Out of One, Many* | how well a conditioned model emulates a specific population |
+| Agent Experience (AX) | Biilmann (Netlify) | the experience of a product whose users are agents |
+
+Three collisions to avoid: "agent usability testing" already means an agent
+*simulating* a human, which is the opposite of the agent-population case; `cheat()` in
+BrowserGym means the reference solution, not misbehavior; and "reward hacking" refers
+to attacking a grader, which is a different failure from routing around a UI.

package/docs/product/{cineguessr-3player-external-public.md → lobby-trivia-3player-external-public.md}

@@ -1,6 +1,6 @@
 # Example: 3 mobile personas share ONE public multiplayer lobby at once
 
-`humanish/labs/cineguessr-3player.yaml` is a worked example of the EXTERNAL-PUBLIC shared-world route
+`humanish/labs/lobby-trivia-3player.yaml` is a worked example of the EXTERNAL-PUBLIC shared-world route
 (#164 phase 2): three mobile-LAYOUT personas play the SAME multiplayer lobby on a REAL public
 deployment simultaneously, watched in ONE Observer. Point `subject.appUrl` + `subject.publicTarget` at
 a public deployment YOU own/operate.
@@ -38,14 +38,14 @@ CSS-viewport rendering is the #221 CDP-device-emulation upgrade.
 Dry-run (the default, $0 — proves the plumbing + the honesty contract, no sandboxes, no tokens):
 
 ```
-humanish lab run cineguessr-3player          # or: humanish watch cineguessr-3player
+humanish lab run lobby-trivia-3player          # or: humanish watch lobby-trivia-3player
 ```
 
 Live (opens 3 real mobile-layout seats against the public app):
 
 ```
 # flip scenario.mode to live in the lab (or override), then:
-humanish watch cineguessr-3player --env-file .env.local   # OPENAI_API_KEY + E2B_API_KEY
+humanish watch lobby-trivia-3player --env-file .env.local   # OPENAI_API_KEY + E2B_API_KEY
 ```
 
 Watch it from a phone (today): serve the run directory's Observer through an authed edge —
@@ -69,7 +69,7 @@ No one run implies adoption, scale, or repeatability.
 
 ## Before a live run against a third-party site
 
-The verified cineguessr lobby mechanics (createLobby/joinLobby → `/lobby/CODE`, the 6-char
+The verified lobby-trivia lobby mechanics (createLobby/joinLobby → `/lobby/CODE`, the 6-char
 `ABCDEFGHJKLMNPQRSTUVWXYZ23456789` alphabet, non-auto-join on a direct visit) are read from current
 source and could change on the next deploy. The two coupling points to re-verify are the lobby-code
 regex (`/\/lobby\/([A-Z2-9]{6})(?:$|[/?#])/`, tolerant of a locale prefix and query/hash) and the

package/docs/ramp/README.md

@@ -2,7 +2,7 @@
 
 Status: public-safe contributor and agent ramp.
 
-Package/source version in this tree: `0.36.0` (2026-08-05). The containment boundary introduced in
+Package/source version in this tree: `0.37.0` (2026-08-07). The containment boundary introduced in
 `0.15.1` remains in force: managed run and output paths bind to validated
 physical filesystem identities, and stored provider IDs are evidence, not
 cleanup authority. The bundled OSS meta-lab is dry-run only until

package/docs/release/0.20.0-external-public-shared-world.md

@@ -18,7 +18,7 @@ asserts (byte-stable; a snapshot regression pins it).
   creates the shared lobby, the orchestrator extracts `/lobby/CODE` from its CDP-observed URL and
   threads the code into the follower JOIN missions; followers fail closed WITHOUT opening on
   `HUMANISH_CONCURRENT_SHARED_WORLD_LAB_HANDOFF_TIMEOUT`.
-- **3-mobile cineguessr example** (`humanish/labs/cineguessr-3player.yaml` + an init template): 3
+- **3-mobile lobby-trivia example** (`humanish/labs/lobby-trivia-3player.yaml` + an init template): 3
   mobile-LAYOUT personas play the same multiplayer lobby on a real public app at once. Dry-run $0.
 - **Honest evidence class.** Attribution stays `shared-world`; every strength claim degrades honestly
   and is asserted-ABSENT by verify: `subject.state.provenance == external-public` (not seeded);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "humanish",
-  "version": "0.36.0",
+  "version": "0.37.0",
   "description": "Open-source-safe CLI for persona simulation, observer review, and public-safe feedback drafts.",
   "author": "Daniel G Wilson <daniel@danielgwilson.com>",
   "keywords": [