monkeybot 3.1.0__tar.gz → 3.3.0__tar.gz

{monkeybot-3.1.0 → monkeybot-3.3.0}/CHANGELOG.md

@@ -6,6 +6,47 @@ the project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+## [browser v0.7.0] - 2026-09-10
+
+#### Added
+
+- Announces the chat it is acting for to the Spaces in-app bridge (`Monkeybot.setChatScope`) so tabs stay attached to their chat; silently no-op on older apps.
+
+#### Fixed
+
+- Chat-switch tab drop now fires on real tool calls (`browser_harness()` reuse no longer forgets the last scoped chat).
+
+## [core v3.3.0] - 2026-09-11
+
+### Breaking
+
+- `run_command` now runs inside a deny-by-default OS filesystem jail (macOS `sandbox-exec`; Linux mount-namespace, unvalidated on a real host as part of this change) on top of the existing binary/path allowlist. The user's home directory is denied by default; the workspace, artifacts, memory palace, and OS temp dirs stay read+write, and skills/the interpreter prefix/granted folders stay read-only, including when they sit inside the denied home directory (the normal case for a desktop app). This closes a real gap: the old argv-string path screen never caught indirect path expressions a shell or interpreter builds at runtime (`bash -c 'cp "$HOME/Desktop/x" .'`, or a Python `os.path.expanduser(...)` call), so a command that depended on reading/writing somewhere under `$HOME` outside the workspace — by accident or on purpose — will now fail where it previously succeeded. `git` keeps working via a narrow, explicit `~/.gitconfig`/`~/.config/git/ignore` read carve-out (needed even for `--version`); `gh` does not — its own auth config is credential-adjacent and is not carved out, matching the credential-path denylist already established for `computer_*` tools. If the OS jail mechanism itself is unavailable on a host (older kernels without unprivileged user namespaces, some hardened/CI Linux configs), `run_command` falls back to running **unconfined** (logged once, not per call) rather than refusing every command — a deliberate difference from the existing memory-off hide mechanism, which still fails closed, since this is default-on hardening for every deployment rather than a narrow, deliberate opt-out.
+
+### Added
+
+- MCP tool calls now carry `_meta.monkeybot` (`thread_id`, `request_id`, `run_id`) so servers can attribute work to a chat. Requires `mcp>=1.27.1` so `ClientSession.call_tool(meta=)` is always available.
+- **Command grants:** an unlisted `run_command` binary now surfaces a "Needs your OK" approval card (`Decision.grant_key`/`grant_kind` on `CommandTierInspector`) instead of silently falling through to a hard `SecurityError` at the executor. "Allow once" covers the rest of the turn; "Always allow" writes a durable rule to a new home-level store, `grants.json` (`core/tools/grant_store.py`), shared by every agent on the machine — deliberately separate from the per-agent, `computer_*`-only `approvals.json`. Same on-disk contract (JSON, `0600`, atomic replace, cross-process exclusive-create locking matched with the Electron side).
+- **Folder read grants:** `read_file`/`load_file`/`glob`/`grep` on an absolute path outside the workspace now ask via a new `PathGrantInspector` rather than rejecting outright — denying credential paths and anything outside `$HOME` hard, never promotable. An approved folder is read in place through `WorkspaceFileService.extra_read_roots` (symlink-escape-protected, read-only — never widens what `write_file`/`replace_in_file` can touch); the same granted folders feed the `run_command` jail's read-only set so the shell tool and the file tools agree on what's reachable. Fixes the failure mode where the model, unable to read a file outside its workspace and told only "use a relative path," would narrate copying the file into the workspace to route around the boundary — tool descriptions, the harness prompt, and the `workspace_error_envelope` hint for this case now say to ask for access instead, explicitly calling relocating-to-route-around-the-boundary a policy violation.
+- Settings → Privacy & access gained **Allowed commands** and **Folder access** sections listing/revoking these new grants, alongside the existing per-agent `computer_*` rules.
+
+## [browser v0.6.0] - 2026-09-07
+
+#### Added
+
+- `browser_login` now returns fail-closed errors from the Spaces sealed-login path, scopes grants and errors per run (`MONKEYBOT_RUN_ID` / `MONKEYBOT_RUN_LABEL`), and surfaces TOTP outcome (`mfa`: `none` / `completed` / `needed`) plus injection mode (`mode`: `keystroke` / `network`).
+- `browser_passkey` requests a Spaces-saved passkey with the same allowlisted error surface as `browser_login`. Live passkey sites have not been verified yet.
+
+#### Fixed
+
+- Credential-broker JSON 413 responses fail closed instead of leaking a body, and thinking text is redacted on that path. Tool descriptions are `cleandoc`'d so the golden surface matches CI.
+
+## [core v3.2.0] - 2026-09-07
+
+#### Added
+
+- Opt-in background verifier (`verifier:` in YAML, defaults off, no env mapping). A SQLite goal ledger plus an off-loop classifier record the user's intent; an observe-only progress tracker emits `VerifierVerdict` events; verdicts persist as pinned `verifierVerdict` history rows through compaction. When `max_severity` allows it, the ladder can nudge on the next system extra, replan by emptying tools for one inner turn, or block mutating tools while `parallel_safe` reads still run. Firestore/Postgres have no ledger (fail closed). See `docs/verifier-agent-design.md`.
+- Credential-egress detection: new assistant text and tool results are scanned against Spaces' `/json/scan` before they reach the LLM, with a separate scan of outbound tool arguments (terminal, file write, browser navigation/input, `web_*`) and a local-SQLite-only trace-value scan. A hit is redacted in place; a routine turn ends visibly and an interactive chat continues with the secret withheld. User-authored text is never scanned. Emits `CredentialEgressBlocked`.
+
 ## [core v3.1.0] - 2026-09-05
 
 ### Breaking

{monkeybot-3.1.0 → monkeybot-3.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: monkeybot
-Version: 3.1.0
+Version: 3.3.0
 Summary: monkeybot agent harness — FastAPI SSE gateway, owned loop, SQLite, and MCP.
 License: MIT
 License-File: LICENSE
@@ -9,7 +9,7 @@ Requires-Dist: aiosqlite>=0.20.0
 Requires-Dist: fastapi>=0.115.0
 Requires-Dist: google-genai>=1.0.0
 Requires-Dist: httpx>=0.27.0
-Requires-Dist: mcp<2,>=1.0.0
+Requires-Dist: mcp<2,>=1.27.1
 Requires-Dist: numpy<2.5,>=1.26.0
 Requires-Dist: pydantic>=2.10.0
 Requires-Dist: python-dotenv>=1.0.0

{monkeybot-3.1.0 → monkeybot-3.3.0}/docs/browser-mcp.md

@@ -189,20 +189,81 @@ other ephemeral-workspace targets they reset when the instance is recycled. Use
 the memory storage contract for durable knowledge rather than adding a workspace
 synchronization layer.
 
+## In-app bridge extensions
+
+When the Spaces in-app browser is bound, each tool call may announce which chat
+it is acting for so new tabs stay attached to that chat. AgentCore and Browser
+Use Cloud backends never send this.
+
+### MCP request meta
+
+The gateway includes `_meta` on every `call_tool`:
+
+```json
+{ "monkeybot": { "thread_id": "<gateway thread id>", "request_id": "<turn request id>", "run_id": "<MONKEYBOT_RUN_ID or null>" } }
+```
+
+Servers that ignore `_meta` are unaffected. Older gateways send nothing; this
+server then announces `chatKey` as `null`.
+
+### `Monkeybot.setChatScope`
+
+On every in-app tool call, after the harness binds and before other CDP
+commands, the backend sends:
+
+```json
+{ "id": 7, "method": "Monkeybot.setChatScope", "params": { "chatKey": "<thread_id or null>" } }
+```
+
+`chatKey` is the gateway `thread_id` from `_meta.monkeybot`, which equals the
+app's session ID. The bridge stores it on the WebSocket client until changed. An
+older app replies with an error; this server treats that as unsupported, stops
+sending the method for the rest of the connection, and continues normally.
+Rebinding the in-app daemon (including after Spaces `warmBrowserOnLaunch`)
+clears that disabled flag so a newer app gets a fresh attempt.
+
 ## Tools
 
-Navigation, interaction, screenshots, tabs, waits, playbooks (`browser_list_playbooks`, `browser_read_playbook`, `browser_write_playbook`, `browser_run_playbook`), `browser_login` for Spaces-saved passwords (returns `{ok, loggedIn, origin}` — never the password), and `browser_stop` for daemon cleanup.
+Navigation, interaction, screenshots, tabs, waits, playbooks (`browser_list_playbooks`, `browser_read_playbook`, `browser_write_playbook`, `browser_run_playbook`), `browser_login` for Spaces-saved passwords (returns `{ok, loggedIn, origin, mfa?, mode?}` — never the password), and `browser_stop` for daemon cleanup.
 
 ### `browser_login` targets the focused tab
 
 A sealed login runs in the tab the **user** has focused, because typing the credential requires Spaces to detach CDP from that tab first. Ordinary `browser_*` calls address tabs by CDP session id, so after `browser_switch_tab` — or if the user clicks another tab — the tab the agent is driving and the tab that receives the password are different ones.
 
-Pass `expected_origin` to make the bridge refuse (`focused tab is on a different origin`) instead of signing in somewhere unintended, and check the returned `origin`. A saved credential is scoped to its own origin and must be marked for agent use, so a mismatch can never disclose another site's password — but it can still submit a login the user did not ask for.
+Pass `expected_origin` to make the bridge refuse (`focused tab is on a different origin`) instead of signing in somewhere unintended, and check the returned `origin`. A saved credential is scoped to its own origin and needs its own grant (`Never` / `Ask me` / `Always`) before the agent can use it, so a mismatch can never disclose another site's password — but it can still submit a login the user did not ask for.
 
 A Spaces build predating `expectedOrigin` support ignores it and echoes no `origin`. Since that login cannot be verified after the fact, `browser_login` reports `in-app browser could not verify the origin` rather than a confirmed success; update Spaces to get the check.
 
 Errors raised by browser tools are scrubbed of `?token=` values before they reach the agent: the in-app CDP token grants full control of the user's browser, and browser-harness echoes its endpoint in daemon log tails.
 
+### `browser_login` fails closed
+
+Spaces verifies its own scrub before returning: after typing the password (and a one-time code, if one was needed and available — see below) and submitting, it hashes every editable value left on the page and confirms none of them match what it just typed. If that check cannot be completed — the login did not clearly succeed or fail, or a same-origin iframe could not be checked — Spaces navigates the tab to the site's origin root itself and `browser_login` returns `login needs your attention` instead of guessing. Do not retry the call or attempt to type the password yourself when you see this error; tell the user to finish signing in in the Spaces browser. Calling `browser_get_text` or `browser_get_elements` afterward is safe either way: on this outcome, the page has already been navigated away from any form that held the secret.
+
+### One-time codes (TOTP)
+
+If the page shows a one-time-code field after the password step, `browser_login`'s result carries an `mfa` field:
+
+- `"none"` — no OTP field appeared; nothing to report.
+- `"completed"` — Spaces had a stored authenticator seed for this credential, generated the code, typed and submitted it itself. The password never left Spaces and neither did the code.
+- `"needed"` — an OTP field appeared but no seed is stored. `browser_login` also returns the error `mfa needs your attention`; tell the user to add an authenticator for this site in Spaces' "Connected sites" panel (paste the `otpauth://` URI or the base32 secret from the site's QR setup) or finish signing in themselves. Do not retry.
+
+### How the password reaches the request
+
+`browser_login`'s result also carries a `mode` field: `"keystroke"` (the password was typed into the page, the default) or `"network"` (Spaces typed a harmless placeholder and rewrote the real password into the outgoing request itself, so the password never touched the page's DOM or JS). Both behave identically from the agent's side — this is informational only, set per credential in Spaces' "Connected sites" panel ("Auto" tries network first and remembers if a site doesn't support it).
+
+### Grants and "ask" mode
+
+Each saved credential has an agent-access grant: `Never` (default for imported passwords — the agent cannot use it at all), `Ask me` (the user is prompted every time, unless the request happens again within an already-approved run), or `Always` (a standing grant, replacing the old "Agent" checkbox). Grants can also be scoped to one run — an "Allow once" approval covers only the routine or chat that asked for it, and expires after 30 minutes or one use, whichever comes first.
+
+In `Ask me` mode, `browser_login` blocks — up to 150s — while Spaces shows the user a prompt ("*Run* wants to sign in to *origin* as *username*") with "Allow once", "Always allow", and "Don't allow" buttons, gated by a click plus Touch ID on macOS. Three outcomes reach the agent:
+
+- `waiting for your approval` — the user did not respond within the window, or another approval was already pending. Do not retry; tell the user their approval is (or was) needed.
+- `agent access denied for this site` — the user declined, or the credential's grant is `Never`.
+- `grant expired` — a scoped grant that once covered this exact run lapsed with no `Ask me` fallback to re-prompt through.
+
+Every credential use — granted, denied, or needing attention — is recorded in Spaces' "Connected sites" panel (reachable from the Saved passwords strip) along with which run used it.
+
 ### Default: indexed DOM interaction (no screenshots needed)
 
 `browser_get_elements`, `browser_click_by_index`, `browser_input_by_index`, and `browser_select_by_index` give the agent a **text-based, indexed** view of the page instead of screenshot + pixel coordinates — ported from [alibaba/page-agent](https://github.com/alibaba/page-agent)'s DOM-extraction engine (MIT licensed; see `src/browser_mcp/dom_indexing.py` and `src/browser_mcp/assets/` for provenance).

{monkeybot-3.1.0 → monkeybot-3.3.0}/docs/features.md

@@ -259,20 +259,20 @@ Each section follows: **Purpose** · **Key files** · **How it works** · **Depe
 
 **Purpose:** Default `ToolExecutorPort` — built-ins, MCP, custom tools, subagents.
 
-**Key files:** `core/tools/core_tool_executor.py`, `terminal.py`, `workspace_service.py`, `patch.py`, `sandbox_executor.py`, `spill_inventory.py`
+**Key files:** `core/tools/core_tool_executor.py`, `terminal.py`, `workspace_service.py`, `patch.py`, `sandbox_executor.py`, `spill_inventory.py`, `fs_isolation.py`, `grant_store.py`
 
 **Built-in tools** (from `context._core_tool_defs`):
 
 | Tool | Role |
 |------|------|
-| `read_file` / `write_file` | `skills/...` read-only or workspace-relative paths |
+| `read_file` / `write_file` | `skills/...` read-only or workspace-relative paths; `read_file`/`load_file` also accept an absolute path outside the workspace once the user grants that folder (see §5) — read in place, never copied in |
 | `replace_in_file` | Unique (or `replace_all`) substring edit; exact then light fuzzy match |
-| `glob` / `grep` | Path discovery / content regex search (prefer over shell) |
+| `glob` / `grep` | Path discovery / content regex search (prefer over shell); `root` accepts a granted external folder the same way `read_file` does |
 | `apply_patch` | Multi-file Codex-style Add/Update/Delete/Move; fail-closed before any write |
 | `search_memory` | Keyword search in memory tree |
 | `search` / `recall` | Local knowledge index (workspace + notes); **parallel-safe**. Gateway owns writes; subagents open the index **read-only**. Harness-as-library (Pattern B/C) callers can opt into read-only via `MONKEYBOT_KNOWLEDGE_READ_ONLY=1`; the gateway always ignores this flag and stays the writer |
 | `list_skills` | Skill discovery |
-| `run_command` | Allowlisted shell (host or OpenSandbox) |
+| `run_command` | Allowlisted **and OS-confined** shell (host or OpenSandbox) — see the workspace jail below |
 | `task` | Subagent subprocess (parent only) |
 | `enable_mcp` / `disable_mcp` | Catalog connect / disconnect (success includes status) |
 | `enable_loops` | Progressive advertise scheduled-loop tools |
@@ -296,6 +296,8 @@ Each section follows: **Purpose** · **Key files** · **How it works** · **Depe
 - Nested `task` disabled inside subagents.
 - Custom tools must not collide with core or MCP names.
 - `MONKEYBOT_KNOWLEDGE_READ_ONLY` (default off) opens the knowledge index read-only for `create_harness_deps` (Pattern B/C) callers; subagent workers are always read-only regardless of the flag. The gateway SSE app is the sole writer per workspace and ignores this flag, logging a warning if it is set.
+- **The workspace jail:** `run_command` runs under a deny-by-default OS filesystem policy (`fs_isolation.py::JailRoots`/`jailed_argv`), not just the `allowed_path_prefixes` argv screen (which a shell or interpreter defeats trivially — `bash -c 'cp "$HOME/Desktop/x" .'` never contains a literal `~`/`/` token an argv scanner can catch). The jail denies the user's home directory by default, then allows the workspace/artifacts/memory-palace/OS-temp-dirs back on top (read+write) and skills/interpreter-prefix/granted folders (read-only) — including when those sit inside the denied home directory, the normal case for a desktop app. Implemented for macOS `sandbox-exec`; the Linux mount-namespace path follows the same structure but is unvalidated on a real Linux host — both fail *closed* (refuse to run) if the OS mechanism can't be established, except that the jail itself (unlike the older memory-hide mechanism) fails *open*, once, with a logged warning, if neither OS mechanism is available at all — a default-on hardening that must not turn into "run_command stops working" on unsupported hosts.
+- **Folder read grants:** `read_file`/`load_file`/`glob`/`grep` on an absolute path outside the workspace ask via `PathGrantInspector` (§5) rather than dead-ending; an approved folder is read in place through `WorkspaceFileService.extra_read_roots` — nothing is copied into the workspace, and the same granted folders feed the jail's read-only set above so `run_command` and the file tools agree on what's reachable.
 
 ---
 
@@ -303,13 +305,14 @@ Each section follows: **Purpose** · **Key files** · **How it works** · **Depe
 
 **Purpose:** Pre-flight policy gates; supports user confirmation via SSE.
 
-**Key files:** `core/tools/inspector.py`, `core/tools/permission.py`, `core/tools/loop_inspector.py`
+**Key files:** `core/tools/inspector.py`, `core/tools/permission.py`, `core/tools/loop_inspector.py`, `core/tools/path_grant_inspector.py`, `core/tools/grant_store.py`
 
 **Inspectors wired at gateway startup (order matters — first deny/confirm wins):**
-1. `CommandTierInspector` — `command_allowlist.yaml` deny-regex preflight; execution allowlists stay on the executor
-2. `RulesInspector` — `MONKEYBOT_TOOL_DENIED_PATTERNS` substring deny list (backward compat)
-3. `PermissionInspector` — `permissions.yaml` last-match-wins `allow` / `ask` / `deny` ruleset (`PERMISSION_CONFIG`)
-4. `LoopStartInspector` — `start_loop` always asks for confirmation (rich plan preview)
+1. `CommandTierInspector` — `command_allowlist.yaml` deny-regex preflight (hard, never promotable), then a binary allow/ask gate: a binary already on the static allowlist, in `grants.json`, or granted "Allow once" earlier this turn is allowed and the chain continues (`permissions.yaml` can still ask/deny a finer-grained pattern on top, unchanged); any other binary asks (`confirm`, carrying `grant_key`=the binary) instead of silently falling through to a hard rejection at the executor
+2. `PathGrantInspector` — for `read_file`/`load_file`/`glob`/`grep`, an absolute/`~` path inside the workspace is a no-op (allow); on the credential denylist or outside `$HOME` it's a hard deny; already covered by a `grants.json`/turn-scoped folder grant it's allow; otherwise it asks (`confirm`, `grant_key`=the containing folder, `grant_kind="path"`)
+3. `RulesInspector` — `MONKEYBOT_TOOL_DENIED_PATTERNS` substring deny list (backward compat)
+4. `PermissionInspector` — `permissions.yaml` last-match-wins `allow` / `ask` / `deny` ruleset (`PERMISSION_CONFIG`)
+5. `LoopStartInspector` — `start_loop` always asks for confirmation (rich plan preview)
 
 **Permission ruleset (`permissions.yaml`):**
 - Rules are ordered; **last match wins** (OpenCode-style).
@@ -318,15 +321,17 @@ Each section follows: **Purpose** · **Key files** · **How it works** · **Depe
 - `default:` applies when nothing matches (shipped default: `allow`).
 - Session approvals: POST tool-confirmation with `{approved: true, always: true}` remembers tool+resource for the rest of the session (`SessionBus.session_approvals`).
 
-**Decision kinds:** `allow` | `deny` | `confirm` (requires `ctx.sse_bus` for pending UI response).
+**Decision kinds:** `allow` | `deny` | `confirm` (requires `ctx.sse_bus` for pending UI response). A `confirm` may carry `grant_key`/`grant_kind` (`"command"` | `"path"`) — what an "Allow once"/"Always allow" click actually remembers (a binary name or a folder, never the full resource string `resource_for_call()` would otherwise use, which would make a durable rule cover exactly one invocation). `tool_dispatch.py` routes an "always" approval to `ctx.grants_persist` (writing `grants.json`, shared by every agent) when `grant_kind` is set, instead of the `computer_*`-only `ctx.approvals_persist`; either way, "Allow once" also records the grant on `ctx.turn_command_grants`/`ctx.turn_path_grants` so a repeat call later in the same turn does not ask again even without a durable write.
 
-**Hard constraints underneath soft asks:** `allowed_commands` / `allowed_path_prefixes` + sandbox still enforce at execution time — permission `ask`/`allow` cannot bypass them.
+**`grants.json`** (`core/tools/grant_store.py`) is a home-level store — one file shared by every agent on the machine, deliberately distinct from the per-agent, `computer_*`-only `approvals.json` (§21). Same on-disk contract: JSON, mode `0600`, atomic replace, cross-process exclusive-create locking matched byte-for-byte with the Electron side (`electron/main/grants.ts`).
+
+**Hard constraints underneath soft asks:** `allowed_commands` / `allowed_path_prefixes` + sandbox still enforce at execution time — permission `ask`/`allow` cannot bypass them. Beyond the argv allowlist, `run_command` additionally runs inside a deny-by-default OS filesystem jail (§4) that a shell/interpreter cannot argue its way around by rewriting a path.
 
 **Invariants:**
 - Missing allowlist file → tier inspector skipped (logged); executor falls back to code defaults.
 - Missing `permissions.yaml` → soft ruleset disabled (logged).
-- Default deny patterns block package installs (`pip install`, `npm install`, etc.).
-- Confirm without `sse_bus` → deny.
+- Default deny patterns block package installs (`pip install`, `npm install`, etc.) and are never promotable to a confirm by any grant.
+- Confirm without `sse_bus` → deny (this is why a subagent, which has no interactive session, can use an already-durably-granted command/folder transparently but can never ask for a new one).
 
 ---
 
@@ -652,7 +657,7 @@ Before spawning the gateway, the CLI probes that interpreter for MonkeyBot `>=3.
 
 **How it works:**
 - Off by default, macOS-only: `computer.enabled: true` in `monkeybot.yaml` → `MONKEYBOT_COMPUTER_TOOLS` env, combined with a hard `sys.platform == "darwin"` check (`should_enable_computer_tools()`). The desktop app sets the env var when it spawns a gateway with the feature turned on in Settings; no other deployment should ever set it.
-- **Hard security boundary lives in the tool bodies, not in `permissions.yaml`** (`safety.py`): every path is resolved and validated against the user's home directory after following symlinks; a fixed denylist blocks credential directories (`.ssh`, `.aws`, keychains, browser profiles, the app's own config) both at their canonical location *and* by directory-name anywhere in the tree; filenames matching credential patterns (`.env`, `*.pem`, `id_rsa*`, …) are always refused; `computer_open` refuses any path/app that would make `open` execute code (`.command`, `.app`, `.sh`, Terminal, script editors, …); trash never hard-deletes. `permissions.yaml` is fail-open (a broken file silently disables it), so none of this can depend on it.
+- **Hard security boundary lives in the tool bodies, not in `permissions.yaml`** (`safety.py`): every path is resolved and validated against the user's home directory after following symlinks; a fixed denylist blocks credential directories (`.ssh`, `.aws`, keychains, browser profiles, the app's own config) both at their canonical location *and* by directory-name anywhere in the tree; filenames matching credential patterns (`.env`, `*.pem`, `id_rsa*`, …) are always refused; `computer_open` refuses any path/app that would make `open` execute code (`.command`, `.app`, `.sh`, Terminal, script editors, …); trash never hard-deletes. `permissions.yaml` is fail-open (a broken file silently disables it), so none of this can depend on it. `_denied_dirs()` also always includes `MONKEYBOT_WORKSPACE_ROOT` (not only under `MONKEYBOT_APP_HOME`, which only the desktop app sets) — a CLI-scaffolded agent's workspace can sit anywhere under `$HOME` the user chose, and `computer_move`/`computer_trash` must not be able to relocate a file straight into or out of it, one click away from smuggling exactly what the workspace boundary (§4) exists to prevent. `computer_move`'s own description now says so, so the model doesn't reach for it as a `read_file`/`run_command` workaround.
 - **Every `computer_*` call asks by default**, via a built-in baseline rule (`COMPUTER_BASELINE_RULES` in `permissions.py`) — not a line in `permissions.yaml`, so it can't be silently defeated by a missing/broken config file. `ComputerAwarePermissionInspector` layers, last-match-wins: baseline `ask` < durable approvals overlay `allow` < the user's `permissions.yaml` (highest authority — a hand-written `deny` always wins).
 - **"Always allow" is durable and narrow**: `monkeybot_config/approvals.json` (machine-written, JSON — deliberately *not* appended into `permissions.yaml`, which is a comment-heavy file the app's Advanced settings save wholesale) stores one exact `(tool, resource)` pair per rule. Mutating tools (`computer_move`, `computer_trash`) are excluded from `ALWAYS_SCOPE` — their resource is the *source* path only, so an "always" rule would cover any destination — every call to them asks. The inspector re-`stat`s the overlay file on every check and reloads on change; on a genuine change it also clears the session-level approval cache, so a revoke in the app's Settings takes effect immediately rather than waiting for a gateway restart.
 - Registered via the standard `extra_tools` extension point in both the SSE (`gateway/sse/app.py`) and realtime (`gateway/realtime/routes.py`) gateways — same mechanism as `web_search`/`todo_list`. Never registered for subagents (`subagent_worker.py` builds its own `extra_tools` independently and never imports this package).

monkeybot-3.3.0/docs/live-evals.md

@@ -0,0 +1,112 @@
+# Live evals
+
+Run real-provider smoke scenarios against a live monkeybot gateway, score them, and optionally compare to a baseline. Complements deterministic loop evals under `tests/evals/` (fake provider / pytest).
+
+**Fixture agent:** [`evals/smoke_agent/`](../evals/smoke_agent/)  
+**Workflow:** [`.github/workflows/live-eval-smoke.yml`](../.github/workflows/live-eval-smoke.yml)  
+**Suite:** [`evals/suites/smoke.yaml`](../evals/suites/smoke.yaml)
+
+## Local run
+
+```bash
+# Terminal 1 — gateway
+cd evals/smoke_agent
+uv sync
+export OLLAMA_API_KEY=…          # ollama.com/settings/keys; extra is monkeybot[ollama]
+# model.provider / model.name come from monkeybot_config/monkeybot.yaml (YAML-only)
+export PORT=8787
+export SANDBOX_ENABLED=false
+uv run -m monkeybot.gateway.main
+
+# Terminal 2 — scorecard (repo root)
+uv sync --extra evals
+export AGENT_URL=http://127.0.0.1:8787
+export JUDGE_PROVIDER=fake       # deepeval's GPTModel against ollama.com times out with no scores
+export EVAL_SCENARIO_DELAY_SEC=10   # space scenarios out if the model gets rate-limited
+uv run python -m evals.report \
+  --suite smoke \
+  --baseline evals/baselines/smoke.json
+```
+
+Useful flags:
+
+| Flag | Effect |
+|---|---|
+| `--fail-on-regression` | Exit non-zero on scenario errors/failures (and baseline drift when a baseline is present) |
+| `--require-baseline` | Fail if the baseline file is missing |
+| `--update-baseline` | Rewrite the baseline from this run (commit deliberately) |
+| `--scenario-delay-sec` (or `EVAL_SCENARIO_DELAY_SEC`) | Sleep this many seconds between scenarios — a scenario itself fires several requests in a tight burst, so pacing is applied between scenarios, not within one. Use when a low-rpm model gets provider-side rate-limited. |
+
+Runs land under `evals/runs/` (gitignored). Diff two artifacts with `uv run python -m evals.diff <old.json> <new.json>`.
+
+## Drift suite (paired verifier off / on)
+
+The drift suite (`evals/suites/drift.yaml`, scenarios under `evals/scenarios/drift/`) is **not** in smoke. It exists to measure verifier precision/recall once Phase 2+ lands. Cases `supersession` and `preempt` assert `max_verdicts: 0` with the verifier on — they are the precision half.
+
+The gateway discovers `monkeybot_config/monkeybot.yaml` from cwd, so a second filename is never loaded unless you point `MONKEYBOT_CONFIG` at it (bootstrap pointer, not a YAML value).
+
+```bash
+# Terminal 1a — verifier off (default smoke config)
+cd evals/smoke_agent
+uv sync
+export OLLAMA_API_KEY=…
+export PORT=8787
+export SANDBOX_ENABLED=false
+uv run -m monkeybot.gateway.main
+
+# Terminal 2a
+cd ../..   # repo root
+uv sync --extra evals
+export AGENT_URL=http://127.0.0.1:8787
+export EVAL_VERIFIER_MODE=off
+uv run python -m evals.report --suite drift
+# note the run id printed / the new file under evals/runs/
+
+# Stop the gateway, then boot with the verifier-on file:
+cd evals/smoke_agent
+export MONKEYBOT_CONFIG=./monkeybot_config/monkeybot.verifier-on.yaml
+uv run -m monkeybot.gateway.main
+
+# Terminal 2b
+export EVAL_VERIFIER_MODE=on
+uv run python -m evals.report --suite drift
+uv run python -m evals.diff evals/runs/<off-run>.json evals/runs/<on-run>.json
+```
+
+Until a consumer reads `verifier:` (Phase 1+), the on-config is a parsed no-op and the on/off pair should be identical besides the YAML digest.
+
+## Smoke baseline (pre-verifier)
+
+Committed at [`evals/baselines/smoke.json`](../evals/baselines/smoke.json); readable summary in [`evals/baselines/smoke.md`](../evals/baselines/smoke.md). Agent is `ollama-cloud` / `glm-5.3-flash`. Judge scores are empty (`JUDGE_PROVIDER=fake`) — deepeval's OpenAI-compatible judge against Ollama Cloud timed out with no metric scores.
+
+```bash
+cd evals/smoke_agent && uv sync
+OLLAMA_API_KEY=… PORT=8787 SANDBOX_ENABLED=false uv run -m monkeybot.gateway.main &
+cd ../.. && AGENT_URL=http://127.0.0.1:8787 JUDGE_PROVIDER=fake \
+  uv run python -m evals.report --suite smoke --update-baseline
+```
+
+Do not add `--require-baseline` to CI until that workflow uses the same provider as this file. The GitHub Action still keys off `NVIDIA_API_KEY`.
+
+## What the smoke suite covers
+
+Scenarios under `evals/scenarios/` assert both judge metrics and harness telemetry (e.g. `required_tools`). Current smoke membership is the explicit list in `evals/suites/smoke.yaml` (tools, skills, subagents, memory, multi-turn, mcp — `context/summarization_trigger` is currently commented out, see `evals/TEST_COVERAGE.md`). MCP scenarios run against an in-process fixture MCP server (`evals/smoke_agent/fixture_mcp_server.py`, no network calls) wired up in `evals/smoke_agent/monkeybot_config/mcp.json` — see `evals/scenarios/mcp/`.
+
+## CI
+
+Live eval does **not** run on every feature PR. Triggers (see the workflow):
+
+- PR from `develop` → `main` (release gate; uses `--fail-on-regression`)
+- PR that changes `uv.lock`, `cli/uv.lock`, or `evals/smoke_agent/uv.lock`
+- Push to `main` (report-only)
+- Manual `workflow_dispatch`
+
+CI still requires repository secret `NVIDIA_API_KEY` (legacy agent + judge on build.nvidia.com). Local smoke uses `OLLAMA_API_KEY` and `monkeybot.yaml`'s `ollama-cloud` / `glm-5.3-flash`. The scorecard is posted to the run summary and as a PR comment.
+
+## vs `tests/evals/`
+
+| | `tests/evals/` | `evals/` (this doc) |
+|---|---|---|
+| Driver | `loop.run` + fake provider | Live gateway HTTP + SSE |
+| CI | Every push/PR via pytest | Selective (above) |
+| Catches | Deterministic loop/tool wiring | Real provider + gateway/SSE behavior |