monkeybot 3.1.0__tar.gz → 3.2.0__tar.gz

{monkeybot-3.1.0 → monkeybot-3.2.0}/CHANGELOG.md

@@ -6,6 +6,24 @@ the project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Browser MCP
+
+#### Added
+
+- `browser_login` now returns fail-closed errors from the Spaces sealed-login path, scopes grants and errors per run (`MONKEYBOT_RUN_ID` / `MONKEYBOT_RUN_LABEL`), and surfaces TOTP outcome (`mfa`: `none` / `completed` / `needed`) plus injection mode (`mode`: `keystroke` / `network`).
+- `browser_passkey` requests a Spaces-saved passkey with the same allowlisted error surface as `browser_login`. Live passkey sites have not been verified yet.
+
+#### Fixed
+
+- Credential-broker JSON 413 responses fail closed instead of leaking a body, and thinking text is redacted on that path. Tool descriptions are `cleandoc`'d so the golden surface matches CI.
+
+## [core v3.2.0] - 2026-09-07
+
+#### Added
+
+- Opt-in background verifier (`verifier:` in YAML, defaults off, no env mapping). A SQLite goal ledger plus an off-loop classifier record the user's intent; an observe-only progress tracker emits `VerifierVerdict` events; verdicts persist as pinned `verifierVerdict` history rows through compaction. When `max_severity` allows it, the ladder can nudge on the next system extra, replan by emptying tools for one inner turn, or block mutating tools while `parallel_safe` reads still run. Firestore/Postgres have no ledger (fail closed). See `docs/verifier-agent-design.md`.
+- Credential-egress detection: new assistant text and tool results are scanned against Spaces' `/json/scan` before they reach the LLM, with a separate scan of outbound tool arguments (terminal, file write, browser navigation/input, `web_*`) and a local-SQLite-only trace-value scan. A hit is redacted in place; a routine turn ends visibly and an interactive chat continues with the secret withheld. User-authored text is never scanned. Emits `CredentialEgressBlocked`.
+
 ## [core v3.1.0] - 2026-09-05
 
 ### Breaking

{monkeybot-3.1.0 → monkeybot-3.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: monkeybot
-Version: 3.1.0
+Version: 3.2.0
 Summary: monkeybot agent harness — FastAPI SSE gateway, owned loop, SQLite, and MCP.
 License: MIT
 License-File: LICENSE

{monkeybot-3.1.0 → monkeybot-3.2.0}/docs/browser-mcp.md

@@ -191,18 +191,46 @@ synchronization layer.
 
 ## Tools
 
-Navigation, interaction, screenshots, tabs, waits, playbooks (`browser_list_playbooks`, `browser_read_playbook`, `browser_write_playbook`, `browser_run_playbook`), `browser_login` for Spaces-saved passwords (returns `{ok, loggedIn, origin}` — never the password), and `browser_stop` for daemon cleanup.
+Navigation, interaction, screenshots, tabs, waits, playbooks (`browser_list_playbooks`, `browser_read_playbook`, `browser_write_playbook`, `browser_run_playbook`), `browser_login` for Spaces-saved passwords (returns `{ok, loggedIn, origin, mfa?, mode?}` — never the password), and `browser_stop` for daemon cleanup.
 
 ### `browser_login` targets the focused tab
 
 A sealed login runs in the tab the **user** has focused, because typing the credential requires Spaces to detach CDP from that tab first. Ordinary `browser_*` calls address tabs by CDP session id, so after `browser_switch_tab` — or if the user clicks another tab — the tab the agent is driving and the tab that receives the password are different ones.
 
-Pass `expected_origin` to make the bridge refuse (`focused tab is on a different origin`) instead of signing in somewhere unintended, and check the returned `origin`. A saved credential is scoped to its own origin and must be marked for agent use, so a mismatch can never disclose another site's password — but it can still submit a login the user did not ask for.
+Pass `expected_origin` to make the bridge refuse (`focused tab is on a different origin`) instead of signing in somewhere unintended, and check the returned `origin`. A saved credential is scoped to its own origin and needs its own grant (`Never` / `Ask me` / `Always`) before the agent can use it, so a mismatch can never disclose another site's password — but it can still submit a login the user did not ask for.
 
 A Spaces build predating `expectedOrigin` support ignores it and echoes no `origin`. Since that login cannot be verified after the fact, `browser_login` reports `in-app browser could not verify the origin` rather than a confirmed success; update Spaces to get the check.
 
 Errors raised by browser tools are scrubbed of `?token=` values before they reach the agent: the in-app CDP token grants full control of the user's browser, and browser-harness echoes its endpoint in daemon log tails.
 
+### `browser_login` fails closed
+
+Spaces verifies its own scrub before returning: after typing the password (and a one-time code, if one was needed and available — see below) and submitting, it hashes every editable value left on the page and confirms none of them match what it just typed. If that check cannot be completed — the login did not clearly succeed or fail, or a same-origin iframe could not be checked — Spaces navigates the tab to the site's origin root itself and `browser_login` returns `login needs your attention` instead of guessing. Do not retry the call or attempt to type the password yourself when you see this error; tell the user to finish signing in in the Spaces browser. Calling `browser_get_text` or `browser_get_elements` afterward is safe either way: on this outcome, the page has already been navigated away from any form that held the secret.
+
+### One-time codes (TOTP)
+
+If the page shows a one-time-code field after the password step, `browser_login`'s result carries an `mfa` field:
+
+- `"none"` — no OTP field appeared; nothing to report.
+- `"completed"` — Spaces had a stored authenticator seed for this credential, generated the code, typed and submitted it itself. The password never left Spaces and neither did the code.
+- `"needed"` — an OTP field appeared but no seed is stored. `browser_login` also returns the error `mfa needs your attention`; tell the user to add an authenticator for this site in Spaces' "Connected sites" panel (paste the `otpauth://` URI or the base32 secret from the site's QR setup) or finish signing in themselves. Do not retry.
+
+### How the password reaches the request
+
+`browser_login`'s result also carries a `mode` field: `"keystroke"` (the password was typed into the page, the default) or `"network"` (Spaces typed a harmless placeholder and rewrote the real password into the outgoing request itself, so the password never touched the page's DOM or JS). Both behave identically from the agent's side — this is informational only, set per credential in Spaces' "Connected sites" panel ("Auto" tries network first and remembers if a site doesn't support it).
+
+### Grants and "ask" mode
+
+Each saved credential has an agent-access grant: `Never` (default for imported passwords — the agent cannot use it at all), `Ask me` (the user is prompted every time, unless the request happens again within an already-approved run), or `Always` (a standing grant, replacing the old "Agent" checkbox). Grants can also be scoped to one run — an "Allow once" approval covers only the routine or chat that asked for it, and expires after 30 minutes or one use, whichever comes first.
+
+In `Ask me` mode, `browser_login` blocks — up to 150s — while Spaces shows the user a prompt ("*Run* wants to sign in to *origin* as *username*") with "Allow once", "Always allow", and "Don't allow" buttons, gated by a click plus Touch ID on macOS. Three outcomes reach the agent:
+
+- `waiting for your approval` — the user did not respond within the window, or another approval was already pending. Do not retry; tell the user their approval is (or was) needed.
+- `agent access denied for this site` — the user declined, or the credential's grant is `Never`.
+- `grant expired` — a scoped grant that once covered this exact run lapsed with no `Ask me` fallback to re-prompt through.
+
+Every credential use — granted, denied, or needing attention — is recorded in Spaces' "Connected sites" panel (reachable from the Saved passwords strip) along with which run used it.
+
 ### Default: indexed DOM interaction (no screenshots needed)
 
 `browser_get_elements`, `browser_click_by_index`, `browser_input_by_index`, and `browser_select_by_index` give the agent a **text-based, indexed** view of the page instead of screenshot + pixel coordinates — ported from [alibaba/page-agent](https://github.com/alibaba/page-agent)'s DOM-extraction engine (MIT licensed; see `src/browser_mcp/dom_indexing.py` and `src/browser_mcp/assets/` for provenance).

monkeybot-3.2.0/docs/live-evals.md

@@ -0,0 +1,112 @@
+# Live evals
+
+Run real-provider smoke scenarios against a live monkeybot gateway, score them, and optionally compare to a baseline. Complements deterministic loop evals under `tests/evals/` (fake provider / pytest).
+
+**Fixture agent:** [`evals/smoke_agent/`](../evals/smoke_agent/)  
+**Workflow:** [`.github/workflows/live-eval-smoke.yml`](../.github/workflows/live-eval-smoke.yml)  
+**Suite:** [`evals/suites/smoke.yaml`](../evals/suites/smoke.yaml)
+
+## Local run
+
+```bash
+# Terminal 1 — gateway
+cd evals/smoke_agent
+uv sync
+export OLLAMA_API_KEY=…          # ollama.com/settings/keys; extra is monkeybot[ollama]
+# model.provider / model.name come from monkeybot_config/monkeybot.yaml (YAML-only)
+export PORT=8787
+export SANDBOX_ENABLED=false
+uv run -m monkeybot.gateway.main
+
+# Terminal 2 — scorecard (repo root)
+uv sync --extra evals
+export AGENT_URL=http://127.0.0.1:8787
+export JUDGE_PROVIDER=fake       # deepeval's GPTModel against ollama.com times out with no scores
+export EVAL_SCENARIO_DELAY_SEC=10   # space scenarios out if the model gets rate-limited
+uv run python -m evals.report \
+  --suite smoke \
+  --baseline evals/baselines/smoke.json
+```
+
+Useful flags:
+
+| Flag | Effect |
+|---|---|
+| `--fail-on-regression` | Exit non-zero on scenario errors/failures (and baseline drift when a baseline is present) |
+| `--require-baseline` | Fail if the baseline file is missing |
+| `--update-baseline` | Rewrite the baseline from this run (commit deliberately) |
+| `--scenario-delay-sec` (or `EVAL_SCENARIO_DELAY_SEC`) | Sleep this many seconds between scenarios — a scenario itself fires several requests in a tight burst, so pacing is applied between scenarios, not within one. Use when a low-rpm model gets provider-side rate-limited. |
+
+Runs land under `evals/runs/` (gitignored). Diff two artifacts with `uv run python -m evals.diff <old.json> <new.json>`.
+
+## Drift suite (paired verifier off / on)
+
+The drift suite (`evals/suites/drift.yaml`, scenarios under `evals/scenarios/drift/`) is **not** in smoke. It exists to measure verifier precision/recall once Phase 2+ lands. Cases `supersession` and `preempt` assert `max_verdicts: 0` with the verifier on — they are the precision half.
+
+The gateway discovers `monkeybot_config/monkeybot.yaml` from cwd, so a second filename is never loaded unless you point `MONKEYBOT_CONFIG` at it (bootstrap pointer, not a YAML value).
+
+```bash
+# Terminal 1a — verifier off (default smoke config)
+cd evals/smoke_agent
+uv sync
+export OLLAMA_API_KEY=…
+export PORT=8787
+export SANDBOX_ENABLED=false
+uv run -m monkeybot.gateway.main
+
+# Terminal 2a
+cd ../..   # repo root
+uv sync --extra evals
+export AGENT_URL=http://127.0.0.1:8787
+export EVAL_VERIFIER_MODE=off
+uv run python -m evals.report --suite drift
+# note the run id printed / the new file under evals/runs/
+
+# Stop the gateway, then boot with the verifier-on file:
+cd evals/smoke_agent
+export MONKEYBOT_CONFIG=./monkeybot_config/monkeybot.verifier-on.yaml
+uv run -m monkeybot.gateway.main
+
+# Terminal 2b
+export EVAL_VERIFIER_MODE=on
+uv run python -m evals.report --suite drift
+uv run python -m evals.diff evals/runs/<off-run>.json evals/runs/<on-run>.json
+```
+
+Until a consumer reads `verifier:` (Phase 1+), the on-config is a parsed no-op and the on/off pair should be identical besides the YAML digest.
+
+## Smoke baseline (pre-verifier)
+
+Committed at [`evals/baselines/smoke.json`](../evals/baselines/smoke.json); readable summary in [`evals/baselines/smoke.md`](../evals/baselines/smoke.md). Agent is `ollama-cloud` / `glm-5.3-flash`. Judge scores are empty (`JUDGE_PROVIDER=fake`) — deepeval's OpenAI-compatible judge against Ollama Cloud timed out with no metric scores.
+
+```bash
+cd evals/smoke_agent && uv sync
+OLLAMA_API_KEY=… PORT=8787 SANDBOX_ENABLED=false uv run -m monkeybot.gateway.main &
+cd ../.. && AGENT_URL=http://127.0.0.1:8787 JUDGE_PROVIDER=fake \
+  uv run python -m evals.report --suite smoke --update-baseline
+```
+
+Do not add `--require-baseline` to CI until that workflow uses the same provider as this file. The GitHub Action still keys off `NVIDIA_API_KEY`.
+
+## What the smoke suite covers
+
+Scenarios under `evals/scenarios/` assert both judge metrics and harness telemetry (e.g. `required_tools`). Current smoke membership is the explicit list in `evals/suites/smoke.yaml` (tools, skills, subagents, memory, multi-turn, mcp — `context/summarization_trigger` is currently commented out, see `evals/TEST_COVERAGE.md`). MCP scenarios run against an in-process fixture MCP server (`evals/smoke_agent/fixture_mcp_server.py`, no network calls) wired up in `evals/smoke_agent/monkeybot_config/mcp.json` — see `evals/scenarios/mcp/`.
+
+## CI
+
+Live eval does **not** run on every feature PR. Triggers (see the workflow):
+
+- PR from `develop` → `main` (release gate; uses `--fail-on-regression`)
+- PR that changes `uv.lock`, `cli/uv.lock`, or `evals/smoke_agent/uv.lock`
+- Push to `main` (report-only)
+- Manual `workflow_dispatch`
+
+CI still requires repository secret `NVIDIA_API_KEY` (legacy agent + judge on build.nvidia.com). Local smoke uses `OLLAMA_API_KEY` and `monkeybot.yaml`'s `ollama-cloud` / `glm-5.3-flash`. The scorecard is posted to the run summary and as a PR comment.
+
+## vs `tests/evals/`
+
+| | `tests/evals/` | `evals/` (this doc) |
+|---|---|---|
+| Driver | `loop.run` + fake provider | Live gateway HTTP + SSE |
+| CI | Every push/PR via pytest | Selective (above) |
+| Catches | Deterministic loop/tool wiring | Real provider + gateway/SSE behavior |