monkeybot 2.2.0__tar.gz → 3.0.0__tar.gz

monkeybot-3.0.0/.github/workflows/live-eval-smoke.yml

@@ -0,0 +1,213 @@
+name: Live eval smoke
+
+# Runs the live eval smoke suite (see docs/live-evals.md) and posts a Markdown
+# scorecard. LLM-judge calls cost money and take minutes, so this does not run on
+# ordinary feature-branch -> develop PRs. Trigger points:
+#   (a) any PR from develop into main (release-promotion gate; --fail-on-regression)
+#   (b) any PR that touches a dependency lockfile (catches silent SDK breakage,
+#       e.g. provider request/response shape changes, that code review won't see)
+#   (c) every push to main (scores the new release; report-only, never fails the build)
+#   (d) manual: Actions tab -> Live eval smoke -> Run workflow
+# The scorecard always lands on the workflow run's Summary page; PR events also get it
+# as a PR comment. The run JSON is uploaded as an artifact — download two of them and
+# compare with `uv run python -m evals.diff <old.json> <new.json>`.
+#
+# Cost/time budget: smoke suite is ~11 scenarios x 2-5 turns x (agent call + judge call),
+# paced by EVAL_SCENARIO_DELAY_SEC=150 between scenarios to stay under the current
+# model's provider rate limit. Expect roughly $0.75-3.00 and ~30-45 minutes per run
+# (10 delays x 150s = 25 min of pacing alone, plus actual call time) — this is not
+# stuck if it runs long; see timeout-minutes below.
+#
+# Secrets required: NVIDIA_API_KEY only (https://build.nvidia.com — free tier). The agent
+# and the judge both run on NVIDIA-hosted models through the same key, pinned to two
+# *different* models below so the judge isn't grading its own output and scoring doesn't
+# silently drift.
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  gate:
+    runs-on: ubuntu-latest
+    outputs:
+      run: ${{ steps.decide.outputs.run }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - id: decide
+        env:
+          EVENT_NAME: ${{ github.event_name }}
+          BASE_REF: ${{ github.base_ref }}
+          HEAD_REF: ${{ github.head_ref }}
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          run=false
+          # Push to main and manual dispatch always run; PRs are gated below.
+          if [[ "$EVENT_NAME" != "pull_request" ]]; then
+            run=true
+          else
+            if [[ "$BASE_REF" == "main" && "$HEAD_REF" == "develop" ]]; then
+              run=true
+            fi
+            changed="$(git diff --name-only "$BASE_SHA" "$HEAD_SHA")"
+            for f in uv.lock cli/uv.lock evals/smoke_agent/uv.lock; do
+              if grep -qxF "$f" <<<"$changed"; then
+                run=true
+              fi
+            done
+          fi
+          echo "run=$run" >> "$GITHUB_OUTPUT"
+
+  secrets:
+    needs: gate
+    if: needs.gate.outputs.run == 'true'
+    runs-on: ubuntu-latest
+    outputs:
+      available: ${{ steps.check.outputs.available }}
+    steps:
+      - name: Require live-eval secrets
+        id: check
+        env:
+          NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
+        run: |
+          if [[ -z "$NVIDIA_API_KEY" ]]; then
+            echo "::warning::NVIDIA_API_KEY is not configured; skipping live-eval smoke (https://build.nvidia.com)"
+            echo "available=false" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          echo "available=true" >> "$GITHUB_OUTPUT"
+
+  live-eval:
+    needs: [gate, secrets]
+    if: needs.gate.outputs.run == 'true' && needs.secrets.outputs.available == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 60 # 11 scenarios x ~150s inter-scenario delay (see EVAL_SCENARIO_DELAY_SEC) + run time,
+    # plus headroom for checkout/uv sync/gateway boot and slow judge or subagent turns
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v5
+        with:
+          version: "latest"
+
+      - name: Sync root deps (evals/report.py needs deepeval, langfuse)
+        run: uv sync --extra evals
+
+      - name: Sync smoke_agent deps
+        working-directory: evals/smoke_agent
+        run: uv sync
+
+      - name: Boot monkeybot gateway (evals/smoke_agent workspace)
+        working-directory: evals/smoke_agent
+        env:
+          NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
+          # Pin agent model for CI; env beats monkeybot.yaml.
+          MODEL_PROVIDER: nvidia
+          MODEL_NAME: thinkingmachines/inkling
+          # smoke_agent monkeybot.yaml sets runtime.port: 8080; the health check and
+          # AGENT_URL below expect 8787, so pin PORT explicitly (env beats yaml).
+          PORT: "8787"
+          SANDBOX_ENABLED: "false" # no Docker socket in this runner
+          # No MODEL_CONTEXT_WINDOW override: that was only needed for
+          # context/summarization_trigger.yaml (see evals/suites/smoke.yaml), which is
+          # currently excluded from smoke pending a retune against thinkingmachines/inkling.
+          # Re-add MODEL_CONTEXT_WINDOW: "8000" here alongside it if it comes back.
+        run: |
+          uv run -m monkeybot.gateway.main > /tmp/gateway.log 2>&1 &
+          echo $! > /tmp/gateway.pid
+          for i in $(seq 1 30); do
+            if curl -sf http://127.0.0.1:8787/health > /dev/null; then
+              echo "gateway healthy"
+              exit 0
+            fi
+            sleep 2
+          done
+          echo "gateway did not become healthy" >&2
+          cat /tmp/gateway.log >&2
+          exit 1
+
+      - name: Run smoke suite report
+        env:
+          AGENT_URL: http://127.0.0.1:8787
+          # Judge on a different NVIDIA-hosted model than the agent (same key).
+          JUDGE_PROVIDER: nvidia
+          JUDGE_MODEL: nvidia/nemotron-3-ultra-550b-a55b
+          NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
+          # Scorecard metadata: report the model the gateway actually ran.
+          MODEL_PROVIDER: nvidia
+          MODEL_NAME: thinkingmachines/inkling
+          EVENT_NAME: ${{ github.event_name }}
+          # thinkingmachines/inkling gets rate-limited by NVIDIA even at low concurrency;
+          # each scenario itself makes several requests in quick succession, so space
+          # scenarios out rather than trying to throttle within one.
+          EVAL_SCENARIO_DELAY_SEC: "150"
+        run: |
+          # PRs are a gate and must block on regressions; main pushes and manual runs
+          # just score the build and always report. --fail-on-regression always blocks on
+          # this run's own scenario errors/failures. It does NOT require evals/baselines/
+          # smoke.json to exist (that's the separate --require-baseline flag, intentionally
+          # omitted below) — no trusted baseline has been committed yet. Once one has
+          # (`uv run python -m evals.report --suite smoke --update-baseline`, committed),
+          # add --require-baseline here too so a missing/deleted file can't silently skip
+          # the token/cost/latency drift checks.
+          flags=""
+          if [[ "$EVENT_NAME" == "pull_request" ]]; then
+            flags="--fail-on-regression"
+          fi
+          uv run python -m evals.report \
+            --suite smoke \
+            --baseline evals/baselines/smoke.json \
+            $flags \
+            | tee eval-report.md
+
+      - name: Publish report to run summary
+        if: always()
+        run: |
+          if [[ -f eval-report.md ]]; then
+            cat eval-report.md >> "$GITHUB_STEP_SUMMARY"
+          else
+            echo '`FAIL` — no report produced; see the job logs.' >> "$GITHUB_STEP_SUMMARY"
+          fi
+
+      - name: Post report as PR comment
+        if: always() && github.event_name == 'pull_request'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const fs = require("fs");
+            const body = fs.existsSync("eval-report.md")
+              ? fs.readFileSync("eval-report.md", "utf8")
+              : "## monkeybot Eval Scorecard\n\n`FAIL` — the report step did not complete " +
+                "(gateway boot or the report command itself errored before writing a report). " +
+                "See the workflow run logs for details.";
+            await github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body,
+            });
+
+      - name: Upload run + report artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: live-eval-smoke-${{ github.run_id }}
+          path: |
+            evals/runs/
+            eval-report.md
+          if-no-files-found: ignore
+
+      - name: Stop gateway
+        if: always()
+        working-directory: evals/smoke_agent
+        run: kill "$(cat /tmp/gateway.pid)" 2>/dev/null || true

monkeybot-3.0.0/.github/workflows/publish-release.yml

@@ -0,0 +1,172 @@
+name: Publish release
+
+# Fires whenever main moves - i.e. right after an admin merges a release PR.
+# Branch protection on main (require PR + restrict merges to admins) is the
+# actual access control; this tags + GitHub-Releases + Trusted-Publishes to PyPI.
+#
+# Also supports workflow_dispatch republish for already-tagged packages (e.g.
+# after fixing a Trusted Publisher or sandbox image failure).
+#
+# PyPI Pending Trusted Publishers (one per project — environments must differ):
+#   monkeybot:             Owner human-plus-machine / Repo monkeybot /
+#                          Workflow publish-release.yml / Environment pypi
+#   monkeybot-cli:         Owner human-plus-machine / Repo monkeybot /
+#                          Workflow publish-release.yml / Environment pypi-cli
+#   monkeybot-browser-mcp: Owner human-plus-machine / Repo monkeybot /
+#                          Workflow publish-release.yml / Environment pypi-browser
+#
+# GitHub: create Environments named exactly `pypi`, `pypi-cli`, and `pypi-browser`.
+on:
+  push:
+    branches: [main]
+    paths:
+      - "pyproject.toml"
+      - "cli/pyproject.toml"
+      - "integrations/browser-mcp/pyproject.toml"
+      - "docker/sandbox.Dockerfile"
+      - "CHANGELOG.md"
+  workflow_dispatch:
+    inputs:
+      packages:
+        description: "Comma-separated packages to (re)publish even if already tagged (core,browser,cli)"
+        required: true
+        type: string
+        default: "browser,cli"
+
+permissions:
+  contents: write
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    outputs:
+      packages: ${{ steps.release.outputs.packages }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Configure git identity
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+
+      - id: release
+        run: python3 scripts/release.py publish
+        env:
+          GH_TOKEN: ${{ github.token }}
+          FORCE_PACKAGES: ${{ github.event.inputs.packages || '' }}
+
+  # Core before CLI (separate jobs / environments — required for two pending publishers).
+  pypi-core:
+    needs: publish
+    if: contains(format(',{0},', needs.publish.outputs.packages), ',core,')
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/monkeybot
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build core
+        run: uv build --out-dir dist/core .
+      - name: Publish core to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/core
+          skip-existing: true
+
+  pypi-browser:
+    needs: publish
+    if: contains(format(',{0},', needs.publish.outputs.packages), ',browser,')
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi-browser
+      url: https://pypi.org/p/monkeybot-browser-mcp
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build browser MCP
+        run: uv build --out-dir dist/browser integrations/browser-mcp
+      - name: Publish browser MCP to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/browser
+          skip-existing: true
+
+  pypi-cli:
+    needs: [publish, pypi-core, pypi-browser]
+    # Run when cli is in the release set. Core/browser may be skipped on a
+    # cli-only republish; treat skipped as success. A failed browser publish
+    # still blocks cli because monkeybot-cli depends on monkeybot-browser-mcp.
+    if: |
+      always() &&
+      needs.publish.result == 'success' &&
+      contains(format(',{0},', needs.publish.outputs.packages), ',cli,') &&
+      (needs.pypi-core.result == 'success' || needs.pypi-core.result == 'skipped') &&
+      (needs.pypi-browser.result == 'success' || needs.pypi-browser.result == 'skipped')
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi-cli
+      url: https://pypi.org/p/monkeybot-cli
+    permissions:
+      id-token: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build cli
+        run: uv build --out-dir dist/cli cli
+      - name: Publish cli to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/cli
+          skip-existing: true
+
+  sandbox-image:
+    needs: publish
+    if: contains(format(',{0},', needs.publish.outputs.packages), ',core,')
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - id: version
+        run: |
+          python - <<'PY' >> "$GITHUB_OUTPUT"
+          import re
+          from pathlib import Path
+
+          text = Path("pyproject.toml").read_text(encoding="utf-8")
+          match = re.search(r'^version = "([^"]+)"', text, re.M)
+          if match is None:
+              raise SystemExit("no version field in pyproject.toml")
+          print(f"value={match.group(1)}")
+          PY
+      - uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: docker/sandbox.Dockerfile
+          push: true
+          tags: ghcr.io/human-plus-machine/monkeybot-sandbox:${{ steps.version.outputs.value }}

monkeybot-3.0.0/.gitignore

@@ -0,0 +1,71 @@
+# Internal (not shipped in OSS)
+internal/
+
+# Environment
+.env
+.env.*
+!.env.example
+
+# Local scaffolded harness config (defaults ship in monkeybot-cli scaffold_defaults/)
+/monkeybot_config/
+
+# Live eval run artifacts (only evals/baselines/ is versioned)
+/evals/runs/
+
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+.Python
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+
+# uv
+.venv/
+.uv/
+
+# Data (SQLite DB + memory files — never commit)
+data/
+/memory/
+.monkeybot/
+*.sqlite3
+workspace/.monkeybot/
+*.sqlite
+*.sqlite.writer-pid
+
+# Testing & coverage
+.pytest_cache/
+.coverage
+htmlcov/
+.mypy_cache/
+.ruff_cache/
+.deepeval/
+
+# macOS
+.DS_Store
+
+# IDEs
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Docker
+docker/data/
+
+# Planning Documents
+.prt/
+.monkeymode/
+
+# Reference clones (read-only)
+code/
+
+# Local venv for evals dependency experiments
+.evals-venv/
+
+# monkeythink
+.monkeythink/

monkeybot-3.0.0/BACKLOG.md

@@ -0,0 +1,62 @@
+# monkeybot Backlog
+
+---
+
+## Up Next — Major Priorities
+
+### 1. Chat Integration
+- **Google Chat gateway** — incoming webhook + event handler for Google Chat spaces.
+- **Slack gateway** — Slack Events API / socket mode integration.
+
+### 2. Sandbox Workspace Protection
+- Add a hard-coded deny layer inside `SandboxExecutor` that blocks `run_command` from targeting harness-owned paths: `.monkeybot/`, `monkeybot.yaml`, `*.env`, `.agents/`, `config/`.
+- Path-level policy at the executor — distinct from `command_allowlist.yaml` which is command-level. Agent should retain free rw access to `./code/`, `./data/`, etc.
+
+### 3. CLI-managed optional service dependencies
+- `monkeybot chat` / `run` only spawn the gateway process; they don't start the Docker-backed optional services some `monkeybot.yaml` configs require: OpenSandbox (`sandbox.enabled`), Firestore emulator (`paths.db_url: firestore://...`), or the observability stack (Phoenix/Langfuse/OTel collector). Those used to be orchestrated by a demo `run.sh`; the CLI alone still can't fully run an agent that uses those features.
+- Decide and implement: should the CLI (`doctor`, `run`, `chat`) detect these from config and actually start/stop the containers, or just validate/warn with remediation hints while the user manages Docker manually?
+- Open sub-questions if going the "CLI starts services" route: where do per-project Docker assets live (`opensandbox.docker.toml`, OTel collector yaml, compose files) — agent-project-owned files referenced from `monkeybot.yaml`, or CLI-shipped generic templates; and is the sandbox worker image (project-specific extra deps) something the CLI should build, or stay manual/documented.
+
+---
+
+## Do Later
+
+### Connectors
+- **CLI gateway** — interactive stdin/stdout beyond the existing `monkeybot chat` / `talk` clients.
+- **Webhook gateway** — generic HTTP + HMAC ingress for third-party chat products.
+- **Expanded `python -m monkeybot` surface** — additional operator subcommands (`usage` / `schedule`, etc.) as needed.
+
+### Cloud Deployments
+- **GCP server** — GCE / GKE deployment option (docs + examples exist; no IaC).
+- **AWS serverless** — Lambda + API Gateway (example handler exists; not a full deploy path).
+- **AWS server** — EC2 / ECS deployment option.
+- **Docker CI image matrix** — multi-extras build matrix in CI.
+
+### Infra
+- **Postgres as production default** — backend exists; gateway still defaults to SQLite; decide when/if this becomes the default.
+- **Firestore usage aggregation doesn't scale** — `FirestoreUsageStore.summary()`/`breakdown()` (agent-wide, no `thread_id`) stream the entire `turn_usage` collection into memory and aggregate in Python. Fine at small scale; revisit with a Firestore aggregation query or a scheduled rollup doc if `/usage` traffic or data volume grows.
+- **Knowledge ANN index (sqlite-vec)** — deliberately deferred. The numpy matrix cache in `sqlite_vector.py` holds through ~50–100k chunks and query cost is embed-API bound, not local ANN. Revisit only if measured ANN p95 exceeds ~20–50 ms or the resident matrix causes RAM pressure; the true scale escape hatch is `store.type: pgvector`, not `sqlite-vec` packaging.
+
+### MCP
+- **MCP distro linkage** — confirm scaffolded `monkeybot.yaml` paths (`paths.mcp_config`, `paths.skills_path`) match deployment; smoke-test against real MCP servers beyond the bundled examples.
+
+### Tooling *(intentionally dropped — not completed)*
+
+Product decision to stop tracking these as active backlog items (not claiming they shipped):
+
+- **File-op tool audit** — Keep `write_file`; no plan to replace it with `create_file` + `find_and_replace` only.
+- **Custom subagents** — Named profiles with per-persona skills / MCP selections remain out of scope for now; workers continue to load the global skills path and `MCP_CONFIG`.
+
+### Future platforms *(not scheduled)*
+
+Moved from README — longer-term / speculative integrations without active implementation:
+
+- **Azure OpenAI** — Azure-hosted OpenAI models
+- **Azure Blob Storage** — memory backend (`[azure]` extra; docs note as planned)
+- **Azure Key Vault** — secrets for Azure deployments
+- **AWS Secrets Manager** — secret resolver at runtime
+- **GCP Secret Manager** — finish `secrets.provider: gcp_secret_manager` wiring (schema exists; resolver not shipped)
+- **Microsoft Teams** — Teams bot interface
+- **Telegram** — Telegram bot interface
+- **DynamoDB** — checkpointer / job storage backend
+- **Cosmos DB** — Azure-native persistence options