@animalabs/connectome-host 0.3.1

package/docs/webui-deployment.md

@@ -0,0 +1,219 @@
+# WebUI Deployment
+
+The `webui` module is a plain HTTP + WebSocket server. It owns no
+TLS, no outer authentication, no certificate management. Those are the
+reverse proxy's job. This is intentional — it keeps the host code small
+and lets the deployer pick whatever auth and cert mechanism the org
+already uses.
+
+This guide covers the typical deployment shape: a remote VM accessed
+over a VPN, with Caddy fronting the host process.
+
+## Module config
+
+In your recipe:
+
+```json
+{
+  "modules": {
+    "webui": {
+      "port": 7340,
+      "host": "127.0.0.1"
+    }
+  }
+}
+```
+
+The defaults are `0.0.0.0:7340` — connectome deployments are remote, not
+local, so the module binds all interfaces by default. Because that is a
+non-loopback bind, the module **refuses to start unless `basicAuth` is
+configured**: any recipe that enables webui must supply credentials. There
+is no unauthenticated escape hatch — even behind a reverse proxy, app-level
+Basic-Auth is required. Set `host: "127.0.0.1"` to bind loopback-only for
+local development, which skips the auth requirement.
+
+Front the module with a reverse proxy (Caddy/nginx) to terminate TLS; the
+proxy handles transport security while Basic-Auth gates the app layer.
+
+Add Basic-Auth credentials sourced from `.env` (never literal in the recipe):
+
+```json
+{
+  "modules": {
+    "webui": {
+      "host": "0.0.0.0",
+      "basicAuth": {
+        "username": "${WEBUI_USER}",
+        "password": "${WEBUI_PASS}"
+      }
+    }
+  }
+}
+```
+
+`${VAR}` substitution happens at recipe-load time, so credentials live in
+your gitignored `.env` rather than committed to the recipe.
+
+## Caddy
+
+Front Caddy as a system service. The `Caddyfile` for a single recipe:
+
+```caddy
+admin.example.internal {
+    encode gzip
+
+    # Basic authentication, swap for forward_auth/oauth/etc as needed.
+    basic_auth {
+        admin $2a$14$...your-bcrypted-password...
+    }
+
+    # The WebUI ships static assets and a /ws upgrade. Caddy's reverse_proxy
+    # handles WebSocket upgrades transparently — no extra config needed.
+    reverse_proxy 127.0.0.1:7340
+}
+```
+
+If multiple recipes run on one VM, give each a distinct port and a
+distinct Caddy block (subdomain or path-based routing both work).
+
+Caddy auto-issues TLS certificates for any public hostname. For
+**VPN-only DNS** (where Let's Encrypt can't reach the host), use a
+private CA — Caddy supports ACME with internal issuers via the
+`tls internal` directive:
+
+```caddy
+admin.vpn.internal {
+    tls internal
+    reverse_proxy 127.0.0.1:7340
+    # ...
+}
+```
+
+## systemd unit
+
+`/etc/systemd/system/connectome-host.service`:
+
+```ini
+[Unit]
+Description=connectome-host (admin recipe)
+After=network.target
+
+[Service]
+Type=simple
+User=connectome
+WorkingDirectory=/opt/connectome-host
+Environment=ANTHROPIC_API_KEY=...
+EnvironmentFile=/etc/connectome-host/env
+ExecStart=/home/connectome/.bun/bin/bun src/index.ts recipes/admin.json --headless
+Restart=on-failure
+RestartSec=5
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
+```
+
+`--headless` keeps the daemon alive without a TTY. The webui module
+serves the admin UI; the headless IPC socket is a separate channel
+(intended for parent-process supervision; not used by the WebUI).
+
+## Tailscale-style hostnames
+
+If the VM is on Tailscale, MagicDNS gives you a stable name like
+`admin.tailnet-name.ts.net`. Caddy with `tls internal` works there
+directly. No DNS or firewall holes needed beyond the existing tailnet.
+
+## Verifying the deployment
+
+After bringing the host up:
+
+```sh
+# Health check — should return 200 and HTML.
+curl -fsSL https://admin.example.internal/
+
+# WebSocket upgrade — open the URL in a browser. The connection-state
+# pill in the header should turn green within a second of page load.
+```
+
+The header pill is your primary signal:
+
+- green: connected, traces flowing
+- amber: reconnecting
+- red: socket unreachable (the SPA shows a banner with elapsed time)
+
+## Debug: context preview
+
+`GET /debug/context` returns the **membrane-normalized request that would be
+emitted if the agent were activated right now** — the compiled messages,
+system prompt, model config, and filtered tool set. Useful for inspecting
+exactly what the model would see.
+
+**Transparent by default.** The endpoint is side-effect-free: it runs no
+inference, writes nothing to Chronicle, and contacts no external MCPL server.
+It leaves the system state untouched, so you can poll it freely. The cost is
+that it omits the *dynamically-gathered* injections (lessons / retrieval /
+MCPL context), because gathering those is not transparent.
+
+Query params:
+
+- `agent=<name>` — defaults to the recipe's root agent
+- `injections=1` — opt into full fidelity: gather the dynamic injections too.
+  **Not transparent** — this can run inference (e.g. the retrieval module's
+  Haiku calls cost tokens) and fires MCPL `beforeInference` hooks (whose paired
+  `afterInference` is never sent, so a stateful server may be left half-open).
+- `pretty=1` — pretty-print the JSON
+
+The response includes `"transparent": true|false` so callers can confirm which
+mode ran.
+
+```sh
+# Transparent preview (default) — behind the proxy, with the same Basic-Auth
+# as the rest of the surface:
+curl -fsSL -u "$WEBUI_USER:$WEBUI_PASS" \
+  'https://admin.example.internal/debug/context?pretty=1'
+
+# Full-fidelity preview (spends tokens, fires MCPL hooks):
+curl -fsSL -u "$WEBUI_USER:$WEBUI_PASS" \
+  'https://admin.example.internal/debug/context?injections=1&pretty=1'
+```
+
+It is gated by the same Basic-Auth as every other route — note the response
+contains the full system prompt and conversation, so treat it as sensitive.
+
+See [`debug-context-api.md`](./debug-context-api.md) for the full reference:
+response shape, `jq` recipes, and the transparency contract.
+
+## Multi-VM admin
+
+The module has no built-in cross-VM aggregation. For a fleet of admin
+hosts, the simplest pattern is a Caddy block per instance with a
+landing page or path prefix:
+
+```caddy
+admin.example.internal {
+    handle /vm-a/* {
+        reverse_proxy 10.0.0.10:7340
+    }
+    handle /vm-b/* {
+        reverse_proxy 10.0.0.11:7340
+    }
+    # ...
+}
+```
+
+A future iteration may add a dedicated aggregator service that speaks
+the same WS protocol; the module's wire shape is forward-compatible
+with that pattern.
+
+## Troubleshooting
+
+- **`refuses to bind ... without auth`**: the default bind is `0.0.0.0`,
+  which requires `basicAuth`. Either add credentials, or switch `host` to
+  `127.0.0.1` for a loopback-only (local-dev) bind.
+- **WebUI bundle not found at .../dist/web**: run `bun install` at the
+  package root; the `postinstall` script builds the SPA. For dev, run
+  `bun run build:web` from the package root.
+- **Disconnect banner stuck**: the host process may have crashed. Check
+  `journalctl -u connectome-host -n 100` (or the equivalent for your
+  init system). The SPA reconnects automatically once the host is back.

package/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "@animalabs/connectome-host",
+  "version": "0.3.1",
+  "description": "General-purpose agent TUI host with recipe-based configuration",
+  "type": "module",
+  "scripts": {
+    "start": "bun src/index.ts",
+    "dev": "bun --watch src/index.ts",
+    "test": "bun test",
+    "build:web": "cd web && bun install && bun run build",
+    "postinstall": "test -d web && cd web && bun install && bun run build || true"
+  },
+  "dependencies": {
+    "@animalabs/agent-framework": "^0.6.0",
+    "@animalabs/chronicle": "^0.2.0",
+    "@animalabs/context-manager": "^0.5.5",
+    "@animalabs/membrane": "^0.5.68",
+    "@opentui/core": "^0.1.82"
+  },
+  "devDependencies": {
+    "@types/bun": "^1.3.13",
+    "@types/node": "^22.0.0",
+    "typescript": "^5.7.0"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/anima-research/connectome-host.git"
+  },
+  "homepage": "https://github.com/anima-research/connectome-host#readme",
+  "bugs": {
+    "url": "https://github.com/anima-research/connectome-host/issues"
+  }
+}

package/recipes/SETUP.md

@@ -0,0 +1,308 @@
+# Knowledge Miner Setup Guide
+
+A step-by-step guide to setting up ConnectomeHost with the knowledge-miner recipe for extracting structured knowledge from your organization's Zulip, Notion, and GitLab.
+
+## What you get
+
+An AI research agent that can:
+- **Read** your team's Zulip conversations, Notion docs, and GitLab issues/MRs/code
+- **Fork** parallel sub-agents to investigate multiple sources simultaneously
+- **Extract** persistent lessons — tagged, scored knowledge that survives across sessions
+- **Write** reports and analysis documents to disk
+- **Cross-reference** information across all three platforms
+
+## Prerequisites
+
+| Tool | Why | Install |
+|------|-----|---------|
+| [Node.js](https://nodejs.org/) 20+ | Runtime | `nvm install 20` or download from nodejs.org |
+| [Bun](https://bun.sh/) | App runner | `curl -fsSL https://bun.sh/install \| bash` |
+| [Anthropic API key](https://console.anthropic.com/) | LLM access | Sign up at console.anthropic.com |
+
+## Step 1: Install ConnectomeHost
+
+```bash
+git clone https://github.com/anima-research/connectome-host.git
+cd connectome-host
+npm install
+```
+
+## Step 2: Set up your data sources
+
+You need credentials for each platform you want to connect. You can start with just one and add more later — the agent adapts to whatever tools are available.
+
+### Zulip
+
+You need a `.zuliprc` file with your bot or user credentials.
+
+1. In your Zulip organization, go to **Settings > Personal > API key**
+2. Download the `.zuliprc` file, or create one manually:
+
+```ini
+[api]
+email=your-bot@your-org.zulipchat.com
+key=YOUR_ZULIP_API_KEY
+site=https://your-org.zulipchat.com
+```
+
+3. Place it in the connectome-host directory:
+
+```bash
+cp ~/Downloads/.zuliprc ./.zuliprc
+```
+
+4. Install the Zulip MCP server:
+
+```bash
+git clone https://github.com/anima-research/zulip_mcp.git ../zulip_mcp
+cd ../zulip_mcp && npm install && npm run build && cd -
+```
+
+### GitLab
+
+Works with both gitlab.com and self-hosted GitLab instances.
+
+1. Go to your GitLab instance > **User Settings > Access Tokens**
+2. Create a personal access token with scopes: `read_api`, `read_repository`
+   - Add `api` scope if you want write access (creating issues, comments)
+3. Note your token and your GitLab API URL
+
+No separate installation needed — the recipe uses `npx` to run `@zereight/mcp-gitlab` on demand.
+
+### Notion (optional, via an MCP server)
+
+If you want the agent to read your Notion workspace, point the recipe at any MCP server that exposes Notion search and page-read tools. The recipe's template entry is named `syncntn` after the particular Notion MCP adapter it was developed against, but the key is just a label — any Notion MCP server works, as long as its exposed tool names match what the system prompt references (`syncntn--search_pages`, `syncntn--get_page_markdown`, and friends). If your server uses different tool names, either rename the MCP key and update the prompt, or skip Notion entirely by removing the block.
+
+Typical setup:
+
+1. Install and start your Notion MCP server somewhere the recipe can launch it.
+2. Note any configuration it needs (API credentials, workspace ID, storage URL).
+3. Fill those values into the recipe's `syncntn` env block in Step 3 below.
+
+Don't have a Notion MCP server? Remove the `syncntn` entry from the recipe — the agent will adapt and work with whatever sources remain.
+
+### DuckDuckGo web search (optional, but enabled by default)
+
+The recipe ships with [`nickclyde/duckduckgo-mcp-server`](https://github.com/nickclyde/duckduckgo-mcp-server) wired as `ddg`. It gives the miner one public source — handy when a question has a regulatory, standards-body, or vendor-spec component the internal sources can't answer on their own. The agent emits `[WEB: <url>]` for any claim backed by a web hit, so the citation chain stays auditable.
+
+Install once as a sibling of `connectome-host/`:
+
+```bash
+cd ..
+git clone https://github.com/nickclyde/duckduckgo-mcp-server.git
+cd duckduckgo-mcp-server
+python3 -m venv .venv
+.venv/bin/pip install -e .
+cd ../connectome-host
+```
+
+The recipe expects the entry-point script at `../duckduckgo-mcp-server/.venv/bin/duckduckgo-mcp-server`. No API key needed. Don't want public-web access? Remove the `ddg` block from the recipe.
+
+## Step 3: Configure the recipe
+
+Copy the template recipe and fill in your credentials:
+
+```bash
+cp recipes/knowledge-miner.json my-recipe.json
+```
+
+Edit `my-recipe.json` and replace the placeholder values in `mcpServers`:
+
+```jsonc
+{
+  "mcpServers": {
+    "zulip": {
+      "command": "node",
+      "args": ["../zulip_mcp/build/index.js"],
+      "env": {
+        "ENABLE_ZULIP": "true",
+        "ENABLE_DISCORD": "false",
+        "ZULIP_RC_PATH": "./.zuliprc"          // path to your .zuliprc
+      }
+    },
+    "syncntn": {
+      "command": "../syncntn/services/mcp/start_mcp_local.sh",
+      "env": {
+        "STORAGE_URL": "http://localhost:8000",
+        "WORKSPACE_ID": "YOUR_WORKSPACE_ID"    // <-- replace this
+      }
+    },
+    "gitlab": {
+      "command": "npx",
+      "args": ["-y", "@zereight/mcp-gitlab"],
+      "env": {
+        "GITLAB_PERSONAL_ACCESS_TOKEN": "YOUR_GITLAB_TOKEN",  // <-- replace
+        "GITLAB_API_URL": "https://gitlab.example.com/api/v4" // <-- replace
+      }
+    },
+    "ddg": {
+      "command": "../duckduckgo-mcp-server/.venv/bin/duckduckgo-mcp-server"
+      // no creds; public web. Remove this block to disable web search.
+    }
+  }
+}
+```
+
+**Don't need all three?** Just remove the server entries you don't have. The agent works with any combination.
+
+## Step 4: Set your API key
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+Or add it to a `.env` file in the project directory.
+
+## Step 5: Run
+
+```bash
+bun src/index.ts my-recipe.json
+```
+
+On subsequent runs, just `bun src/index.ts` — the recipe is remembered.
+
+## Using the agent
+
+Once running, you'll see a terminal interface. Type natural language requests:
+
+```
+> Map out the key architectural decisions made in the last month across Zulip and GitLab
+
+> What does the team's Notion say about the deployment process? Cross-reference with
+  recent Zulip discussions about deploy failures.
+
+> Find all open GitLab issues tagged "tech-debt" and check if any were discussed in Zulip.
+  Create a summary report.
+```
+
+The agent will:
+1. **Scout** available sources (list Zulip streams, search Notion, browse GitLab)
+2. **Fork** sub-agents to read specific threads, pages, and issues in parallel
+3. **Synthesize** findings and cross-reference across platforms
+4. **Extract** lessons and optionally write reports to `./output/`
+
+### Useful commands
+
+| Command | What it does |
+|---------|-------------|
+| `Tab` | Toggle fleet view — see sub-agents working in parallel |
+| `/lessons` | Show all extracted knowledge, sorted by confidence |
+| `/status` | Agent state, session info |
+| `/undo` | Roll back the last agent turn |
+| `/mcp list` | Show connected data sources |
+| `/newtopic [context]` | Reset context window for a new topic (compresses old context) |
+| `/session new` | Start a fresh session (lessons persist) |
+| `Esc` | Interrupt the agent mid-turn |
+
+### Tips
+
+- **Start broad, then narrow.** Ask "what streams/projects exist?" before diving deep.
+- **Let it fork.** The agent is designed to run 2-4 sub-agents in parallel. Don't micromanage.
+- **Check lessons periodically.** `/lessons` shows what's been extracted. You can ask the agent to revise or merge lessons.
+- **Use the workspace.** Ask the agent to "write a report" and it produces files in `./output/`.
+- **Switch topics cleanly.** Use `/newtopic` when changing research direction — it compresses old context and frees up the context window. Cheaper than a new session, and lessons carry over either way.
+
+## Customization
+
+### Using only some sources
+
+Remove MCP server entries from the recipe for sources you don't have. The system prompt automatically adapts — the agent only uses tools that are available.
+
+### Overriding servers at runtime
+
+Instead of editing the recipe, you can override server config in `mcpl-servers.json`:
+
+```bash
+# Add or override a server (persists across restarts)
+/mcp add gitlab npx -y @zereight/mcp-gitlab
+/mcp env gitlab GITLAB_PERSONAL_ACCESS_TOKEN=glpat-xxx GITLAB_API_URL=https://gitlab.myco.com/api/v4
+```
+
+Changes require a restart to take effect.
+
+### Read-only GitLab
+
+If you want the agent to only read from GitLab (no issue creation, no MR comments):
+
+```json
+"gitlab": {
+  "command": "npx",
+  "args": ["-y", "@zereight/mcp-gitlab"],
+  "env": {
+    "GITLAB_PERSONAL_ACCESS_TOKEN": "YOUR_TOKEN",
+    "GITLAB_API_URL": "https://gitlab.example.com/api/v4",
+    "GITLAB_READ_ONLY_MODE": "true"
+  }
+}
+```
+
+### Limiting GitLab tools
+
+To only expose certain tool categories (e.g., issues and merge requests, not pipelines or wiki):
+
+```json
+"env": {
+  "GITLAB_TOOLSETS": "issues,merge_requests,search,projects"
+}
+```
+
+## Reviewing knowledge quality
+
+After the Knowledge Miner produces documents, run the Reviewer agent for a quality audit.
+
+### Step 1: Export lessons
+
+In the Knowledge Miner session:
+```
+/export
+```
+
+Or just quit — lessons are auto-exported on exit. This creates `./output/lessons-export.json` and `./output/lessons-export.md`.
+
+### Step 2: Run the Reviewer
+
+```bash
+bun src/index.ts recipes/knowledge-reviewer.json
+```
+
+The Reviewer reads the exported lessons and documents from `./output/` (mounted as read-only `input/`). It produces:
+- **Critic findings** per document — internal contradictions, unsupported claims, missing markers
+- **SME checklist** — a focused list of items for domain experts to verify
+
+Ask it:
+```
+> Review all documents in input/. Generate the SME checklist.
+```
+
+### Step 3: Human review
+
+Open `./review-output/sme-checklist.md`. It contains a prioritized list:
+- **High risk** — `[GEN]` claims (general knowledge, no source)
+- **Medium risk** — `[INF]` claims on system boundaries
+- **Knowledge gaps** — `❓` markers needing expert input
+- **Unmarked suspicious claims** — the most dangerous: plausible but unsourced
+
+A domain expert can complete this checklist in 10-20 minutes without reading the full document.
+
+### The confidence markers
+
+The Knowledge Miner tags document claims with `[SRC]`, `[WEB]`, `[INF]`, `[GEN]`, or `❓`. The Reviewer audits these. If the Miner missed markers (unmarked claims that look like general knowledge), the Reviewer flags them.
+
+- `[SRC: source]` — direct quote from an **internal** system (Zulip, Notion, GitLab)
+- `[WEB: url]` — quote from a public web page via the DuckDuckGo MCP; the URL is the citation
+- `[INF]` — inferred from multiple sources
+- `[GEN]` — general domain knowledge with no citation (prefer `[WEB]` when checkable)
+- `❓` — knowledge gap; needs a human expert
+
+Internal `[SRC]` always wins over `[WEB]` when both speak to the same org-specific term — the miner is instructed to keep the two meanings visibly separate, and the reviewer flags collisions.
+
+## Troubleshooting
+
+| Problem | Fix |
+|---------|-----|
+| `ANTHROPIC_API_KEY not set` | `export ANTHROPIC_API_KEY=sk-ant-...` |
+| Zulip tools not appearing | Check `.zuliprc` path and that zulip_mcp is built |
+| GitLab 401 errors | Verify your token has the right scopes and hasn't expired |
+| Notion tools not appearing or connection refused | Make sure your Notion MCP server is running and reachable at the command/URL the recipe expects |
+| Agent seems stuck | Press `Esc` to interrupt, then ask it to try a different approach |
+| Sub-agents not returning | Press `Tab` to check fleet view — they may still be working |