mercury-agent 0.4.5

package/docs/extensions.md

@@ -0,0 +1,375 @@
+# Extensions
+
+Mercury's extension system lets you add CLIs, skills, background jobs, lifecycle hooks, config keys, and dashboard widgets — all in TypeScript.
+
+Extensions **cannot register HTTP routes** on the host API today. Features that need authenticated host endpoints (for example TradeStation **order placement** with a two-step confirm flow) are implemented as routes under `src/core/routes/` and gated with the same permission name as the extension (`tradestation`). A future `mercury.route()` API could move such handlers next to the extension source.
+
+## Structure
+
+Extensions live in `.mercury/extensions/*/`. Each directory is an extension:
+
+```
+.mercury/extensions/
+├── napkin/
+│   ├── index.ts           # Required — setup function
+│   ├── skill/SKILL.md     # Optional — agent skill
+│   └── package.json       # Optional — dependencies
+└── gws/
+    └── index.ts
+```
+
+The extension **name** is the directory name.
+
+## Setup Function
+
+Every extension exports a default function that receives the `MercuryExtensionAPI`:
+
+```typescript
+import type { MercuryExtensionAPI } from "mercury-agent";
+
+export default function(mercury: MercuryExtensionAPI) {
+  // Declare what this extension provides
+  mercury.cli({ name: "napkin", install: "bun add -g napkin-ai" });
+  mercury.permission({ defaultRoles: ["admin", "member"] });
+  mercury.skill("./skill");
+
+  mercury.on("workspace_init", async ({ workspace, containerWorkspace }) => {
+    mkdirSync(join(workspace, "entities"), { recursive: true });
+  });
+
+  mercury.job("cleanup", {
+    interval: 3600_000,
+    run: async (ctx) => { /* ... */ },
+  });
+
+  mercury.config("enabled", {
+    description: "Enable napkin for this space",
+    default: "true",
+  });
+
+  mercury.widget({
+    label: "Napkin Status",
+    render: (ctx) => `<p>Last run: ${mercury.store.get("last-run") ?? "never"}</p>`,
+  });
+}
+```
+
+All declarations are optional — use only what you need.
+
+## API Reference
+
+### `mercury.cli(opts)`
+
+Declare a CLI tool to install in the container image.
+
+```typescript
+mercury.cli({ name: "napkin", install: "bun add -g napkin-ai" });
+```
+
+- `name` — binary name (should match extension name)
+- `install` — shell command run as a Dockerfile `RUN` step
+
+Mercury auto-generates a derived Docker image with all extension CLIs installed. The agent calls them directly in bash. Permission enforcement is handled by a built-in pi extension that blocks denied CLIs based on the caller's role.
+
+Can be called multiple times for extensions that need several tools (e.g., media needs ffmpeg, imagemagick, and yt-dlp).
+
+### `mercury.permission(opts)`
+
+Register this extension's permission and set default roles.
+
+```typescript
+mercury.permission({ defaultRoles: ["admin", "member"] });
+```
+
+- Permission name = extension name (e.g., `napkin`)
+- `defaultRoles` — roles that get this permission by default
+- `admin` always gets all permissions automatically
+- Per-space overrides in `space_config` take precedence
+
+Can only be called once per extension.
+
+### `mercury.env(def)`
+
+Declare an environment variable this extension needs. Only injected into containers when the caller has permission for this extension. Claimed vars are excluded from the blind `MERCURY_*` passthrough, preventing credential leakage to unprivileged callers.
+
+```typescript
+mercury.env({ from: "MERCURY_GH_TOKEN" });                    // injected as GH_TOKEN
+mercury.env({ from: "MERCURY_GH_TOKEN", as: "GITHUB_TOKEN" }); // custom container name
+```
+
+- `from` — env var name as set in `.env` (e.g. `MERCURY_GH_TOKEN`)
+- `as` — (optional) name inside the container. Defaults to `from` with `MERCURY_` prefix stripped
+
+Can be called multiple times for multiple env vars.
+
+### `mercury.skill(relativePath)`
+
+Register a skill directory for agent discovery.
+
+```typescript
+mercury.skill("./skill");
+```
+
+The directory must contain a `SKILL.md` file in pi's [skill format](https://agentskills.io/specification). Mercury copies the entire skill directory into `.mercury/global/skills/<name>/`, which is mounted into containers at `/home/node/.pi/agent/skills/<name>/`. Pi discovers it automatically.
+
+Skills can contain multiple files — scripts, references, assets — not just SKILL.md. The agent uses relative paths from SKILL.md to access them.
+
+### `mercury.requires(capabilities)`
+
+Declare that this extension's skill (or CLI workflows) needs certain model capabilities (`tools`, `vision`, etc.). If **no** leg in `MERCURY_MODEL_CHAIN` satisfies **all** listed flags, the extension skill is not copied into the global skills directory and Mercury logs a startup warning.
+
+```typescript
+mercury.requires(["tools"]); // e.g. PDF / scripting extensions
+```
+
+Capability keys: `tools`, `vision`, `audio_input`, `audio_output`, `extended_thinking`.
+
+Host configuration:
+
+- Built-in prefix map for common model IDs
+- Optional `.mercury/model-capabilities.yaml` for per-model overrides
+- Optional `MERCURY_MODEL_CAPABILITIES` JSON env var (applies to every leg)
+
+### `mercury.on(event, handler)`
+
+Subscribe to lifecycle events.
+
+```typescript
+mercury.on("workspace_init", async (event, ctx) => {
+  // event.workspace — absolute host path
+  // event.containerWorkspace — container-relative path (e.g. /spaces/main)
+  mkdirSync(join(event.workspace, "my-dir"), { recursive: true });
+});
+```
+
+#### Events
+
+| Event | When | Can mutate? |
+|-------|------|-------------|
+| `startup` | After extensions loaded, runtime ready | No |
+| `shutdown` | Mercury shutting down | No |
+| `workspace_init` | Space workspace created/ensured | No |
+| `before_container` | About to spawn container | Yes |
+| `after_container` | Container finished | Yes |
+
+Both `workspace_init` and `before_container` events include:
+- `workspace` — absolute host path (for file operations on the host)
+- `containerWorkspace` — container-relative path, e.g. `/spaces/main` (for env vars passed into the container)
+
+`before_container` also includes:
+- `attachments` — incoming `MessageAttachment[]` (e.g. voice, images), if any
+
+#### `before_container` mutations
+
+```typescript
+mercury.on("before_container", async (event, ctx) => {
+  return {
+    systemPrompt: "Extra instructions...",  // appended to system prompt
+    promptAppend: "Transcript or extra user text...", // appended to user prompt (newline-joined across handlers)
+    env: { MY_VAR: event.containerWorkspace + "/data" },  // container-relative paths
+    block: { reason: "Rate limited" },       // prevent container from running
+  };
+});
+```
+
+The user message stored in the DB and passed to the container uses the original prompt plus any `promptAppend` from hooks (after `before_container` runs).
+
+#### Extension context helpers
+
+`ctx.hasCallerPermission(spaceId, callerId, permission)` returns whether the caller has a built-in or extension-registered permission in that space. Use it in hooks to mirror container RBAC (e.g. only transcribe voice for members who have your extension permission).
+
+#### `after_container` mutations
+
+```typescript
+mercury.on("after_container", async (event, ctx) => {
+  return {
+    reply: event.reply + "\n\n_Powered by Mercury_",  // transform reply
+    suppress: true,                                     // don't send reply
+  };
+});
+```
+
+### `mercury.job(name, def)`
+
+Register a background job that runs on the host.
+
+```typescript
+// Interval-based
+mercury.job("cleanup", {
+  interval: 3600_000,  // every hour
+  run: async (ctx) => { /* ... */ },
+});
+
+// Cron-based
+mercury.job("daily-report", {
+  cron: "0 9 * * *",  // 9am daily
+  run: async (ctx) => { /* ... */ },
+});
+```
+
+Must specify either `interval` or `cron`, not both. Jobs are started after extensions load and stopped on shutdown. Errors are caught and logged — a failing job never crashes Mercury.
+
+### `mercury.config(key, def)`
+
+Register a per-space config key.
+
+```typescript
+mercury.config("enabled", {
+  description: "Enable napkin for this space",
+  default: "true",
+  validate: (v) => v === "true" || v === "false",
+});
+```
+
+Keys are namespaced to the extension: the above registers `napkin.enabled`. Users set it via `mrctl config set napkin.enabled false`.
+
+### `mercury.widget(def)`
+
+Register a dashboard widget.
+
+```typescript
+mercury.widget({
+  label: "KB Distillation",
+  render: (ctx) => {
+    const lastRun = mercury.store.get("last-run") ?? "never";
+    return `<div>Last run: ${lastRun}</div>`;
+  },
+});
+```
+
+Widgets render HTML fragments in the dashboard overview. Errors show a placeholder — never crash the dashboard.
+
+### `mercury.store`
+
+Scoped key-value store for persistent state.
+
+```typescript
+mercury.store.set("last-run", Date.now().toString());
+mercury.store.get("last-run");     // "1709654400000"
+mercury.store.delete("last-run");  // true
+mercury.store.list();              // [{ key: "last-run", value: "..." }]
+```
+
+Each extension sees only its own keys. Backed by the `extension_state` SQLite table.
+
+## Extension Context
+
+Event handlers and job runners receive a `MercuryExtensionContext`:
+
+```typescript
+interface MercuryExtensionContext {
+  readonly db: Db;          // Database access
+  readonly config: AppConfig; // Mercury configuration
+  readonly log: Logger;      // Logger scoped to the extension
+  hasCallerPermission(spaceId: string, callerId: string, permission: string): boolean;
+}
+```
+
+## Container Integration
+
+### Skill Files
+
+Skills can include anything the agent needs:
+
+```
+napkin/skill/
+├── SKILL.md              # Required: frontmatter + instructions
+├── scripts/              # Helper scripts
+│   └── search.js
+├── references/           # Detailed docs loaded on-demand
+│   └── api-reference.md
+└── assets/
+    └── template.json
+```
+
+All files are copied into the container mount. Relative paths from SKILL.md work.
+
+### Derived Image
+
+Extensions that declare `mercury.cli()` get their tools installed in a derived Docker image:
+
+```
+FROM mercury-agent:latest
+RUN bun add -g napkin-ai        # from napkin extension
+RUN pip install some-tool       # from another extension
+```
+
+Mercury builds this image on startup (cached by content hash). If no extensions declare CLIs, the base image is used unchanged.
+
+### Agent Discovery
+
+The agent discovers extension CLIs via skills (SKILL.md files) and invokes them through `mrctl`:
+
+```bash
+napkin search "query"           # called directly, RBAC enforced at bash level
+```
+
+## Built-in Commands vs Extensions
+
+`mrctl` has two types of commands:
+
+| Type | Examples | How it works |
+|------|----------|--------------|
+| **Built-in** | `tasks`, `roles`, `permissions`, `config`, `spaces`, `conversations`, `stop`, `compact` | HTTP calls to host API via `mrctl` |
+| **Extension** | `napkin`, `pinchtab`, any custom | Called directly in bash, RBAC enforced by pi extension |
+
+Built-in names are reserved — extension registration fails on collision.
+
+## Permissions
+
+One permission per extension, named after the extension. Extensions declare which roles get it by default:
+
+```typescript
+mercury.permission({ defaultRoles: ["admin", "member"] });
+```
+
+- `admin` always gets all permissions
+- Per-space overrides via `mrctl permissions set <role> prompt,napkin,...`
+- See [permissions.md](permissions.md) for the full RBAC system
+
+## Installation
+
+### `mercury add`
+
+Install extensions from local paths, npm, or git:
+
+```bash
+mercury add ./path/to/extension         # local directory
+mercury add npm:mercury-ext-napkin      # npm package
+mercury add git:github.com/user/ext     # git repo
+```
+
+Mercury copies the extension to `.mercury/extensions/<name>/`, installs dependencies if `package.json` is present, copies skills to the global dir, and validates the extension loads correctly.
+
+### `mercury remove`
+
+```bash
+mercury remove napkin
+```
+
+Removes the extension directory and its installed skill. Restart Mercury to apply.
+
+### `mercury extensions list`
+
+```bash
+mercury extensions list    # or: mercury ext list
+```
+
+Shows all installed extensions (user + built-in) with features and descriptions.
+
+## Examples
+
+See [`examples/extensions/`](../examples/extensions/) for complete, working extensions ranging from minimal (charts — CLI + skill) to full-featured (napkin — hooks, jobs, config, widgets, KB distillation).
+
+## Types
+
+All types are in `src/extensions/types.ts`:
+
+- `MercuryExtensionAPI` — setup API surface
+- `MercuryExtensionContext` — runtime context for hooks/jobs
+- `MercuryEvents` — all lifecycle events
+- `EventHandler<E>` / `EventResult<E>` — typed handlers with mutation support
+- `ExtensionMeta` — collected metadata after setup
+- `ExtensionStore` — scoped key-value store
+- `JobDef`, `ConfigDef`, `WidgetDef`, `CliDef`, `PermissionDef` — definitions
+- `ExtensionSetupFn` — the default export signature

package/docs/graceful-shutdown.md

@@ -0,0 +1,62 @@
+# Graceful Shutdown
+
+On `SIGTERM` or `SIGINT`, mercury tears down all components in order instead of exiting abruptly.
+
+## Sequence
+
+```
+SIGTERM/SIGINT received
+  │
+  ├─1─► Stop scheduler (clear poll timer)
+  ├─2─► Cancel all pending queue entries
+  ├─3─► Kill running containers (docker kill)
+  ├─4─► Wait for active work to drain (up to 8s)
+  ├─5─► Disconnect adapters (WhatsApp socket, etc.)
+  ├─6─► Stop HTTP server
+  ├─7─► Close SQLite database
+  └─8─► exit(0)
+
+  Second signal → force exit(1)
+  10s timeout  → force exit(1)
+```
+
+## Why this order
+
+1. **Scheduler first** — prevents new work from being created while we're shutting down.
+2. **Cancel pending queue entries** — no point starting queued work we'll just kill.
+3. **Kill containers** — uses `docker kill` for reliable termination. Falls back to `SIGKILL` if the docker command fails. Containers are labeled with `mercury.managed=true` for identification (see [container-lifecycle.md](./container-lifecycle.md)).
+4. **Wait for drain** — gives active container runs a chance to finish cleanly (up to 8s).
+5. **Disconnect adapters** — closes the WhatsApp socket, Slack/Discord connections. Done after containers so in-flight replies can still be posted.
+6. **Stop HTTP server** — stops accepting new webhook/API requests.
+7. **Close DB last** — everything above may still write to the database (message storage, task updates), so the DB connection stays open until the very end.
+
+## Safety mechanisms
+
+| Mechanism | Behavior |
+|-----------|----------|
+| **Double-signal** | Second SIGINT/SIGTERM forces immediate `exit(1)` |
+| **Global timeout** | If cleanup takes longer than 10s, forced `exit(1)` |
+| **Idempotent** | `shutdown()` is guarded by a `shuttingDown` flag — calling it twice is a no-op |
+| **Hook errors** | Individual shutdown hook failures are logged and swallowed — remaining cleanup continues |
+
+## API
+
+### `MercuryCoreRuntime`
+
+```ts
+core.installSignalHandlers()     // trap SIGTERM + SIGINT
+core.onShutdown(async () => {})  // register cleanup callback
+await core.shutdown(timeoutMs?)  // trigger shutdown manually (default 10s)
+core.isShuttingDown              // boolean
+```
+
+### Component methods used during shutdown
+
+| Component | Method | What it does |
+|-----------|--------|-------------|
+| `TaskScheduler` | `stop()` | Clears the poll timer |
+| `SpaceQueue` | `cancelAll()` | Drops all pending entries, returns count |
+| `SpaceQueue` | `waitForActive(ms)` | Resolves when active count hits 0 or timeout |
+| `AgentContainerRunner` | `killAll()` | Kill all running containers via `docker kill` |
+| `Db` | `close()` | Closes SQLite connection |
+| `WhatsAppBaileysAdapter` | `shutdown()` | Ends the Baileys socket |

package/docs/kb-distillation.md

@@ -0,0 +1,77 @@
+# KB Distillation
+
+KB distillation is **extension-based** in Mercury (not built-in).
+
+As of v0.3.x, Mercury no longer ships a built-in `kb-distill` extension. The recommended approach is to use a user-installed extension (for example, `napkin`) that runs a background job and writes distilled knowledge into each space vault.
+
+---
+
+## Recommended Setup
+
+Use the real example extension at:
+
+- `examples/extensions/napkin/`
+
+It demonstrates:
+
+- `workspace_init` hook to scaffold vault structure
+- `before_container` hook to set `NAPKIN_VAULT`
+- background job (`distill`) for periodic extraction
+- dashboard widget + extension store state
+
+---
+
+## How Distillation Works (Extension Pattern)
+
+Typical flow:
+
+1. Read messages from `state.db` per `space_id`
+2. Export messages to `.messages/YYYY-MM-DD.jsonl`
+3. Detect changed files (hash compare)
+4. Run `pi` with a distillation prompt against changed files
+5. Update vault files incrementally (append/update, not destructive rewrite)
+
+This keeps runs idempotent and avoids re-processing unchanged days.
+
+---
+
+## Data Layout (napkin example)
+
+```text
+.mercury/spaces/<space-id>/
+├── .messages/
+│   └── YYYY-MM-DD.jsonl
+└── knowledge/
+    ├── people/
+    ├── projects/
+    ├── references/
+    ├── daily/
+    └── templates/
+```
+
+Message export format:
+
+```json
+{"ts":1709123456,"role":"ambient","content":"Alice: Great idea!"}
+{"ts":1709123457,"role":"user","content":"What do you think about X?"}
+{"ts":1709123458,"role":"assistant","content":"I think..."}
+```
+
+---
+
+## Configuration
+
+With the napkin example extension:
+
+- `MERCURY_KB_DISTILL_INTERVAL_MS=0` → disabled (default)
+- `MERCURY_KB_DISTILL_INTERVAL_MS=3600000` → check every hour
+
+You can also expose interval as extension config keys (see `examples/extensions/napkin/index.ts`).
+
+---
+
+## Notes
+
+- Distillation behavior depends on the installed extension implementation
+- Mercury core provides hooks, jobs, DB access, and workspace isolation
+- Distillation logic/prompt lives in the extension, not in core Mercury

package/docs/media/overview.md

@@ -0,0 +1,140 @@
+# Media Handling
+
+Mercury downloads and processes media attachments from chat platforms, saving them to space workspaces and passing them to pi for processing. Models can also produce files via the `outbox/` directory.
+
+## Supported Platforms
+
+| Platform | Ingress | Egress | Details |
+|----------|---------|--------|---------|
+| WhatsApp | ✅ Baileys socket | ✅ image/video/audio/document | [whatsapp.md](./whatsapp.md) |
+| Discord | ✅ CDN URL download | ✅ channel.send() with files | Via `DiscordBridge` |
+| Slack | ✅ URL download (auth'd) | ✅ files.uploadV2 | Via `SlackBridge` |
+
+## Architecture
+
+```
+┌──────────────┐     ┌──────────────┐     ┌──────────────┐     ┌─────────┐
+│   Platform   │────▶│   Bridge     │────▶│   Runtime    │────▶│   pi    │
+│              │     │ (normalize)  │     │ (store/pass) │     │ (view)  │
+└──────────────┘     └──────────────┘     └──────────────┘     └─────────┘
+                            │                    │
+                            ▼                    ▼
+                     ┌──────────────┐     ┌──────────────┐
+                     │  Workspace   │     │  Workspace   │
+                     │   /inbox/    │     │   /outbox/   │
+                     └──────────────┘     └──────────────┘
+```
+
+## Media Types
+
+All platforms map to these generic types defined in `src/types.ts`:
+
+```typescript
+type MediaType = "image" | "video" | "audio" | "voice" | "document";
+
+interface MessageAttachment {
+  path: string;        // Local file path
+  type: MediaType;     // Generic type
+  mimeType: string;    // MIME type (e.g., "image/jpeg")
+  filename?: string;   // Original filename if available
+  sizeBytes?: number;  // File size in bytes
+}
+```
+
+## Configuration
+
+| Env Variable | Default | Description |
+|--------------|---------|-------------|
+| `MERCURY_MEDIA_ENABLED` | `true` | Enable/disable media downloads |
+| `MERCURY_MEDIA_MAX_SIZE_MB` | `10` | Max file size to download (MB) |
+
+## Storage
+
+### Ingress (inbox/)
+
+Incoming media files are saved to the space workspace:
+
+```
+.mercury/spaces/<space_id>/inbox/<timestamp>-<type>.<ext>
+```
+
+Example:
+```
+.mercury/spaces/whatsapp_123456_g_us/inbox/
+├── 1709012345-image.jpg
+├── 1709012400-voice.ogg
+└── 1709012500-document.pdf
+```
+
+### Egress (outbox/)
+
+The model writes files to `outbox/` during a container run. After exit, the runtime scans for files with `mtime >= startTime` and attaches them to the reply:
+
+```
+.mercury/spaces/<space_id>/outbox/
+├── chart.png
+└── summary.pdf
+```
+
+Previous outbox files are not deleted — only new or modified files are sent. See [pipeline.md](../pipeline.md) for details.
+
+## Database Schema
+
+Attachments are stored as JSON in the `messages.attachments` column:
+
+```sql
+ALTER TABLE messages ADD COLUMN attachments TEXT;
+```
+
+```json
+[
+  {
+    "path": "/Users/.../media/1709012345-image.jpg",
+    "type": "image",
+    "mimeType": "image/jpeg",
+    "sizeBytes": 12345
+  }
+]
+```
+
+## Prompt Format
+
+Attachments are passed to pi as XML:
+
+```xml
+<attachments>
+  <attachment type="image" path="/spaces/xxx/media/123-image.jpg" mime="image/jpeg" size="12345" />
+</attachments>
+
+@mercury what's in this image?
+```
+
+Reply context includes media info:
+
+```xml
+<reply_to name="John" jid="123@wa" message_id="ABC" media_type="image" media_mime="image/jpeg">
+Check out this sunset!
+</reply_to>
+```
+
+## pi Capabilities
+
+| Media Type | pi Support |
+|------------|------------|
+| Images (jpg, png, gif, webp) | ✅ Can view via `read` tool |
+| Voice/Audio | ❌ Cannot play — needs transcription |
+| Video | ❌ Cannot play — could extract frames |
+| Documents (txt, code) | ✅ Can read text-based files |
+| Documents (pdf, docx) | ❌ Cannot read binary formats |
+
+## Voice transcription
+
+Voice and audio attachments are not playable inside pi. Install the **`voice-transcribe`** extension (see dashboard catalog or `examples/extensions/voice-transcribe/`) to append a text transcript before the agent runs.
+
+- **Default (`voice-transcribe.provider=local`)**: runs Python on the Mercury host; install deps from the extension’s `requirements.txt`. Set `voice-transcribe.local_engine` to `transformers` (default, e.g. [mike249/whisper-tiny-he-2](https://huggingface.co/mike249/whisper-tiny-he-2)) or `faster_whisper` for [CTranslate2](https://github.com/OpenNMT/CTranslate2) models on the Hub (e.g. [ivrit-ai](https://huggingface.co/ivrit-ai) `*-ct2` repos). See the extension skill for `MERCURY_VOICE_FW_COMPUTE_TYPE` and `MERCURY_VOICE_LANGUAGE`.
+- **API (`voice-transcribe.provider=api`)**: uses the [Hugging Face Inference API](https://huggingface.co/docs/api-inference) with `MERCURY_HF_TOKEN` — choose a model that has a Hub Inference Provider.
+
+## Future Enhancements
+
+- [ ] Video frame extraction
+- [ ] PDF text extraction