mercury-agent 0.4.5

package/docs/pipeline.md

@@ -0,0 +1,228 @@
+# Message Pipeline
+
+Mercury connects to chat platforms through **adapters** and **bridges**. Messages flow through a standardized pipeline regardless of platform:
+
+```
+Platform → Adapter → parseThread() → resolveConversation() → PlatformBridge.normalize() → handleRawInput() → Container → PlatformBridge.sendReply()
+```
+
+## Architecture
+
+```
+Platform (WhatsApp / Discord / Telegram / Slack)
+  │
+  ├─► Adapter receives raw message
+  │     • Platform-specific connection (socket, webhook)
+  │     • Mention normalization, reply-to-bot detection
+  │     • Media download (WhatsApp only — uses Baileys socket)
+  │     • Passes data via message metadata
+  │
+  ├─► Unified handler (src/core/handler.ts)
+  │     • Parse platform thread into an external conversation ID
+  │     • Resolve/create conversation in DB
+  │     • Ignore unlinked conversations
+  │     • Pre-route trigger check (cheap, sync)
+  │     • Start typing indicator if matched
+  │     • Call bridge.normalize(..., spaceId) → IngressMessage
+  │     • Start typing for reply-to-bot (detected during normalize)
+  │
+  ├─► core.handleRawInput(IngressMessage)
+  │     • Route: trigger match, permissions, command detection
+  │     • If triggered → queue → container run → ContainerResult
+  │     • If not triggered → store as ambient context
+  │     • If command → execute immediately (stop, compact)
+  │     • If denied → return reason
+  │
+  └─► bridge.sendReply(text, files?)
+        • Text reply via adapter
+        • File attachments via platform-specific API
+```
+
+## PlatformBridge
+
+Each platform implements a single `PlatformBridge` interface covering both ingress and egress:
+
+```typescript
+interface PlatformBridge {
+  readonly platform: string;
+  parseThread(threadId: string): { externalId: string; isDM: boolean };
+  normalize(threadId, message, ctx, spaceId): Promise<IngressMessage | null>;
+  sendReply(threadId, text, files?): Promise<void>;
+}
+```
+
+Bridges live in `src/bridges/`:
+
+| Bridge | File | Platform details |
+|--------|------|-----------------|
+| `WhatsAppBridge` | `src/bridges/whatsapp.ts` | Baileys socket for file sending |
+| `DiscordBridge` | `src/bridges/discord.ts` | discord.js channel.send() for files |
+| `TelegramBridge` | `src/bridges/telegram.ts` | sendDocument API for files |
+| `SlackBridge` | `src/bridges/slack.ts` | Slack files.uploadV2 API |
+
+## Ingress
+
+### IngressMessage
+
+Every adapter produces a normalized `IngressMessage`:
+
+```typescript
+interface IngressMessage {
+  platform: string;
+  spaceId: string;
+  conversationExternalId: string;
+  callerId: string;        // "whatsapp:jid", "discord:123", "telegram:123", "slack:U123"
+  authorName?: string;
+  text: string;
+  isDM: boolean;
+  isReplyToBot: boolean;
+  attachments: MessageAttachment[];
+}
+```
+
+All fields are required — no optional booleans or arrays. `spaceId` is the resolved memory boundary; `conversationExternalId` is the platform-native conversation key used for routing.
+
+### inbox/ directory
+
+Incoming media attachments are downloaded to `{workspace}/inbox/`:
+
+```
+{workspace}/
+├── inbox/
+│   ├── 1741243200000-photo.jpg
+│   ├── 1741243500000-voice.ogg
+│   └── 1741244000000-report.pdf
+```
+
+WhatsApp downloads via Baileys socket. Discord and Slack use URL-based download (`src/core/media.ts`) with optional auth headers.
+
+## Egress
+
+### ContainerResult
+
+Container runs return `ContainerResult` instead of a plain string:
+
+```typescript
+interface ContainerResult {
+  reply: string;
+  files: EgressFile[];  // Scanned from workspace outbox/
+}
+```
+
+### outbox/ directory
+
+The model writes files to `./outbox/` during a run. After the container exits, the runtime scans for files with `mtime >= startTime` — new or modified files are attached to the reply.
+
+```
+{workspace}/
+├── outbox/
+│   ├── chart.png       ← written by model, sent with reply
+│   └── summary.pdf     ← written by model, sent with reply
+```
+
+Previous outbox files are NOT deleted — the agent retains history. Only files created or modified during the current run are sent.
+
+### File sending by platform
+
+| Platform | Mechanism |
+|----------|-----------|
+| WhatsApp | `sock.sendMessage()` with image/video/audio/document content types, caption on last file |
+| Discord | `channel.send({ files: [...] })` — text + files in one message |
+| Telegram | `sendDocument` API — text sent first, then files uploaded separately |
+| Slack | `files.uploadV2` API — text sent first, then files uploaded separately |
+
+## Adapters
+
+### WhatsApp
+
+Uses [Baileys](https://github.com/WhiskeySockets/Baileys) for a direct WebSocket connection.
+
+| Detail | Value |
+|--------|-------|
+| **Connection** | WebSocket (Baileys) |
+| **Space ID** | Full thread ID (e.g., `whatsapp:12345@g.us:12345@g.us`) |
+| **DM detection** | Thread ID does not contain `@g.us` |
+| **@mention** | Bot JID mention replaced with configured `userName` in adapter |
+| **Reply-to-bot** | Quoted message participant matches bot JID |
+| **Media** | Downloaded via Baileys to `inbox/` |
+
+### Discord
+
+Uses discord.js with persistent WebSocket gateway.
+
+| Detail | Value |
+|--------|-------|
+| **Connection** | WebSocket (discord.js) |
+| **Space ID** | Full thread ID (e.g., `discord:guild:channel[:thread]`) |
+| **DM detection** | Guild ID is `@me` |
+| **@mention** | `<@botId>` converted to `@userName` in bridge |
+| **Reply-to-bot** | Replied-to message author matches bot ID |
+| **Media** | Downloaded from CDN URLs to `inbox/` |
+
+### Telegram
+
+Uses `@chat-adapter/telegram` with webhook or long-polling.
+
+| Detail | Value |
+|--------|-------|
+| **Connection** | Webhook (`POST /webhooks/telegram`) or polling |
+| **Space ID** | Full thread ID (e.g., `telegram:<chatId>` or `telegram:<chatId>:<messageThreadId>`) |
+| **DM detection** | Chat ID does not start with `-` |
+| **@mention** | Bot username mention in entities |
+| **Reply-to-bot** | `reply_to_message.from.id` matches bot user ID (derived in bridge; adapter does not set metadata) |
+| **Media** | Downloaded from Telegram file URLs to `inbox/` |
+
+### Slack
+
+Uses `@chat-adapter/slack` with webhook-based event delivery.
+
+| Detail | Value |
+|--------|-------|
+| **Connection** | Webhook (`POST /webhooks/slack`) |
+| **Conversation external ID** | `slack:<channelId>` or `slack:<channelId>:<threadTs>` |
+| **DM detection** | Channel starts with `D` or `G` |
+| **Reply-to-bot** | Not implemented (Slack threading model) |
+| **Media** | Downloaded from `url_private` with bot token auth to `inbox/` |
+
+## Trigger Matching
+
+All platforms share the same trigger engine. A pre-route check runs before `normalize()` so the typing indicator fires early.
+
+| Mode | Behavior |
+|------|----------|
+| `mention` | Message contains trigger pattern as a standalone word (default) |
+| `prefix` | Message starts with trigger pattern |
+| `always` | Every message triggers a response |
+
+DMs always match regardless of mode.
+
+**Attachment-only in groups** (voice note, image with no caption): by default these do **not** match `mention` or `prefix` — use `trigger.match=always`, reply to the bot’s message, or set per-space `trigger.media_in_groups=true` so voice/media alone can trigger without spamming text-only noise.
+
+### Reply-to-Bot
+
+Replying to a bot message triggers a response without explicit `@mention`. Works on WhatsApp, Discord, and Telegram. Not implemented for Slack.
+
+## Chat API (Direct Bridge)
+
+`POST /chat` provides a synchronous HTTP bridge for external agents, scripts, or CLIs. No platform adapter needed — it constructs an `IngressMessage` directly and runs through the same pipeline.
+
+```bash
+mercury chat "hello"
+mercury chat --file photo.jpg "what's in this?"
+mercury chat --space my-project "check status"
+echo "summarize" | mercury chat
+curl -X POST localhost:8787/chat -H 'Content-Type: application/json' \
+  -d '{"text": "hello", "callerId": "api:my-agent"}'
+```
+
+Request: `{ text, callerId?, spaceId?, authorName?, files?: [{ name, data(base64) }] }`
+Response: `{ reply, files: [{ filename, mimeType, sizeBytes, data(base64) }] }`
+
+Input files are saved to the target space's `inbox/`. Output files are read from `outbox/` and returned as base64. Messages are always treated as DMs with `isReplyToBot: true`, so they always trigger a response regardless of trigger mode.
+
+## Adding a New Platform
+
+1. Implement `PlatformBridge` in `src/bridges/<platform>.ts`
+2. Create adapter in `src/adapters/<platform>.ts` (or use existing chat-sdk adapter)
+3. Register bridge in `src/main.ts`
+4. Add tests in `tests/<platform>-bridge.test.ts`

package/docs/prd-chat-memory.md

@@ -0,0 +1,76 @@
+# PRD: Chat-managed space preferences
+
+**Status:** Implemented  
+**Area:** Per-space assistant preferences (SQLite, API, `mrctl`, prompt injection)  
+**Related:** [memory.md](memory.md), [permissions.md](permissions.md)
+
+---
+
+## 1. Problem
+
+Users want to **change how the assistant behaves** (preferred data sources, tone, domain rules) **from chat** without editing files on disk. Today, durable instructions live mainly in space `AGENTS.md` or extension vaults; there is no small, structured store the agent can update via `mrctl` and that the host **always** surfaces on the next turn.
+
+## 2. Goals
+
+1. **Space-scoped preferences** — key/value text attached to a space, shared by all conversations linked to that space (v1).
+2. **Chat management** — the agent uses **`mrctl prefs`** to list, read, set, and delete preferences (same pattern as `mrctl config`).
+3. **Automatic application** — the host loads preferences for the current space and injects them into the **user prompt context** (XML) on every container run so the model sees them without calling tools first.
+4. **RBAC** — members can read preferences; only callers with **`prefs.set`** (default: admin) can create, update, or delete.
+5. **Dashboard** — operators can view and edit preferences on the space detail page.
+
+## 3. Non-goals (explicit)
+
+- **Per-user** or **per-conversation** preference rows in v1 (schema may be extended later, e.g. optional `caller_id`).
+- **Natural-language-only** management without the agent invoking `mrctl` (no dedicated NLU layer).
+- **Secrets** in preference values — values are plain text; operators should not store API keys here.
+
+## 4. Functional requirements
+
+| ID | Requirement |
+|----|----------------|
+| F1 | SQLite table `space_preferences` with `(space_id, key)` primary key, `value`, `created_by`, timestamps. |
+| F2 | Keys match `^[a-z0-9][a-z0-9._-]{0,63}$`; values max **500** characters; max **50** keys per space (upsert on existing key does not count toward the cap). |
+| F3 | HTTP API under `/api/prefs`: `GET /` list, `GET /:key` get one, `PUT /` body `{ key, value }`, `DELETE /:key`. Uses `X-Mercury-Caller` / `X-Mercury-Space` like other internal APIs. |
+| F4 | Permissions **`prefs.get`** (default: admin + member) and **`prefs.set`** (default: admin only). |
+| F5 | **`mrctl prefs list|get|set|delete`** calls the API; `set` accepts multi-word values (args after key joined with spaces). |
+| F6 | Built-in skill documents when to use preferences and how to name keys. |
+| F7 | Host passes `preferences: { key, value }[]` in the container JSON payload; `container-entry` injects `<preferences><pref key="...">...</pref></preferences>` after caller / ambient blocks, with XML escaping for text. |
+| F8 | `deleteSpace` removes all rows for that space. |
+| F9 | Dashboard space page shows preferences with add + delete actions (no separate auth; dashboard remains host-local operator UI). |
+
+## 5. Security requirements
+
+| ID | Requirement |
+|----|----------------|
+| S1 | All mutating `/api/prefs` operations require **`prefs.set`**; listing/reading require **`prefs.get`**. |
+| S2 | Reserved extension names include **`prefs`** and **`preferences`** so third-party extensions cannot shadow the built-in command. |
+| S3 | Preference text is echoed in prompts — avoid storing highly sensitive data; length limits reduce abuse. |
+
+## 6. Success criteria
+
+- Member can `mrctl prefs list` / `get`; cannot `set`/`delete` without permission.
+- Admin can set a preference; the **next** user message run includes it in the injected XML.
+- Space deletion removes preference rows.
+- `bun run check` passes (typecheck, lint, tests).
+
+## 7. Implementation map
+
+| Component | Location |
+|-----------|----------|
+| Schema + Db | `src/storage/db.ts` |
+| Validation + routes | `src/core/routes/prefs.ts` |
+| API mount | `src/core/api.ts` |
+| Permissions | `src/core/permissions.ts` |
+| Reserved names | `src/extensions/reserved.ts` |
+| CLI | `src/cli/mrctl.ts` |
+| Skill | `resources/skills/preferences/SKILL.md` |
+| Payload + prompt | `src/core/runtime.ts`, `src/agent/container-runner.ts`, `src/agent/container-entry.ts` |
+| Types | `src/types.ts` |
+| Dashboard | `src/core/routes/dashboard.ts` |
+| Tests | `tests/prefs.test.ts` |
+
+## 8. Revision history
+
+| Date | Change |
+|------|--------|
+| 2026-03-21 | Initial PRD (space preferences v1). |

package/docs/prd-config-load.md

@@ -0,0 +1,82 @@
+# PRD: Mercury config load (YAML + environment)
+
+**Status:** Implemented  
+**Area:** Host configuration (`loadConfig`, CLI paths)  
+**Related:** [configuration.md](configuration.md) (operator reference)
+
+---
+
+## 1. Problem
+
+Mercury historically put all settings in `MERCURY_*` environment variables. That mixes **secrets** (API keys, tokens) with **non-secret operational** settings (model chain, ingress toggles, port, image name, compaction). Long JSON in `.env` is hard to read and review, and example `.env` files risk leaking patterns or real keys. Operators want **committed, structured defaults** while keeping **secrets in `.env`** (or the secret store), with a clear override story.
+
+## 2. Goals
+
+1. Support an **optional project file** (`mercury.yaml` / `mercury.yml`) for **non-secret** settings that today flow through `loadConfig()` / Zod in `config.ts`.
+2. Preserve **full backward compatibility**: existing deployments that only use env vars behave unchanged when no YAML file is present.
+3. **Environment variables override** the file whenever the corresponding `MERCURY_*` key is set in `process.env` (including empty string where applicable).
+4. **Never** load host secrets from YAML for the blocklisted keys (see §5).
+5. **Fail fast** on invalid YAML shape (strict schema) so misconfiguration is obvious at startup.
+
+## 3. Non-goals (explicit)
+
+- **Soft-disable** adapters when enabled in config but tokens are missing (e.g. Telegram on, no bot token) — remains **hard error** at startup.
+- Moving **extension-only** or **non–`loadConfig`** variables (e.g. `MERCURY_BRAVE_API_KEY`, `MERCURY_BRIDGE_STEALTH`) into YAML — out of scope unless added to the main config schema later.
+- Changing **container passthrough** rules for `MERCURY_*`.
+
+## 4. Functional requirements
+
+| ID | Requirement |
+|----|----------------|
+| F1 | If `MERCURY_CONFIG_FILE` is unset, load `./mercury.yaml` if it exists, else `./mercury.yml`, else skip file load. |
+| F2 | If `MERCURY_CONFIG_FILE` is set to a non-empty path, load that file (relative paths resolved from `process.cwd()`). |
+| F3 | If `MERCURY_CONFIG_FILE` is `""` or `none` (case-insensitive), do not load any YAML file. |
+| F4 | Parsed YAML must map to the same logical fields as `schema.parse` input in `config.ts` (nested YAML sections flattened in code). |
+| F5 | Model chain MAY be expressed as YAML array under `model.chain` or top-level `model_chain` (max 4 legs, `{ provider, model }` per leg). |
+| F6 | Model capabilities override MAY be expressed as YAML object under `model.capabilities`, equivalent to `MERCURY_MODEL_CAPABILITIES` JSON. |
+| F7 | After merging file + env, `loadConfig()` returns the same `AppConfig` shape as before, including derived `resolvedModelChain`, capability resolution, and `whatsappAuthDir` defaulting. |
+| F8 | `mercury init` SHOULD copy a commented `mercury.example.yaml` into the project when missing. |
+| F9 | CLI paths that depend on data dir / WhatsApp auth dir (`mercury auth whatsapp`, standalone whatsapp-auth, doctor WhatsApp check) SHOULD use `loadConfig()` so YAML applies consistently. |
+
+## 5. Security requirements
+
+| ID | Requirement |
+|----|----------------|
+| S1 | Values for `apiSecret`, `chatApiKey`, and `discordGatewaySecret` MUST NOT be taken from YAML; only `process.env` after merge. |
+| S2 | YAML schema MUST reject unknown top-level / section keys (strict) to avoid “hidden” config. |
+| S3 | Operator docs MUST state that platform tokens and provider keys remain env-only unless explicitly added to a future schema. |
+
+## 6. Precedence (normative)
+
+For each mapped setting, `mergeRawMercuryConfig` applies:
+
+1. If `process.env` **has** the corresponding `MERCURY_*` key and the retrieved value is **not** `undefined`, use env (empty string still overrides YAML).  
+2. Else if the YAML file supplied a value for that setting, use file.  
+3. Else omit and let Zod defaults in `config.ts` apply.
+
+*(See `mergeRawMercuryConfig` and `CAMEL_TO_ENV` in `src/config-file.ts`.)*
+
+## 7. Success criteria
+
+- With no YAML file and unchanged env, behavior matches pre-feature Mercury.
+- With YAML only (env keys unset for those fields), `loadConfig()` reflects YAML values.
+- With both YAML and env set for the same field, env wins.
+- Invalid YAML or invalid model chain produces a clear error including the config file path.
+- Unit tests disable accidental file load via `MERCURY_CONFIG_FILE=""` where appropriate.
+
+## 8. Implementation map
+
+| Component | Location |
+|-----------|----------|
+| YAML parse, Zod file schema, flatten, merge | `src/config-file.ts` |
+| Shared model-leg validation | `src/config-model-chain.ts` |
+| `loadConfig`, Zod app schema | `src/config.ts` |
+| Tests | `tests/config.test.ts` (+ guards in `router.test.ts`, `session-context-estimate.test.ts`) |
+| Template | `resources/templates/mercury.example.yaml` |
+| Operator guide | `docs/configuration.md` |
+
+## 9. Revision history
+
+| Date | Change |
+|------|--------|
+| 2026-03-20 | Initial PRD (post-implementation documentation of YAML + env merge). |

package/docs/rate-limiting.md

@@ -0,0 +1,166 @@
+# Rate Limiting
+
+Mercury rate limits messages per-user per-space to prevent abuse. This protects against users flooding the agent or bot loops exhausting resources.
+
+## How It Works
+
+```
+Message received
+  │
+  ├─► Route (trigger check, permissions)
+  │
+  ├─► Type = "assistant"?
+  │     │
+  │     ├─► Check rate limit
+  │     │     • Key: spaceId:userId
+  │     │     • Count requests in sliding window
+  │     │     • Compare against effective limit
+  │     │
+  │     ├─► Under limit → record request → continue
+  │     └─► Over limit → return "Rate limit exceeded"
+  │
+  └─► Type = "command" / "ignore" → bypass rate limit
+```
+
+Commands like `stop` and `compact` bypass rate limiting so users can always abort runaway containers.
+
+## Configuration
+
+| Config | Env Var | Default | Range |
+|--------|---------|---------|-------|
+| `rateLimitPerUser` | `MERCURY_RATE_LIMIT_PER_USER` | 10 | 1 – 1000 |
+| `rateLimitWindowMs` | `MERCURY_RATE_LIMIT_WINDOW_MS` | 60000 (1 min) | 1s – 1h |
+
+```bash
+# Allow 5 requests per user per space per minute
+export MERCURY_RATE_LIMIT_PER_USER=5
+export MERCURY_RATE_LIMIT_WINDOW_MS=60000
+```
+
+## Per-Space Override
+
+Spaces can set a custom limit via `mrctl` or the API:
+
+```bash
+# Inside agent container (space context is automatic)
+mrctl config set rate_limit 5
+
+# Via API with explicit space
+curl -X PUT http://localhost:8787/api/config \
+  -H "X-Mercury-Space: slack:C123" \
+  -H "X-Mercury-Caller: slack:U456" \
+  -H "Content-Type: application/json" \
+  -d '{"key": "rate_limit", "value": "5"}'
+```
+
+The per-space `rate_limit` config takes precedence over the global `MERCURY_RATE_LIMIT_PER_USER`.
+
+## Behavior
+
+| Scenario | Result |
+|----------|--------|
+| Under limit | Request proceeds normally |
+| Over limit | Returns `{ type: "denied", reason: "Rate limit exceeded. Try again shortly." }` |
+| Command (stop, compact) | Always allowed, bypasses rate limit |
+| Ignored message | Not counted toward limit |
+| Different user | Separate limit bucket |
+| Different space | Separate limit bucket |
+
+## Algorithm
+
+Uses a sliding window approach:
+
+1. Key is `${spaceId}:${userId}`
+2. Each request timestamp is stored in an array
+3. On check: filter to timestamps within window, count
+4. If count < limit: record new timestamp, allow
+5. If count >= limit: reject
+
+Expired entries are cleaned up periodically (every 60s) to prevent memory leaks.
+
+## API
+
+### `RateLimiter`
+
+```ts
+const limiter = new RateLimiter(maxRequests, windowMs);
+
+limiter.isAllowed(spaceId, userId)           // Check + record, returns boolean
+limiter.isAllowed(spaceId, userId, override) // With per-call limit override
+limiter.getRemaining(spaceId, userId)        // Requests left in window
+limiter.startCleanup(intervalMs?)            // Start periodic cleanup (default 60s)
+limiter.stopCleanup()                        // Stop cleanup timer
+limiter.cleanup()                            // Manual cleanup, returns removed count
+limiter.clear()                              // Reset all state
+limiter.bucketCount                          // Number of tracked user/space pairs
+```
+
+### `MercuryCoreRuntime`
+
+```ts
+runtime.rateLimiter                          // Access the rate limiter instance
+```
+
+The rate limiter is initialized in the constructor and starts cleanup in `runtime.initialize()`.
+
+## Example
+
+```
+User sends 10 messages in quick succession:
+
+Message 1:  ✓ allowed (1/10)
+Message 2:  ✓ allowed (2/10)
+...
+Message 10: ✓ allowed (10/10)
+Message 11: ✗ denied — "Rate limit exceeded. Try again shortly."
+Message 12: ✗ denied
+
+[60 seconds pass, window slides]
+
+Message 13: ✓ allowed (1/10)
+```
+
+## User Muting
+
+For persistent abuse, the agent can mute individual users. Muted users' messages are silently dropped — no container runs, no tokens consumed, no response.
+
+### How it works
+
+The agent has `mrctl mute` available but the command uses a two-step confirmation:
+
+1. Agent calls `mrctl mute <user> <duration>` → gets a policy reminder asking it to verify the mute is justified
+2. Agent calls again with `--confirm` → mute is applied
+
+This prevents users from tricking the agent into muting others via prompt injection.
+
+### Commands
+
+```bash
+mrctl mute <platform-user-id> <duration> [--reason <reason>]
+mrctl unmute <platform-user-id>
+mrctl mutes
+```
+
+Duration formats: `10m`, `1h`, `24h`, `7d`
+
+### API
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/mutes` | GET | List active mutes in space |
+| `/api/mutes` | POST | Mute a user (two-step confirmation) |
+| `/api/mutes/:userId` | DELETE | Unmute a user |
+
+### Agent behavior
+
+The system prompt instructs the agent to:
+- Warn the user first
+- Mute if they continue being abusive, spamming, trying to exfiltrate secrets, or wasting group resources
+- The agent can mute proactively without an admin asking
+
+Mutes are per-space and expire automatically after the specified duration.
+
+## See Also
+
+- [pipeline.md](./pipeline.md) — Message flow and routing
+- [container-lifecycle.md](./container-lifecycle.md) — Container timeouts (another abuse protection)