@hybridaione/hybridclaw 0.4.2 → 0.5.0

package/AGENTS.md

@@ -1,5 +1,84 @@
 # AGENTS.md
 
+## Scope
+
+This file is the canonical repo-level instruction set for coding agents working
+in HybridClaw.
+
+- Follow this file first.
+- If a deeper directory contains its own `AGENTS.md`, that file overrides this
+  one for its subtree.
+- Keep `CLAUDE.md` aligned with this file. `CLAUDE.md` should only carry
+  tool-specific deltas.
+
+## Project Map
+
+- `src/` core CLI, gateway, providers, auth, audit, scheduler, and runtime
+  wiring
+- `container/` sandboxed runtime, tool executor, provider adapters, and
+  container build inputs
+- `skills/` bundled `SKILL.md` skills plus any supporting scripts or reference
+  material
+- `templates/` runtime workspace bootstrap files seeded into agent workspaces
+- `tests/` Vitest suites across unit, integration, e2e, and live coverage
+- `docs/` static site assets and maintainer/development reference docs
+
+## Working Rules
+
+- Keep changes focused. Prefer targeted fixes over broad refactors unless the
+  task requires wider movement.
+- Match the existing TypeScript + ESM patterns already used in the touched area.
+- Update tests and docs when behavior, commands, or repo workflows change.
+- Treat existing uncommitted changes as user work unless you created them.
+- Do not rename or relocate files in `templates/` without updating
+  `src/workspace.ts` and the workspace bootstrap tests.
+
+## Setup And Commands
+
+Prerequisites:
+
+- Node.js 22 (matches CI)
+- npm
+- Docker when working on container-mode behavior or image builds
+
+Common commands:
+
+```bash
+npm install
+npm run setup
+npm run build
+npm run typecheck
+npm run lint
+npm run check
+npm run test:unit
+npm run test:integration
+npm run test:e2e
+npm run test:live
+npm run release:check
+npm --prefix container run lint
+npm --prefix container run release:check
+```
+
+## Testing Expectations
+
+- Docs-only changes: keep links and commands accurate; runtime tests are usually
+  unnecessary.
+- `src/` changes: run `npm run typecheck`, `npm run lint`, and the relevant
+  Vitest suites.
+- `container/` changes: run `npm --prefix container run lint`, `npm run build`,
+  and targeted tests that exercise the runtime boundary.
+- Release or packaging changes: run both release checks and verify versioned
+  docs stay aligned.
+- If you skip a relevant check, state that explicitly in your handoff.
+
+## Documentation Hierarchy
+
+- `README.md` is the end-user and product entry point.
+- `CONTRIBUTING.md` is the human contributor quickstart.
+- `docs/development/` holds deeper maintainer and runtime reference docs.
+- `templates/*.md` are product runtime workspace seed files, not repo
+  contributor onboarding docs.
+
 ## Bump Release
 
 When the user says "bump release":
@@ -10,10 +89,12 @@ When the user says "bump release":
    - `package-lock.json` (root `version` and `packages[""]`)
    - `container/package.json`
    - `container/package-lock.json` (root `version` and `packages[""]`)
-   - any user-facing version text (for example `src/tui.ts` banner).
-3. Move `CHANGELOG.md` release notes from `Unreleased` to the new version heading (or create one).
+   - any user-facing version text (for example `src/tui.ts` banner)
+3. Move `CHANGELOG.md` release notes from `Unreleased` to the new version
+   heading (or create one).
 4. Update `README.md` "latest tag" link/text if present.
 5. Commit with a release chore message (for example `chore: release vX.Y.Z`).
-6. Create annotated git tag `vX.Y.Z`.
-7. Push commit and tag.
-8. Always create/publish a GitHub Release entry for the tag (tags alone do not update the Releases list).
+6. Create an annotated git tag `vX.Y.Z`.
+7. Push the commit and tag.
+8. Always create or publish a GitHub Release entry for the tag. Tags alone do
+   not update the Releases list.

package/CHANGELOG.md

@@ -2,6 +2,48 @@
 
 ## [Unreleased]
 
+## [0.5.0](https://github.com/HybridAIOne/hybridclaw/tree/v0.5.0)
+
+### Added
+
+- **Model Context Protocol support**: Added runtime `mcpServers` config plus container MCP client loading so HybridClaw can expose configured MCP servers as namespaced tools, with TUI `/mcp list|add|toggle|remove|reconnect` management commands.
+- **Discord slash command control plane**: Added global Discord slash commands for status, approvals, compaction, channel policy, model/bot selection, RAG, Ralph loop, MCP management, usage, export, sessions, audit, and scheduling, with private approval responses.
+- **Bundled office document skills**: Added `docx`, `xlsx`, `pptx`, and `office-workflows` bundled skills plus shared office helper scripts for OOXML pack/unpack, tracked-change cleanup, spreadsheet import/recalc, and presentation thumbnail QA.
+- **Authenticated artifact downloads**: Added gateway `/api/artifact` serving for generated agent artifacts and cached Discord media so the web chat can render previews and download generated office outputs safely.
+
+### Changed
+
+- **Runtime capability guidance**: Prompt/tool summaries now group MCP tools cleanly and add office-file guardrails so models avoid fake binary placeholders and follow document QA workflows.
+- **Discord delivery workflow**: The Discord `message` tool now supports native local-file uploads via `filePath`, and runtime delivery/register flows better handle workspace files, `/discord-media-cache`, and DM-visible global slash commands.
+- **Documentation and examples**: README, runtime docs, and built-in web/chat surfaces now document MCP setup, bundled office skills, and artifact handling for the new workflows.
+
+## [0.4.3](https://github.com/HybridAIOne/hybridclaw/tree/v0.4.3)
+
+### Added
+
+- **Manual session compaction command**: Added built-in `/compact` support across gateway, TUI, and Discord to archive older transcript history, summarize it into high-confidence session memory, and preserve a recent conversation tail for active context.
+- **Bundled PDF workflow support**: Added a built-in `pdf` skill plus Node-based PDF tooling for text extraction, page rendering, fillable form inspection/filling, and non-fillable overlay workflows, with current-turn PDF context injection for explicit file paths and Discord attachments.
+- **Skill installer commands**: Added `hybridclaw skill list` and `hybridclaw skill install <skill> [install-id]` so bundled skills can advertise optional dependency installers.
+- **Container bind path config**: Added `container.binds` support alongside validated host/container path aliasing so configured external directories can be used safely from sandboxed tools and PDF workflows.
+- **Published coverage badge**: CI now generates and publishes a coverage badge JSON artifact for the README badge and release-health visibility.
+
+### Changed
+
+- **Attachment and media routing**: Gateway/media prompt assembly now distinguishes image attachments from document attachments, prefers current-turn local files for PDFs, and limits native vision injection to actual image inputs.
+- **Contributor documentation structure**: Promoted `AGENTS.md` to the canonical repo-level agent guide, slimmed `CONTRIBUTING.md` into a contributor quickstart, and moved deeper maintainer/runtime references into `docs/development/`.
+- **Host runtime workspace setup**: Host-mode agent workspaces now link package `node_modules`, while runtime path handling and workspace globbing understand configured extra mounts and local scratch paths more reliably.
+- **Release metadata and docs alignment**: The published package now declares `Node 22.x`, README badges point at maintained badge sources, and the docs landing page tracks the current tagged release/version requirements.
+- **Regression coverage**: Added focused unit coverage for memory chunking, gateway startup/health flows, Discord delivery chunking, PDF context handling, and compaction paths.
+
+### Fixed
+
+- **Compaction archive path exposure**: `/compact` responses now show a safe archive reference instead of leaking absolute host filesystem paths in user-facing output.
+- **Workspace bootstrap lifecycle**: `BOOTSTRAP.md` is now removed once onboarding is effectively complete and is not recreated on subsequent starts.
+- **Codex device-code activation flow**: Device-code login now falls back to the default activation URL and tolerates nested pending/authorization error payloads from the auth service.
+- **Runtime-home migration false positive**: Launching HybridClaw from `~/.hybridclaw` no longer treats the runtime `data/` directory as a legacy current-working-directory migration target.
+- **Heartbeat proactive queue cleanup**: Local proactive delivery now drops orphaned heartbeat queue rows instead of trying to route them as real outbound messages.
+- **Coverage badge publishing permissions**: CI now has the repository permissions needed to update the published coverage badge without failing the main workflow.
+
 ## [0.4.2](https://github.com/HybridAIOne/hybridclaw/tree/v0.4.2)
 
 ### Added

package/README.md

@@ -1,5 +1,13 @@
 # HybridClaw
 
+[![CI](https://github.com/HybridAIOne/hybridclaw/actions/workflows/ci.yml/badge.svg)](https://github.com/HybridAIOne/hybridclaw/actions/workflows/ci.yml)
+[![coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/HybridAIOne/hybridclaw/gh-pages/badge/coverage.json)](https://github.com/HybridAIOne/hybridclaw/actions/workflows/ci.yml)
+[![npm](https://img.shields.io/npm/v/@hybridaione/hybridclaw)](https://www.npmjs.com/package/@hybridaione/hybridclaw)
+[![Node](https://img.shields.io/badge/node-22.x-5FA04E?logo=node.js&logoColor=white)](https://nodejs.org/en/download)
+[![License](https://img.shields.io/github/license/HybridAIOne/hybridclaw)](https://github.com/HybridAIOne/hybridclaw/blob/main/LICENSE)
+[![Docs](https://img.shields.io/badge/docs-github%20pages-blue)](https://hybridaione.github.io/hybridclaw/)
+[![Powered by HybridAI](https://img.shields.io/badge/powered%20by-HybridAI-blueviolet)](https://hybridai.one)
+
 <img width="540" height="511" alt="image" src="docs/hero.png" />
 
 Personal AI assistant bot for Discord, powered by [HybridAI](https://hybridai.one).
@@ -11,6 +19,9 @@ npm install -g @hybridaione/hybridclaw
 hybridclaw onboarding
 ```
 
+Prerequisites: Node.js 22. Docker is recommended when you want the default
+container sandbox.
+
 ## HybridAI Advantage
 
 - Security-focused foundation
@@ -50,6 +61,9 @@ hybridclaw gateway
 # Or run gateway in foreground in this terminal
 hybridclaw gateway start --foreground
 
+# For stdio MCP servers that rely on host tools like `docker` or `npx`
+hybridclaw gateway start --foreground --sandbox=host
+
 # If DISCORD_TOKEN is set, gateway auto-connects to Discord.
 
 # Start terminal adapter (optional, in a second terminal)
@@ -129,11 +143,86 @@ HybridClaw creates `~/.hybridclaw/config.json` on first run and hot-reloads most
 - Start from `config.example.json` (reference).
 - Runtime state lives under `~/.hybridclaw/` (`config.json`, `credentials.json`, `data/hybridclaw.db`, audit/session files).
 - HybridClaw does not keep runtime state in the current working directory. If `./.env` exists, supported secrets are migrated once into `~/.hybridclaw/credentials.json`.
-- `container.*` controls execution isolation, including `sandboxMode`, `memory`, `memorySwap`, `cpus`, `network`, and additional mounts.
+- `container.*` controls execution isolation, including `sandboxMode`, `memory`, `memorySwap`, `cpus`, `network`, `binds`, and additional mounts.
+- Use `container.binds` for explicit host-to-container mounts in `host:container[:ro|rw]` format. Mounted paths appear inside the sandbox under `/workspace/extra/<container>`.
+- `mcpServers.*` declares Model Context Protocol servers that HybridClaw connects to per session and exposes as namespaced tools (`server__tool`).
+- `mcpServers.*.env` and `mcpServers.*.headers` are currently written to `~/.hybridclaw/config.json` as plain text. Use low-privilege tokens only, set `chmod 700 ~/.hybridclaw && chmod 600 ~/.hybridclaw/config.json`, and prefer `host` sandbox mode for stdio MCP servers that depend on host-installed tools.
 - Keep HybridAI secrets in `~/.hybridclaw/credentials.json` (`HYBRIDAI_API_KEY` required for HybridAI models, `DISCORD_TOKEN` optional). Codex OAuth sessions are stored separately in `~/.hybridclaw/codex-auth.json`.
 - Trust-model acceptance is stored in `~/.hybridclaw/config.json` under `security.*` and is required before runtime starts.
 - See [TRUST_MODEL.md](./TRUST_MODEL.md) for onboarding acceptance policy and [SECURITY.md](./SECURITY.md) for technical security guidelines.
-- For advanced configuration, audit/observability details, skills internals, agent tools, and developer docs, see [CONTRIBUTING.md](./CONTRIBUTING.md).
+- For contributor workflow, see [CONTRIBUTING.md](./CONTRIBUTING.md). For deeper runtime, skills, release, and maintainer reference docs, see [docs/development/README.md](./docs/development/README.md).
+
+## TUI MCP Quickstart
+
+For stdio MCP servers that use host binaries such as `docker`, `node`, or
+`npx`, start the gateway in host mode:
+
+```bash
+hybridclaw gateway start --foreground --sandbox=host
+hybridclaw tui
+```
+
+In the TUI, use the MCP slash commands directly:
+
+```text
+/mcp list
+/mcp add filesystem {"transport":"stdio","command":"npx","args":["-y","@modelcontextprotocol/server-filesystem","/Users/you/project"],"enabled":true}
+/mcp toggle filesystem
+/mcp reconnect filesystem
+/mcp remove filesystem
+```
+
+Once a server is enabled, its tools appear in prompts as namespaced tool names
+such as `filesystem__read_file` or `github__list_issues`.
+
+## Bundled Skills
+
+- `pdf` is bundled and supports text extraction, page rendering, fillable form inspection/filling, and non-fillable overlay workflows.
+- `xlsx` is bundled for spreadsheet creation, formula-safe editing, CSV/TSV cleanup, and LibreOffice-backed recalculation.
+- `docx` is bundled for Word document creation plus OOXML unpack/edit/pack workflows, comments, and tracked-change cleanup.
+- `pptx` is bundled for presentation creation with `pptxgenjs`, template-preserving OOXML edits, and thumbnail-based visual QA.
+- `office-workflows` is bundled for cross-format tasks such as CSV to XLSX cleanup and XLSX to PPTX or DOCX deliverables coordinated with delegation.
+- Use `hybridclaw skill list` to inspect available installers and `hybridclaw skill install pdf [install-id]` when a bundled skill advertises optional setup helpers.
+
+## Optional Office Dependencies
+
+When you run HybridClaw in the default container sandbox, the bundled office image already includes the main office tooling. These installs matter primarily for `--sandbox=host` workflows or when you want the same capabilities on your local machine.
+
+What they unlock:
+
+- LibreOffice (`soffice`) enables Office-to-PDF export, PPTX visual QA, and XLSX formula recalculation.
+- Poppler (`pdftoppm`) enables slide/page thumbnail rendering for PPTX visual QA.
+- Pandoc improves document conversion workflows around DOCX and Markdown.
+
+macOS:
+
+```bash
+brew install --cask libreoffice
+brew install poppler pandoc
+```
+
+Ubuntu / Debian:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y libreoffice poppler-utils pandoc
+```
+
+Fedora:
+
+```bash
+sudo dnf install -y libreoffice poppler-utils pandoc
+```
+
+Verify availability:
+
+```bash
+sh -lc 'command -v soffice >/dev/null 2>&1 || command -v libreoffice >/dev/null 2>&1 && echo soffice_ok'
+sh -lc 'command -v pdftoppm >/dev/null 2>&1 && echo pdftoppm_ok'
+sh -lc 'command -v pandoc >/dev/null 2>&1 && echo pandoc_ok'
+```
+
+Without these tools, the office skills still create and edit `.docx`, `.xlsx`, and `.pptx` files, but some higher-quality QA and conversion paths are skipped.
 
 ## Commands
 
@@ -145,6 +234,7 @@ CLI runtime commands:
 - `hybridclaw gateway stop` — Stop managed gateway backend process
 - `hybridclaw gateway status` — Show lifecycle/API status
 - `hybridclaw gateway <command...>` — Send a command to a running gateway (for example `sessions`, `bot info`)
+- `hybridclaw gateway compact` — Archive older session history into semantic memory while preserving a recent active context tail
 - `hybridclaw tui` — Start terminal client connected to gateway
 - `hybridclaw onboarding` — Run HybridAI account/API key onboarding
 - `hybridclaw hybridai login [--device-code|--browser|--import]` — Store HybridAI API credentials via browser-assisted, headless/manual, or env-import flows
@@ -153,6 +243,8 @@ CLI runtime commands:
 - `hybridclaw codex login [--device-code|--browser|--import]` — Authenticate OpenAI Codex via OAuth or one-time Codex CLI import
 - `hybridclaw codex status` — Show stored Codex auth state, token mask, expiry, and source
 - `hybridclaw codex logout` — Clear stored Codex credentials
+- `hybridclaw skill list` — Show skills and any declared installer options
+- `hybridclaw skill install <skill> [install-id]` — Run a declared skill dependency installer
 - `hybridclaw update [status|--check] [--yes]` — Check for updates and upgrade global npm installs (source checkouts get git-based update instructions)
 - `hybridclaw audit ...` — Verify and inspect structured audit trail (`recent`, `search`, `approvals`, `verify`, `instructions`)
 - `hybridclaw audit instructions [--sync]` — Compare runtime instruction copies under `~/.hybridclaw/instructions/` against installed sources and restore shipped defaults when needed
@@ -163,6 +255,7 @@ In Discord, use `!claw help` to see all commands. Key ones:
 - `!claw bot set <id>` — Set chatbot for this channel
 - `!claw model set <name>` — Set model for this channel
 - `!claw rag on/off` — Toggle RAG
+- `!claw compact` — Archive older history into session memory and keep a recent working tail
 - `!claw clear` — Clear conversation history
 - `!claw audit recent [n]` — Show recent structured audit events
 - `!claw audit verify [sessionId]` — Verify audit hash chain integrity
@@ -170,6 +263,11 @@ In Discord, use `!claw help` to see all commands. Key ones:
 - `!claw audit approvals [n] [--denied]` — Show policy approval decisions
 - `!claw usage [summary|daily|monthly|model [daily|monthly] [agentId]]` — Show token/cost aggregates
 - `!claw export session [sessionId]` — Export session snapshot as JSONL
+- `!claw mcp list` — List configured MCP servers
+- `!claw mcp add <name> <json>` — Add or update an MCP server config
 - `!claw schedule add "<cron>" <prompt>` — Add cron scheduled task
 - `!claw schedule add at "<ISO time>" <prompt>` — Add one-shot task
 - `!claw schedule add every <ms> <prompt>` — Add interval task
+
+In the TUI, use `/compact` for session compaction and `/mcp ...` for runtime
+MCP management.

package/config.example.json

@@ -1,5 +1,5 @@
 {
-  "version": 7,
+  "version": 8,
   "security": {
     "trustModelAccepted": false,
     "trustModelAcceptedAt": "",
@@ -89,10 +89,38 @@
     "cpus": "1",
     "network": "bridge",
     "timeoutMs": 300000,
+    "binds": [],
     "additionalMounts": "",
     "maxOutputBytes": 10485760,
     "maxConcurrent": 5
   },
+  "mcpServers": {
+    "filesystem": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": [
+        "-y",
+        "@modelcontextprotocol/server-filesystem",
+        "/absolute/path/to/a/workspace"
+      ],
+      "enabled": false
+    },
+    "tavily": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": ["-y", "tavily-mcp@latest"],
+      "env": {
+        "TAVILY_API_KEY": "replace-me"
+      },
+      "enabled": false
+    },
+    "playwright": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": ["-y", "@playwright/mcp@latest"],
+      "enabled": false
+    }
+  },
   "web": {
     "search": {
       "provider": "auto",

package/container/Dockerfile

@@ -1,10 +1,25 @@
 FROM node:20-slim
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
-    ripgrep git curl python3 python3-pip \
+    ripgrep git curl python3 python3-pip poppler-utils qpdf pandoc \
+    libreoffice-calc libreoffice-impress libreoffice-writer \
     && rm -rf /var/lib/apt/lists/*
 
-RUN python3 -m pip install --no-cache-dir --break-system-packages uv
+RUN python3 -m pip install --no-cache-dir --break-system-packages \
+    uv \
+    pypdf==5.4.0 \
+    pdfplumber==0.11.6 \
+    pdf2image==1.17.0 \
+    reportlab==4.4.4 \
+    pillow==11.3.0
+
+RUN npm install -g \
+    docx@9.5.1 \
+    pptxgenjs@4.0.1 \
+    exceljs@4.4.0 \
+    csv-parse@6.1.0 \
+    iconv-lite@0.7.0
+ENV NODE_PATH=/usr/local/lib/node_modules:/app/node_modules
 
 WORKDIR /app