nodal-agents 0.1.4 → 0.1.6

package/README.md

@@ -1,81 +1,211 @@
-# Nodal-Agents
-
-Local-first AI agent platform. Self-host on Mac, PC, Linux, VPS, or NAS in 3 commands.
-
-## Install in 30 seconds (Docker)
-
-Works on any host with Docker — Mac, PC, Linux, VPS, Synology / Unraid / TrueNAS NAS, Raspberry Pi.
-
-```bash
-curl -O https://raw.githubusercontent.com/Kwintspiracy/nodal-agents/main/docker-compose.yml
-docker compose up -d
-```
-
-Open <http://localhost:3000> — Nodal-Agents is running. Configure your LLM provider from **Settings → LLM keys** in the dashboard.
-
-Data (config, postgres, logs) lives in the `nodal-data` Docker volume. To wipe everything: `docker compose down -v`.
-
-The image is published to `ghcr.io/kwintspiracy/nodal-agents:latest` (multi-arch: amd64 + arm64). To pin a version, replace `latest` with a release tag (e.g. `v0.1.0`).
-
-### Build the image locally
-
-If you'd rather build from source instead of pulling the published image:
-
-```bash
-git clone https://github.com/Kwintspiracy/nodal-agents.git
-cd nodal-agents
-docker compose build
-docker compose up -d
-```
-
-(Comment the `image:` line and uncomment `build: .` in `docker-compose.yml`.)
-
-## Developer setup (monorepo)
-
-```bash
-pnpm install
-pnpm typecheck
-pnpm lint
-pnpm test
-pnpm --filter nodal-agents dev   # CLI in tsx watch mode
-```
-
-## Monorepo structure
-
-- `apps/web` — Next.js dashboard (UI)
-- `apps/runner` — Agent runtime (HTTP API, job execution, cron ticker)
-- `apps/cli` — `nodal-agents` install + ops command
-- `packages/db` — Drizzle schema + migrations + client (only place that imports postgres)
-- `packages/shared` — Zod types and constants shared across web + runner
-- `packages/llm` — Vercel AI SDK wrapper, multi-provider abstraction
-- `packages/tools` — Tool registration + execution + approval gates
-- `packages/memory` — Agent memory CRUD + embeddings
-- `packages/orchestration` — Router + Planner patterns (delegation, task board)
-- `packages/runner-adapters` — Connectors: notion, drive, gmail, sheets, etc.
-- `packages/delivery` — Telegram, email, future Slack/Discord
-- `packages/auth` — Pluggable auth provider (local-trust default, better-auth opt-in, bearer-token for LAN)
-
-## Architecture rules (enforced by `dependency-cruiser`)
-
-- `apps/*` may import `packages/*`, never the reverse
-- `apps/web` and `apps/runner` cannot import each other (DB or HTTP only)
-- Only `packages/db` may import postgres clients (`pg`, `postgres`, `drizzle-orm`)
-- `packages/runner-adapters/*` may only import from `packages/tools` and `packages/shared`
-- No circular dependencies
-
-Run `pnpm deps:check` locally before pushing.
-
-## Stack
-
-- **Runtime:** Node 22+, TypeScript strict
-- **Monorepo:** pnpm workspaces + Turborepo
-- **DB:** embedded-postgres (local + Docker), Drizzle ORM, pgvector
-- **Validation:** Zod
-- **HTTP server:** Hono (runner)
-- **LLM:** Vercel AI SDK
-- **Auth:** local-trust (single-user loopback) / local-auth (better-auth, multi-user LAN) / bearer-token
-- **Tests:** Vitest (unit), Playwright (e2e), dependency-cruiser (architecture)
-
-## License
-
-TBD.
+# Nodal-Agents
+
+> **Your AI agents. Your data. Your machine.**
+> Self-hosted, local-first AI agent platform — install in two commands.
+
+[![npm](https://img.shields.io/npm/v/nodal-agents?color=cb3837&label=npm&logo=npm)](https://www.npmjs.com/package/nodal-agents)
+[![Node](https://img.shields.io/badge/node-22%2B-339933?logo=node.js&logoColor=white)](https://nodejs.org)
+[![TypeScript](https://img.shields.io/badge/typescript-strict-3178c6?logo=typescript&logoColor=white)](https://www.typescriptlang.org)
+
+Build and orchestrate AI agents on your own hardware. Multi-LLM,
+multi-channel, multi-connector — **no SaaS lock-in, no per-token markup,
+no cloud roundtrip.** Runs on any machine with Node 22+ — Mac, PC, Linux.
+
+---
+
+## Why Nodal-Agents
+
+| | |
+| --- | --- |
+| 🏠 &nbsp;**Local-first** | Single binary, embedded Postgres, zero cloud dependency. Your conversations, memory, and credentials stay on your machine. |
+| 🔌 &nbsp;**LLM-agnostic** | Anthropic, OpenAI, Google, Groq, Mistral, OpenRouter, or any OpenAI-compatible local model (LM Studio, Ollama). Per-agent key, swap providers without code changes. |
+| 🧠 &nbsp;**Memory that compounds** | Persistent facts (entity-scoped, auto-injected into every job) and chat-thread continuity (your agent remembers what it said 30 seconds ago — and what it said yesterday). |
+| 🤝 &nbsp;**Orchestrators that finish** | Router and planner orchestrators delegate to specialist sub-agents. Hard guards against runaway loops — turn caps, chain caps, per-slug delegation budgets, smart retry on side-effect-free failures. |
+| 🔧 &nbsp;**Multi-instance connectors** | Gmail perso *and* Gmail boulot on the same install. OAuth *and* API-key supported. Active list + Marketplace UI in the dashboard. |
+| 📡 &nbsp;**MCP support** | Connect Streamable HTTP MCP servers — per-job tool discovery, tool whitelisting, multi-instance. |
+| 💬 &nbsp;**Telegram out of the box** | Long-polling, multi-agent routing (`/ask <slug>`), group-chat filters, conversation continuity, delegation gracefulness on Telegram. |
+| ⚙️ &nbsp;**Real engineering** | TypeScript strict, dependency-cruiser-enforced architecture, full unit + integration suite, Playwright e2e, idempotent migrations, encryption at rest for keys. |
+
+---
+
+## Install
+
+```bash
+npm install -g nodal-agents
+nodal-agents up
+```
+
+Open <http://localhost:3000>. The CLI spawns an embedded Postgres on a
+free port, applies migrations, seeds the system skills, and starts the
+runner (`:3001`) and dashboard (`:3000`). Configure your LLM provider
+from **Settings → LLM Keys** in the dashboard.
+
+> Requires Node 22+. No external Postgres, no Redis, no cloud config.
+> Data lives at `~/.nodalai/` — wipe with `rm -rf ~/.nodalai`.
+
+To stop the stack: `nodal-agents down`.
+
+### Build from source
+
+```bash
+git clone https://github.com/Kwintspiracy/nodal-agents.git
+cd nodal-agents
+pnpm install
+pnpm --filter nodal-agents exec tsx src/index.ts --dev
+```
+
+Dev mode runs `next dev` so the dashboard hot-reloads on file changes.
+
+---
+
+## How it works
+
+```
+   ┌─────────────┐    Telegram /     ┌────────────────────────────────┐
+   │   Channel   │   Dashboard  ───▶ │         Runner (Hono)          │
+   │  (telegram, │    POST /api  ◀── │  • Job queue + executor        │
+   │   web …)    │                   │  • Anti-runaway guards          │
+   └─────────────┘                   │  • Per-agent tool whitelist     │
+                                     │  • Memory auto-injection        │
+                                     │  • Session-thread continuity    │
+                                     └─────────┬──────────┬────────────┘
+                                               │          │
+                                       ┌───────▼───┐  ┌───▼─────────────┐
+                                       │    LLM    │  │  Connectors /   │
+                                       │  client   │  │  MCP servers    │
+                                       │ (multi-   │  │  (Gmail, Drive, │
+                                       │ provider) │  │   Notion, Cogni │
+                                       └───────────┘  │   Cortex …)     │
+                                                      └─────────────────┘
+```
+
+Every agent is a row in Postgres — personality, skills, connectors,
+memory budget, team assignments live in the database. The runtime is
+generic: **zero hardcoded agent metadata.** Adding capabilities means
+inserting rows, not editing code.
+
+A user message via Telegram becomes an `agent_jobs` row. The runner
+loads the agent's prior chat-thread history, injects relevant
+persistent memories, dispatches to the LLM, executes any tool calls
+emitted, and finalizes via `telegram_send_message` + `return_result`.
+Delegations create child jobs that resume the parent on completion.
+
+---
+
+## Dashboard
+
+| Route | Purpose |
+| --- | --- |
+| `/agents` | Create, edit, assign skills + connectors + MCP servers to agents. |
+| `/jobs` | Live job stream — task, agent, status, full transcript, tool I/O. |
+| `/connectors` | Active connector instances + Marketplace (multi-instance, OAuth or API-key). |
+| `/mcp` | Active MCP servers + Marketplace (HTTP transport). |
+| `/memories` | Persistent facts per entity — search, edit, archive. |
+| `/skills` | Procedural memory blocks — shipped with the product, user-overridable. |
+| `/logs` | Tool-call audit — input/output JSON per call, filterable by tool name. |
+| `/approvals` | Human-in-the-loop gates for risky tools. |
+| `/automations` | Cron-scheduled agent triggers. |
+| `/settings` | LLM keys, auth mode, network (loopback / LAN), bot tokens. |
+
+---
+
+## Stack
+
+| Layer | Tech |
+| --- | --- |
+| Runtime | Node 22+, TypeScript strict (no `any`, no `@ts-ignore` without comment) |
+| Monorepo | pnpm workspaces + Turborepo |
+| Database | embedded-postgres (Win / Mac / Linux), Drizzle ORM, idempotent migrations |
+| Validation | Zod everywhere |
+| HTTP server | Hono (runner), Next.js 16 (dashboard) |
+| LLM | Vercel AI SDK — multi-provider with retry + timeout + tolerant parsing |
+| Auth | local-trust (single-user loopback) / better-auth (multi-user LAN) / bearer-token |
+| Encryption | AES-256-GCM at rest for API keys, master key in `~/.nodalai/secrets.key` |
+| Tests | Vitest (unit + integration), Playwright (e2e), dependency-cruiser (architecture) |
+
+---
+
+## Monorepo
+
+```
+apps/
+├── cli              nodal-agents CLI: install, up, down, ops
+├── runner           Hono server: job execution, cron, channel pollers
+└── web              Next.js dashboard
+
+packages/
+├── db               Drizzle schema + migrations + client (only postgres importer)
+├── shared           Zod types + constants shared across web + runner
+├── llm              Vercel AI SDK wrapper, retry, timeout, native tool-call parsers
+├── tools            Tool registration + execution + approval gates
+├── memory           Persistent memory CRUD + sanitation + dedup + auto-injection
+├── orchestration    Router / Planner patterns, delegation, chain counters
+├── adapters         Connector packages (gmail, drive, sheets, docs, notion,
+│                    airtable, apify, firecrawl, tavily, MCP)
+├── runner-adapters  Adapter registry, agent ↔ tool wiring
+├── delivery         Telegram, email
+├── auth             Pluggable auth provider
+├── catalog          Shipped system skills (telegram-responder, obsidian,
+│                    research-scope-discipline, claude-html-design)
+└── secrets          AES-256-GCM key vault
+```
+
+---
+
+## Architecture rules (enforced by `dependency-cruiser`)
+
+- `apps/*` may import `packages/*` — never the reverse.
+- `apps/web` and `apps/runner` cannot import each other (DB or HTTP only).
+- Only `packages/db` may import `postgres` / `drizzle-orm` / `pg`.
+- `packages/runner-adapters/*` may only import from `packages/tools` and
+  `packages/shared`.
+- No circular dependencies.
+
+```bash
+pnpm deps:check   # runs locally and in CI before every release
+```
+
+---
+
+## Status
+
+**Current release:** `0.1.5` on npm `latest`. Used daily by the
+maintainer, stable enough for personal production. Pre-1.0 — breaking
+changes are still possible between minors.
+
+### Shipped and working
+
+- Multi-LLM provider routing with per-agent keys
+- Persistent memory (sanitation, dedup, importance ranking, auto-injection,
+  feedback loop)
+- Session-thread continuity on chat channels (Telegram today)
+- Orchestrator (router + planner) with delegation chains and anti-runaway
+  caps
+- Multi-instance connectors with OAuth (Gmail, Drive, Sheets, Docs, Notion,
+  Airtable) and API-key (Notion, Airtable, Apify, Firecrawl, Tavily)
+- MCP Streamable HTTP catalog with API-key auth
+- Telegram delivery (long-poll, group filters, multi-agent routing,
+  delegation gracefulness)
+- Approval gates for risky tools
+- Cron scheduling
+- Encryption at rest for LLM keys + MCP keys
+- Embedded Postgres distribution via npm (no external DB to install)
+
+### On the roadmap (genuine, not vaporware)
+
+- **MCP OAuth flow** → unlocks Linear, Notion remote, GitHub remote,
+  Atlassian, Sentry, and the rest of the SaaS-as-MCP ecosystem.
+- **MCP stdio transport** → unlocks the Anthropic reference servers
+  (filesystem, sqlite, brave-search, GitHub stdio…) and the community
+  catalog.
+- **pgvector binaries bundled in the npm pack** → semantic memory search
+  active out-of-the-box. Today, installs without pgvector fall back to
+  keyword search (which works, just less smart for cross-vocabulary
+  recall).
+- Data migration tools from the legacy Python stack.
+
+---
+
+## License
+
+TBD.

package/cli.js

@@ -12990,7 +12990,7 @@ var init_mcp = __esm({
         check("mcp_servers_transport_check", sql`${table.transport} IN ('http','stdio')`),
         check(
           "mcp_servers_auth_scheme_check",
-          sql`${table.authScheme} IN ('header','query') OR ${table.authScheme} IS NULL`
+          sql`${table.authScheme} IN ('header','query','bearer') OR ${table.authScheme} IS NULL`
         )
         // Multi-instance brique (migration 0017): the (entity_id, slug) UNIQUE
         // index was dropped to allow multiple instances of the same MCP server
@@ -13464,10 +13464,11 @@ function buildEnvForWeb(config, databaseUrl) {
     // readConfig() guarantees this is set (auto-mints on first read).
     NEXT_SERVER_ACTIONS_ENCRYPTION_KEY: config.serverActionsKey ?? "",
     // The Next.js standalone server reads HOSTNAME for its listener bind
-    // (defaults to '0.0.0.0' when unset). Docker sets HOSTNAME to the
-    // container ID, which Next interprets as the hostname to bind on —
-    // resolving via /etc/hosts to 127.0.0.1 only, breaking port forwarding
-    // and the `localhost:3000` healthcheck. Pin to the loopback/lan choice.
+    // (defaults to '0.0.0.0' when unset). Some host environments set
+    // HOSTNAME to an arbitrary identifier (e.g. a runner ID), which Next
+    // would interpret as the hostname to bind on — usually resolving via
+    // /etc/hosts to 127.0.0.1 only, breaking port forwarding and the
+    // `localhost:3000` healthcheck. Pin to the loopback/lan choice.
     HOSTNAME: bind
   };
   if (config.llm) {

package/migrations/0018_mcp_servers_bearer_auth.sql

@@ -0,0 +1,12 @@
+-- Migration 0018 — relax mcp_servers.auth_scheme to allow 'bearer'.
+--
+-- Brique A v3 (MCP catalog enrichment): adds the 'bearer' auth scheme so
+-- catalog entries for servers expecting `Authorization: Bearer <token>`
+-- (Stripe, OpenAI, etc.) can be represented and connected without a
+-- bespoke per-server hack. The adapter injects the literal "Bearer "
+-- prefix when this scheme is selected; authParamName is ignored.
+
+ALTER TABLE "mcp_servers" DROP CONSTRAINT IF EXISTS "mcp_servers_auth_scheme_check";
+--> statement-breakpoint
+ALTER TABLE "mcp_servers" ADD CONSTRAINT "mcp_servers_auth_scheme_check"
+  CHECK ("auth_scheme" IN ('header','query','bearer') OR "auth_scheme" IS NULL);

package/migrations/meta/_journal.json

@@ -1,132 +1,139 @@
-{
-  "version": "7",
-  "dialect": "postgresql",
-  "entries": [
-    {
-      "idx": 0,
-      "version": "7",
-      "when": 1777221376983,
-      "tag": "0000_flashy_clea",
-      "breakpoints": true
-    },
-    {
-      "idx": 1,
-      "version": "7",
-      "when": 1777222639218,
-      "tag": "0001_exotic_shooting_star",
-      "breakpoints": true
-    },
-    {
-      "idx": 2,
-      "version": "7",
-      "when": 1777222668100,
-      "tag": "0002_gray_devos",
-      "breakpoints": true
-    },
-    {
-      "idx": 3,
-      "version": "7",
-      "when": 1778116559502,
-      "tag": "0003_stormy_thaddeus_ross",
-      "breakpoints": true
-    },
-    {
-      "idx": 4,
-      "version": "7",
-      "when": 1778141889404,
-      "tag": "0004_married_deadpool",
-      "breakpoints": true
-    },
-    {
-      "idx": 5,
-      "version": "7",
-      "when": 1778237574443,
-      "tag": "0005_tense_hedge_knight",
-      "breakpoints": true
-    },
-    {
-      "idx": 6,
-      "version": "7",
-      "when": 1778341607214,
-      "tag": "0006_credentials_table",
-      "breakpoints": true
-    },
-    {
-      "idx": 7,
-      "version": "7",
-      "when": 1778421836079,
-      "tag": "0007_free_speed",
-      "breakpoints": true
-    },
-    {
-      "idx": 8,
-      "version": "7",
-      "when": 1778459305708,
-      "tag": "0008_curious_thor_girl",
-      "breakpoints": true
-    },
-    {
-      "idx": 9,
-      "version": "7",
-      "when": 1778860776473,
-      "tag": "0009_agent_memory_token_budget",
-      "breakpoints": true
-    },
-    {
-      "idx": 10,
-      "version": "7",
-      "when": 1778940762250,
-      "tag": "0010_agent_workspace_root",
-      "breakpoints": true
-    },
-    {
-      "idx": 11,
-      "version": "7",
-      "when": 1779000000000,
-      "tag": "0011_agent_jobs_failed_delegations_count",
-      "breakpoints": true
-    },
-    {
-      "idx": 12,
-      "version": "7",
-      "when": 1779100000000,
-      "tag": "0012_agent_jobs_last_failed_delegation_slug",
-      "breakpoints": true
-    },
-    {
-      "idx": 13,
-      "version": "7",
-      "when": 1779200000000,
-      "tag": "0013_cascade_agent_fks",
-      "breakpoints": true
-    },
-    {
-      "idx": 14,
-      "version": "7",
-      "when": 1779400000000,
-      "tag": "0014_mcp_server_credentials",
-      "breakpoints": true
-    },
-    {
-      "idx": 15,
-      "version": "7",
-      "when": 1779500000000,
-      "tag": "0015_agent_mcp_servers_unique",
-      "breakpoints": true
-    },
-    {
-      "idx": 16,
-      "version": "7",
-      "when": 1779600000000,
-      "tag": "0016_drop_connector_entity_slug_unique",
-      "breakpoints": true
-    },
-    {
-      "idx": 17,
-      "version": "7",
-      "when": 1779700000000,
-      "tag": "0017_drop_mcp_server_entity_slug_unique",
-      "breakpoints": true
-    }
-  ]
-}
+{
+  "version": "7",
+  "dialect": "postgresql",
+  "entries": [
+    {
+      "idx": 0,
+      "version": "7",
+      "when": 1777221376983,
+      "tag": "0000_flashy_clea",
+      "breakpoints": true
+    },
+    {
+      "idx": 1,
+      "version": "7",
+      "when": 1777222639218,
+      "tag": "0001_exotic_shooting_star",
+      "breakpoints": true
+    },
+    {
+      "idx": 2,
+      "version": "7",
+      "when": 1777222668100,
+      "tag": "0002_gray_devos",
+      "breakpoints": true
+    },
+    {
+      "idx": 3,
+      "version": "7",
+      "when": 1778116559502,
+      "tag": "0003_stormy_thaddeus_ross",
+      "breakpoints": true
+    },
+    {
+      "idx": 4,
+      "version": "7",
+      "when": 1778141889404,
+      "tag": "0004_married_deadpool",
+      "breakpoints": true
+    },
+    {
+      "idx": 5,
+      "version": "7",
+      "when": 1778237574443,
+      "tag": "0005_tense_hedge_knight",
+      "breakpoints": true
+    },
+    {
+      "idx": 6,
+      "version": "7",
+      "when": 1778341607214,
+      "tag": "0006_credentials_table",
+      "breakpoints": true
+    },
+    {
+      "idx": 7,
+      "version": "7",
+      "when": 1778421836079,
+      "tag": "0007_free_speed",
+      "breakpoints": true
+    },
+    {
+      "idx": 8,
+      "version": "7",
+      "when": 1778459305708,
+      "tag": "0008_curious_thor_girl",
+      "breakpoints": true
+    },
+    {
+      "idx": 9,
+      "version": "7",
+      "when": 1778860776473,
+      "tag": "0009_agent_memory_token_budget",
+      "breakpoints": true
+    },
+    {
+      "idx": 10,
+      "version": "7",
+      "when": 1778940762250,
+      "tag": "0010_agent_workspace_root",
+      "breakpoints": true
+    },
+    {
+      "idx": 11,
+      "version": "7",
+      "when": 1779000000000,
+      "tag": "0011_agent_jobs_failed_delegations_count",
+      "breakpoints": true
+    },
+    {
+      "idx": 12,
+      "version": "7",
+      "when": 1779100000000,
+      "tag": "0012_agent_jobs_last_failed_delegation_slug",
+      "breakpoints": true
+    },
+    {
+      "idx": 13,
+      "version": "7",
+      "when": 1779200000000,
+      "tag": "0013_cascade_agent_fks",
+      "breakpoints": true
+    },
+    {
+      "idx": 14,
+      "version": "7",
+      "when": 1779400000000,
+      "tag": "0014_mcp_server_credentials",
+      "breakpoints": true
+    },
+    {
+      "idx": 15,
+      "version": "7",
+      "when": 1779500000000,
+      "tag": "0015_agent_mcp_servers_unique",
+      "breakpoints": true
+    },
+    {
+      "idx": 16,
+      "version": "7",
+      "when": 1779600000000,
+      "tag": "0016_drop_connector_entity_slug_unique",
+      "breakpoints": true
+    },
+    {
+      "idx": 17,
+      "version": "7",
+      "when": 1779700000000,
+      "tag": "0017_drop_mcp_server_entity_slug_unique",
+      "breakpoints": true
+    },
+    {
+      "idx": 18,
+      "version": "7",
+      "when": 1779800000000,
+      "tag": "0018_mcp_servers_bearer_auth",
+      "breakpoints": true
+    }
+  ]
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "nodal-agents",
-  "version": "0.1.4",
+  "version": "0.1.6",
   "description": "Local-first AI agent platform — self-host on Mac, PC, Linux, VPS, or NAS.",
   "license": "MIT",
   "repository": {
@@ -52,7 +52,6 @@
     "@mendable/firecrawl-js": "^4.22.0",
     "@tavily/core": "^0.7.0",
     "apify-client": "^2.23.0",
-    "exceljs": "^4.4.0",
     "pdf-parse": "^2.4.5",
     "mammoth": "^1.9.0",
     "@napi-rs/canvas": "^0.1.80",