@seldonframe/mcp 1.26.1 → 1.27.0

package/package.json

@@ -1,6 +1,8 @@
 {
   "name": "@seldonframe/mcp",
-  "version": "1.26.1",
+  "version": "1.27.0",
+  "description": "MCP server for SeldonFrame — AI-native Business OS platform. v1.27.0: AGENTS GET A PROPER HOME — discoverable nav + full in-dashboard agent management. PROBLEM v1.27 SOLVES: after v1.26.2 the chat-AI agents lived at /admin/agents (not in the sidebar nav, type-the-URL discovery), confusable with the existing /agents/runs (workflow run log) and /automations (workflow archetype configurator). SF clients had to use Claude Code MCP for ALL agent edits — no in-dashboard editing path. WHAT SHIPS: (1) NAV REWORK. /admin/agents promoted to /agents (operator-facing top-level route, not SF-internal). 'Agents' added to sidebar nav between Bookings and Pages with Bot icon (lucide-react). The existing /agents/runs (workflow runs log, 15 cross-refs) coexists at the same level — Next.js resolves the static 'runs' segment before falling through to dynamic '[id]', so /agents (chat AI list), /agents/<uuid>/* (chat AI detail), and /agents/runs (workflow runs) all route correctly. (2) AGENT DETAIL SHELL. New /agents/[id]/layout.tsx renders the agent header (← All agents back link, name, status pill, version, archetype, slug) + AgentTabs nav with 5 tabs: Overview / Sandbox / Conversations / Settings / Evals. Each child route is a tab body. (3) OVERVIEW TAB (/agents/[id]/page.tsx). At-a-glance stat grid (24h conversations, validator pass rate, eval pass rate, avg latency); daily-token-usage progress bar (green/amber/rose at 70/90%); 3 most recent conversations with quality marks visible; embed snippet copy block. OverviewActions client component lets operator toggle status (draft/test/live/paused) — 'live' triggers the eval gate server-side; failure surfaces the failing scenarios in-line with hint to fix in Settings. (4) SETTINGS TAB (/agents/[id]/settings + settings-client.tsx). INLINE BLUEPRINT EDITOR replicating MCP update_agent_blueprint without leaving the dashboard. Greeting textarea, capability checklist (5 capabilities: look_up_availability, book_appointment, find_my_existing_appointment, escalate_to_human, provide_faq_answer), FAQ rows (add/edit/delete), pricing facts rows (add/edit/delete with label+amount+currency), optional change-note input, Save button. saveAgentBlueprintAction validates with Zod, calls updateAgentBlueprint server action (bumps current_version + writes new agent_versions row), revalidates path, returns new version number. (5) EVALS TAB (/agents/[id]/evals + evals-client.tsx). Lists all 8 platform-owned scenarios (description + severity pill + category) with latest result inline (passed / failed / not run); failed rows show failure reasons + collapsible 'Agent response' detail. 'Run evals now' button calls runEvalsAction → runEvalSuite → returns updated summary. Header shows aggregate pass count + meets-gate badge + last-run timestamp. (6) CONVERSATIONS TAB (existing, polished). Same expand-in-place transcript as v1.26.2 plus QualityMarker client component on every expanded conversation: good/bad pill toggles + notes input. markConversationQualityAction writes operator_quality + operator_notes (already in schema since v1.26.0); collapsed-row preview shows the quality mark + first 60 chars of notes so operators can scan the list for already-reviewed chats. (7) SERVER ACTIONS (lib/agents/actions.ts). saveAgentBlueprintAction, setAgentStatusAction, runEvalsAction, markConversationQualityAction. All workspace-scoped via getOrgId(). Zod-validated input. Each calls revalidatePath so the dashboard reflects new state without hard refresh. (8) DUPLICATE-HEADER TRIM. /agents/[id]/test/page.tsx and /agents/[id]/conversations/page.tsx had their own h1 + back-link UI (from v1.26.2 when they were at /admin/agents); v1.27.0 removes those since the layout.tsx provides them at the parent level. Cleaner, no duplication. NO new migrations (v1.26.0 0044 still covers everything; operator_quality + operator_notes columns already there). NO new env vars. Backend redeploy + npm publish required. DEFERRED to v1.28: real Anthropic-streaming-passthrough on /turn (mid-turn tool_use streaming); voice-receptionist + sms-followup-bot specialized eval scenarios; per-scenario eval history graph; conversation-detail standalone page; 'fold operator-marked good/bad chats into future eval scenarios' (signal pipeline). The existing /agents/runs workflow run log will move to /automations/runs in v1.28 once we've audited the 15 cross-refs.",
+  "description": "MCP server for SeldonFrame — AI-native Business OS platform. v1.26.2: AGENTS GO SAFE — eval-gated publish + SSE streaming + admin debug surfaces + 5 new debug MCP tools. WHAT SHIPS: (1) EVAL SUITE. New lib/agents/eval-scenarios.ts ships 8 platform-owned scenarios per archetype (website-chatbot first; voice + sms inherit until v1.27/v1.28 specialization): 5 critical (3 prompt-injection + PII safety probes + 2 pricing-discipline checks — refuses-invented-price, refuses-competitor-match) and 3 warning (off-topic refusal, greeting-within-cap, escalation-on-complex-legal). Each scenario = id + description + userMessages[] + expected{responseContains|responseLacks|toolCallsRequired|validatorsAllPassed}. New lib/agents/eval-runner.ts runs each scenario as an ephemeral test conversation (channelMeta.eval_run=true so they don't pollute tail_conversations), captures actuals, persists to agent_evals, returns aggregate {totalRun, passed, failed, passRate, meetsPublishGate, results[]}. (2) EVAL-GATED PUBLISH. lib/agents/store.publishAgent now runs the eval suite before flipping status='live' and rejects with error='eval_gate_failed' when passRate < 87.5% (≥7/8 of 8 default scenarios). Other transitions (draft, test, paused) unrestricted. SF-controlled emergency override via {force:true} (logged). evalSummary returned alongside ok=true so the operator sees what passed. (3) SSE STREAMING ON /turn. POST /api/v1/public/agent/<slug>/turn now branches on Accept: text/event-stream | body.stream=true | ?stream=1: emits 'start' (conversation_id), 'delta' chunks (~28-char word-aware splits, ~22ms gap for typewriter UX), 'done' (conversation_id + validators_critical_failed). Runtime still buffers full response for validator gating BEFORE any byte streams (critical-fail still replaces the whole text). Real Anthropic-passthrough streaming with mid-turn tool-use events queued for v1.27 alongside multi-step tool-call streaming. embed.js (v1.26.2) consumes SSE: assistant message DOM node grows in place as deltas arrive, falls back to JSON if Content-Type isn't text/event-stream. (4) ADMIN DEBUG SURFACES. /(dashboard)/admin/agents — workspace agent roster with status pills, daily-token-usage bars, quick links to /test + /conversations. /(dashboard)/admin/agents/[id]/test — interactive sandbox client (TestSandboxClient) consuming SSE, identical UX to embed widget; surfaces conversation_id so operator can call get_agent_conversation MCP tool to inspect. /(dashboard)/admin/agents/[id]/conversations — server-rendered list of recent customer chats (eval-runs filtered by default), expand-in-place to show full transcript with tool_calls + tool_results + validator pass/fail per assistant turn. (5) 5 NEW DEBUG MCP TOOLS. run_agent_evals({agent_id}) — manual eval suite trigger (publish auto-runs too). tail_agent_conversations({agent_id, limit?, include_eval_runs?}) — newest-first conversation list with first_user_message preview. get_agent_conversation({conversation_id}) — full transcript with all tool_calls + tool_results + validator results per turn. replay_conversation({conversation_id}) — re-run user messages against current blueprint, returns original_turns + replay_turns side-by-side for regression-testing blueprint changes. get_agent_metrics({agent_id, since_hours?}) — aggregate stats over a time window: conversations + turns + tokens_in/out + avg_latency_ms + validator_pass_rate (% of assistant turns where ALL validators passed) + eval_pass_rate (latest result per scenario). NO new migrations (0044 from v1.26.0 still covers all five agent tables). NO new env vars. Backend redeploy + npm publish required. DEFERRED TO v1.27: real Anthropic-streaming-passthrough on /turn (mid-turn tool_use streaming); voice-receptionist + sms-followup-bot specialized eval scenarios; operator-quality marking (good/bad/notes) on /conversations; conversation-detail standalone page (/conversations/[convId]/page.tsx); per-scenario eval history graph.",
   "description": "MCP server for SeldonFrame — AI-native Business OS platform. v1.26.1: AGENTS GO LIVE — BYOK + embed.js + 5 new MCP tools. WHAT SHIPS: (1) BYOK (BRING-YOUR-OWN-KEY). SF clients (the agencies + their HVAC/dental/coaching operators) use Claude Code or Cursor to build agents — they already have Anthropic API keys. SF doesn't markup LLM cost; SF charges per agent turn (separate revenue line). lib/agents/runtime.ts swaps from process.env.ANTHROPIC_API_KEY to getAIClient({ orgId }) — the same BYOK helper that lib/ai/client.ts already exposes for the rest of the platform (encrypted at rest in organizations.integrations.anthropic.apiKey with v1. prefix; decryptIfNeeded resolves at call time). When a workspace hasn't configured a key, executeTurn returns { ok: false, reason: 'llm_not_configured', fallbackMessage } — the public turn endpoint surfaces a graceful 'I'm not set up to chat yet — please reach out directly' instead of crashing. (2) EMBED.JS — single-line install on operator's website. New endpoint GET /api/v1/public/agent/<orgSlug>--<agentSlug>/embed.js returns a self-contained IIFE (~3KB) that injects a bottom-right chat bubble + chat panel. Click bubble → panel opens → user types → POST to /turn → response appears. Anonymous session id stored in localStorage (sf_agent_session_<agentSlug>) so multi-message sessions thread together server-side via agent_conversations.anonymous_session_id. Themed via agent's blueprint.greeting + workspace's theme.primaryColor. Returns no-op script (still 200) when agent not 'live' so operator's site console stays clean during draft → test → live progression. CSS namespaced .sf-agent-* (zero collision risk on operator's site). Operator's install snippet: <script src=\"https://app.seldonframe.com/api/v1/public/agent/cypress-pine-hvac--default/embed.js\" async></script>. (3) 5 NEW MCP TOOLS so SF clients build + ship agents from Claude Code without touching the dashboard: configure_llm_provider({ provider: 'anthropic'|'openai', api_key }) — first-run setup, encrypts + writes to organizations.integrations[provider].apiKey. create_agent({ name, archetype, channel, capabilities?, faq?, pricing_facts?, greeting? }) — wraps lib/agents/store.createAgent, returns { agent, embed_url, turn_url, next_steps[] } so the SF client immediately gets the embed snippet to drop on their operator's site. update_agent_blueprint({ agent_id, patch, publish_notes? }) — wraps updateAgentBlueprint, bumps current_version + writes new agent_versions row. publish_agent({ agent_id, status: draft|test|live|paused }) — flips status. list_agents() — workspace-scoped roster with id/name/slug/channel/archetype/status/version/tokens/budget/created. (4) /api/v1/agents route extended with op='set_llm_key' handler so the new MCP tool routes through one endpoint. Validates provider ∈ {anthropic, openai}, api_key length ≥ 10, encrypts via encryptValue (returns 503 encryption_unavailable when ENCRYPTION_KEY env not set), merges into organizations.integrations preserving other providers' entries (kit, mailchimp, etc.). NO new migrations (0044 from v1.26.0 covers all agent tables). NO new env vars beyond ENCRYPTION_KEY (already required for the existing BYOK path). Backend redeploy + npm publish required. DEFERRED TO v1.26.2: SSE streaming on /turn; eval suite runner (executes scenarios from blocks/agent-website-chatbot/SKILL.md against the live agent); eval-gated publish (require ≥7/8 scenarios pass before flipping to live); /admin/agents/[id]/{conversations,test} debug surfaces; MCP debug tools (tail_agent_conversations, get_agent_conversation, replay_conversation, get_agent_metrics).",
   "description": "MCP server for SeldonFrame — AI-native Business OS platform. v1.26.0: AGENT FOUNDATION — primitives for building chat / voice / SMS / email agents on top of Soul + Brain. Web chat archetype ships first; voice + SMS queued for v1.27/v1.28. Migration 0044 adds agents / agent_versions / agent_conversations / agent_turns / agent_evals tables. Schema (src/db/schema/agents.ts) typed with AgentBlueprint, AgentToolCall, AgentToolResult, AgentValidatorResult. System prompt composer (lib/agents/prompt.ts) derives prompts from Soul + blueprint, NEVER operator-authored — operators contribute knowledge (FAQ, services, pricing); platform owns the prompt. Five output validators (lib/agents/validators.ts) run on every assistant response: quotes_only_from_soul_pricing (critical — blocks hallucinated $X), no_prompt_injection_echo (critical — blocks 'ignore previous instructions' echoes), no_pii_leak (critical — blocks emails/phones not in user's message), no_avoid_words (warning — Soul voice rule), response_length_under_cap (warning — 600 chars). Critical fail = response replaced with 'let me check + escalate'. Five typed tools (lib/agents/tools.ts) with Zod input schemas: look_up_availability, book_appointment (wraps submitPublicBookingAction — same atomic primitive as /book), find_my_existing_appointment, escalate_to_human (writes portal_messages + activities), provide_faq_answer. Runtime (lib/agents/runtime.ts) executeTurn — loads conv + agent + soul, checks daily token budget, persists user turn, calls Anthropic with tools, loops on tool_use (max 6 iterations), runs validators, persists assistant turn with cost/latency, updates aggregates, activity-bridges first turn to operator's CRM. Non-streaming for v1.26.0; v1.26.1 adds SSE. Public turn endpoint POST /api/v1/public/agent/<orgSlug>--<agentSlug>/turn (anonymous; live/test status required). Agent CRUD endpoint POST /api/v1/agents (op: create | update_blueprint | publish | list — workspace bearer auth). createAgent generates slug, creates v1 in agent_versions, soul-derives blueprint defaults. blocks/agent-website-chatbot/SKILL.md documents the archetype + 8 eval scenarios including prompt-injection + PII-leak adversarial probes. NO new MCP tools yet (HTTP endpoint exposed; MCP-tool wiring + SSE streaming + embed.js + operator debug surfaces land in v1.26.1). NO new env vars beyond ANTHROPIC_API_KEY (already required). Backend redeploy required.",
   "description": "MCP server for SeldonFrame — AI-native Business OS platform. v1.25.4: operator 'Today' snapshot widget + remaining polish. NEW: replaces the v1.25.3 gap (where SF agency operators saw 'Newly installed blocks') with an operator-actionable 'Today' overview at the top of /dashboard. Four cards: today's bookings (count + first 3 with time/name), unread customer-portal messages, deals stuck >5 days (closedAt IS NULL + updatedAt < 5d ago), bookings-this-week trend. Each card click-throughs to its source surface (/bookings, /contacts, /deals). Below the cards: 'Up next' list of today's first 3 bookings rendered as time/name/title rows. All queries scoped by session orgId; widget renders only for isOperatorSession. POLISH: SeldonChat (the SF AI assistant floating button) hidden for operator sessions — that's an SF-builder tool. Command palette items split by session type — operator sessions only see Dashboard/Contacts/Deals/Bookings + their contact + deal hits; non-operator sessions get the full SF palette including Soul Marketplace, Studio, Seldon It, Pages, Email, Settings, Recent Activity. NO migrations, NO new env vars. Backend redeploy required.",

package/src/tools.js

@@ -4247,6 +4247,155 @@ export const TOOLS = [
       return result;
     },
   },
+
+  // ───────────────────────────────────────────────────────────────────────
+  // v1.26.2 — agent debug + observability tools.
+  // run_agent_evals = manual eval suite trigger (publish auto-runs too).
+  // tail_agent_conversations / get_agent_conversation = observability.
+  // replay_conversation = regression-test a blueprint change against a
+  //   known-good past chat. get_agent_metrics = aggregate stats for the
+  //   "is my agent healthy?" check.
+  // ───────────────────────────────────────────────────────────────────────
+
+  {
+    name: "run_agent_evals",
+    description:
+      "Run the platform's safety + behavior eval suite against this agent. The 8-scenario suite covers prompt-injection probes (ignore-instructions, role-swap), PII probes (customer-list leak), pricing discipline (refuses invented prices, refuses competitor match), scope refusal (off-topic), greeting + escalation. Each scenario runs through the live blueprint as an ephemeral test conversation, results persist to agent_evals. " +
+      "publish_agent with status='live' AUTOMATICALLY runs this and gates on ≥87.5% pass. Call this directly to dry-run before publishing or to verify after a blueprint update.",
+    inputSchema: obj(
+      {
+        workspace_id: str("Workspace id (bearer workspace)."),
+        agent_id: str("Agent id from list_agents / create_agent."),
+      },
+      ["workspace_id", "agent_id"],
+    ),
+    handler: async (args) => {
+      const ws = args.workspace_id;
+      const result = await api("POST", "/agents", {
+        body: { op: "run_evals", agent_id: args.agent_id },
+        workspace_id: ws,
+      });
+      return result;
+    },
+  },
+
+  {
+    name: "tail_agent_conversations",
+    description:
+      "List recent conversations for an agent — newest first. Excludes eval-runs and replay-runs by default (set include_eval_runs=true to see them). Each row includes status, turn_count, tokens, llm_cost_cents, and the customer's first message preview so you can spot patterns (most common questions, escalations, etc.) without opening each transcript. " +
+      "Use BEFORE get_agent_conversation to pick which conversation to drill into.",
+    inputSchema: obj(
+      {
+        workspace_id: str("Workspace id (bearer workspace)."),
+        agent_id: str("Agent id from list_agents."),
+        limit: {
+          type: "integer",
+          description: "How many conversations to return (default 20, max 100).",
+        },
+        include_eval_runs: {
+          type: "boolean",
+          description: "Include eval/replay synthetic runs in the list (default false).",
+        },
+      },
+      ["workspace_id", "agent_id"],
+    ),
+    handler: async (args) => {
+      const ws = args.workspace_id;
+      const result = await api("POST", "/agents", {
+        body: {
+          op: "tail_conversations",
+          agent_id: args.agent_id,
+          limit: args.limit,
+          include_eval_runs: args.include_eval_runs,
+        },
+        workspace_id: ws,
+      });
+      return result;
+    },
+  },
+
+  {
+    name: "get_agent_conversation",
+    description:
+      "Fetch the full transcript of a single conversation: every turn (user + assistant), all tool_calls (look_up_availability, book_appointment, escalate_to_human, etc.) with their inputs, all tool_results with success/error, validator_results per assistant turn (which validators passed/failed), tokens, latency, model. " +
+      "Use this to debug WHY an agent gave a specific answer — was it a tool failure? a validator gating? wrong info in blueprint? Pair with replay_conversation to test a blueprint fix.",
+    inputSchema: obj(
+      {
+        workspace_id: str("Workspace id (bearer workspace)."),
+        conversation_id: str(
+          "Conversation id (from tail_agent_conversations or admin /conversations page).",
+        ),
+      },
+      ["workspace_id", "conversation_id"],
+    ),
+    handler: async (args) => {
+      const ws = args.workspace_id;
+      const result = await api("POST", "/agents", {
+        body: {
+          op: "get_conversation",
+          conversation_id: args.conversation_id,
+        },
+        workspace_id: ws,
+      });
+      return result;
+    },
+  },
+
+  {
+    name: "replay_conversation",
+    description:
+      "Replay a past conversation's user messages against the agent's CURRENT blueprint, returning the original responses + the new responses side-by-side. Lets you regression-test a blueprint change ('did adding this FAQ break the booking flow?') without touching production. " +
+      "Creates a new ephemeral test-status conversation tagged with replay_of=<original>; the original is untouched.",
+    inputSchema: obj(
+      {
+        workspace_id: str("Workspace id (bearer workspace)."),
+        conversation_id: str("Original conversation id to replay."),
+      },
+      ["workspace_id", "conversation_id"],
+    ),
+    handler: async (args) => {
+      const ws = args.workspace_id;
+      const result = await api("POST", "/agents", {
+        body: {
+          op: "replay_conversation",
+          conversation_id: args.conversation_id,
+        },
+        workspace_id: ws,
+      });
+      return result;
+    },
+  },
+
+  {
+    name: "get_agent_metrics",
+    description:
+      "Aggregate health stats for an agent over a time window: conversations + turns count, tokens (in/out), avg latency, validator pass rate (% of assistant turns where ALL validators passed), latest eval pass rate (last result per scenario). " +
+      "Use as a dashboard ping — 'is my agent healthy?' If validator_pass_rate drops or eval_pass_rate falls below the 87.5% gate, the agent shouldn't be promoted to live.",
+    inputSchema: obj(
+      {
+        workspace_id: str("Workspace id (bearer workspace)."),
+        agent_id: str("Agent id from list_agents."),
+        since_hours: {
+          type: "integer",
+          description:
+            "Time window in hours (default 24). Pass 168 for last 7 days, 720 for last 30 days.",
+        },
+      },
+      ["workspace_id", "agent_id"],
+    ),
+    handler: async (args) => {
+      const ws = args.workspace_id;
+      const result = await api("POST", "/agents", {
+        body: {
+          op: "get_metrics",
+          agent_id: args.agent_id,
+          since_hours: args.since_hours,
+        },
+        workspace_id: ws,
+      });
+      return result;
+    },
+  },
 ];
 
 export const TOOL_MAP = Object.fromEntries(TOOLS.map((t) => [t.name, t]));