@ema.co/mcp-toolkit 2026.1.25 → 2026.1.26-4

package/docs/proposals/self-contained-guidance.md

@@ -1,427 +0,0 @@
-# Self-Contained MCP Guidance System
-
-> **ID**: PROP-2026-01-21-guidance  
-> **Status**: ✅ Complete  
-> **Created**: 2026-01-21  
-> **Completed**: 2026-01-21  
-> **Author**: Architecture Review  
-> **Implemented-by**: feat/agent-coordination branch
-
-## Implementation Tracking
-
-| Step | Status | Commit | Test | Notes |
-|------|--------|--------|------|-------|
-| Step 1: guidance.ts | ✅ Done | `pending` | `test/guidance.test.ts` | Centralized guidance content |
-| Step 2: Server instructions | ✅ Done | `pending` | - | Add to init response |
-| Step 3: Tool descriptions | ✅ Done | `pending` | - | Import TOOL_GUIDANCE into tools-v2.ts |
-| Step 4: Response tips | ✅ Done | `pending` | - | _tip + _next_step in responses |
-| Step 5: Session onboarding | ✅ Done | `pending` | - | env handler getting_started |
-| Step 6: Workflow guidance | ✅ Done | `pending` | - | analyze workflow_guidance |
-| Step 7: Self-ref prompts | ✅ Done | `pending` | - | toolkit_onboard → usage-guide |
-| Step 8: Usage guide resource | ✅ Done | `pending` | - | ema://docs/usage-guide + 3 more |
-| Step 9: Remove externals | ✅ Done | `pending` | - | Legacy redirect + comment cleanup |
-| Step 10: Documentation | ✅ Done | `pending` | - | README + mcp-tools-guide |
-
-## Change Log
-
-| Date | Change | By |
-|------|--------|-----|
-| 2026-01-21 | Created proposal | Architecture Review |
-| 2026-01-21 | Added implementation tracking | sp-cursor |
-| 2026-01-21 | All 10 steps implemented | sp-cursor |
-
----
-
-## Problem Statement
-
-External customers installing only `npx @ema.co/mcp-toolkit@latest` don't have access to:
-- `.cursor/rules/` files (50+ rules)
-- `.cursor/agents/` definitions
-- `AGENTS.md` / `CLAUDE.md` guidance
-
-This means they start with zero context about Ema patterns, leading to:
-- Misuse of tools (wrong order, missing steps)
-- Anti-patterns (no Fallback, skipping analysis)
-- Inconsistent experiences across different IDEs
-
-## Design Principles
-
-1. **IDE Agnostic**: Works identically in Cursor, Claude Desktop, any MCP client
-2. **Progressive Disclosure**: Start minimal, provide more on request
-3. **Context-Aware**: Different guidance for different operations
-4. **Single Source of Truth**: All guidance lives in MCP package, not external files
-5. **Fail-Safe**: Even if guidance is ignored, tool responses guide correct usage
-
-## Architecture
-
-```
-┌─────────────────────────────────────────────────────────┐
-│                    MCP Toolkit                          │
-├─────────────────────────────────────────────────────────┤
-│  Layer 1: Server Instructions (on init)                 │
-│  - Core workflow patterns                               │
-│  - Essential rules (~300 tokens)                        │
-├─────────────────────────────────────────────────────────┤
-│  Layer 2: Tool Descriptions (on tools/list)             │
-│  - Per-tool usage patterns                              │
-│  - Common mistakes to avoid                             │
-├─────────────────────────────────────────────────────────┤
-│  Layer 3: Contextual Tips (in tool responses)           │
-│  - Next-step suggestions                                │
-│  - Warning when doing something risky                   │
-├─────────────────────────────────────────────────────────┤
-│  Layer 4: Prompts (on explicit request)                 │
-│  - Detailed workflows                                   │
-│  - Step-by-step guides                                  │
-├─────────────────────────────────────────────────────────┤
-│  Layer 5: Resources (on explicit request)               │
-│  - Reference data (catalogs, schemas)                   │
-│  - Templates                                            │
-└─────────────────────────────────────────────────────────┘
-```
-
-## Implementation Steps
-
-### Step 1: Create Guidance Content Module
-
-**Files:** `src/sdk/guidance.ts`
-
-Create a single source of truth for all guidance content:
-
-```typescript
-// src/sdk/guidance.ts
-
-export const SERVER_INSTRUCTIONS = `
-# Ema MCP Toolkit
-
-## Workflow Pattern
-1. **Discover**: persona() to list AI Employees
-2. **Understand**: persona(id="...", analyze=true) before any changes
-3. **Preview**: Always use preview=true before deploying
-4. **Deploy**: Only after reviewing preview
-
-## Critical Rules
-- NEVER modify a workflow without analyzing first
-- ALWAYS include Fallback in categorizers  
-- Use workflow_spec for modifications, not raw workflow_def
-
-## When Stuck
-Use the toolkit_onboard prompt for comprehensive guidance.
-`;
-
-export const TOOL_GUIDANCE: Record<string, ToolGuidance> = {
-  persona: {
-    quickTip: "Use analyze=true to understand before modifying",
-    commonMistakes: ["Modifying without analysis", "Missing preview step"],
-    nextSteps: {
-      list: "Get details with persona(id='...')",
-      get: "Analyze with persona(id='...', analyze=true)",
-      analyze: "Build workflow_spec from issues, then update",
-    }
-  },
-  // ... other tools
-};
-
-export const CONTEXTUAL_WARNINGS: Record<string, string> = {
-  update_without_analysis: "⚠️ Consider running analyze first",
-  deploy_without_preview: "⚠️ Use preview=true to review changes first",
-};
-```
-
-**Checkpoint:** `feat(sdk): add centralized guidance module`
-
----
-
-### Step 2: Add Server Instructions to Initialization
-
-**Files:** `src/mcp/server.ts`
-
-Add instructions to Server constructor:
-
-```typescript
-import { SERVER_INSTRUCTIONS } from "../sdk/guidance.js";
-
-const server = new Server(
-  { name: TOOLKIT_NAME, version: TOOLKIT_VERSION },
-  { 
-    capabilities: { tools: {}, prompts: {}, resources: {} },
-    instructions: SERVER_INSTRUCTIONS  // NEW
-  }
-);
-```
-
-**Checkpoint:** `feat(mcp): add server instructions for automatic guidance`
-
----
-
-### Step 3: Enhance Tool Descriptions
-
-**Files:** `src/mcp/tools-v2.ts`
-
-Update TOOLS array with enhanced descriptions:
-
-```typescript
-{
-  name: "persona",
-  description: `Manage Ema AI Employees (Personas).
-
-**Quick Start:**
-- List all: persona()
-- Get one: persona(id="...")
-- Analyze before changes: persona(id="...", analyze=true)
-- Update: persona(id="...", update={workflow_spec: {...}})
-
-**Important:** Always analyze before modifying. Use preview=true.`,
-  inputSchema: { ... }
-}
-```
-
-**Checkpoint:** `feat(mcp): enhance tool descriptions with usage guidance`
-
----
-
-### Step 4: Add Contextual Tips to Tool Responses
-
-**Files:** `src/mcp/handlers-consolidated.ts`
-
-Add `_guidance` field to tool responses:
-
-```typescript
-// In persona handler
-if (mode === 'list') {
-  return {
-    personas: results,
-    _guidance: {
-      next_step: "Get details: persona(id='<id>')",
-      tip: "Use analyze=true to check for issues before modifying"
-    }
-  };
-}
-
-if (mode === 'get' && !args.analyze) {
-  return {
-    persona: result,
-    _guidance: {
-      next_step: "Analyze: persona(id='...', analyze=true)",
-      warning: needsAnalysis ? "This persona has potential issues" : null
-    }
-  };
-}
-```
-
-**Checkpoint:** `feat(mcp): add contextual guidance to tool responses`
-
----
-
-### Step 5: Create Session Onboarding
-
-**Files:** `src/mcp/handlers-consolidated.ts`
-
-First tool call includes onboarding summary:
-
-```typescript
-let sessionInitialized = false;
-
-function maybeAddOnboarding(response: any): any {
-  if (!sessionInitialized) {
-    sessionInitialized = true;
-    return {
-      ...response,
-      _onboarding: {
-        message: "Welcome to Ema MCP Toolkit!",
-        quick_reference: {
-          list_personas: "persona()",
-          analyze_persona: "persona(id='...', analyze=true)",
-          available_prompts: "prompts/list for detailed workflows",
-          help: "Use toolkit_onboard prompt for full guide"
-        }
-      }
-    };
-  }
-  return response;
-}
-```
-
-**Checkpoint:** `feat(mcp): add session onboarding to first response`
-
----
-
-### Step 6: Add Workflow State Guidance
-
-**Files:** `src/sdk/guidance.ts`, `src/mcp/handlers-consolidated.ts`
-
-Detect workflow state and provide specific guidance:
-
-```typescript
-export function getWorkflowGuidance(workflow: any): WorkflowGuidance {
-  const issues = analyzeWorkflow(workflow);
-  
-  if (issues.critical.length > 0) {
-    return {
-      status: "needs_attention",
-      message: `${issues.critical.length} critical issues found`,
-      suggested_action: "Review issues and build workflow_spec to fix"
-    };
-  }
-  
-  if (!workflow.nodes?.some(n => n.category === "Fallback")) {
-    return {
-      status: "warning", 
-      message: "Missing Fallback category",
-      suggested_action: "Add Fallback handling for unmatched intents"
-    };
-  }
-  
-  return { status: "healthy" };
-}
-```
-
-**Checkpoint:** `feat(sdk): add workflow state guidance detection`
-
----
-
-### Step 7: Update Prompts to be Self-Referential
-
-**Files:** `src/mcp/prompts.ts`
-
-Ensure prompts reference other MCP capabilities:
-
-```typescript
-// In toolkit_onboard prompt
-text: `
-## Available Tools
-- persona: Manage AI Employees
-- catalog: Browse actions, templates, patterns
-- sync: Cross-environment sync
-
-## Available Prompts  
-- requirements_clarify: Get help with ambiguous requirements
-- workflow_extend: Step-by-step workflow modification
-- workflow_review: Comprehensive workflow analysis
-
-## Available Resources
-- ema://catalog/agents-summary: Action catalog
-- ema://rules/anti-patterns: Common mistakes
-- ema://docs/getting-started: Quick start guide
-
-All guidance is built into this MCP - no external files needed.
-`
-```
-
-**Checkpoint:** `feat(mcp): update prompts with self-referential guidance`
-
----
-
-### Step 8: Add Comprehensive Usage Guide Resource
-
-**Files:** `src/mcp/resources.ts`
-
-```typescript
-{
-  uri: "ema://docs/usage-guide",
-  name: "docs/usage-guide",
-  description: "Complete usage guide for Ema MCP Toolkit",
-  mimeType: "text/markdown",
-  generate: async () => generateUsageGuide()
-}
-```
-
-**Checkpoint:** `feat(mcp): add comprehensive usage guide resource`
-
----
-
-### Step 9: Remove External Dependencies
-
-**Files:** Audit all prompts and resources
-
-- Remove references to `.cursor/rules/`
-- Remove references to `AGENTS.md`
-- Replace with MCP-native alternatives
-
-**Checkpoint:** `refactor(mcp): remove external guidance dependencies`
-
----
-
-### Step 10: Documentation Update
-
-**Files:** `README.md`, `docs/mcp-tools-guide.md`
-
-```markdown
-## Self-Contained Guidance
-
-The Ema MCP Toolkit includes all guidance internally:
-
-- **Automatic**: Server instructions and tool tips guide usage
-- **On-Demand**: Prompts and resources for detailed workflows
-- **Contextual**: Response tips based on what you're doing
-
-No external rules files or IDE configuration required.
-```
-
-**Checkpoint:** `docs: document self-contained guidance system`
-
----
-
-## Affected Files
-
-| File | Change | Description |
-|------|--------|-------------|
-| `src/sdk/guidance.ts` | New | Centralized guidance content |
-| `src/mcp/server.ts` | Modify | Add server instructions |
-| `src/mcp/tools-v2.ts` | Modify | Enhanced tool descriptions |
-| `src/mcp/handlers-consolidated.ts` | Modify | Contextual response tips |
-| `src/mcp/prompts.ts` | Modify | Self-referential prompts |
-| `src/mcp/resources.ts` | Modify | Add usage guide resource |
-| `README.md` | Modify | Document self-contained nature |
-
-## Testing Strategy
-
-**Unit Tests:**
-- Guidance content is valid markdown
-- Contextual tips match workflow states
-- No external file references in guidance
-
-**Integration Tests:**
-- Server instructions in init response
-- Response tips present and accurate
-- Resources generate valid content
-
-**Manual Testing:**
-1. Fresh `npx @ema.co/mcp-toolkit@latest` in Claude Desktop
-2. Verify onboarding on first call
-3. Verify guidance in responses
-4. Complete full workflow without external docs
-
-## Risks and Mitigations
-
-| Risk | Mitigation |
-|------|------------|
-| Guidance becomes stale | Generate from code, not static strings |
-| Too much guidance bloats responses | Progressive disclosure, keep tips short |
-| Clients ignore server instructions | Multiple layers of guidance |
-| Token budget exceeded | Measure token counts, optimize |
-
-## Open Questions
-
-- [ ] Token budget for tool descriptions before UX degrades?
-- [ ] Track guidance effectiveness (what tips are followed)?
-- [ ] Different guidance levels (beginner vs advanced)?
-
-## Alternatives Considered
-
-### MCP Server Instructions Only
-- Pro: Automatic, standard MCP
-- Con: Cursor support uncertain, single blob
-
-### Onboarding Prompt Only
-- Pro: Works now, detailed
-- Con: Requires user action
-
-### Generated Rules File
-- Pro: Full fidelity with internal rules
-- Con: Requires user action, version drift
-
-### Hybrid (Selected)
-- Multiple layers for resilience
-- Progressive disclosure
-- Works across all clients

package/docs/proto-sdk-generation.md

@@ -1,242 +0,0 @@
-# Proto SDK Generation
-
-This document describes how the MCP toolkit generates and maintains TypeScript clients from Ema's protocol buffer definitions.
-
-## Overview
-
-The MCP toolkit uses [Connect-ES](https://connectrpc.com/docs/web) to generate typed TypeScript clients from proto definitions. This provides type-safe access to gRPC services that aren't exposed via REST/OpenAPI.
-
-```
-protos/service/**/*.proto
-        │
-        │ buf generate
-        ▼
-src/sdk/generated/protos/
-├── service/workflows/v1/
-│   ├── action_registry_connect.ts  ← Service definition
-│   ├── action_registry_pb.ts       ← Message types
-│   ├── dashboards_connect.ts
-│   └── dashboards_pb.ts
-└── ...
-
-        │
-        │ wrapped by
-        ▼
-src/sdk/grpc-client.ts              ← GrpcClient class
-```
-
-## Quick Start
-
-### Regenerate Proto SDK
-
-```bash
-npm run generate:protos
-```
-
-This command:
-1. Reads `buf.gen.yaml` for codegen configuration
-2. Generates TypeScript files from `../ema-repos/protos/service/`
-3. Outputs to `src/sdk/generated/protos/`
-
-### Use the GrpcClient
-
-```typescript
-import { GrpcClient } from '@ema.co/mcp-toolkit';
-
-const client = new GrpcClient({
-  name: 'dev',
-  baseUrl: 'https://dev.ema.co',
-  bearerToken: process.env.EMA_DEV_BEARER_TOKEN,
-});
-
-// List all actions (typed response)
-const actions = await client.listActions({ includeDocumentation: true });
-
-// Get dashboard rows (typed response)  
-const rows = await client.getDashboardRows('dashboard-123', { limit: 50 });
-```
-
-## Architecture
-
-### Why Both OpenAPI and Proto?
-
-Ema's API surface is split:
-
-| Protocol | Services | Generated Client |
-|----------|----------|------------------|
-| REST/OpenAPI | Personas, Templates, Data Sources | `EmaClientAdapter` |
-| gRPC | Actions, Dashboards, Workflows | `GrpcClient` |
-
-The MCP toolkit needs both clients to cover the full API surface.
-
-### Source of Truth
-
-```
-protos/service/**/*.proto  ← Single source of truth for gRPC
-        │
-        ├── Go services (goservice, workflow-engine)
-        ├── Python services (ema-backend)
-        ├── TypeScript clients (ema-app, mcp-toolkit)
-        │
-        ▼
-All generated, never hand-written
-```
-
-## Files
-
-| File | Purpose |
-|------|---------|
-| `buf.gen.yaml` | Codegen configuration |
-| `src/sdk/generated/protos/` | Generated Connect-ES clients (DO NOT EDIT) |
-| `src/sdk/grpc-client.ts` | Typed wrapper with retry/auth |
-| `.github/workflows/sync-protos.yml` | Automated weekly sync |
-
-## Dependencies
-
-### Runtime
-```json
-{
-  "@connectrpc/connect": "^1.4.0",
-  "@connectrpc/connect-node": "^1.4.0",
-  "@bufbuild/protobuf": "^1.10.0"
-}
-```
-
-### Development
-```json
-{
-  "@bufbuild/buf": "^1.34.0",
-  "@bufbuild/protoc-gen-es": "^1.10.0",
-  "@connectrpc/protoc-gen-connect-es": "^1.4.0"
-}
-```
-
-## Automation
-
-### GitHub Workflow
-
-The `sync-protos.yml` workflow runs weekly to:
-
-1. Checkout the latest `protos` repository
-2. Regenerate TypeScript clients
-3. Create a PR if files changed
-4. Verify build and tests pass
-
-**Manual trigger:**
-```bash
-gh workflow run sync-protos.yml
-```
-
-### Local Development
-
-When protos change:
-
-```bash
-# Pull latest protos
-cd ../ema-repos/protos && git pull
-
-# Regenerate
-cd ../ema-mcp-toolkit && npm run generate:protos
-
-# Verify
-npm run build && npm test
-```
-
-## Troubleshooting
-
-### "Plugin not found" Error
-
-```
-Failure: plugin connect-es: could not find protoc plugin
-```
-
-**Fix:** Ensure dev dependencies are installed:
-```bash
-npm install
-```
-
-### "Cannot find module" Import Error
-
-Generated files use `.js` extensions for ESM compatibility:
-```typescript
-// Correct
-import { ActionManager } from './generated/protos/service/workflows/v1/action_registry_connect.js';
-
-// Incorrect (will fail at runtime)
-import { ActionManager } from './generated/protos/service/workflows/v1/action_registry_connect';
-```
-
-### Type Errors After Regeneration
-
-If types break after regeneration:
-
-1. Check if proto field names changed
-2. Update `grpc-client.ts` wrapper accordingly
-3. Run `npm run build` to verify
-
-### gRPC Call Fails with 401
-
-**Fix:** Check bearer token:
-```typescript
-const client = new GrpcClient({
-  name: 'dev',
-  baseUrl: 'https://dev.ema.co',
-  bearerToken: process.env.EMA_DEV_BEARER_TOKEN, // Must be valid
-});
-```
-
-### gRPC Call Returns 404 or 415
-
-**HTTP 404 - Wrong base URL:**
-- Use `api.dev.ema.co` (not `staging.ema.co` or `dev.ema.co`)
-- The gRPC endpoints are served from the `api.*` subdomain
-
-**HTTP 415 - Wrong content type:**
-- **Root cause**: Ema backend is a pure gRPC server, not a Connect server
-- **Wrong**: `application/proto` (Connect binary format)
-- **Wrong**: `application/json` (Connect JSON format)  
-- **Correct**: `application/grpc-web+proto` (gRPC-web format)
-
-**Solution**: Use `createGrpcWebTransport` instead of `createConnectTransport`:
-```typescript
-import { createGrpcWebTransport } from '@connectrpc/connect-node';
-
-const transport = createGrpcWebTransport({
-  baseUrl: 'https://api.dev.ema.co',
-  httpVersion: '1.1',
-});
-```
-
-## Extending the Client
-
-To add new gRPC methods:
-
-1. **Check the generated files** - See if the method exists in `*_connect.ts`
-
-2. **Add to GrpcClient**:
-```typescript
-// In src/sdk/grpc-client.ts
-async myNewMethod(param: string) {
-  const request = new MyNewRequest({ field: param });
-  return this.myService.myNewMethod(request);
-}
-```
-
-3. **Export types** (if needed):
-```typescript
-// In src/sdk/grpc-client.ts
-export type { MyNewResponse } from './generated/protos/...';
-```
-
-4. **Add to SDK exports** (if public):
-```typescript
-// In src/sdk/index.ts
-export { type MyNewResponse as GrpcMyNewResponse } from "./grpc-client.js";
-```
-
-## Related Documentation
-
-- [Connect-ES Documentation](https://connectrpc.com/docs/web)
-- [Buf CLI Documentation](https://buf.build/docs/bsr/overview)
-- [Ema Protos Repository](https://github.com/Ema-Unlimited/protos)
-- [ema-app buf.gen.yaml](../../../ema-repos/ema-app/buf.gen.yaml) - Reference implementation