@ema.co/mcp-toolkit 2026.1.27 → 2026.1.28-2

package/docs/rule-format-specification.md

@@ -0,0 +1,346 @@
+# Rule Format Specification for LLM Consumption
+
+**Date**: 2026-01-27  
+**Purpose**: Define structured rule format for MCP resources and LLM prompts
+
+---
+
+## Design Principles
+
+1. **LLM-Friendly**: Easy for LLM to parse and understand
+2. **Structured**: Both JSON (programmatic) and Markdown (human-readable)
+3. **Complete**: Includes logic, rules, examples, remediation
+4. **Referenceable**: Each rule has unique ID
+5. **Actionable**: Rules guide workflow generation, not just validation
+
+---
+
+## Rule Structure
+
+### JSON Format (Programmatic)
+
+```typescript
+export interface ValidationRule {
+  // Identification
+  id: string;                    // Unique rule ID: "required_output_all_paths"
+  name: string;                  // Human-readable name: "Required Output on All Paths"
+  version: string;               // Rule version: "1.0.0"
+  
+  // Logic (Algorithmic)
+  logic: {
+    algorithm: string;           // Algorithm description
+    steps: string[];             // Step-by-step algorithm
+    complexity: "O(P*A)" | ...;  // Time complexity
+  };
+  
+  // Rules (Constraints)
+  rules: {
+    constraint: string;           // Human-readable constraint
+    severity: "critical" | "warning" | "info";
+    applies_to: string[];        // What it applies to: ["named_results", "paths"]
+  }[];
+  
+  // Examples
+  examples: {
+    bad: {
+      description: string;
+      workflow_snippet: string;  // JSON snippet
+      why_bad: string;
+    };
+    good: {
+      description: string;
+      workflow_snippet: string;  // JSON snippet
+      why_good: string;
+    };
+  };
+  
+  // Remediation
+  remediation: {
+    steps: string[];             // How to fix
+    common_fixes: string[];      // Common solutions
+    prevention: string;          // How to avoid in generation
+  };
+  
+  // Metadata
+  source: "go_validator" | "mcp_extension";
+  error_type?: string;           // Go error type if from Go validator
+}
+```
+
+### Markdown Format (LLM-Friendly)
+
+```markdown
+## Rule: Required Output on All Paths
+
+**Rule ID**: `required_output_all_paths`  
+**Severity**: Critical  
+**Applies To**: Named results, execution paths
+
+### Logic
+
+**Algorithm**: Enumerate all execution paths. For each path, check if all named results are produced.
+
+**Steps**:
+1. Enumerate all paths from trigger to completion
+2. For each path, collect all outputs produced
+3. For each named result, check if it's in path outputs
+4. If missing on any path → error
+
+**Complexity**: O(P * R) where P = paths, R = named results
+
+### Rules
+
+**Constraint**: Every execution path must produce all named results.
+
+**Why**: Named results are contract guarantees. If a path doesn't produce a required output, the contract is violated.
+
+### Examples
+
+**Bad Pattern**:
+```json
+{
+  "path": ["trigger", "categorizer", "billing_branch", "search"],
+  "named_result": "response_with_sources",
+  "produces": []  // Missing response
+}
+```
+
+**Why Bad**: Path ends without producing required output.
+
+**Good Pattern**:
+```json
+{
+  "path": ["trigger", "categorizer", "billing_branch", "search", "respond_with_sources"],
+  "named_result": "response_with_sources",
+  "produces": ["response_with_sources"]  // ✅ Produced
+}
+```
+
+**Why Good**: All paths produce required output.
+
+### Remediation
+
+**How to Fix**:
+1. Identify the path missing the output
+2. Add a node that produces the required output
+3. Connect the node's output to WORKFLOW_OUTPUT
+
+**Common Fixes**:
+- Add `respond_with_sources` node after search
+- Add `call_llm` node for text generation
+- Add `fixed_response` for static responses
+
+**Prevention** (During Generation):
+- Always ensure every categorizer branch has a response node
+- Check that all paths end with nodes connected to WORKFLOW_OUTPUT
+- Verify resultMappings include all required outputs
+```
+
+---
+
+## Rule Collection Format
+
+### MCP Resource Format
+
+```typescript
+// ema://validation/rules
+
+export interface ValidationRulesResource {
+  version: string;
+  rules: ValidationRule[];
+  summary: {
+    total_rules: number;
+    critical_rules: number;
+    warning_rules: number;
+    info_rules: number;
+  };
+}
+```
+
+### LLM Prompt Format
+
+```markdown
+# Workflow Validation Rules
+
+Before generating or modifying workflows, ensure these rules are followed:
+
+## Critical Rules (Must Never Violate)
+
+### 1. Required Output on All Paths
+**Rule**: Every execution path must produce all named results.
+
+**Algorithm**: 
+1. Enumerate all paths from trigger
+2. Check each path produces all named results
+3. Report error if any path missing any result
+
+**Example Bad**: Path ends without response node
+**Example Good**: All paths end with response → WORKFLOW_OUTPUT
+
+**Prevention**: Always add response nodes to every categorizer branch.
+
+[... more rules ...]
+
+## Self-Check Checklist
+
+After generating a workflow, verify:
+- [ ] All paths produce all named results
+- [ ] No multiple writers on same path
+- [ ] Every path has response or abstain
+- [ ] Category hierarchy is valid
+```
+
+---
+
+## Rule Deconstruction
+
+### For LLM Consumption
+
+Rules are deconstructed into:
+
+1. **Logic** (Algorithmic):
+   - How to detect the violation
+   - Step-by-step algorithm
+   - Complexity analysis
+
+2. **Rules** (Constraints):
+   - What must be true
+   - Why it matters
+   - When it applies
+
+3. **Examples** (Patterns):
+   - Bad patterns to avoid
+   - Good patterns to follow
+   - Why each is bad/good
+
+4. **Remediation** (Fixes):
+   - How to fix violations
+   - Common solutions
+   - Prevention during generation
+
+---
+
+## Rule IDs
+
+### Naming Convention
+
+```
+{error_type}_{scope}_{constraint}
+```
+
+**Examples**:
+- `required_output_all_paths`
+- `single_writer_per_output`
+- `response_or_abstain_required`
+- `category_hierarchy_valid`
+
+### Rule Reference Format
+
+```
+ema://validation/rules#{rule_id}
+```
+
+---
+
+## Integration Points
+
+### 1. MCP Resource
+
+**URI**: `ema://validation/rules`
+
+**Format**: Markdown (LLM-friendly) with JSON metadata
+
+**Access**: Via `resources.ts` handler
+
+### 2. Generation Prompts
+
+**Location**: `workflow_generate` prompt, `workflow_intent.ts` system prompt
+
+**Format**: Markdown rules section
+
+**Purpose**: LLM self-validates during generation
+
+### 3. Validation Tool
+
+**Location**: `workflow(mode="validate")` tool
+
+**Format**: Returns errors with rule references
+
+**Purpose**: Post-generation validation
+
+---
+
+## Example: Complete Rule
+
+### JSON Format
+
+```json
+{
+  "id": "required_output_all_paths",
+  "name": "Required Output on All Paths",
+  "version": "1.0.0",
+  "logic": {
+    "algorithm": "Enumerate all execution paths. For each path, check if all named results are produced.",
+    "steps": [
+      "1. Enumerate all paths from trigger to completion",
+      "2. For each path, collect all outputs produced",
+      "3. For each named result, check if it's in path outputs",
+      "4. If missing on any path → error"
+    ],
+    "complexity": "O(P * R)"
+  },
+  "rules": [{
+    "constraint": "Every execution path must produce all named results",
+    "severity": "critical",
+    "applies_to": ["named_results", "paths"]
+  }],
+  "examples": {
+    "bad": {
+      "description": "Path ends without response",
+      "workflow_snippet": "{\"path\": [\"trigger\", \"categorizer\", \"billing\"], \"produces\": []}",
+      "why_bad": "Path ends without producing required output"
+    },
+    "good": {
+      "description": "All paths produce response",
+      "workflow_snippet": "{\"path\": [\"trigger\", \"categorizer\", \"billing\", \"respond\"], \"produces\": [\"response_with_sources\"]}",
+      "why_good": "All paths produce required output"
+    }
+  },
+  "remediation": {
+    "steps": [
+      "Identify the path missing the output",
+      "Add a node that produces the required output",
+      "Connect the node's output to WORKFLOW_OUTPUT"
+    ],
+    "common_fixes": [
+      "Add respond_with_sources node after search",
+      "Add call_llm node for text generation",
+      "Add fixed_response for static responses"
+    ],
+    "prevention": "Always ensure every categorizer branch has a response node"
+  },
+  "source": "go_validator",
+  "error_type": "RESULT_ERROR_TYPE_REQUIRED_OUTPUT_NOT_PRODUCED"
+}
+```
+
+### Markdown Format
+
+See example in "Markdown Format (LLM-Friendly)" section above.
+
+---
+
+## Implementation Checklist
+
+- [ ] Define TypeScript interfaces
+- [ ] Create rule extraction from Go code
+- [ ] Generate JSON format
+- [ ] Generate Markdown format
+- [ ] Create MCP resource handler
+- [ ] Integrate into generation prompts
+- [ ] Test rule consumption by LLM
+
+---
+
+**Status**: ✅ Format designed  
+**Next**: Implement rule extraction and generation