@stephen-lord/docs 1.0.0 → 1.0.1

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/tools/evaluation/generate-rubric.md

@@ -0,0 +1,189 @@
+# Generate Rubric Tool
+
+## Purpose
+
+Automatically generate a scoring rubric for a given evaluation criterion. Creates detailed descriptions for each score level to ensure consistent evaluation.
+
+## Tool Definition
+
+```typescript
+import { tool } from "ai";
+import { z } from "zod";
+
+export const generateRubric = tool({
+  description: `Generate a scoring rubric for an evaluation criterion.
+Creates detailed descriptions for each score level.
+Use when you need to establish consistent evaluation standards.`,
+
+  parameters: z.object({
+    criterionName: z.string()
+      .describe("Name of the criterion (e.g., 'Factual Accuracy')"),
+    
+    criterionDescription: z.string()
+      .describe("What this criterion measures"),
+    
+    scale: z.enum(["1-3", "1-5", "1-10"]).default("1-5")
+      .describe("Scoring scale to use"),
+    
+    domain: z.string().optional()
+      .describe("Domain context (e.g., 'medical writing', 'code review')"),
+    
+    includeExamples: z.boolean().default(true)
+      .describe("Include example text for each score level"),
+    
+    strictness: z.enum(["lenient", "balanced", "strict"]).default("balanced")
+      .describe("How strictly to define score boundaries")
+  }),
+
+  execute: async (input) => {
+    return generateRubricWithLLM(input);
+  }
+});
+```
+
+## Input Schema
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| criterionName | string | Yes | Name of criterion |
+| criterionDescription | string | Yes | What criterion measures |
+| scale | enum | No | Scoring scale (default: 1-5) |
+| domain | string | No | Domain for context |
+| includeExamples | boolean | No | Include examples (default: true) |
+| strictness | enum | No | Score boundary strictness |
+
+## Output Schema
+
+```typescript
+interface GeneratedRubric {
+  success: boolean;
+  
+  criterion: {
+    name: string;
+    description: string;
+  };
+  
+  scale: {
+    min: number;
+    max: number;
+    type: string;
+  };
+  
+  levels: {
+    score: number;
+    label: string;        // e.g., "Excellent", "Poor"
+    description: string;  // Detailed description
+    characteristics: string[];  // Key characteristics
+    example?: string;     // Example text at this level
+  }[];
+  
+  scoringGuidelines: string[];
+  
+  edgeCases: {
+    situation: string;
+    guidance: string;
+  }[];
+  
+  metadata: {
+    domain: string | null;
+    strictness: string;
+    generationTimeMs: number;
+  };
+}
+```
+
+## Usage Example
+
+```typescript
+const rubric = await generateRubric.execute({
+  criterionName: "Code Readability",
+  criterionDescription: "How easy the code is to read and understand",
+  scale: "1-5",
+  domain: "code review",
+  includeExamples: true,
+  strictness: "balanced"
+});
+
+// Result:
+// {
+//   criterion: {
+//     name: "Code Readability",
+//     description: "How easy the code is to read and understand"
+//   },
+//   scale: { min: 1, max: 5, type: "1-5" },
+//   levels: [
+//     {
+//       score: 1,
+//       label: "Poor",
+//       description: "Code is extremely difficult to understand...",
+//       characteristics: [
+//         "No meaningful variable names",
+//         "Deeply nested logic without explanation",
+//         "No comments on complex sections"
+//       ],
+//       example: "function x(a,b,c){return a?b+c:c-b;}"
+//     },
+//     {
+//       score: 5,
+//       label: "Excellent",
+//       description: "Code is immediately understandable...",
+//       characteristics: [
+//         "Self-documenting variable and function names",
+//         "Appropriate comments explaining 'why'",
+//         "Clear logical structure"
+//       ],
+//       example: "function calculateShippingCost(weight, distance, expedited) {\n  // Base rate plus per-mile charge\n  const baseCost = weight * BASE_RATE_PER_KG;\n  ..."
+//     },
+//     ...
+//   ],
+//   scoringGuidelines: [
+//     "Focus on clarity for someone unfamiliar with the codebase",
+//     "Consider both naming and structure",
+//     "Comments should explain 'why', not 'what'"
+//   ],
+//   edgeCases: [
+//     {
+//       situation: "Code uses domain-specific abbreviations",
+//       guidance: "Accept if abbreviations are standard in the domain"
+//     }
+//   ]
+// }
+```
+
+## Rubric Templates
+
+### Factual Accuracy (1-5)
+```
+5: All claims factually correct, properly sourced
+4: Minor factual issues, non-critical
+3: Some factual errors, main points correct
+2: Multiple factual errors affecting reliability
+1: Fundamentally incorrect or misleading
+```
+
+### Clarity (1-5)
+```
+5: Immediately understandable, well-structured
+4: Clear with minor ambiguities
+3: Generally clear, some confusing sections
+2: Difficult to follow, unclear organization
+1: Incomprehensible or incoherent
+```
+
+### Completeness (1-5)
+```
+5: Addresses all aspects comprehensively
+4: Covers main points, minor gaps
+3: Addresses core requirements, notable gaps
+2: Missing significant required elements
+1: Fails to address the question
+```
+
+## Implementation Notes
+
+1. **Domain Adaptation**: Rubrics should reflect domain-specific expectations
+2. **Boundary Clarity**: Clear distinctions between adjacent scores
+3. **Example Quality**: Examples should be realistic, not strawmen
+4. **Edge Case Coverage**: Anticipate common ambiguous situations
+5. **Calibration**: Test rubric against known samples before use
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/tools/evaluation/pairwise-compare.md

@@ -0,0 +1,182 @@
+# Pairwise Compare Tool
+
+## Purpose
+
+Compare two LLM responses and determine which one better satisfies the given criteria. More reliable for subjective evaluations than direct scoring.
+
+## Tool Definition
+
+```typescript
+import { tool } from "ai";
+import { z } from "zod";
+
+export const pairwiseCompare = tool({
+  description: `Compare two responses and select the better one.
+Use for subjective evaluations like tone, persuasiveness, or writing style.
+More reliable than direct scoring for preferences.
+Returns winner selection with detailed comparison.`,
+
+  parameters: z.object({
+    responseA: z.string()
+      .describe("First response to compare"),
+    
+    responseB: z.string()
+      .describe("Second response to compare"),
+    
+    prompt: z.string()
+      .describe("The original prompt both responses address"),
+    
+    context: z.string().optional()
+      .describe("Additional context if relevant"),
+    
+    criteria: z.array(z.string())
+      .describe("Aspects to compare on, e.g., ['clarity', 'engagement', 'accuracy']"),
+    
+    allowTie: z.boolean().default(true)
+      .describe("Whether to allow a tie verdict"),
+    
+    swapPositions: z.boolean().default(true)
+      .describe("Evaluate twice with positions swapped to reduce position bias")
+  }),
+
+  execute: async (input) => {
+    if (input.swapPositions) {
+      return evaluateWithPositionSwap(input);
+    }
+    return evaluatePairwise(input);
+  }
+});
+```
+
+## Input Schema
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| responseA | string | Yes | First response |
+| responseB | string | Yes | Second response |
+| prompt | string | Yes | Original prompt |
+| context | string | No | Additional context |
+| criteria | string[] | Yes | Comparison dimensions |
+| allowTie | boolean | No | Allow tie verdict (default: true) |
+| swapPositions | boolean | No | Swap positions to reduce bias (default: true) |
+
+## Output Schema
+
+```typescript
+interface PairwiseCompareResult {
+  success: boolean;
+  
+  winner: "A" | "B" | "TIE";
+  confidence: number; // 0-1
+  
+  comparison: {
+    criterion: string;
+    winner: "A" | "B" | "TIE";
+    reasoning: string;
+    aStrength: string;
+    bStrength: string;
+  }[];
+  
+  overallReasoning: string;
+  
+  differentiators: {
+    aAdvantages: string[];
+    bAdvantages: string[];
+  };
+  
+  // If swapPositions was true
+  positionConsistency?: {
+    firstPassWinner: "A" | "B" | "TIE";
+    secondPassWinner: "A" | "B" | "TIE";
+    consistent: boolean;
+  };
+  
+  metadata: {
+    evaluationTimeMs: number;
+    positionsSwapped: boolean;
+  };
+}
+```
+
+## Usage Example
+
+```typescript
+const result = await pairwiseCompare.execute({
+  responseA: "Exercise improves cardiovascular health, builds muscle, and boosts mental wellbeing...",
+  
+  responseB: "Working out regularly has many benefits. You'll feel better and look better...",
+  
+  prompt: "Explain the benefits of regular exercise",
+  
+  criteria: ["accuracy", "specificity", "engagement", "completeness"],
+  
+  allowTie: true,
+  swapPositions: true
+});
+
+// Result:
+// {
+//   winner: "A",
+//   confidence: 0.85,
+//   comparison: [
+//     {
+//       criterion: "accuracy",
+//       winner: "A",
+//       reasoning: "Response A uses specific medical terminology...",
+//       aStrength: "Mentions cardiovascular, muscle, mental health",
+//       bStrength: "General but not incorrect"
+//     },
+//     ...
+//   ],
+//   ...
+// }
+```
+
+## Position Swap Algorithm
+
+To mitigate position bias:
+
+```typescript
+async function evaluateWithPositionSwap(input) {
+  // First pass: Original order
+  const pass1 = await evaluate({
+    first: input.responseA,
+    second: input.responseB,
+    ...input
+  });
+  
+  // Second pass: Swapped order
+  const pass2 = await evaluate({
+    first: input.responseB,
+    second: input.responseA,
+    ...input
+  });
+  
+  // Reconcile results
+  const pass2Adjusted = pass2.winner === "A" ? "B" : pass2.winner === "B" ? "A" : "TIE";
+  
+  if (pass1.winner === pass2Adjusted) {
+    return {
+      ...pass1,
+      positionConsistency: { consistent: true, ... }
+    };
+  } else {
+    // Inconsistent - return tie or lower confidence
+    return {
+      winner: "TIE",
+      confidence: 0.5,
+      positionConsistency: { consistent: false, ... },
+      ...
+    };
+  }
+}
+```
+
+## Implementation Notes
+
+1. **Position Bias Mitigation**: Always use `swapPositions: true` for production
+2. **Criteria Order**: Order criteria by importance for better focus
+3. **Tie Handling**: Consider domain - some tasks should rarely tie
+4. **Confidence Calibration**: Lower confidence when evaluations are close
+5. **Length Considerations**: Note if one response is significantly longer
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/tools/index.md

@@ -0,0 +1,141 @@
+# Tools Index
+
+Tools provide specific capabilities that agents can use to accomplish tasks.
+
+## Tool Categories
+
+### Evaluation Tools
+**Path**: `tools/evaluation/`
+
+Tools for assessing LLM output quality.
+
+| Tool | Purpose | Approval |
+|------|---------|----------|
+| `directScore` | Score response against criteria | No |
+| `pairwiseCompare` | Compare two responses | No |
+| `generateRubric` | Generate scoring rubric | No |
+| `extractCriteria` | Extract criteria from task | No |
+
+---
+
+### Research Tools
+**Path**: `tools/research/`
+
+Tools for gathering and processing information.
+
+| Tool | Purpose | Approval |
+|------|---------|----------|
+| `webSearch` | Search the web | No |
+| `readUrl` | Extract content from URL | No |
+| `extractClaims` | Identify claims in text | No |
+| `verifyClaim` | Cross-reference a claim | No |
+| `synthesize` | Combine findings | No |
+
+---
+
+### Orchestration Tools
+**Path**: `tools/orchestration/`
+
+Tools for managing multi-agent workflows.
+
+| Tool | Purpose | Approval |
+|------|---------|----------|
+| `delegateToAgent` | Route task to agent | No |
+| `parallelExecution` | Run tasks concurrently | No |
+| `waitForCompletion` | Wait for async tasks | No |
+| `synthesizeResults` | Combine agent outputs | No |
+| `handleError` | Manage failures | No |
+
+## Tool Design Patterns
+
+### Standard Tool Structure
+
+```typescript
+export const toolName = tool({
+  description: "Clear description of what tool does",
+  
+  parameters: z.object({
+    // Required parameters first
+    requiredParam: z.string().describe("What this parameter is for"),
+    
+    // Optional parameters with defaults
+    optionalParam: z.number().default(10)
+      .describe("What this parameter controls")
+  }),
+  
+  // Approval for dangerous operations
+  needsApproval: false, // or true, or function
+  
+  // Strict mode for guaranteed schema compliance
+  strict: true,
+  
+  execute: async (input) => {
+    try {
+      const result = await performOperation(input);
+      return { success: true, data: result };
+    } catch (error) {
+      return {
+        success: false,
+        error: {
+          code: error.code ?? "UNKNOWN",
+          message: error.message,
+          retryable: isRetryable(error)
+        }
+      };
+    }
+  },
+  
+  // Optional: control what model sees
+  toModelOutput: (result) => ({
+    summary: result.data.summary,
+    truncated: result.data.full.length > 5000
+  })
+});
+```
+
+### Error Response Pattern
+
+```typescript
+interface ToolError {
+  code: string;       // Machine-readable error code
+  message: string;    // Human-readable message
+  retryable: boolean; // Whether retry might help
+  details?: object;   // Additional context
+}
+
+interface ToolResult<T> {
+  success: boolean;
+  data?: T;
+  error?: ToolError;
+  metadata: {
+    executionTimeMs: number;
+    [key: string]: any;
+  };
+}
+```
+
+## Adding New Tools
+
+1. Determine category or create new: `tools/<category>/`
+2. Create tool file: `tools/<category>/<tool-name>.md`
+3. Define:
+   - Purpose and description
+   - Input parameters with Zod schema
+   - Output schema
+   - Error codes
+   - Usage examples
+4. Update this index
+5. Assign to relevant agents
+
+## Tool Selection Guidelines
+
+### When Agent Needs To...
+
+| Action | Tool Category | Suggested Tools |
+|--------|---------------|-----------------|
+| Assess quality | Evaluation | directScore, pairwiseCompare |
+| Find information | Research | webSearch, readUrl |
+| Verify facts | Research | verifyClaim, extractClaims |
+| Coordinate work | Orchestration | delegateToAgent |
+| Wait for results | Orchestration | waitForCompletion |
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/tools/orchestration/delegate-to-agent.md

@@ -0,0 +1,171 @@
+# Delegate to Agent Tool
+
+## Purpose
+
+Route a task to a specialized agent for execution. Handles context passing, result collection, and error management.
+
+## Tool Definition
+
+```typescript
+import { tool } from "ai";
+import { z } from "zod";
+
+export const delegateToAgent = tool({
+  description: `Delegate a task to a specialized agent.
+Use when a subtask requires specific capabilities.
+Pass complete context needed for the agent to succeed.`,
+
+  parameters: z.object({
+    agentName: z.enum(["evaluator", "researcher", "writer", "analyst"])
+      .describe("Name of the agent to delegate to"),
+    
+    task: z.string()
+      .describe("Clear description of what the agent should do"),
+    
+    context: z.object({
+      previousOutputs: z.array(z.string()).optional()
+        .describe("Outputs from prior steps this agent needs"),
+      
+      documents: z.array(z.string()).optional()
+        .describe("Relevant documents or data"),
+      
+      constraints: z.array(z.string()).optional()
+        .describe("Requirements or limitations to observe")
+    }).optional(),
+    
+    expectedOutput: z.object({
+      format: z.enum(["text", "json", "markdown", "structured"])
+        .describe("Expected output format"),
+      
+      schema: z.string().optional()
+        .describe("JSON schema if format is structured"),
+      
+      maxLength: z.number().optional()
+        .describe("Maximum length constraint")
+    }).optional(),
+    
+    timeout: z.number().default(60000)
+      .describe("Timeout in milliseconds")
+  }),
+
+  execute: async (input) => {
+    return executeAgentDelegation(input);
+  }
+});
+```
+
+## Input Schema
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| agentName | enum | Yes | Target agent |
+| task | string | Yes | Task description |
+| context | object | No | Context and dependencies |
+| expectedOutput | object | No | Output requirements |
+| timeout | number | No | Timeout ms (default: 60000) |
+
+## Output Schema
+
+```typescript
+interface DelegationResult {
+  success: boolean;
+  
+  agentName: string;
+  task: string;
+  
+  output: {
+    content: string | object;
+    format: string;
+  };
+  
+  execution: {
+    startTime: string;
+    endTime: string;
+    durationMs: number;
+    tokenUsage: {
+      prompt: number;
+      completion: number;
+    };
+  };
+  
+  error?: {
+    code: string;
+    message: string;
+    retryable: boolean;
+  };
+}
+```
+
+## Available Agents
+
+### Evaluator Agent
+```typescript
+await delegateToAgent.execute({
+  agentName: "evaluator",
+  task: "Evaluate the quality of this response against accuracy and clarity criteria",
+  context: {
+    documents: [responseToEvaluate],
+    constraints: ["Use 1-5 scale", "Include justification"]
+  },
+  expectedOutput: { format: "structured" }
+});
+```
+
+### Researcher Agent
+```typescript
+await delegateToAgent.execute({
+  agentName: "researcher",
+  task: "Research current best practices for LLM evaluation",
+  context: {
+    constraints: ["Focus on 2024 publications", "Include citations"]
+  },
+  expectedOutput: { format: "markdown" }
+});
+```
+
+### Writer Agent
+```typescript
+await delegateToAgent.execute({
+  agentName: "writer",
+  task: "Write an executive summary of these research findings",
+  context: {
+    previousOutputs: [researchFindings],
+    constraints: ["Maximum 500 words", "Non-technical audience"]
+  },
+  expectedOutput: { format: "text", maxLength: 2500 }
+});
+```
+
+### Analyst Agent
+```typescript
+await delegateToAgent.execute({
+  agentName: "analyst",
+  task: "Analyze the trade-offs between direct scoring and pairwise comparison",
+  context: {
+    documents: [evaluationData],
+    constraints: ["Quantitative where possible"]
+  },
+  expectedOutput: { format: "structured" }
+});
+```
+
+## Error Handling
+
+```typescript
+const errorCodes = {
+  "AGENT_NOT_FOUND": "Specified agent does not exist",
+  "TIMEOUT": "Agent did not complete within timeout",
+  "CONTEXT_TOO_LARGE": "Context exceeds agent's capacity",
+  "INVALID_OUTPUT": "Agent output did not match expected format",
+  "AGENT_ERROR": "Agent encountered an error during execution"
+};
+```
+
+## Implementation Notes
+
+1. **Context Optimization**: Compress context if needed before passing
+2. **Timeout Handling**: Set realistic timeouts per agent type
+3. **Retry Logic**: Implement retries for transient failures
+4. **Audit Trail**: Log all delegations for traceability
+5. **Resource Management**: Track token usage across delegations
+