@stephen-lord/docs 1.0.0 → 1.0.1

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/skills/context-fundamentals/context-fundamentals.md

@@ -0,0 +1,114 @@
+# Context Fundamentals Skill
+
+## Overview
+
+Context engineering is the systematic approach to managing what information an LLM receives and how it processes that information. Effective context management directly impacts output quality, consistency, and task success rates.
+
+## Core Principles
+
+### 1. Context Window Management
+
+The context window is finite. Every token counts. Prioritize information by relevance and recency.
+
+**Strategies:**
+- Summarize historical conversation turns
+- Use retrieval to inject only relevant context
+- Implement context compression for long documents
+
+### 2. Information Hierarchy
+
+Structure context to guide model attention:
+
+```
+1. System Instructions (highest priority)
+   └── Role definition
+   └── Task constraints
+   └── Output format requirements
+
+2. Relevant Context (dynamic)
+   └── Retrieved documents
+   └── User-specific data
+   └── Recent conversation history
+
+3. User Input (current request)
+   └── Query or instruction
+   └── Any inline context
+```
+
+### 3. Context Relevance
+
+Not all context is equally useful. Apply relevance filtering:
+
+- **Temporal Relevance**: Recent information often outweighs older data
+- **Semantic Relevance**: Use embeddings to surface related content
+- **Task Relevance**: Only include information needed for current task
+
+## Context Types
+
+### Static Context
+- System prompts
+- Role definitions
+- Tool descriptions
+- Format specifications
+
+### Dynamic Context
+- Retrieved documents (RAG)
+- Conversation history
+- User preferences
+- Session state
+
+### Ephemeral Context
+- Current tool outputs
+- Intermediate reasoning steps
+- Scratchpad content
+
+## Best Practices
+
+1. **Explicit Over Implicit**: State requirements clearly rather than relying on inference
+2. **Structured Formatting**: Use consistent delimiters and sections
+3. **Redundancy Removal**: Avoid repeating information across context sections
+4. **Source Attribution**: Mark where context comes from for traceability
+5. **Freshness Signals**: Indicate when information was last updated
+
+## Context Patterns
+
+### RAG Integration Pattern
+```
+[System Instructions]
+You are a helpful assistant. Use the provided context to answer questions.
+Only use information from the context. If unsure, say so.
+
+[Retrieved Context]
+<document source="doc1.pdf" date="2024-01-15">
+  Content here...
+</document>
+
+[User Query]
+{user_input}
+```
+
+### Multi-Turn Context Pattern
+```
+[System Instructions]
+...
+
+[Conversation History]
+Summary of earlier turns: {summary}
+
+Recent exchanges:
+User: {recent_user_1}
+Assistant: {recent_assistant_1}
+User: {recent_user_2}
+Assistant: {recent_assistant_2}
+
+[Current Turn]
+User: {current_input}
+```
+
+## Metrics
+
+- **Context Utilization Rate**: How much of provided context is used in response
+- **Context Relevance Score**: Semantic similarity between context and response
+- **Context Compression Ratio**: Original size vs. compressed size
+- **Information Retention**: Key facts preserved after summarization
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/skills/index.md

@@ -0,0 +1,79 @@
+# Skills Index
+
+Skills are foundational knowledge modules that inform the design and implementation of agents, tools, and prompts.
+
+## Available Skills
+
+### LLM Evaluator
+**Path**: `skills/llm-evaluator/llm-evaluator.md`
+
+Covers LLM-as-a-Judge evaluation methodology including:
+- Scoring approaches (direct, pairwise, reference-based)
+- Evaluation metrics (classification, correlation)
+- Known biases and mitigation strategies
+- Implementation patterns
+
+**Key Takeaways**:
+- Use direct scoring for objective evaluations
+- Use pairwise comparison for subjective preferences
+- Always mitigate position bias
+- Prefer classification metrics for interpretability
+
+### Context Fundamentals
+**Path**: `skills/context-fundamentals/context-fundamentals.md`
+
+Covers context engineering principles including:
+- Context window management
+- Information hierarchy
+- Context types (static, dynamic, ephemeral)
+- Relevance filtering
+
+**Key Takeaways**:
+- Structure context by priority
+- Be explicit over implicit
+- Remove redundancy
+- Signal freshness of information
+
+### Tool Design
+**Path**: `skills/tool-design/tool-design.md`
+
+Covers agent tool design best practices including:
+- Single responsibility principle
+- Input/output schemas
+- Error handling patterns
+- AI SDK 6 features (approval, strict mode, examples)
+
+**Key Takeaways**:
+- Clear, validated schemas
+- Predictable output structure
+- Graceful error handling
+- Consider approval for dangerous tools
+
+## Skill Application Matrix
+
+| Skill | Agents | Tools | Prompts |
+|-------|--------|-------|---------|
+| LLM Evaluator | Evaluator | directScore, pairwiseCompare | evaluation/* |
+| Context Fundamentals | All | All (context params) | All (context handling) |
+| Tool Design | All (tool selection) | All | orchestrator-prompt |
+
+## Adding New Skills
+
+1. Create skill directory: `skills/<skill-name>/`
+2. Create main file: `skills/<skill-name>/<skill-name>.md`
+3. Include:
+   - Overview and purpose
+   - Core principles
+   - Practical patterns
+   - Implementation examples
+   - References
+4. Update this index
+
+## Skill Development Guidelines
+
+- Focus on principles that transfer across implementations
+- Include concrete examples and patterns
+- Reference authoritative sources
+- Keep content actionable, not just theoretical
+- Update as understanding evolves
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/skills/llm-evaluator/llm-evaluator.md

@@ -0,0 +1,77 @@
+# LLM-Evaluator Skill
+
+## Overview
+
+LLM-Evaluators (LLM-as-a-Judge) are large language models designed to evaluate the quality of another LLM's response to an instruction or query. This skill provides the foundational knowledge for building evaluation systems.
+
+## Key Considerations
+
+### Baseline Selection
+- **Human Annotators**: Aim for LLM-human correlation to match human-human correlation. LLM-evaluators are orders of magnitude faster and cheaper than human annotation.
+- **Finetuned Classifiers**: Goal is to achieve similar recall and precision as a finetuned classifier. More challenging baseline as these are optimized for specific tasks.
+
+### Scoring Approaches
+
+| Approach | Use Case | Reliability |
+|----------|----------|-------------|
+| **Direct Scoring** | Objective tasks (factuality, toxicity, instruction-following) | More suitable for binary classification |
+| **Pairwise Comparison** | Subjective evaluations (tone, persuasiveness, coherence) | More reliable for preference tasks |
+| **Reference-Based** | Comparing against gold standard | Requires ground truth reference |
+
+### Evaluation Metrics
+
+**Classification Metrics** (Preferred for binary tasks):
+- Recall and Precision
+- F1 Score
+- Cohen's κ (Kappa)
+
+**Correlation Metrics** (For Likert scale tasks):
+- Spearman's ρ (rho)
+- Kendall's τ (tau)
+
+## Known Biases
+
+1. **Position Bias**: LLM-evaluators tend to prefer responses in certain positions during pairwise comparison (usually first position)
+2. **Verbosity Bias**: Favor longer, more verbose responses even if not higher quality
+3. **Self-Enhancement Bias**: LLM-evaluators prefer answers generated by themselves
+
+## Mitigation Strategies
+
+- Swap response positions and average results
+- Normalize for length when evaluating
+- Use a Panel of LLMs (PoLL) instead of single judge
+- Include "don't overthink" instructions
+- Use CoT + n-shot prompts for reliability
+
+## Implementation Pattern
+
+```typescript
+interface EvaluatorConfig {
+  scoringApproach: 'direct' | 'pairwise' | 'reference-based';
+  criteria: EvaluationCriteria[];
+  metrics: MetricType[];
+  useCoT: boolean;
+  nShot: number;
+}
+
+interface EvaluationCriteria {
+  name: string;
+  description: string;
+  rubric: RubricLevel[];
+}
+
+interface RubricLevel {
+  score: number;
+  description: string;
+}
+```
+
+## References
+
+Key papers reviewed:
+- Constitutional AI (Anthropic)
+- G-Eval: NLG Evaluation using GPT-4
+- SelfCheckGPT: Zero-Resource Hallucination Detection
+- Prometheus: Fine-grained Evaluation Capability
+- MT-Bench and Chatbot Arena
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/skills/tool-design/tool-design.md

@@ -0,0 +1,198 @@
+# Agent Tool Design Skill
+
+## Overview
+
+Tools are the foundation of an agent's capabilities. An agent's ability to take meaningful actions depends entirely on how reliably it can generate valid tool inputs, how well those inputs align with user intent, and how effectively tool outputs inform next steps.
+
+## Design Principles
+
+### 1. Single Responsibility
+
+Each tool should do one thing well. Complex operations should be composed from multiple tools.
+
+```typescript
+// Bad: Tool does too much
+const analyzeAndSummarizeAndSend = { ... }
+
+// Good: Separate concerns
+const analyzeDocument = { ... }
+const summarizeContent = { ... }
+const sendEmail = { ... }
+```
+
+### 2. Clear Input Schemas
+
+Use explicit, validated schemas with descriptive field names and constraints.
+
+```typescript
+const searchTool = tool({
+  description: "Search for documents by semantic similarity",
+  parameters: z.object({
+    query: z.string().describe("Natural language search query"),
+    limit: z.number().min(1).max(100).default(10)
+      .describe("Maximum number of results to return"),
+    filters: z.object({
+      dateAfter: z.string().optional()
+        .describe("ISO date string, only return docs after this date"),
+      source: z.enum(["internal", "external", "all"]).default("all")
+    }).optional()
+  }),
+  execute: async (input) => { ... }
+});
+```
+
+### 3. Predictable Output Structure
+
+Return consistent, typed output that the model can reliably parse.
+
+```typescript
+interface ToolResult<T> {
+  success: boolean;
+  data?: T;
+  error?: {
+    code: string;
+    message: string;
+    retryable: boolean;
+  };
+  metadata: {
+    executionTimeMs: number;
+    source?: string;
+  };
+}
+```
+
+### 4. Graceful Error Handling
+
+Tools should never throw unhandled exceptions. Always return structured errors.
+
+```typescript
+execute: async (input) => {
+  try {
+    const result = await performAction(input);
+    return { success: true, data: result };
+  } catch (error) {
+    return {
+      success: false,
+      error: {
+        code: error.code ?? "UNKNOWN_ERROR",
+        message: error.message,
+        retryable: isRetryable(error)
+      }
+    };
+  }
+}
+```
+
+## Tool Categories
+
+### Read-Only Tools
+- Database queries
+- API fetches
+- File reads
+- Search operations
+
+Safe to execute without approval. Return data but don't mutate state.
+
+### State-Modifying Tools
+- Database writes
+- File modifications
+- API POST/PUT/DELETE
+- System configuration changes
+
+May require human approval. Consider `needsApproval` flag.
+
+### Dangerous Tools
+- File deletion
+- Payment processing
+- Production deployments
+- Sending external communications
+
+Should always require approval and audit logging.
+
+## AI SDK 6 Tool Features
+
+### Tool Execution Approval
+```typescript
+const deleteTool = tool({
+  description: "Delete a file from the system",
+  parameters: z.object({
+    path: z.string()
+  }),
+  needsApproval: true, // Requires human approval
+  execute: async ({ path }) => { ... }
+});
+
+// Dynamic approval based on input
+const commandTool = tool({
+  description: "Execute a shell command",
+  parameters: z.object({
+    command: z.string()
+  }),
+  needsApproval: ({ command }) => {
+    return command.includes("rm") || command.includes("delete");
+  },
+  execute: async ({ command }) => { ... }
+});
+```
+
+### Strict Mode
+Enable native strict mode for guaranteed schema compliance:
+```typescript
+const strictTool = tool({
+  description: "...",
+  parameters: schema,
+  strict: true, // Enable strict mode
+  execute: async (input) => { ... }
+});
+```
+
+### Input Examples
+Help the model understand expected input format:
+```typescript
+const complexTool = tool({
+  description: "Create a calendar event",
+  parameters: eventSchema,
+  inputExamples: [
+    {
+      title: "Team Standup",
+      date: "2024-01-15",
+      time: "09:00",
+      duration: 30,
+      attendees: ["alice@example.com", "bob@example.com"]
+    }
+  ],
+  execute: async (input) => { ... }
+});
+```
+
+### toModelOutput
+Control what gets sent back to the model:
+```typescript
+const readFileTool = tool({
+  description: "Read file contents",
+  parameters: z.object({ path: z.string() }),
+  execute: async ({ path }) => {
+    const content = await fs.readFile(path, 'utf-8');
+    return { path, content, size: content.length };
+  },
+  toModelOutput: (result) => {
+    // Only send truncated content to model
+    return {
+      path: result.path,
+      content: result.content.slice(0, 5000),
+      truncated: result.content.length > 5000
+    };
+  }
+});
+```
+
+## Best Practices
+
+1. **Descriptive Names**: Tool names should clearly indicate their function
+2. **Comprehensive Descriptions**: Include usage examples in tool descriptions
+3. **Reasonable Defaults**: Provide sensible defaults for optional parameters
+4. **Idempotency**: Design tools to be safely re-executable when possible
+5. **Timeout Handling**: Implement timeouts for external operations
+6. **Rate Limiting**: Protect against runaway tool execution
+7. **Logging**: Log all tool invocations for debugging and audit
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/src/agents/evaluator.ts

@@ -0,0 +1,112 @@
+import { openai } from '@ai-sdk/openai';
+import { generateText } from 'ai';
+import { config } from '../config/index.js';
+import { 
+  executeDirectScore, 
+  executePairwiseCompare, 
+  executeGenerateRubric,
+  type DirectScoreInput,
+  type PairwiseCompareInput,
+  type GenerateRubricInput
+} from '../tools/evaluation/index.js';
+
+export interface EvaluatorAgentConfig {
+  model?: string;
+  temperature?: number;
+  maxTokens?: number;
+}
+
+export class EvaluatorAgent {
+  private model: string;
+  private temperature: number;
+
+  constructor(agentConfig?: EvaluatorAgentConfig) {
+    this.model = agentConfig?.model || config.openai.model;
+    this.temperature = agentConfig?.temperature || 0.3;
+  }
+
+  /**
+   * Score a response against defined criteria
+   */
+  async score(input: DirectScoreInput) {
+    return executeDirectScore(input);
+  }
+
+  /**
+   * Compare two responses and pick the better one
+   */
+  async compare(input: PairwiseCompareInput) {
+    return executePairwiseCompare(input);
+  }
+
+  /**
+   * Generate a rubric for a criterion
+   */
+  async generateRubric(input: GenerateRubricInput) {
+    return executeGenerateRubric(input);
+  }
+
+  /**
+   * Full evaluation workflow: generate rubric, then score
+   */
+  async evaluateWithGeneratedRubric(
+    response: string,
+    prompt: string,
+    criteria: Array<{ name: string; description: string; weight?: number }>
+  ) {
+    // Generate rubrics for each criterion
+    const rubrics = await Promise.all(
+      criteria.map(c => this.generateRubric({
+        criterionName: c.name,
+        criterionDescription: c.description,
+        scale: '1-5',
+        includeExamples: false,
+        strictness: 'balanced'
+      }))
+    );
+
+    // Build combined rubric
+    const levelDescriptions: Record<string, string> = {};
+    rubrics[0]?.levels?.forEach(level => {
+      levelDescriptions[String(level.score)] = level.description;
+    });
+
+    // Score using generated rubric
+    return this.score({
+      response,
+      prompt,
+      criteria: criteria.map((c) => ({
+        name: c.name,
+        description: c.description,
+        weight: c.weight || 1
+      })),
+      rubric: {
+        scale: '1-5',
+        levelDescriptions
+      }
+    });
+  }
+
+  /**
+   * Chat-based evaluation for custom queries
+   */
+  async chat(userMessage: string) {
+    const result = await generateText({
+      model: openai(this.model),
+      system: `You are an expert evaluator of AI-generated content.
+Your role is to assess quality, identify issues, and provide actionable feedback.
+Be objective, specific, and constructive in your evaluations.`,
+      prompt: userMessage,
+      temperature: this.temperature
+    });
+
+    return {
+      text: result.text,
+      usage: result.usage
+    };
+  }
+}
+
+// Default instance
+export const evaluatorAgent = new EvaluatorAgent();
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/src/agents/index.ts

@@ -0,0 +1,3 @@
+export { EvaluatorAgent, evaluatorAgent } from './evaluator.js';
+export type { EvaluatorAgentConfig } from './evaluator.js';
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/src/config/index.ts

@@ -0,0 +1,18 @@
+import 'dotenv/config';
+
+export const config = {
+  openai: {
+    apiKey: process.env.OPENAI_API_KEY || '',
+    model: process.env.OPENAI_MODEL || 'gpt-4o'
+  },
+  anthropic: {
+    apiKey: process.env.ANTHROPIC_API_KEY || ''
+  }
+} as const;
+
+export function validateConfig(): void {
+  if (!config.openai.apiKey) {
+    throw new Error('OPENAI_API_KEY is required. Create a .env file with your API key.');
+  }
+}
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/llm-as-judge-skills/src/index.ts

@@ -0,0 +1,19 @@
+// Configuration
+export { config, validateConfig } from './config/index.js';
+
+// Tools
+export * from './tools/evaluation/index.js';
+
+// Agents
+export * from './agents/index.js';
+
+// Re-export types for convenience
+export type { 
+  DirectScoreInput, 
+  DirectScoreOutput,
+  PairwiseCompareInput,
+  PairwiseCompareOutput,
+  GenerateRubricInput,
+  GenerateRubricOutput
+} from './tools/evaluation/index.js';
+