agentv 1.5.0 → 2.0.1

package/dist/templates/.claude/skills/agentv-eval-builder/references/composite-evaluator.md

@@ -1,215 +1,215 @@
-# Composite Evaluator Guide
-
-Composite evaluators combine multiple evaluators and aggregate their results. This enables sophisticated evaluation patterns like safety gates, weighted scoring, and conflict resolution.
-
-## Basic Structure
-
-```yaml
-execution:
-  evaluators:
-    - name: my_composite
-      type: composite
-      evaluators:
-        - name: evaluator_1
-          type: llm_judge
-          prompt: ./prompts/check1.md
-        - name: evaluator_2
-          type: code_judge
-          script: uv run check2.py
-      aggregator:
-        type: weighted_average
-        weights:
-          evaluator_1: 0.6
-          evaluator_2: 0.4
-```
-
-## Aggregator Types
-
-### 1. Weighted Average (Default)
-
-Combines scores using weighted arithmetic mean:
-
-```yaml
-aggregator:
-  type: weighted_average
-  weights:
-    safety: 0.3      # 30% weight
-    quality: 0.7     # 70% weight
-```
-
-If weights are omitted, all evaluators have equal weight (1.0).
-
-**Score calculation:**
-```
-final_score = Σ(score_i × weight_i) / Σ(weight_i)
-```
-
-### 2. Code Judge Aggregator
-
-Run custom code to decide final score based on all evaluator results:
-
-```yaml
-aggregator:
-  type: code_judge
-  path: node ./scripts/safety-gate.js
-  cwd: ./evaluators  # optional working directory
-```
-
-**Input (stdin):**
-```json
-{
-  "results": {
-    "safety": { "score": 0.9, "hits": [...], "misses": [...] },
-    "quality": { "score": 0.85, "hits": [...], "misses": [...] }
-  }
-}
-```
-
-**Output (stdout):**
-```json
-{
-  "score": 0.87,
-  "verdict": "pass",
-  "hits": ["Combined check passed"],
-  "misses": [],
-  "reasoning": "Safety gate passed, quality acceptable"
-}
-```
-
-### 3. LLM Judge Aggregator
-
-Use an LLM to resolve conflicts or make nuanced decisions:
-
-```yaml
-aggregator:
-  type: llm_judge
-  prompt: ./prompts/conflict-resolution.md
-```
-
-The `{{EVALUATOR_RESULTS_JSON}}` variable is replaced with the JSON results from all child evaluators.
-
-## Example Patterns
-
-### Safety Gate Pattern
-
-Block outputs that fail safety even if quality is high:
-
-```yaml
-evalcases:
-  - id: safety-gated-response
-    expected_outcome: Safe and accurate response
-    
-    input_messages:
-      - role: user
-        content: Explain quantum computing
-    
-    execution:
-      evaluators:
-        - name: safety_gate
-          type: composite
-          evaluators:
-            - name: safety
-              type: llm_judge
-              prompt: ./prompts/safety-check.md
-            - name: quality
-              type: llm_judge
-              prompt: ./prompts/quality-check.md
-          aggregator:
-            type: code_judge
-            path: ./scripts/safety-gate.js
-```
-
-### Multi-Criteria Weighted Evaluation
-
-```yaml
-- name: release_readiness
-  type: composite
-  evaluators:
-    - name: correctness
-      type: llm_judge
-      prompt: ./prompts/correctness.md
-    - name: style
-      type: code_judge
-      script: uv run style_checker.py
-    - name: security
-      type: llm_judge
-      prompt: ./prompts/security.md
-  aggregator:
-    type: weighted_average
-    weights:
-      correctness: 0.5
-      style: 0.2
-      security: 0.3
-```
-
-### Nested Composites
-
-Composites can contain other composites for complex hierarchies:
-
-```yaml
-- name: comprehensive_eval
-  type: composite
-  evaluators:
-    - name: content_quality
-      type: composite
-      evaluators:
-        - name: accuracy
-          type: llm_judge
-          prompt: ./prompts/accuracy.md
-        - name: clarity
-          type: llm_judge
-          prompt: ./prompts/clarity.md
-      aggregator:
-        type: weighted_average
-        weights:
-          accuracy: 0.6
-          clarity: 0.4
-    - name: safety
-      type: llm_judge
-      prompt: ./prompts/safety.md
-  aggregator:
-    type: weighted_average
-    weights:
-      content_quality: 0.7
-      safety: 0.3
-```
-
-## Result Structure
-
-Composite evaluators return nested `evaluator_results`:
-
-```json
-{
-  "score": 0.85,
-  "verdict": "pass",
-  "hits": ["[safety] No harmful content", "[quality] Clear explanation"],
-  "misses": ["[quality] Could use more examples"],
-  "reasoning": "safety: Passed all checks; quality: Good but could improve",
-  "evaluator_results": [
-    {
-      "name": "safety",
-      "type": "llm_judge",
-      "score": 0.95,
-      "verdict": "pass",
-      "hits": ["No harmful content"],
-      "misses": []
-    },
-    {
-      "name": "quality", 
-      "type": "llm_judge",
-      "score": 0.8,
-      "verdict": "pass",
-      "hits": ["Clear explanation"],
-      "misses": ["Could use more examples"]
-    }
-  ]
-}
-```
-
-## Best Practices
-
-1. **Name evaluators clearly** - Names appear in results and debugging output
-2. **Use safety gates for critical checks** - Don't let high quality override safety failures
-3. **Balance weights thoughtfully** - Consider which aspects matter most for your use case
-4. **Keep nesting shallow** - Deep nesting makes debugging harder
-5. **Test aggregators independently** - Verify your custom aggregation logic with unit tests
+# Composite Evaluator Guide
+
+Composite evaluators combine multiple evaluators and aggregate their results. This enables sophisticated evaluation patterns like safety gates, weighted scoring, and conflict resolution.
+
+## Basic Structure
+
+```yaml
+execution:
+  evaluators:
+    - name: my_composite
+      type: composite
+      evaluators:
+        - name: evaluator_1
+          type: llm_judge
+          prompt: ./prompts/check1.md
+        - name: evaluator_2
+          type: code_judge
+          script: uv run check2.py
+      aggregator:
+        type: weighted_average
+        weights:
+          evaluator_1: 0.6
+          evaluator_2: 0.4
+```
+
+## Aggregator Types
+
+### 1. Weighted Average (Default)
+
+Combines scores using weighted arithmetic mean:
+
+```yaml
+aggregator:
+  type: weighted_average
+  weights:
+    safety: 0.3      # 30% weight
+    quality: 0.7     # 70% weight
+```
+
+If weights are omitted, all evaluators have equal weight (1.0).
+
+**Score calculation:**
+```
+final_score = Σ(score_i × weight_i) / Σ(weight_i)
+```
+
+### 2. Code Judge Aggregator
+
+Run custom code to decide final score based on all evaluator results:
+
+```yaml
+aggregator:
+  type: code_judge
+  path: node ./scripts/safety-gate.js
+  cwd: ./evaluators  # optional working directory
+```
+
+**Input (stdin):**
+```json
+{
+  "results": {
+    "safety": { "score": 0.9, "hits": [...], "misses": [...] },
+    "quality": { "score": 0.85, "hits": [...], "misses": [...] }
+  }
+}
+```
+
+**Output (stdout):**
+```json
+{
+  "score": 0.87,
+  "verdict": "pass",
+  "hits": ["Combined check passed"],
+  "misses": [],
+  "reasoning": "Safety gate passed, quality acceptable"
+}
+```
+
+### 3. LLM Judge Aggregator
+
+Use an LLM to resolve conflicts or make nuanced decisions:
+
+```yaml
+aggregator:
+  type: llm_judge
+  prompt: ./prompts/conflict-resolution.md
+```
+
+The `{{EVALUATOR_RESULTS_JSON}}` variable is replaced with the JSON results from all child evaluators.
+
+## Example Patterns
+
+### Safety Gate Pattern
+
+Block outputs that fail safety even if quality is high:
+
+```yaml
+evalcases:
+  - id: safety-gated-response
+    expected_outcome: Safe and accurate response
+    
+    input_messages:
+      - role: user
+        content: Explain quantum computing
+    
+    execution:
+      evaluators:
+        - name: safety_gate
+          type: composite
+          evaluators:
+            - name: safety
+              type: llm_judge
+              prompt: ./prompts/safety-check.md
+            - name: quality
+              type: llm_judge
+              prompt: ./prompts/quality-check.md
+          aggregator:
+            type: code_judge
+            path: ./scripts/safety-gate.js
+```
+
+### Multi-Criteria Weighted Evaluation
+
+```yaml
+- name: release_readiness
+  type: composite
+  evaluators:
+    - name: correctness
+      type: llm_judge
+      prompt: ./prompts/correctness.md
+    - name: style
+      type: code_judge
+      script: uv run style_checker.py
+    - name: security
+      type: llm_judge
+      prompt: ./prompts/security.md
+  aggregator:
+    type: weighted_average
+    weights:
+      correctness: 0.5
+      style: 0.2
+      security: 0.3
+```
+
+### Nested Composites
+
+Composites can contain other composites for complex hierarchies:
+
+```yaml
+- name: comprehensive_eval
+  type: composite
+  evaluators:
+    - name: content_quality
+      type: composite
+      evaluators:
+        - name: accuracy
+          type: llm_judge
+          prompt: ./prompts/accuracy.md
+        - name: clarity
+          type: llm_judge
+          prompt: ./prompts/clarity.md
+      aggregator:
+        type: weighted_average
+        weights:
+          accuracy: 0.6
+          clarity: 0.4
+    - name: safety
+      type: llm_judge
+      prompt: ./prompts/safety.md
+  aggregator:
+    type: weighted_average
+    weights:
+      content_quality: 0.7
+      safety: 0.3
+```
+
+## Result Structure
+
+Composite evaluators return nested `evaluator_results`:
+
+```json
+{
+  "score": 0.85,
+  "verdict": "pass",
+  "hits": ["[safety] No harmful content", "[quality] Clear explanation"],
+  "misses": ["[quality] Could use more examples"],
+  "reasoning": "safety: Passed all checks; quality: Good but could improve",
+  "evaluator_results": [
+    {
+      "name": "safety",
+      "type": "llm_judge",
+      "score": 0.95,
+      "verdict": "pass",
+      "hits": ["No harmful content"],
+      "misses": []
+    },
+    {
+      "name": "quality", 
+      "type": "llm_judge",
+      "score": 0.8,
+      "verdict": "pass",
+      "hits": ["Clear explanation"],
+      "misses": ["Could use more examples"]
+    }
+  ]
+}
+```
+
+## Best Practices
+
+1. **Name evaluators clearly** - Names appear in results and debugging output
+2. **Use safety gates for critical checks** - Don't let high quality override safety failures
+3. **Balance weights thoughtfully** - Consider which aspects matter most for your use case
+4. **Keep nesting shallow** - Deep nesting makes debugging harder
+5. **Test aggregators independently** - Verify your custom aggregation logic with unit tests

package/dist/templates/.claude/skills/agentv-eval-builder/references/custom-evaluators.md

@@ -8,20 +8,47 @@ Code evaluators receive input via stdin and write output to stdout, both as JSON
 
 ### Input Format (via stdin)
 
+Wire format uses snake_case for cross-language compatibility:
+
 ```json
 {
   "question": "string describing the task/question",
   "expected_outcome": "expected outcome description",
   "reference_answer": "gold standard answer (optional)",
   "candidate_answer": "generated code/text from the agent",
-  "guideline_paths": ["path1", "path2"],
+  "guideline_files": ["path1", "path2"],
   "input_files": ["file1", "file2"],
   "input_messages": [{"role": "user", "content": "..."}],
-  "output_messages": [{"role": "assistant", "content": "...", "tool_calls": [...]}]
+  "output_messages": [
+    {
+      "role": "assistant",
+      "content": "...",
+      "tool_calls": [
+        {
+          "tool": "search",
+          "input": { "query": "..." },
+          "output": { "results": [...] },
+          "id": "call_123",
+          "timestamp": "2024-01-15T10:30:00Z"
+        }
+      ]
+    }
+  ],
+  "trace_summary": {
+    "event_count": 5,
+    "tool_names": ["fetch", "search"],
+    "tool_calls_by_name": { "search": 2, "fetch": 1 },
+    "error_count": 0,
+    "token_usage": { "input": 1000, "output": 500 },
+    "cost_usd": 0.0015,
+    "duration_ms": 3500
+  }
 }
 ```
 
-The `output_messages` array contains the full agent execution trace with tool calls, enabling custom validation of agent behavior.
+**Key fields:**
+- `output_messages` - Full agent execution trace with tool calls (use `tool_calls[].input` for arguments)
+- `trace_summary` - Lightweight summary with execution metrics (counts only, no tool arguments)
 
 ### Output Format (to stdout)
 
@@ -125,6 +152,96 @@ if __name__ == "__main__":
     main()
 ```
 
+## TypeScript Code Evaluator Template (with SDK)
+
+The optional `@agentv/core` SDK provides type-safe payload parsing with camelCase properties (`candidateAnswer` vs `candidate_answer`).
+
+**Execution:** Keep evaluators as `.ts` files and run via Node loaders like `npx --yes tsx ./evaluators/my-check.ts` so users don't need Bun after `npm install -g agentv`.
+
+**Without SDK:** Skip the import and parse JSON from stdin directly (similar to the Python template above).
+
+```typescript
+/**
+ * Example TypeScript code evaluator using the AgentV SDK
+ *
+ * Run with: npx --yes tsx ./evaluators/example-check.ts
+ *
+ * The SDK provides:
+ * - Type-safe CodeJudgePayload interface with all fields
+ * - camelCase properties (candidateAnswer, expectedOutcome, etc.)
+ * - Automatic conversion from snake_case wire format
+ */
+
+import { readCodeJudgePayload } from '@agentv/core';
+
+try {
+  // Read and parse stdin with automatic snake_case → camelCase conversion
+  const payload = readCodeJudgePayload();
+
+  // Type-safe camelCase access to all fields
+  const { candidateAnswer, expectedOutcome, inputFiles, guidelineFiles } = payload;
+
+  // Your validation logic here
+  const hits: string[] = [];
+  const misses: string[] = [];
+
+  // Example: Check if answer contains expected outcome
+  if (candidateAnswer.includes(expectedOutcome)) {
+    hits.push('Answer matches expected outcome');
+  } else {
+    misses.push('Answer does not match expected outcome');
+  }
+
+  // Example: Check attachment mentions
+  const attachments = [...guidelineFiles, ...inputFiles];
+  for (const filePath of attachments) {
+    const fileName = filePath.split('/').pop() ?? filePath;
+    if (candidateAnswer.includes(fileName)) {
+      hits.push(`Mentions attachment: ${fileName}`);
+    } else {
+      misses.push(`Missing attachment: ${fileName}`);
+    }
+  }
+
+  // Calculate score
+  const totalChecks = hits.length + misses.length;
+  const score = totalChecks === 0 ? 0 : hits.length / totalChecks;
+
+  // Build result
+  const result = {
+    score,
+    hits,
+    misses,
+    reasoning: `Passed ${hits.length}/${totalChecks} checks`
+  };
+
+  console.log(JSON.stringify(result, null, 2));
+
+} catch (error) {
+  const message = error instanceof Error ? error.message : String(error);
+  console.log(JSON.stringify({
+    score: 0,
+    hits: [],
+    misses: [`Error: ${message}`],
+    reasoning: 'Evaluator error'
+  }, null, 2));
+  process.exit(1);
+}
+```
+
+**TypeScript SDK Benefits:**
+- **Type-safe**: `CodeJudgePayload` interface with all fields typed
+- **camelCase**: Idiomatic TypeScript naming (`candidateAnswer` vs `candidate_answer`)
+- **Automatic conversion**: Handles snake_case wire format → camelCase objects
+- **Compile-time safety**: Catch typos and missing fields before runtime
+
+**Available in SDK:**
+- `readCodeJudgePayload()`: Read stdin and convert to camelCase (recommended)
+- `parseCodeJudgePayload(jsonString)`: Parse JSON string and convert to camelCase
+- `CodeJudgePayload`: TypeScript interface for type safety
+
+**See also:** `examples/features/code-judge-sdk/` for complete working examples
+
 ## LLM Judge Prompt Template
 
 LLM judges use markdown prompts to guide evaluation. AgentV automatically handles the output format, so focus your prompt on evaluation criteria and guidelines.
@@ -189,11 +306,22 @@ You can customize this template in your eval file using the `evaluatorTemplate`
 execution:
   evaluators:
     - name: my_validator
-      type: code
+      type: code_judge
       script: uv run my_validator.py
       cwd: ./evaluators
 ```
 
+TypeScript evaluators use the same structure but invoke `tsx` (or another Node-compatible loader) so they work everywhere:
+
+```yaml
+execution:
+  evaluators:
+    - name: csv_guardrail
+      type: code_judge
+      script: npx --yes tsx ./evaluators/check-csv.ts
+      cwd: ./evaluators
+```
+
 ### Command Line Testing
 
 Test your evaluator locally:
@@ -214,3 +342,12 @@ echo '{
 #   "reasoning": "..."
 # }
 ```
+
+```bash
+# TypeScript (uses tsx loader under Node)
+echo '{
+  "candidate_answer": "test output here",
+  "question": "test task",
+  "expected_outcome": "expected result"
+}' | npx --yes tsx ./evaluators/check-csv.ts
+```

package/dist/templates/.claude/skills/agentv-eval-builder/references/eval-schema.json

@@ -4,11 +4,6 @@
   "description": "Schema for YAML evaluation files with conversation flows, multiple evaluators, and execution configuration",
   "type": "object",
   "properties": {
-    "$schema": {
-      "type": "string",
-      "description": "Schema identifier",
-      "enum": ["agentv-eval-v2"]
-    },
     "description": {
       "type": "string",
       "description": "Description of what this eval suite covers"
@@ -37,7 +32,16 @@
               },
               "type": {
                 "type": "string",
-                "enum": ["code", "llm_judge"],
+                "enum": [
+                  "code",
+                  "llm_judge",
+                  "composite",
+                  "tool_trajectory",
+                  "field_accuracy",
+                  "latency",
+                  "cost",
+                  "token_usage"
+                ],
                 "description": "Evaluator type: 'code' for scripts/regex/keywords, 'llm_judge' for LLM-based evaluation"
               },
               "script": {