agentv 1.6.1 → 2.0.2

package/README.md

@@ -59,6 +59,14 @@ bun run build
 bun test
 ```
 
+5. (Optional) Install example dependencies:
+
+```bash
+bun run examples:install
+```
+
+This step is required if you want to run the examples in the `examples/` directory, as they are self-contained packages with their own dependencies.
+
 You are now ready to start development. The monorepo contains:
 
 - `packages/core/` - Core evaluation engine
@@ -77,9 +85,8 @@ You are now ready to start development. The monorepo contains:
 
 ## Quick Start
 
-You can use the following examples as a starting point.
-- [Simple Example](docs/examples/simple/README.md): A minimal working example to help you get started fast.
-- [Showcase](docs/examples/showcase/README.md): A collection of advanced use cases and real-world agent evaluation scenarios.
+You can use the following examples as a starting point:
+- [Examples](examples/README.md): Feature demonstrations and real-world showcase examples
 
 ### Validating Eval Files
 
@@ -129,9 +136,6 @@ agentv eval --target vscode_projectx --targets "path/to/targets.yaml" --eval-id
 - `--agent-timeout SECONDS`: Timeout in seconds for agent response polling (default: 120)
 - `--max-retries COUNT`: Maximum number of retries for timeout cases (default: 2)
 - `--cache`: Enable caching of LLM responses (default: disabled)
-- `--dump-prompts`: Save all prompts to `.agentv/prompts/` directory
-- `--dump-traces`: Write trace files to `.agentv/traces/` directory
-- `--include-trace`: Include full trace in result output (verbose)
 - `--workers COUNT`: Parallel workers for eval cases (default: 3; target `workers` setting used when provided)
 - `--verbose`: Verbose output
 
@@ -297,45 +301,13 @@ Code evaluators receive input via stdin and write output to stdout as JSON.
 - Score range: `0.0` to `1.0` (float)
 - `hits` and `misses` are optional but recommended for debugging
 
-### Code Evaluator Script Template
-
-```python
-#!/usr/bin/env python3
-import json
-import sys
-
-def evaluate(input_data):
-    # Extract only the fields you need
-    candidate_answer = input_data.get("candidate_answer", "")
-    
-    # Your validation logic here
-    score = 0.0  # to 1.0
-    hits = ["successful check 1", "successful check 2"]
-    misses = ["failed check 1"]
-    reasoning = "Explanation of score"
-    
-    return {
-        "score": score,
-        "hits": hits,
-        "misses": misses,
-        "reasoning": reasoning
-    }
-
-if __name__ == "__main__":
-    try:
-        input_data = json.loads(sys.stdin.read())
-        result = evaluate(input_data)
-        print(json.dumps(result, indent=2))
-    except Exception as e:
-        error_result = {
-            "score": 0.0,
-            "hits": [],
-            "misses": [f"Evaluator error: {str(e)}"],
-            "reasoning": f"Evaluator error: {str(e)}"
-        }
-        print(json.dumps(error_result, indent=2))
-        sys.exit(1)
-```
+### Code Evaluator Templates
+
+Custom evaluators can be written in any language. For complete templates and examples:
+
+- **Python template**: See `apps/cli/src/templates/.claude/skills/agentv-eval-builder/references/custom-evaluators.md`
+- **TypeScript template (with SDK)**: See `apps/cli/src/templates/.claude/skills/agentv-eval-builder/references/custom-evaluators.md`
+- **Working examples**: See [examples/features/code-judge-sdk](examples/features/code-judge-sdk)
 
 ### LLM Judge Template Structure
 
@@ -408,7 +380,7 @@ agentv generate rubrics evals/my-eval.yaml --target openai:gpt-4o
   - `borderline`: Score ≥ 0.6 and all required rubrics met
   - `fail`: Score < 0.6 or any required rubric failed
 
-For complete examples and detailed patterns, see [examples/features/evals/rubric/](examples/features/evals/rubric/).
+For complete examples and detailed patterns, see [examples/features/rubric/](examples/features/rubric/).
 
 ## Advanced Configuration