ciagent 0.4.2__tar.gz → 0.5.1__tar.gz

ciagent-0.5.1/CHANGELOG.md

@@ -0,0 +1,87 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.5.0] - 2026-03-01
+
+### Added
+
+#### Three-Layer Evaluation Engine
+- **Correctness layer** (Layer 1 — hard fail): keyword matching, LLM-as-a-judge, safety checks, hallucination checks, regex/exact match, JSON schema validation
+- **Path layer** (Layer 2 — soft warn): tool trajectory validation, loop detection (default `max_loops=3`), routing assertions, handoff expectations
+- **Cost layer** (Layer 3 — soft warn): token budgets, cost caps, LLM call limits, latency thresholds
+- `runner.py` orchestrates all three layers per query
+- `parallel.py` for parallel query execution across specs
+
+#### OR-Logic Keywords
+- `any_expected_in_answer` field — at least one keyword must match (complementing `expected_in_answer` which requires all)
+
+#### LLM Judge Enhancements
+- `context_file` support in `JudgeRubric` — doc-grounded judging against reference documents
+- `refutes_premise` flag — injects built-in premise-correction rubric for trick questions
+
+#### Span Assertions
+- `SpanAssertionSpec` schema model for span-level assertions
+- `Span.attributes: dict[str, Any]` — OTel-style span-level data propagation
+- Span-level LLM judge support
+
+#### Mock Testing Mode
+- `agentci test --mock` — generates synthetic traces, zero API cost
+- `mock_runner.py` — synthetic trace generation from spec expectations
+- `--golden-file` flag — load Q&A pairs from JSON/CSV for mock mode
+
+#### Cost Estimator
+- `cost_estimator.py` — pre-execution cost estimates with pricing table
+- Cost estimate shown before live test runs; `--yes`/`-y` skips confirmation
+
+#### CLI Improvements
+- `agentci init --generate` — AI-assisted spec generation with guided interview
+- `agentci doctor` — health-check command (spec, runner, API keys, deps, CI)
+- Scan-first flow: auto-scan project before questions, show summary
+- Agent type auto-detected via `_detect_agent_type_from_code()`
+- Skeleton template generation with TODO placeholders for zero-API-key usage
+- Context-aware "Next Steps" based on mode (mock vs live)
+- Non-interactive flags: `--kb-path`, `--mode`
+
+#### Diff Engine
+- Three-tier diff engine for baseline comparison
+- 11 `DiffType` categories including `ROUTING_CHANGED`, `GUARDRAILS_CHANGED`
+- `agentci diff` CLI command
+
+#### Reporting
+- GitHub annotations with budget cap (`MAX_INLINE_ANNOTATIONS = 10` for warnings; errors uncapped)
+- JSON output format
+- Prometheus metrics export
+
+#### Trace Helpers
+- `Trace.called(tool)` / `never_called(tool)` / `loop_count(tool)` — readable assertion helpers
+- `Trace.cost_under(usd)` / `llm_calls_under(n)` — budget assertion helpers
+- `langgraph_trace(agent_name)` — context manager shortcut for LangGraph
+- `TraceContext.attach(state)` — alias for `attach_langgraph_state()`
+
+#### Adapters
+- OpenAI Agents SDK adapter (`openai_agents.py`)
+- LangGraph adapter (`langgraph.py`)
+
+#### Other
+- `python-dotenv` added as core dependency
+- Deep KB sampling: 2000 chars/file for spec generation
+- Progressive spec building: smoke queries (3) then full queries (10-12)
+- Pytest plugin entry point (`pytest11: agentci`)
+- GUARDRAIL span type, HANDOFF span type
+
+### Changed
+- `PathSpec.max_loops` now defaults to `3` (was `None`)
+- Development status upgraded from Alpha to Beta
+- Package version bumped to 0.5.0
+
+## [0.4.1] - 2026-02-20
+
+### Added
+- Initial project structure and core models
+- Basic trace capture and assertion framework
+- CLI scaffolding with `agentci init`
+- PyPI publishing as `ciagent`

{ciagent-0.4.2 → ciagent-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ciagent
-Version: 0.4.2
+Version: 0.5.1
 Summary: Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production.
 Project-URL: Documentation, https://github.com/agentci-org/agentci#readme
 Project-URL: Repository, https://github.com/agentci-org/agentci.git
@@ -8,7 +8,7 @@ Project-URL: Issues, https://github.com/agentci-org/agentci/issues
 License: Apache-2.0
 License-File: LICENSE
 Keywords: agents,ai,ci-cd,llm,regression,testing
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Framework :: Pytest
 Classifier: Intended Audience :: Developers
 Classifier: Topic :: Software Development :: Testing
@@ -23,7 +23,10 @@ Requires-Dist: python-dotenv>=0.19
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: rich>=13.0
 Provides-Extra: all
-Requires-Dist: agentci[anthropic,jsonschema,langgraph,openai]; extra == 'all'
+Requires-Dist: anthropic>=0.20; extra == 'all'
+Requires-Dist: jsonschema>=4.0; extra == 'all'
+Requires-Dist: langgraph>=0.1; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
 Provides-Extra: anthropic
 Requires-Dist: anthropic>=0.20; extra == 'anthropic'
 Provides-Extra: dev
@@ -219,11 +222,87 @@ Clone the [DemoAgents](https://github.com/agentci-org/DemoAgents) repo and try t
 ## CLI
 
 ```bash
-agentci init         # Generate CI config (GitHub Actions + pre-push hooks)
-agentci run          # Run all tests (pytest-compatible)
-agentci record       # Record golden baseline traces
-agentci diff         # Diff current run against baseline
-agentci report       # Generate HTML report from results
+# ── Setup & Scaffolding ──────────────────────────────────────────────
+agentci init              # Scaffold GitHub Actions + pre-push hooks
+agentci init --generate   # Guided interview: auto-scan project, generate agentci_spec.yaml
+agentci init --generate --mode mock --golden-file qa.json  # Zero-API-key spec generation
+agentci doctor            # Health check: spec, deps, API keys, KB, CI workflow
+agentci validate          # Validate agentci_spec.yaml against schema (no execution)
+agentci bootstrap         # Quick setup from a queries file + runner path
+
+# ── Testing & Evaluation ─────────────────────────────────────────────
+agentci test              # Run 3-layer evaluation (Correctness → Path → Cost)
+agentci test --mock       # Zero-cost synthetic traces — no API keys needed
+agentci test --yes        # Skip cost-estimate confirmation (CI-friendly)
+agentci test --workers 4  # Parallel execution
+agentci eval              # Standalone correctness evaluation (no golden baselines)
+
+# ── Golden Baselines ─────────────────────────────────────────────────
+agentci record            # Record a golden baseline trace from a live run
+agentci save --agent my-agent --version v1 --trace-file trace.json  # Save versioned baseline
+agentci baselines --agent my-agent  # List saved baseline versions
+agentci diff --agent my-agent --baseline v1 --compare v2  # Diff two versions
+
+# ── Reporting ─────────────────────────────────────────────────────────
+agentci run               # Legacy test runner (pytest-compatible)
+agentci report            # Generate HTML report from results JSON
+```
+
+## 3-Layer Evaluation Engine
+
+`agentci test` evaluates every query through three layers:
+
+| Layer | Severity | Purpose | Example |
+|-------|----------|---------|---------|
+| **Correctness** | Hard FAIL (blocks CI) | Answer quality | `expected_in_answer`, `llm_judge`, `not_in_answer`, `regex_match`, `json_schema` |
+| **Path** | Soft WARN (annotation only) | Tool trajectory | `expected_tools`, `min_tool_recall`, `max_loops` |
+| **Cost** | Soft WARN (annotation only) | Efficiency budget | `max_llm_calls`, `max_tokens`, `max_cost_usd` |
+
+Layer 1 failures block CI (exit code 1). Layers 2-3 produce GitHub annotations but let CI pass.
+
+```yaml
+# agentci_spec.yaml — one query with all three layers
+queries:
+  - query: "How do I install AgentCI?"
+    correctness:
+      any_expected_in_answer: ["pip install", "ciagent"]
+      llm_judge:
+        - rule: "Response provides clear installation instructions"
+          threshold: 0.7
+    path:
+      expected_tools: [retrieve_docs]
+      min_tool_recall: 1.0
+    cost:
+      max_llm_calls: 8
+```
+
+## Mock Mode
+
+Test your spec structure with zero API keys and zero cost:
+
+```bash
+agentci test --mock
+```
+
+Mock mode generates synthetic traces that match your spec expectations. Use it to:
+- Validate spec structure before spending on live runs
+- Run in CI without API keys for fast feedback
+- Iterate on assertions without waiting for LLM responses
+
+## Cost Estimation
+
+Before live test runs, AgentCI shows a cost estimate:
+
+```bash
+$ agentci test
+Estimated cost: $0.12 – $0.35 (12 queries)
+Proceed? [y/N] y
+```
+
+Skip the prompt in CI with `--yes` or `-y`:
+
+```bash
+agentci test --yes  # no confirmation prompt
 ```
 
 ## CI/CD
@@ -298,13 +377,18 @@ When you diff a current run against a golden baseline, AgentCI flags these chang
 
 ## Status
 
-AgentCI is in **early release** (v0.1.x). The core trace model, diffing engine, assertion library, mock system, and CLI are stable. The API may evolve based on community feedback.
+AgentCI is in **beta** (v0.5.0). The core trace model, diffing engine, 3-layer evaluation engine, mock system, and CLI are stable. The API may evolve based on community feedback.
 
 **What's here today:**
+- 3-layer evaluation engine (Correctness / Path / Cost)
 - Trace capture for LangGraph, Anthropic, and OpenAI Agents SDK
-- Golden baseline recording and regression diffing
+- Golden baseline recording, versioned saves, and regression diffing
 - LLM mocking for all three frameworks (zero API keys)
-- LLM-as-judge assertions for subjective output grading
+- LLM-as-judge with composite rubrics, ensemble judging, and doc-grounded evaluation
+- Mock mode (`agentci test --mock`) for zero-cost spec validation
+- Guided spec generation (`agentci init --generate`) with auto-scan and KB sampling
+- Health check (`agentci doctor`) for environment validation
+- Cost estimation before live runs
 - Pytest plugin for native integration
 - GitHub Actions generator via `agentci init`
 

{ciagent-0.4.2 → ciagent-0.5.1}/README.md

@@ -179,11 +179,87 @@ Clone the [DemoAgents](https://github.com/agentci-org/DemoAgents) repo and try t
 ## CLI
 
 ```bash
-agentci init         # Generate CI config (GitHub Actions + pre-push hooks)
-agentci run          # Run all tests (pytest-compatible)
-agentci record       # Record golden baseline traces
-agentci diff         # Diff current run against baseline
-agentci report       # Generate HTML report from results
+# ── Setup & Scaffolding ──────────────────────────────────────────────
+agentci init              # Scaffold GitHub Actions + pre-push hooks
+agentci init --generate   # Guided interview: auto-scan project, generate agentci_spec.yaml
+agentci init --generate --mode mock --golden-file qa.json  # Zero-API-key spec generation
+agentci doctor            # Health check: spec, deps, API keys, KB, CI workflow
+agentci validate          # Validate agentci_spec.yaml against schema (no execution)
+agentci bootstrap         # Quick setup from a queries file + runner path
+
+# ── Testing & Evaluation ─────────────────────────────────────────────
+agentci test              # Run 3-layer evaluation (Correctness → Path → Cost)
+agentci test --mock       # Zero-cost synthetic traces — no API keys needed
+agentci test --yes        # Skip cost-estimate confirmation (CI-friendly)
+agentci test --workers 4  # Parallel execution
+agentci eval              # Standalone correctness evaluation (no golden baselines)
+
+# ── Golden Baselines ─────────────────────────────────────────────────
+agentci record            # Record a golden baseline trace from a live run
+agentci save --agent my-agent --version v1 --trace-file trace.json  # Save versioned baseline
+agentci baselines --agent my-agent  # List saved baseline versions
+agentci diff --agent my-agent --baseline v1 --compare v2  # Diff two versions
+
+# ── Reporting ─────────────────────────────────────────────────────────
+agentci run               # Legacy test runner (pytest-compatible)
+agentci report            # Generate HTML report from results JSON
+```
+
+## 3-Layer Evaluation Engine
+
+`agentci test` evaluates every query through three layers:
+
+| Layer | Severity | Purpose | Example |
+|-------|----------|---------|---------|
+| **Correctness** | Hard FAIL (blocks CI) | Answer quality | `expected_in_answer`, `llm_judge`, `not_in_answer`, `regex_match`, `json_schema` |
+| **Path** | Soft WARN (annotation only) | Tool trajectory | `expected_tools`, `min_tool_recall`, `max_loops` |
+| **Cost** | Soft WARN (annotation only) | Efficiency budget | `max_llm_calls`, `max_tokens`, `max_cost_usd` |
+
+Layer 1 failures block CI (exit code 1). Layers 2-3 produce GitHub annotations but let CI pass.
+
+```yaml
+# agentci_spec.yaml — one query with all three layers
+queries:
+  - query: "How do I install AgentCI?"
+    correctness:
+      any_expected_in_answer: ["pip install", "ciagent"]
+      llm_judge:
+        - rule: "Response provides clear installation instructions"
+          threshold: 0.7
+    path:
+      expected_tools: [retrieve_docs]
+      min_tool_recall: 1.0
+    cost:
+      max_llm_calls: 8
+```
+
+## Mock Mode
+
+Test your spec structure with zero API keys and zero cost:
+
+```bash
+agentci test --mock
+```
+
+Mock mode generates synthetic traces that match your spec expectations. Use it to:
+- Validate spec structure before spending on live runs
+- Run in CI without API keys for fast feedback
+- Iterate on assertions without waiting for LLM responses
+
+## Cost Estimation
+
+Before live test runs, AgentCI shows a cost estimate:
+
+```bash
+$ agentci test
+Estimated cost: $0.12 – $0.35 (12 queries)
+Proceed? [y/N] y
+```
+
+Skip the prompt in CI with `--yes` or `-y`:
+
+```bash
+agentci test --yes  # no confirmation prompt
 ```
 
 ## CI/CD
@@ -258,13 +334,18 @@ When you diff a current run against a golden baseline, AgentCI flags these chang
 
 ## Status
 
-AgentCI is in **early release** (v0.1.x). The core trace model, diffing engine, assertion library, mock system, and CLI are stable. The API may evolve based on community feedback.
+AgentCI is in **beta** (v0.5.0). The core trace model, diffing engine, 3-layer evaluation engine, mock system, and CLI are stable. The API may evolve based on community feedback.
 
 **What's here today:**
+- 3-layer evaluation engine (Correctness / Path / Cost)
 - Trace capture for LangGraph, Anthropic, and OpenAI Agents SDK
-- Golden baseline recording and regression diffing
+- Golden baseline recording, versioned saves, and regression diffing
 - LLM mocking for all three frameworks (zero API keys)
-- LLM-as-judge assertions for subjective output grading
+- LLM-as-judge with composite rubrics, ensemble judging, and doc-grounded evaluation
+- Mock mode (`agentci test --mock`) for zero-cost spec validation
+- Guided spec generation (`agentci init --generate`) with auto-scan and KB sampling
+- Health check (`agentci doctor`) for environment validation
+- Cost estimation before live runs
 - Pytest plugin for native integration
 - GitHub Actions generator via `agentci init`
 

{ciagent-0.4.2 → ciagent-0.5.1}/pyproject.toml

@@ -1,13 +1,13 @@
 [project]
 name = "ciagent"
-version = "0.4.2"
+version = "0.5.1"
 description = "Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production."
 readme = "README.md"
 license = {text = "Apache-2.0"}
 requires-python = ">=3.10"
 keywords = ["ai", "agents", "testing", "llm", "ci-cd", "regression"]
 classifiers = [
-    "Development Status :: 3 - Alpha",
+    "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
     "Topic :: Software Development :: Testing",
     "Framework :: Pytest",
@@ -37,7 +37,7 @@ openai = ["openai>=1.0"]
 anthropic = ["anthropic>=0.20"]
 langgraph = ["langgraph>=0.1"]
 jsonschema = ["jsonschema>=4.0"]
-all = ["agentci[openai,anthropic,langgraph,jsonschema]"]
+all = ["ciagent[openai,anthropic,langgraph,jsonschema]"]
 dev = ["pytest>=8.0", "ruff", "mypy"]
 
 [project.scripts]

{ciagent-0.4.2 → ciagent-0.5.1}/src/agentci/capture.py

@@ -8,6 +8,7 @@ object. The developer doesn't change their agent code at all.
 Phase 1: Patch openai.ChatCompletion and anthropic.Messages
 Phase 2: Add OTEL span emission for interop with Arize/Langfuse
 """
+from __future__ import annotations
 
 import time
 import contextvars