ciagent 0.5.0__tar.gz → 0.6.0__tar.gz

ciagent-0.6.0/.gitignore

@@ -0,0 +1,43 @@
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+venv/
+.venv/
+env/
+.env
+
+# Distribution
+dist/
+build/
+*.egg-info/
+
+# IDEs
+.idea/
+.vscode/
+*.swp
+.DS_Store
+
+# Testing cache
+.pytest_cache/
+
+# Agent CI
+golden/
+test_results/
+
+# Internal planning & development
+Plan_docs/
+phase1.md
+Vision.md
+llms.txt
+demo_ux/
+scripts/
+README_v2.md
+
+# Local config
+agentci.yaml
+agentci_spec.yaml
+
+# Claude context
+.claude/

{ciagent-0.5.0 → ciagent-0.6.0}/CHANGELOG.md

@@ -5,6 +5,42 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.0] - 2026-03-05
+
+### Added
+
+#### `final_output` Auto-Capture
+- `TraceContext._auto_extract_final_output()` called in `__exit__()` — automatically extracts the agent's answer from traces
+- Extraction priority: LangGraph state messages > span `output_data` (string) > span `output_data` (dict with `content`/`message`/`text`/`output` keys) > last LLM call `output_text`
+- Manual `trace.metadata["final_output"]` still takes precedence (no overwrite)
+- LangGraph adapter: auto-sets `final_output` from last AI message in `parse_state()`
+- OpenAI Agents adapter: auto-sets `final_output` from last span output in `on_trace_end()`
+
+#### `agentci calibrate` Command
+- Runs N sample queries against the live agent, measures actual metrics, shows Rich comparison table
+- Updates spec budgets with headroom: +50% for LLM/tool calls, +100% for tokens/cost
+- Flags: `--samples N` (default 2), `--dry-run`, `--yes`, `--spec PATH`
+
+#### Strict Tool Sequence Assertions
+- `PathSpec.expected_tool_sequence: Optional[list[str]]` — strict ordered tool call check
+- Mismatch = WARN (soft warning) with position-level diff via `_format_sequence_diff()`
+
+#### HTML Trace Report
+- Self-contained `report.html.j2` Jinja2 template with dark theme
+- Summary dashboard (pass/fail/warn counts, total cost)
+- Per-query cards with status badges, answer preview, three-layer details
+- Collapsible trace tree with JS toggle
+- Available via `agentci test --format html --output report.html` or `agentci report -i results.json`
+
+### Changed
+- `max_llm_calls` default in spec generator raised from 8 to 10 (better headroom for real agents)
+- `max_llm_calls` fallback in mock runner raised from 3 to 10
+- Calibrate command floor raised from 8 to 10
+- `--format` choices in `test` and `eval` commands now include `html`
+- `--output / -o` option added to `test` and `eval` commands for HTML file path
+- `agentci report` command fully implemented (was stub) — converts JSON results to HTML
+- 22 new tests added (570 total, up from 548 in v0.5.1)
+
 ## [0.5.0] - 2026-03-01
 
 ### Added

{ciagent-0.5.0 → ciagent-0.6.0}/LICENSE

@@ -1,3 +1,5 @@
+Copyright 2025-2026 The AgentCI Authors
+
                                  Apache License
                            Version 2.0, January 2004
                         http://www.apache.org/licenses/

ciagent-0.6.0/PKG-INFO

@@ -0,0 +1,156 @@
+Metadata-Version: 2.4
+Name: ciagent
+Version: 0.6.0
+Summary: Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production.
+Project-URL: Documentation, https://github.com/suniel12/AgentCI#readme
+Project-URL: Repository, https://github.com/suniel12/AgentCI.git
+Project-URL: Issues, https://github.com/suniel12/AgentCI/issues
+License: Apache-2.0
+License-File: LICENSE
+Keywords: agents,ai,ci-cd,llm,regression,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: click>=8.0
+Requires-Dist: jinja2>=3.0
+Requires-Dist: opentelemetry-api>=1.20
+Requires-Dist: opentelemetry-sdk>=1.20
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pytest>=7.0
+Requires-Dist: python-dotenv>=0.19
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Provides-Extra: all
+Requires-Dist: anthropic>=0.20; extra == 'all'
+Requires-Dist: jsonschema>=4.0; extra == 'all'
+Requires-Dist: langgraph>=0.1; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.20; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: jsonschema
+Requires-Dist: jsonschema>=4.0; extra == 'jsonschema'
+Provides-Extra: langgraph
+Requires-Dist: langgraph>=0.1; extra == 'langgraph'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# AgentCI
+
+**Pytest-native regression testing for AI agents.** Catch routing changes, tool call drift, and cost spikes before production.
+
+[![PyPI](https://img.shields.io/pypi/v/ciagent)](https://pypi.org/project/ciagent/)
+[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![AGENTS.md](https://img.shields.io/badge/AGENTS.md-supported-blue)](AGENTS.md)
+
+You changed a prompt. Your agent broke in production. Three days later, a user complained. You had no tests, no diff, no idea what went wrong.
+
+Works with OpenAI, Anthropic, and LangGraph. Runs inside pytest.
+
+## Add to Your Project
+
+```bash
+pip install ciagent
+```
+
+Write your golden queries — what should your agent handle, and what should it refuse?
+
+```yaml
+# agentci_spec.yaml
+# runner: any function that takes a query string and returns a response
+runner: my_app.agent:run_for_agentci
+queries:
+  - query: "How do I install AgentCI?"
+    correctness:
+      any_expected_in_answer: ["pip install", "ciagent"]
+    path:
+      expected_tools: [retrieve_docs]
+    cost:
+      max_llm_calls: 8
+
+  - query: "What's the CEO's favorite restaurant?"
+    correctness:
+      not_in_answer: ["restaurant", "favorite"]
+    path:
+      expected_tools: []  # expect no tools called for out-of-scope queries
+```
+
+Run:
+
+```bash
+agentci test --mock       # start here: zero-cost with synthetic traces
+agentci test              # run live against your real agent
+```
+
+`agentci test` evaluates each query through 3 layers — correctness, path, and cost:
+
+```
+============================================================
+
+Query: How do I install AgentCI?
+Answer: To install AgentCI, you can use pip with the following command:
+        pip install ciagent. Make sure you have Python 3.10 or later.
+
+  ✅  CORRECTNESS: PASS
+       ✓ Found keywords: "pip install ciagent"
+       ✓ LLM judge passed (score: 5 ≥ 0.6)
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Tool precision: 0.500
+       ✓ No loops detected
+  💰  COST: PASS
+       ✓ LLM calls: 8 ≤ max 8
+
+============================================================
+
+Query: What Python version does AgentCI require and what frameworks does it support?
+Answer: AgentCI currently does not specify a required Python version
+        in the provided context, so I don't have that information...
+
+  ❌  CORRECTNESS: FAIL
+       • Expected '3.10' not found in answer
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Loops: 1 ≤ max 3
+  💰  COST: PASS
+       ✓ LLM calls: 4 ≤ max 5
+
+============================================================
+```
+
+Don't have golden queries yet? `agentci init --generate` scans your code and generates a starter spec.
+
+## Demo
+
+Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. AgentCI catches the correctness regression:
+
+![AgentCI Demo](demo/agentci-rag-demo.gif)
+
+## CLI
+
+```bash
+agentci init --generate        # Scan project, generate test spec
+agentci init                   # Generate GitHub Actions workflow + pre-push hook
+agentci test --mock --yes      # Zero-cost synthetic traces, CI-friendly (no keys, no prompts)
+agentci test                   # Run 3-layer evaluation (correctness → path → cost)
+agentci test --format html -o report.html  # HTML report with per-query details
+agentci calibrate              # Measure real agent metrics, auto-tune spec budgets
+agentci doctor                 # Health check: spec, deps, API keys
+agentci record <test>          # Record golden baseline
+agentci diff                   # Diff against baseline
+agentci report -i results.json # Generate HTML report from JSON results
+```
+## Contributing
+
+[GitHub Issues](https://github.com/suniel12/AgentCI/issues)
+[DemoAgents](https://github.com/suniel12/DemoAgents) — working examples for all three frameworks
+
+Apache 2.0. If you build an agent and test it with AgentCI, I'd love to hear about it.
+
+---

ciagent-0.6.0/README.md

@@ -0,0 +1,113 @@
+# AgentCI
+
+**Pytest-native regression testing for AI agents.** Catch routing changes, tool call drift, and cost spikes before production.
+
+[![PyPI](https://img.shields.io/pypi/v/ciagent)](https://pypi.org/project/ciagent/)
+[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![AGENTS.md](https://img.shields.io/badge/AGENTS.md-supported-blue)](AGENTS.md)
+
+You changed a prompt. Your agent broke in production. Three days later, a user complained. You had no tests, no diff, no idea what went wrong.
+
+Works with OpenAI, Anthropic, and LangGraph. Runs inside pytest.
+
+## Add to Your Project
+
+```bash
+pip install ciagent
+```
+
+Write your golden queries — what should your agent handle, and what should it refuse?
+
+```yaml
+# agentci_spec.yaml
+# runner: any function that takes a query string and returns a response
+runner: my_app.agent:run_for_agentci
+queries:
+  - query: "How do I install AgentCI?"
+    correctness:
+      any_expected_in_answer: ["pip install", "ciagent"]
+    path:
+      expected_tools: [retrieve_docs]
+    cost:
+      max_llm_calls: 8
+
+  - query: "What's the CEO's favorite restaurant?"
+    correctness:
+      not_in_answer: ["restaurant", "favorite"]
+    path:
+      expected_tools: []  # expect no tools called for out-of-scope queries
+```
+
+Run:
+
+```bash
+agentci test --mock       # start here: zero-cost with synthetic traces
+agentci test              # run live against your real agent
+```
+
+`agentci test` evaluates each query through 3 layers — correctness, path, and cost:
+
+```
+============================================================
+
+Query: How do I install AgentCI?
+Answer: To install AgentCI, you can use pip with the following command:
+        pip install ciagent. Make sure you have Python 3.10 or later.
+
+  ✅  CORRECTNESS: PASS
+       ✓ Found keywords: "pip install ciagent"
+       ✓ LLM judge passed (score: 5 ≥ 0.6)
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Tool precision: 0.500
+       ✓ No loops detected
+  💰  COST: PASS
+       ✓ LLM calls: 8 ≤ max 8
+
+============================================================
+
+Query: What Python version does AgentCI require and what frameworks does it support?
+Answer: AgentCI currently does not specify a required Python version
+        in the provided context, so I don't have that information...
+
+  ❌  CORRECTNESS: FAIL
+       • Expected '3.10' not found in answer
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Loops: 1 ≤ max 3
+  💰  COST: PASS
+       ✓ LLM calls: 4 ≤ max 5
+
+============================================================
+```
+
+Don't have golden queries yet? `agentci init --generate` scans your code and generates a starter spec.
+
+## Demo
+
+Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. AgentCI catches the correctness regression:
+
+![AgentCI Demo](demo/agentci-rag-demo.gif)
+
+## CLI
+
+```bash
+agentci init --generate        # Scan project, generate test spec
+agentci init                   # Generate GitHub Actions workflow + pre-push hook
+agentci test --mock --yes      # Zero-cost synthetic traces, CI-friendly (no keys, no prompts)
+agentci test                   # Run 3-layer evaluation (correctness → path → cost)
+agentci test --format html -o report.html  # HTML report with per-query details
+agentci calibrate              # Measure real agent metrics, auto-tune spec budgets
+agentci doctor                 # Health check: spec, deps, API keys
+agentci record <test>          # Record golden baseline
+agentci diff                   # Diff against baseline
+agentci report -i results.json # Generate HTML report from JSON results
+```
+## Contributing
+
+[GitHub Issues](https://github.com/suniel12/AgentCI/issues)
+[DemoAgents](https://github.com/suniel12/DemoAgents) — working examples for all three frameworks
+
+Apache 2.0. If you build an agent and test it with AgentCI, I'd love to hear about it.
+
+---

{ciagent-0.5.0 → ciagent-0.6.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ciagent"
-version = "0.5.0"
+version = "0.6.0"
 description = "Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production."
 readme = "README.md"
 license = {text = "Apache-2.0"}
@@ -28,16 +28,16 @@ dependencies = [
 ]
 
 [project.urls]
-Documentation = "https://github.com/agentci-org/agentci#readme"
-Repository = "https://github.com/agentci-org/agentci.git"
-Issues = "https://github.com/agentci-org/agentci/issues"
+Documentation = "https://github.com/suniel12/AgentCI#readme"
+Repository = "https://github.com/suniel12/AgentCI.git"
+Issues = "https://github.com/suniel12/AgentCI/issues"
 
 [project.optional-dependencies]
 openai = ["openai>=1.0"]
 anthropic = ["anthropic>=0.20"]
 langgraph = ["langgraph>=0.1"]
 jsonschema = ["jsonschema>=4.0"]
-all = ["agentci[openai,anthropic,langgraph,jsonschema]"]
+all = ["ciagent[openai,anthropic,langgraph,jsonschema]"]
 dev = ["pytest>=8.0", "ruff", "mypy"]
 
 [project.scripts]

{ciagent-0.5.0 → ciagent-0.6.0}/src/agentci/__init__.py

@@ -1,3 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
 """
 Agent CI — Continuous Integration for AI Agents.
 

ciagent-0.6.0/src/agentci/_version.py

@@ -0,0 +1,3 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
+__version__ = "0.1.0"

ciagent-0.6.0/src/agentci/adapters/__init__.py

@@ -0,0 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
+"""
+Adapter registry and base classes.
+"""

{ciagent-0.5.0 → ciagent-0.6.0}/src/agentci/adapters/base.py

@@ -1,3 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
 """
 Base Adapter class.
 """

{ciagent-0.5.0 → ciagent-0.6.0}/src/agentci/adapters/generic.py

@@ -1,3 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
 """
 Generic Python Function Adapter.
 """

{ciagent-0.5.0 → ciagent-0.6.0}/src/agentci/adapters/langgraph.py

@@ -1,3 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
 """
 LangGraph Adapter.
 """
@@ -61,9 +63,17 @@ class LangGraphAdapter(BaseAdapter):
         Parses a standard LangChain/LangGraph `messages` state list into a Trace.
         """
         trace = Trace(framework="langgraph", graph_state=state)
-        
+
         messages = state.get("messages", [])
-        
+
+        # Extract final output from last AI message
+        for msg in reversed(messages):
+            if getattr(msg, "type", "") == "ai":
+                content = getattr(msg, "content", "")
+                if content:
+                    trace.metadata["final_output"] = str(content)
+                    break
+
         current_span = Span(name="langgraph_execution")
         
         for msg in messages:

{ciagent-0.5.0 → ciagent-0.6.0}/src/agentci/adapters/openai_agents.py

@@ -1,3 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
 """
 OpenAI Agents SDK Adapter for AgentCI.
 
@@ -85,9 +87,23 @@ class AgentCITraceProcessor:
             self._span_map[span_id] = agentci_span
 
     def on_trace_end(self, trace: Any) -> None:
-        """Called when the trace ends. Finalize metrics."""
+        """Called when the trace ends. Finalize metrics and extract output."""
         if self._current_trace is not None:
             self._current_trace.compute_metrics()
+
+            # Auto-extract final output from last agent or generation span
+            if "final_output" not in self._current_trace.metadata:
+                for span in reversed(self._current_trace.spans):
+                    if span.output_data:
+                        self._current_trace.metadata["final_output"] = str(span.output_data)
+                        break
+                    if span.llm_calls:
+                        last_llm = span.llm_calls[-1]
+                        text = getattr(last_llm, "output_text", "")
+                        if text:
+                            self._current_trace.metadata["final_output"] = text
+                            break
+
             self._last_trace = self._current_trace
             self._current_trace = None
 

{ciagent-0.5.0 → ciagent-0.6.0}/src/agentci/assertions.py

@@ -1,3 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
 """
 Built-in assertion evaluators.
 

{ciagent-0.5.0 → ciagent-0.6.0}/src/agentci/baselines.py

@@ -1,3 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
 """
 AgentCI v2 Baseline Manager.
 

{ciagent-0.5.0 → ciagent-0.6.0}/src/agentci/capture.py

@@ -1,3 +1,5 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
 """
 Trace capture via monkey-patching.
 
@@ -8,6 +10,7 @@ object. The developer doesn't change their agent code at all.
 Phase 1: Patch openai.ChatCompletion and anthropic.Messages
 Phase 2: Add OTEL span emission for interop with Arize/Langfuse
 """
+from __future__ import annotations
 
 import time
 import contextvars
@@ -75,17 +78,72 @@ class TraceContext:
     def __exit__(self, *args):
         # Compute duration
         self.trace.total_duration_ms = (time.perf_counter() - self._start_time) * 1000
-        
+
         # Roll up metrics
         self.trace.compute_metrics()
-        
+
+        # Auto-extract final output if not manually set
+        self._auto_extract_final_output()
+
         # Remove patches
         for restore_fn in self._patches:
             restore_fn()
-        
+
         # Clear context
         _active_trace.set(None)
         _active_span.set(None)
+
+    def _auto_extract_final_output(self) -> None:
+        """Auto-extract the agent's final output from the trace.
+
+        Only runs if ``final_output`` has not been manually set in
+        ``trace.metadata``.  Extraction priority:
+
+        1. LangGraph state: last AI message's ``.content``
+        2. Last span's ``output_data`` (string)
+        3. Last span's ``output_data`` dict with common keys
+        4. Last LLM call's ``output_text`` from last span
+        """
+        if "final_output" in self.trace.metadata:
+            return
+
+        # 1. LangGraph state
+        graph_state = getattr(self.trace, "graph_state", None)
+        if graph_state:
+            messages = graph_state.get("messages", [])
+            if messages:
+                last_msg = messages[-1]
+                content = getattr(last_msg, "content", None)
+                if content:
+                    self.trace.metadata["final_output"] = str(content)
+                    return
+
+        # 2-3. Last span output_data
+        if self.trace.spans:
+            last_span = self.trace.spans[-1]
+            output = last_span.output_data
+
+            if output is not None:
+                if isinstance(output, str):
+                    self.trace.metadata["final_output"] = output
+                    return
+                if isinstance(output, dict):
+                    for key in ("content", "message", "text", "output"):
+                        if key in output:
+                            self.trace.metadata["final_output"] = str(output[key])
+                            return
+
+            # 4. Last LLM call output
+            if last_span.llm_calls:
+                last_llm = last_span.llm_calls[-1]
+                # Handle both LLMCall objects and raw dicts
+                if isinstance(last_llm, dict):
+                    text = last_llm.get("content") or last_llm.get("output_text", "")
+                else:
+                    text = getattr(last_llm, "output_text", "")
+                if text:
+                    self.trace.metadata["final_output"] = str(text)
+                    return
     
     def _patch_openai(self):
         """Wrap openai.chat.completions.create to capture LLM calls."""