ciagent 0.8.0__tar.gz → 0.9.0__tar.gz

{ciagent-0.8.0 → ciagent-0.9.0}/CHANGELOG.md

@@ -5,9 +5,109 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0] - 2026-07-06
+
+### Added — retrieval layer 2.5: `retrieval:` assertions (F4, F6 Phase 4)
+- New `retrieval:` block on queries and scenario `per_turn:`/`outcome:`
+  checks — deterministic assertions on the retriever tool's captured
+  `ToolCall.result`: `forbid_empty` (empty retrieval + confident answer →
+  WARN "ungrounded answer"; a refusal passes), `min_results`,
+  `expected_sources`, `facts_in_context` (informational-only in v1),
+  `result_format` hint, `empty_markers`/`refusal_markers` overrides
+- Binding result-interpretation contract: the layer SKIPs on uncaptured or
+  unparseable results — it never guesses. Empty means None/[]/""/whitespace
+  plus spec markers. Retrieval never hard-fails (WARN tier, like path/cost)
+- Fifth stability flip source `retrieval-variance` (single-turn and
+  scenario): verdict flipped, same tool sequence, retrieved set differed →
+  blames the retriever, not the prompt; attribution skips when any run's
+  retriever output wasn't captured
+- Judge audit gains a "judged against EMPTY retrieval" row (console + JSON):
+  the judge graded an answer whose ground truth was already lost
+- Diff engine gains `RETRIEVAL_CHANGED`: retrieved source set changed vs
+  golden, emitted only when a source set is extractable from both traces
+- Mock runner synthesizes retrieval results satisfying the spec — the
+  zero-key path exercises layer 2.5 end to end
+- Adapter capture verification and fixes: langgraph now pairs `ToolMessage`
+  outputs onto tool calls by `tool_call_id`; the openai/anthropic
+  monkey-patches backfill `ToolCall.result` from the next request's
+  `role="tool"` messages / `tool_result` blocks (openai-agents adapter
+  already captured results)
+
+### Added — generative personas, cost guardrails, stability (F6 Phase 3)
+- Scenarios with `persona:`/`goal:` and no `turns:` get user turns from a
+  persona LLM (cheap haiku-class default, `persona_config` spec key) — the
+  finder path. Termination rules unchanged: max_turns / stop_when only; a
+  derailed persona (empty/unusable output) marks the scenario infra-error
+  with partial turns kept rather than silently grading the agent
+- `--max-cost` session budget: hard-aborts mid-conversation at the next turn
+  boundary; partial results are clearly marked and the outcome verdict is
+  never evaluated on a partial conversation. Pre-run cost estimate with a
+  confirm gate; per-scenario cost line
+- `--runs N` scenario stability with new `simulation-variance` flip source:
+  the simulated user said different things across runs — the persona varied,
+  not the agent. `--workers` runs scenarios in parallel (turns stay
+  sequential); a mock persona gives a zero-key generative path
+
+### Added — record + replay: found bug → regression test (F6 Phase 2)
+- `--record` / `--record-dir`: save any driven conversation as a golden
+  envelope at `<baseline_dir>/<agent>/scenarios/<slug>.json`. Recording
+  never prechecks — a FAILED scenario records too (`checks_passed: false`),
+  because converting a found bug into a regression test is the point
+- `--replay`: recorded user turns are fed back verbatim (the persona is
+  never consulted); the scenario spec is embedded in the envelope so goldens
+  are self-contained. Replaying a deterministic agent twice yields
+  byte-identical verdicts (the `scenario_verdict` contract)
+- Conversation-aware diff (`diff_envelopes`): per-turn tool-sequence and
+  answer changes between a golden conversation and a fresh run
+
+### Added — `ciagent simulate` Phase 1: scripted multi-turn scenarios (F6)
+- New `scenarios:` spec block: `turns:` (scripted user messages), `max_turns`,
+  `per_turn:` checks (evaluated on every turn), `outcome:` checks (evaluated
+  once at the END as the verdict — never a stop condition), and explicit
+  `stop_when: {tool_called: X}` early exit. Termination is deterministic only:
+  script exhausted, max_turns, or the stop_when event — no judge, no keywords
+- New `conversation_runner:` spec key — `(messages: list[{role, content}]) →
+  str | Trace`, history passed explicitly, same capture/coercion as `test`
+- `ciagent simulate` CLI: `--mock` runs scenarios on synthetic traces with
+  zero API keys (the CI path); live runs confirm turn count first. Exit codes:
+  0 pass / 1 outcome or per-turn correctness failure / 2 config or agent error
+- Agent exception mid-conversation marks the scenario `infra-error` and keeps
+  the completed turns; conversations convert to schema_version-2 envelopes
+- Spec change: `queries:` is no longer required when `scenarios:` is present
+  (at least one of the two must exist)
+- 18 new tests, incl. the ADR checklist items: both termination causes,
+  agent-raises-mid-conversation, per_turn + outcome evaluation, and
+  outcome-never-stops (a turn-1 keyword match must not end the scenario)
+
+### Added — schema_version + conversation envelope (F6 Phase 0)
+- `schema_version: 1` written into new single-trace baselines; unversioned
+  files read as legacy; envelopes are `schema_version: 2`; newer-than-reader
+  files rejected by name. `ConversationEnvelope`: one loader for envelope,
+  wrapper, and bare-trace shapes (single-turn = 1-turn degenerate case)
+
+### Changed — brand and story
+- **Brand: AgentCI → CIAgent** (display name), standardizing on the `ciagent`
+  package name. Spec/runner filenames (`agentci_spec.yaml`), `AGENTCI_*` env
+  vars, and `AgentCISpec`/`AgentCITraceProcessor` class names unchanged.
+- **BREAKING: Python module renamed `agentci` → `ciagent`** (`from ciagent
+  import ...`); the `agentci` CLI entry point is removed (`ciagent` is the
+  command), and the pytest plugin entry point is now `ciagent`. Flag-day, no
+  deprecation shim (pre-adoption). Motive is a verified collision: the
+  unrelated PyPI package `agentci` 0.1.1 (Agent-CI) installs a top-level
+  `agentci/` module and an `agentci` console script — installing both tools
+  in one environment broke ours.
+- **GitHub repo renamed** to `suniel12/ciagent` (old URLs redirect).
+- **Claude Code plugin identity renamed**: install is now
+  `/plugin marketplace add suniel12/ciagent` + `/plugin install ciagent@ciagent`;
+  skills are `ciagent:onboard` / `ciagent:check`. (The 0.8.0 identity below had
+  zero installs.)
+- **README hero, PyPI description, and GitHub About repositioned to the
+  eval-reliability wedge** ("Your eval score is stable. Your system isn't."),
+  per the strategy ADR; regression testing remains as supporting capability.
+
 ## [0.8.0] - 2026-07-06
 
-### Added — Claude Code plugin (F5: your coding agent onboards and operates AgentCI)
+### Added — Claude Code plugin (F5: your coding agent onboards and operates CIAgent)
 - The repo is now a Claude Code plugin marketplace
   (`/plugin marketplace add suniel12/AgentCI`, `/plugin install agentci@agentci`)
   with two skills: **onboard** (write the runner, record goldens via

{ciagent-0.8.0 → ciagent-0.9.0}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.4
 Name: ciagent
-Version: 0.8.0
-Summary: Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production.
-Project-URL: Documentation, https://github.com/suniel12/AgentCI#readme
-Project-URL: Repository, https://github.com/suniel12/AgentCI.git
-Project-URL: Issues, https://github.com/suniel12/AgentCI/issues
+Version: 0.9.0
+Summary: Find out whether your AI agent's evals are lying to you: stability reports with flip attribution, LLM judge audits, and deterministic checks mined from your KB.
+Project-URL: Documentation, https://github.com/suniel12/ciagent#readme
+Project-URL: Repository, https://github.com/suniel12/ciagent.git
+Project-URL: Issues, https://github.com/suniel12/ciagent/issues
 License: Apache-2.0
 License-File: LICENSE
 Keywords: agents,ai,ci-cd,llm,regression,testing
@@ -41,17 +41,24 @@ Provides-Extra: openai
 Requires-Dist: openai>=1.0; extra == 'openai'
 Description-Content-Type: text/markdown
 
-# AgentCI
+# CIAgent
 
-**Pytest-native regression testing for AI agents.** Catch routing changes, tool call drift, and cost spikes before production.
+**Your eval score is stable. Your system isn't.**
+
+CIAgent tells you whether your agent's evals are lying to you:
+
+- **Run the suite N times** — see which verdicts flip, with the blame attached
+  (`agent-variance` → fix the agent, `judge-flake` → fix the eval)
+- **Audit your LLM judge** against ground truth you already have
+- **Replace judge calls with deterministic fact-checks** mined from your knowledge base
 
 [![PyPI](https://img.shields.io/pypi/v/ciagent)](https://pypi.org/project/ciagent/)
-[![CI](https://github.com/suniel12/AgentCI/actions/workflows/ci.yml/badge.svg)](https://github.com/suniel12/AgentCI/actions/workflows/ci.yml)
+[![CI](https://github.com/suniel12/ciagent/actions/workflows/ci.yml/badge.svg)](https://github.com/suniel12/ciagent/actions/workflows/ci.yml)
 [![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
 [![AGENTS.md](https://img.shields.io/badge/AGENTS.md-supported-blue)](AGENTS.md)
 
-You changed a prompt. Your agent broke in production. Three days later, a user complained. You had no tests, no diff, no idea what went wrong.
-
+Built from a real study: 1 in 7 answers an LLM judge passed were wrong, and deterministic
+checks caught 8 of 8 — methodology and raw numbers in [STUDY.md](STUDY.md).
 Works with OpenAI, Anthropic, and LangGraph. Runs inside pytest.
 
 ## See It in 30 Seconds
@@ -82,80 +89,6 @@ Suite score across 3 runs: 88%  /  88%  /  88%     ← looks stable
 
 The aggregate score is identical every run. Three of the eight verdicts underneath it flipped. A single-run eval score would never tell you — the stability report does, and attributes every flip to the layer that caused it. (The demo simulates a flaky agent; point it at your own with the spec below.)
 
-## Add to Your Project
-
-```bash
-pip install ciagent
-```
-
-Write your golden queries — what should your agent handle, and what should it refuse?
-
-```yaml
-# agentci_spec.yaml
-agent: my-agent
-# runner: any function that takes a query string and returns a response
-runner: my_app.agent:run_for_agentci
-queries:
-  - query: "How do I install AgentCI?"
-    correctness:
-      any_expected_in_answer: ["pip install", "ciagent"]
-    path:
-      expected_tools: [retrieve_docs]
-    cost:
-      max_llm_calls: 8
-
-  - query: "What's the CEO's favorite restaurant?"
-    correctness:
-      not_in_answer: ["restaurant", "favorite"]
-    path:
-      expected_tools: []  # expect no tools called for out-of-scope queries
-```
-
-Run:
-
-```bash
-agentci test --mock       # start here: zero-cost with synthetic traces
-agentci test              # run live against your real agent
-```
-
-`agentci test` evaluates each query through 3 layers — correctness, path, and cost:
-
-```
-============================================================
-
-Query: How do I install AgentCI?
-Answer: To install AgentCI, you can use pip with the following command:
-        pip install ciagent. Make sure you have Python 3.10 or later.
-
-  ✅  CORRECTNESS: PASS
-       ✓ Found keywords: "pip install ciagent"
-       ✓ LLM judge passed (score: 5 ≥ 0.6)
-  📈  PATH: PASS
-       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
-       ✓ Tool precision: 0.500
-       ✓ No loops detected
-  💰  COST: PASS
-       ✓ LLM calls: 8 ≤ max 8
-
-============================================================
-
-Query: What Python version does AgentCI require and what frameworks does it support?
-Answer: AgentCI currently does not specify a required Python version
-        in the provided context, so I don't have that information...
-
-  ❌  CORRECTNESS: FAIL
-       • Expected '3.10' not found in answer
-  📈  PATH: PASS
-       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
-       ✓ Loops: 1 ≤ max 3
-  💰  COST: PASS
-       ✓ LLM calls: 4 ≤ max 5
-
-============================================================
-```
-
-Don't have golden queries yet? `agentci init --generate` scans your code and generates a starter spec.
-
 ## A stable score is not a stable system
 
 Run the identical eval three times and you can get 96% / 95% / 96% — rock solid — while
@@ -163,7 +96,7 @@ individual queries flip verdicts every run. The aggregate holds because the erro
 around. `--runs N` shows what a single run can't:
 
 ```bash
-agentci test --runs 3
+ciagent test --runs 3
 ```
 
 ```
@@ -196,7 +129,7 @@ facts; pass@k/pass^k estimates live in the JSON output, labeled as estimates.
 
 Flaky-but-passing exits 0 so adoption won't break your CI; add `--fail-on-flaky` when
 you're ready to gate on it. Try it with zero API keys:
-`AGENTCI_MOCK_FLAKY=1 agentci test --mock --runs 3`. Details: [docs/stability.md](docs/stability.md).
+`AGENTCI_MOCK_FLAKY=1 ciagent test --mock --runs 3`. Details: [docs/stability.md](docs/stability.md).
 
 ## Audit the judge itself
 
@@ -206,7 +139,7 @@ nothing, passes it. `judge-audit` measures your judge against ground truth you a
 by re-scoring recorded baselines (the agent is never re-run):
 
 ```bash
-agentci judge-audit
+ciagent judge-audit
 ```
 
 1. **Judge vs. deterministic checks** — the disagreement matrix. The row that matters:
@@ -223,7 +156,7 @@ shouldn't be trusted where you can't. Verdict: `TRUSTWORTHY` / `NEEDS CALIBRATIO
 
 Most agent failures that matter involve a hard fact — a product name, a price, a version number. Those are checkable deterministically, for free. And an LLM judge grading against the same context as your agent inherits your agent's blind spots: when retrieval comes up empty, the agent answers from nothing and the judge — reading the same nothing — passes it.
 
-So AgentCI runs deterministic checks first and treats the judge as the last resort, not the default:
+So CIAgent runs deterministic checks first and treats the judge as the last resort, not the default:
 
 1. **Fact checks in code** — `expected_in_answer`, `not_in_answer`, `regex_match`, `json_schema`. Zero LLM calls, zero flakiness, same verdict every run.
 2. **Path checks** — did the agent call the tools it should have? A missing expected tool warns; a forbidden tool fails.
@@ -233,7 +166,7 @@ So AgentCI runs deterministic checks first and treats the judge as the last reso
 Don't write the fact checks by hand — mine them from your knowledge base:
 
 ```bash
-agentci generate-checks
+ciagent generate-checks
 ```
 
 It extracts hard facts (prices, rates, SKUs, "30 days") as variant-set assertions, and
@@ -241,44 +174,122 @@ It extracts hard facts (prices, rates, SKUs, "30 days") as variant-set assertion
 would fail a known-good answer is rejected before you ever see it. One LLM call at
 authoring time; the checks run free forever. Details: [docs/generate-checks.md](docs/generate-checks.md).
 
+## Add to Your Project
+
+```bash
+pip install ciagent
+```
+
+Write your golden queries — what should your agent handle, and what should it refuse?
+
+```yaml
+# agentci_spec.yaml
+agent: my-agent
+# runner: any function that takes a query string and returns a response
+runner: my_app.agent:run_for_agentci
+queries:
+  - query: "How do I install CIAgent?"
+    correctness:
+      any_expected_in_answer: ["pip install", "ciagent"]
+    path:
+      expected_tools: [retrieve_docs]
+    retrieval:
+      tool: retrieve_docs      # assert on what the retriever actually returned
+      forbid_empty: true       # empty retrieval + confident answer = ungrounded
+      expected_sources: [install.md]
+    cost:
+      max_llm_calls: 8
+
+  - query: "What's the CEO's favorite restaurant?"
+    correctness:
+      not_in_answer: ["restaurant", "favorite"]
+    path:
+      expected_tools: []  # expect no tools called for out-of-scope queries
+```
+
+Run:
+
+```bash
+ciagent test --mock       # start here: zero-cost with synthetic traces
+ciagent test              # run live against your real agent
+```
+
+`ciagent test` evaluates each query through 4 layers — correctness, path, retrieval, and cost. The retrieval layer reads the retriever tool's captured result and warns on empty retrievals, missing sources, and count floors — deterministically, and it SKIPs (never guesses) when a result wasn't captured or doesn't parse:
+
+```
+============================================================
+
+Query: How do I install CIAgent?
+Answer: To install CIAgent, you can use pip with the following command:
+        pip install ciagent. Make sure you have Python 3.10 or later.
+
+  ✅  CORRECTNESS: PASS
+       ✓ Found keywords: "pip install ciagent"
+       ✓ LLM judge passed (score: 5 ≥ 0.6)
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Tool precision: 0.500
+       ✓ No loops detected
+  💰  COST: PASS
+       ✓ LLM calls: 8 ≤ max 8
+
+============================================================
+
+Query: What Python version does CIAgent require and what frameworks does it support?
+Answer: CIAgent currently does not specify a required Python version
+        in the provided context, so I don't have that information...
+
+  ❌  CORRECTNESS: FAIL
+       • Expected '3.10' not found in answer
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Loops: 1 ≤ max 3
+  💰  COST: PASS
+       ✓ LLM calls: 4 ≤ max 5
+
+============================================================
+```
+
+Don't have golden queries yet? `ciagent init --generate` scans your code and generates a starter spec.
+
 ## Let your coding agent set it up
 
-AgentCI ships as a Claude Code plugin. Two skills: **onboard** (writes the runner,
+CIAgent ships as a Claude Code plugin. Two skills: **onboard** (writes the runner,
 records golden baselines, generates the spec, verifies it) and **check** (runs the
 right test after every change to your agent and routes failures by flip source).
 
 ```
-/plugin marketplace add suniel12/AgentCI
-/plugin install agentci@agentci
+/plugin marketplace add suniel12/ciagent
+/plugin install ciagent@ciagent
 ```
 
-Then ask your coding agent to "set up AgentCI for this repo." It records goldens with
-`agentci bootstrap --yes` and verifies with `agentci test --runs 3` — no human CLI use
+Then ask your coding agent to "set up CIAgent for this repo." It records goldens with
+`ciagent bootstrap --yes` and verifies with `ciagent test --runs 3` — no human CLI use
 needed. The runner it writes is one function: `(query: str) -> str`; trace capture is
 automatic.
 
 ## Demo
 
-Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. AgentCI catches the correctness regression:
+Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. CIAgent catches the correctness regression:
 
-![AgentCI Demo](demo/agentci-rag-demo.gif)
+![CIAgent Demo](demo/agentci-rag-demo.gif)
 
 ## CLI
 
 ```bash
-agentci init --generate        # Scan project, generate test spec
-agentci init                   # Generate GitHub Actions workflow + pre-push hook
-agentci test --mock --yes      # Zero-cost synthetic traces, CI-friendly (no keys, no prompts)
-agentci test                   # Run 3-layer evaluation (correctness → path → cost)
-agentci test --runs 3          # Stability report: verdict flips + flip-source attribution
-agentci judge-audit            # Audit the LLM judge against checks, retests, hand labels
-agentci generate-checks        # Mine KB facts into deterministic assertions (gated)
-agentci test --format html -o report.html  # HTML report with per-query details
-agentci calibrate              # Measure real agent metrics, auto-tune spec budgets
-agentci doctor                 # Health check: spec, deps, API keys
-agentci record <test>          # Record golden baseline
-agentci diff                   # Diff against baseline
-agentci report -i results.json # Generate HTML report from JSON results
+ciagent init --generate        # Scan project, generate test spec
+ciagent init                   # Generate GitHub Actions workflow + pre-push hook
+ciagent test --mock --yes      # Zero-cost synthetic traces, CI-friendly (no keys, no prompts)
+ciagent test                   # Run 3-layer evaluation (correctness → path → cost)
+ciagent test --runs 3          # Stability report: verdict flips + flip-source attribution
+ciagent judge-audit            # Audit the LLM judge against checks, retests, hand labels
+ciagent generate-checks        # Mine KB facts into deterministic assertions (gated)
+ciagent test --format html -o report.html  # HTML report with per-query details
+ciagent calibrate              # Measure real agent metrics, auto-tune spec budgets
+ciagent doctor                 # Health check: spec, deps, API keys
+ciagent record <test>          # Record golden baseline
+ciagent diff                   # Diff against baseline
+ciagent report -i results.json # Generate HTML report from JSON results
 ```
 ## Docs
 
@@ -295,13 +306,13 @@ agentci report -i results.json # Generate HTML report from JSON results
 
 ## Why not just an LLM judge?
 
-Judge-only evals are expensive, flaky, and blind to their own context. AgentCI is pytest-native regression testing: deterministic checks catch the factual failures, golden traces catch behavioral drift, cost budgets catch spend regressions — and the judge handles only what genuinely needs judgment. Mock mode (`agentci test --mock`) runs the whole suite with zero API keys and zero cost, so it can gate every PR.
+Judge-only evals are expensive, flaky, and blind to their own context. CIAgent is pytest-native regression testing: deterministic checks catch the factual failures, golden traces catch behavioral drift, cost budgets catch spend regressions — and the judge handles only what genuinely needs judgment. Mock mode (`ciagent test --mock`) runs the whole suite with zero API keys and zero cost, so it can gate every PR.
 
 ## Contributing
 
-[GitHub Issues](https://github.com/suniel12/AgentCI/issues) ·
+[GitHub Issues](https://github.com/suniel12/ciagent/issues) ·
 [DemoAgents](https://github.com/suniel12/DemoAgents) — working examples for OpenAI, Anthropic, and LangGraph agents
 
-Apache 2.0. If you build an agent and test it with AgentCI, I'd love to hear about it.
+Apache 2.0. If you build an agent and test it with CIAgent, I'd love to hear about it.
 
 ---