ciagent 0.6.0__tar.gz → 0.7.0__tar.gz

{ciagent-0.6.0 → ciagent-0.7.0}/CHANGELOG.md

@@ -5,6 +5,106 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Changed — stability report hardening (pre-launch fixes from eng review)
+- **Flip attribution now compares per-layer sub-verdicts first**: if every
+  deterministic check returned the same outcome across runs and only the LLM judge's
+  verdict changed, the flip is `judge-flake` even when the answer text was
+  paraphrased (previously mislabeled `agent-variance`)
+- **New flip source `infra-error`**: a judge API failure counted as a fail no longer
+  reads as `judge-flake` — one transient hiccup must not say "fix your rubric"
+- **Console shows observed facts only** (verdict history + pass rate); pass@k /
+  pass^k move to JSON output as clearly labeled estimates (at small k they restate
+  the pass rate)
+- Duplicate query texts in a spec are flagged (they merge into one stability
+  record); queries missing from some runs are marked `partial (k/N runs)`
+- `agentci run` (legacy suite runner) now prints a deprecation warning pointing at
+  `agentci test`; removal planned for 0.9.0
+- Removed stale `src/agentci/_version.py` (said 0.1.0; version comes from package
+  metadata)
+
+## [0.7.0] - 2026-07-05
+
+### Added
+
+#### KB-Derived Fact Checks — `agentci generate-checks`
+Mines the knowledge base for hard facts (prices, rates, SKUs, versions,
+explicit quantities) and proposes them as deterministic assertions on
+existing spec queries. One LLM call at authoring time; the checks run
+deterministically forever at zero cost.
+
+- **Brittleness gate**: every candidate is validated against recorded golden
+  answers before it is offered — a check that would fail a known-good answer
+  is rejected automatically with the failing answer shown
+- Only non-paraphrasable facts; prose facts become variant sets
+  (`any_expected_in_answer`), never single literal strings; only
+  `any_expected_in_answer` / `not_in_answer` / `regex_match` are proposed
+- Candidates without a golden answer are `unvalidated` and never
+  auto-accepted, even with `--yes`; interactive review for everything else
+- Merge never overwrites user-written assertions; `.bak` backup on write;
+  `--dry-run` mode
+- New module `agentci.engine.check_generator`; 21 new tests
+
+#### Judge Audit — `agentci judge-audit`
+Meta-evaluation of the LLM judge against ground truth you already have,
+re-scoring recorded golden baselines (the agent is never re-run):
+
+- **Mode 1 — judge vs. deterministic checks**: independent verdicts on the
+  same recorded answer; reports the disagreement matrix. The killer row:
+  answers the judge PASSED that a deterministic fact-check FAILED
+  (shared-context judge blindness, detected automatically)
+- **Mode 2 — retest stability** (`--repeats K`, default 3): same answer
+  judged K times; verdict flips on identical input are the judge's noise floor
+- **Mode 3 — hand labels** (`--labels FILE`): agreement + Cohen's κ against
+  human review, with standard trust thresholds
+- Scoped claim stated in the report itself: measured on fact-checkable
+  queries; one-directional (disqualifying) signal for judgment-only queries
+- Verdicts: TRUSTWORTHY / NEEDS CALIBRATION / UNRELIABLE (exit 1) /
+  ERROR when the judge never ran (exit 2 — a judge that couldn't run is
+  never scored)
+- `--sample N` cost cap; console + JSON output
+- New module `agentci.engine.judge_audit`; 27 new tests
+
+#### Stability Report — `agentci test --runs N`
+A stable suite score can hide per-query verdict flips: the aggregate holds
+because the errors move around. `--runs N` executes the whole suite N times
+and reports what a single run cannot show:
+
+- Per-query verdict history (✅❌✅), pass rate, pass@k and pass^k estimates
+- Suite score per run printed side by side with the queries that flipped
+- **Flip-source attribution** — every flip is labelled:
+  - `agent-variance`: the agent's output changed → fix the agent
+  - `judge-flake`: same output, the LLM judge changed its verdict → fix the eval
+  - `mixed`: near-identical paraphrase with a judge configured — not guessed
+  Attribution rests on a structural fact: deterministic checks cannot flip on
+  identical output, so identical answer + tools + flipped verdict = judge.
+- Exit semantics: flaky-but-passing exits 0 (warnings only); queries failing
+  in EVERY run exit 1; `--fail-on-flaky` escalates flips to exit 1
+- Works in every format: console section, GitHub `::warning` annotations,
+  `stability` block in JSON, stability card in the HTML report
+- Mock mode support: `AGENTCI_MOCK_FLAKY=1 agentci test --mock --runs 3`
+  demonstrates the report with zero API keys
+- New module `agentci.engine.stability` (`build_stability_report`,
+  `StabilityReport`, `QueryStability`, `FlipSource`); 21 new tests
+
+### Changed
+- **`expected_tools` now asserts by default**: a missing expected tool produces a WARN
+  (tool recall gates at 1.0 unless `min_tool_recall` explicitly loosens it). Previously,
+  without `min_tool_recall`, a recall of 0.0 displayed as PASS with a checkmark.
+- **`expected_tools: []` now asserts that no tools are called**: an explicit empty list
+  produces a WARN if the agent called any tool. Previously it was silently skipped.
+- `agentci doctor` no longer reports numpy as a required dependency (it was never used
+  by AgentCI, so every fresh install showed a false failure).
+
+### Docs
+- README: fixed the quickstart spec example (was missing the required `agent:` field),
+  added CI badge, "Check facts in code" section, docs index
+- Rewrote `docs/quickstart.md`, `docs/ci-cd.md`, `docs/cost-tracking.md`, and
+  `docs/golden-traces.md` to match the current `agentci test` workflow
+- Fixed dead clone URLs in CONTRIBUTING.md and quickstart
+- Removed unused demo GIF variants and cast recordings (6 files, ~2 MB)
+
 ## [0.6.0] - 2026-03-05
 
 ### Added

ciagent-0.7.0/PKG-INFO

@@ -0,0 +1,263 @@
+Metadata-Version: 2.4
+Name: ciagent
+Version: 0.7.0
+Summary: Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production.
+Project-URL: Documentation, https://github.com/suniel12/AgentCI#readme
+Project-URL: Repository, https://github.com/suniel12/AgentCI.git
+Project-URL: Issues, https://github.com/suniel12/AgentCI/issues
+License: Apache-2.0
+License-File: LICENSE
+Keywords: agents,ai,ci-cd,llm,regression,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: click>=8.0
+Requires-Dist: jinja2>=3.0
+Requires-Dist: opentelemetry-api>=1.20
+Requires-Dist: opentelemetry-sdk>=1.20
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pytest>=7.0
+Requires-Dist: python-dotenv>=0.19
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Provides-Extra: all
+Requires-Dist: anthropic>=0.20; extra == 'all'
+Requires-Dist: jsonschema>=4.0; extra == 'all'
+Requires-Dist: langgraph>=0.1; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.20; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: jsonschema
+Requires-Dist: jsonschema>=4.0; extra == 'jsonschema'
+Provides-Extra: langgraph
+Requires-Dist: langgraph>=0.1; extra == 'langgraph'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# AgentCI
+
+**Pytest-native regression testing for AI agents.** Catch routing changes, tool call drift, and cost spikes before production.
+
+[![PyPI](https://img.shields.io/pypi/v/ciagent)](https://pypi.org/project/ciagent/)
+[![CI](https://github.com/suniel12/AgentCI/actions/workflows/ci.yml/badge.svg)](https://github.com/suniel12/AgentCI/actions/workflows/ci.yml)
+[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![AGENTS.md](https://img.shields.io/badge/AGENTS.md-supported-blue)](AGENTS.md)
+
+You changed a prompt. Your agent broke in production. Three days later, a user complained. You had no tests, no diff, no idea what went wrong.
+
+Works with OpenAI, Anthropic, and LangGraph. Runs inside pytest.
+
+## Add to Your Project
+
+```bash
+pip install ciagent
+```
+
+Write your golden queries — what should your agent handle, and what should it refuse?
+
+```yaml
+# agentci_spec.yaml
+agent: my-agent
+# runner: any function that takes a query string and returns a response
+runner: my_app.agent:run_for_agentci
+queries:
+  - query: "How do I install AgentCI?"
+    correctness:
+      any_expected_in_answer: ["pip install", "ciagent"]
+    path:
+      expected_tools: [retrieve_docs]
+    cost:
+      max_llm_calls: 8
+
+  - query: "What's the CEO's favorite restaurant?"
+    correctness:
+      not_in_answer: ["restaurant", "favorite"]
+    path:
+      expected_tools: []  # expect no tools called for out-of-scope queries
+```
+
+Run:
+
+```bash
+agentci test --mock       # start here: zero-cost with synthetic traces
+agentci test              # run live against your real agent
+```
+
+`agentci test` evaluates each query through 3 layers — correctness, path, and cost:
+
+```
+============================================================
+
+Query: How do I install AgentCI?
+Answer: To install AgentCI, you can use pip with the following command:
+        pip install ciagent. Make sure you have Python 3.10 or later.
+
+  ✅  CORRECTNESS: PASS
+       ✓ Found keywords: "pip install ciagent"
+       ✓ LLM judge passed (score: 5 ≥ 0.6)
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Tool precision: 0.500
+       ✓ No loops detected
+  💰  COST: PASS
+       ✓ LLM calls: 8 ≤ max 8
+
+============================================================
+
+Query: What Python version does AgentCI require and what frameworks does it support?
+Answer: AgentCI currently does not specify a required Python version
+        in the provided context, so I don't have that information...
+
+  ❌  CORRECTNESS: FAIL
+       • Expected '3.10' not found in answer
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Loops: 1 ≤ max 3
+  💰  COST: PASS
+       ✓ LLM calls: 4 ≤ max 5
+
+============================================================
+```
+
+Don't have golden queries yet? `agentci init --generate` scans your code and generates a starter spec.
+
+## A stable score is not a stable system
+
+Run the identical eval three times and you can get 96% / 95% / 96% — rock solid — while
+individual queries flip verdicts every run. The aggregate holds because the errors move
+around. `--runs N` shows what a single run can't:
+
+```bash
+agentci test --runs 3
+```
+
+```
+Run 1/3: 18/19 passed
+Run 2/3: 18/19 passed
+Run 3/3: 18/19 passed
+
+────────────────────────────────────────────────────────────
+Stability Report
+────────────────────────────────────────────────────────────
+Suite score across 3 runs: 95%  /  95%  /  95%
+
+⚠️  FLAKY — 2/19 queries flipped verdicts across runs:
+   "What's your return window?"    ✅❌✅  pass_rate=0.67  source: agent-variance (answer changed)
+   "Do you sell gift cards?"       ❌✅✅  pass_rate=0.67  source: judge-flake (same answer, verdict flipped)
+
+   Flip sources: 1 agent-variance (fix the agent) │ 1 judge-flake (fix the eval) │ 0 infra-error (retry) │ 0 mixed
+
+Stability verdict: FLAKY
+```
+
+Every flip is attributed to its source, so it's a routed work item, not a scary number:
+**agent-variance** means the agent produced different output (fix the prompt, retrieval, or
+temperature); **judge-flake** means the output — or every deterministic check's outcome —
+was identical but the LLM judge changed its mind (fix the rubric, or replace the judge with
+a deterministic check); **infra-error** means a judge API call failed (retry, fix nothing).
+Attribution is structural, not guessed: deterministic checks cannot flip on identical
+output, and per-layer sub-verdicts are compared across runs. The console shows observed
+facts; pass@k/pass^k estimates live in the JSON output, labeled as estimates.
+
+Flaky-but-passing exits 0 so adoption won't break your CI; add `--fail-on-flaky` when
+you're ready to gate on it. Try it with zero API keys:
+`AGENTCI_MOCK_FLAKY=1 agentci test --mock --runs 3`. Details: [docs/stability.md](docs/stability.md).
+
+## Audit the judge itself
+
+An LLM judge that shares your agent's context inherits your agent's blind spots: when
+retrieval comes up empty, the agent answers from nothing — and the judge, reading the same
+nothing, passes it. `judge-audit` measures your judge against ground truth you already have,
+by re-scoring recorded baselines (the agent is never re-run):
+
+```bash
+agentci judge-audit
+```
+
+1. **Judge vs. deterministic checks** — the disagreement matrix. The row that matters:
+   answers the judge PASSED that a hard fact-check FAILED.
+2. **Retest stability** — the same answer judged `--repeats` times; flips on identical
+   input are the judge's own noise floor.
+3. **Hand labels** (`--labels`) — agreement + Cohen's κ against your own review.
+
+The claim is deliberately one-directional: a judge that fails where you *can* check it
+shouldn't be trusted where you can't. Verdict: `TRUSTWORTHY` / `NEEDS CALIBRATION` /
+`UNRELIABLE`. Details: [docs/judge-audit.md](docs/judge-audit.md).
+
+## Check facts in code. Save the judge for judgment.
+
+Most agent failures that matter involve a hard fact — a product name, a price, a version number. Those are checkable deterministically, for free. And an LLM judge grading against the same context as your agent inherits your agent's blind spots: when retrieval comes up empty, the agent answers from nothing and the judge — reading the same nothing — passes it.
+
+So AgentCI runs deterministic checks first and treats the judge as the last resort, not the default:
+
+1. **Fact checks in code** — `expected_in_answer`, `not_in_answer`, `regex_match`, `json_schema`. Zero LLM calls, zero flakiness, same verdict every run.
+2. **Path checks** — did the agent call the tools it should have? A missing expected tool warns; a forbidden tool fails.
+3. **Cost budgets** — LLM calls, tokens, dollars per query.
+4. **LLM judge** (`llm_judge` rubrics, optional) — only for answers that genuinely need judgment, evaluated after every deterministic check has run.
+
+Don't write the fact checks by hand — mine them from your knowledge base:
+
+```bash
+agentci generate-checks
+```
+
+It extracts hard facts (prices, rates, SKUs, "30 days") as variant-set assertions, and
+**validates every candidate against your recorded golden answers first** — a check that
+would fail a known-good answer is rejected before you ever see it. One LLM call at
+authoring time; the checks run free forever. Details: [docs/generate-checks.md](docs/generate-checks.md).
+
+## Demo
+
+Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. AgentCI catches the correctness regression:
+
+![AgentCI Demo](demo/agentci-rag-demo.gif)
+
+## CLI
+
+```bash
+agentci init --generate        # Scan project, generate test spec
+agentci init                   # Generate GitHub Actions workflow + pre-push hook
+agentci test --mock --yes      # Zero-cost synthetic traces, CI-friendly (no keys, no prompts)
+agentci test                   # Run 3-layer evaluation (correctness → path → cost)
+agentci test --runs 3          # Stability report: verdict flips + flip-source attribution
+agentci judge-audit            # Audit the LLM judge against checks, retests, hand labels
+agentci generate-checks        # Mine KB facts into deterministic assertions (gated)
+agentci test --format html -o report.html  # HTML report with per-query details
+agentci calibrate              # Measure real agent metrics, auto-tune spec budgets
+agentci doctor                 # Health check: spec, deps, API keys
+agentci record <test>          # Record golden baseline
+agentci diff                   # Diff against baseline
+agentci report -i results.json # Generate HTML report from JSON results
+```
+## Docs
+
+- [Quickstart](docs/quickstart.md) — install to first green run
+- [Stability testing](docs/stability.md) — `--runs N`, flip-source attribution
+- [Judge audit](docs/judge-audit.md) — is your LLM judge lying to you?
+- [Generate checks](docs/generate-checks.md) — mine KB facts into gated assertions
+- [Writing tests](docs/writing-tests.md) — the full spec reference
+- [Cost tracking](docs/cost-tracking.md) — budgets and spike detection
+- [Golden traces](docs/golden-traces.md) — record baselines, diff regressions
+- [CI/CD integration](docs/ci-cd.md) — GitHub Actions setup
+- [LangGraph](docs/langgraph.md) — graph-based agent support
+- [Metrics reference](docs/metrics_reference.md) — every metric, defined
+
+## Why not just an LLM judge?
+
+Judge-only evals are expensive, flaky, and blind to their own context. AgentCI is pytest-native regression testing: deterministic checks catch the factual failures, golden traces catch behavioral drift, cost budgets catch spend regressions — and the judge handles only what genuinely needs judgment. Mock mode (`agentci test --mock`) runs the whole suite with zero API keys and zero cost, so it can gate every PR.
+
+## Contributing
+
+[GitHub Issues](https://github.com/suniel12/AgentCI/issues) ·
+[DemoAgents](https://github.com/suniel12/DemoAgents) — working examples for OpenAI, Anthropic, and LangGraph agents
+
+Apache 2.0. If you build an agent and test it with AgentCI, I'd love to hear about it.
+
+---

ciagent-0.7.0/README.md

@@ -0,0 +1,220 @@
+# AgentCI
+
+**Pytest-native regression testing for AI agents.** Catch routing changes, tool call drift, and cost spikes before production.
+
+[![PyPI](https://img.shields.io/pypi/v/ciagent)](https://pypi.org/project/ciagent/)
+[![CI](https://github.com/suniel12/AgentCI/actions/workflows/ci.yml/badge.svg)](https://github.com/suniel12/AgentCI/actions/workflows/ci.yml)
+[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![AGENTS.md](https://img.shields.io/badge/AGENTS.md-supported-blue)](AGENTS.md)
+
+You changed a prompt. Your agent broke in production. Three days later, a user complained. You had no tests, no diff, no idea what went wrong.
+
+Works with OpenAI, Anthropic, and LangGraph. Runs inside pytest.
+
+## Add to Your Project
+
+```bash
+pip install ciagent
+```
+
+Write your golden queries — what should your agent handle, and what should it refuse?
+
+```yaml
+# agentci_spec.yaml
+agent: my-agent
+# runner: any function that takes a query string and returns a response
+runner: my_app.agent:run_for_agentci
+queries:
+  - query: "How do I install AgentCI?"
+    correctness:
+      any_expected_in_answer: ["pip install", "ciagent"]
+    path:
+      expected_tools: [retrieve_docs]
+    cost:
+      max_llm_calls: 8
+
+  - query: "What's the CEO's favorite restaurant?"
+    correctness:
+      not_in_answer: ["restaurant", "favorite"]
+    path:
+      expected_tools: []  # expect no tools called for out-of-scope queries
+```
+
+Run:
+
+```bash
+agentci test --mock       # start here: zero-cost with synthetic traces
+agentci test              # run live against your real agent
+```
+
+`agentci test` evaluates each query through 3 layers — correctness, path, and cost:
+
+```
+============================================================
+
+Query: How do I install AgentCI?
+Answer: To install AgentCI, you can use pip with the following command:
+        pip install ciagent. Make sure you have Python 3.10 or later.
+
+  ✅  CORRECTNESS: PASS
+       ✓ Found keywords: "pip install ciagent"
+       ✓ LLM judge passed (score: 5 ≥ 0.6)
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Tool precision: 0.500
+       ✓ No loops detected
+  💰  COST: PASS
+       ✓ LLM calls: 8 ≤ max 8
+
+============================================================
+
+Query: What Python version does AgentCI require and what frameworks does it support?
+Answer: AgentCI currently does not specify a required Python version
+        in the provided context, so I don't have that information...
+
+  ❌  CORRECTNESS: FAIL
+       • Expected '3.10' not found in answer
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Loops: 1 ≤ max 3
+  💰  COST: PASS
+       ✓ LLM calls: 4 ≤ max 5
+
+============================================================
+```
+
+Don't have golden queries yet? `agentci init --generate` scans your code and generates a starter spec.
+
+## A stable score is not a stable system
+
+Run the identical eval three times and you can get 96% / 95% / 96% — rock solid — while
+individual queries flip verdicts every run. The aggregate holds because the errors move
+around. `--runs N` shows what a single run can't:
+
+```bash
+agentci test --runs 3
+```
+
+```
+Run 1/3: 18/19 passed
+Run 2/3: 18/19 passed
+Run 3/3: 18/19 passed
+
+────────────────────────────────────────────────────────────
+Stability Report
+────────────────────────────────────────────────────────────
+Suite score across 3 runs: 95%  /  95%  /  95%
+
+⚠️  FLAKY — 2/19 queries flipped verdicts across runs:
+   "What's your return window?"    ✅❌✅  pass_rate=0.67  source: agent-variance (answer changed)
+   "Do you sell gift cards?"       ❌✅✅  pass_rate=0.67  source: judge-flake (same answer, verdict flipped)
+
+   Flip sources: 1 agent-variance (fix the agent) │ 1 judge-flake (fix the eval) │ 0 infra-error (retry) │ 0 mixed
+
+Stability verdict: FLAKY
+```
+
+Every flip is attributed to its source, so it's a routed work item, not a scary number:
+**agent-variance** means the agent produced different output (fix the prompt, retrieval, or
+temperature); **judge-flake** means the output — or every deterministic check's outcome —
+was identical but the LLM judge changed its mind (fix the rubric, or replace the judge with
+a deterministic check); **infra-error** means a judge API call failed (retry, fix nothing).
+Attribution is structural, not guessed: deterministic checks cannot flip on identical
+output, and per-layer sub-verdicts are compared across runs. The console shows observed
+facts; pass@k/pass^k estimates live in the JSON output, labeled as estimates.
+
+Flaky-but-passing exits 0 so adoption won't break your CI; add `--fail-on-flaky` when
+you're ready to gate on it. Try it with zero API keys:
+`AGENTCI_MOCK_FLAKY=1 agentci test --mock --runs 3`. Details: [docs/stability.md](docs/stability.md).
+
+## Audit the judge itself
+
+An LLM judge that shares your agent's context inherits your agent's blind spots: when
+retrieval comes up empty, the agent answers from nothing — and the judge, reading the same
+nothing, passes it. `judge-audit` measures your judge against ground truth you already have,
+by re-scoring recorded baselines (the agent is never re-run):
+
+```bash
+agentci judge-audit
+```
+
+1. **Judge vs. deterministic checks** — the disagreement matrix. The row that matters:
+   answers the judge PASSED that a hard fact-check FAILED.
+2. **Retest stability** — the same answer judged `--repeats` times; flips on identical
+   input are the judge's own noise floor.
+3. **Hand labels** (`--labels`) — agreement + Cohen's κ against your own review.
+
+The claim is deliberately one-directional: a judge that fails where you *can* check it
+shouldn't be trusted where you can't. Verdict: `TRUSTWORTHY` / `NEEDS CALIBRATION` /
+`UNRELIABLE`. Details: [docs/judge-audit.md](docs/judge-audit.md).
+
+## Check facts in code. Save the judge for judgment.
+
+Most agent failures that matter involve a hard fact — a product name, a price, a version number. Those are checkable deterministically, for free. And an LLM judge grading against the same context as your agent inherits your agent's blind spots: when retrieval comes up empty, the agent answers from nothing and the judge — reading the same nothing — passes it.
+
+So AgentCI runs deterministic checks first and treats the judge as the last resort, not the default:
+
+1. **Fact checks in code** — `expected_in_answer`, `not_in_answer`, `regex_match`, `json_schema`. Zero LLM calls, zero flakiness, same verdict every run.
+2. **Path checks** — did the agent call the tools it should have? A missing expected tool warns; a forbidden tool fails.
+3. **Cost budgets** — LLM calls, tokens, dollars per query.
+4. **LLM judge** (`llm_judge` rubrics, optional) — only for answers that genuinely need judgment, evaluated after every deterministic check has run.
+
+Don't write the fact checks by hand — mine them from your knowledge base:
+
+```bash
+agentci generate-checks
+```
+
+It extracts hard facts (prices, rates, SKUs, "30 days") as variant-set assertions, and
+**validates every candidate against your recorded golden answers first** — a check that
+would fail a known-good answer is rejected before you ever see it. One LLM call at
+authoring time; the checks run free forever. Details: [docs/generate-checks.md](docs/generate-checks.md).
+
+## Demo
+
+Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. AgentCI catches the correctness regression:
+
+![AgentCI Demo](demo/agentci-rag-demo.gif)
+
+## CLI
+
+```bash
+agentci init --generate        # Scan project, generate test spec
+agentci init                   # Generate GitHub Actions workflow + pre-push hook
+agentci test --mock --yes      # Zero-cost synthetic traces, CI-friendly (no keys, no prompts)
+agentci test                   # Run 3-layer evaluation (correctness → path → cost)
+agentci test --runs 3          # Stability report: verdict flips + flip-source attribution
+agentci judge-audit            # Audit the LLM judge against checks, retests, hand labels
+agentci generate-checks        # Mine KB facts into deterministic assertions (gated)
+agentci test --format html -o report.html  # HTML report with per-query details
+agentci calibrate              # Measure real agent metrics, auto-tune spec budgets
+agentci doctor                 # Health check: spec, deps, API keys
+agentci record <test>          # Record golden baseline
+agentci diff                   # Diff against baseline
+agentci report -i results.json # Generate HTML report from JSON results
+```
+## Docs
+
+- [Quickstart](docs/quickstart.md) — install to first green run
+- [Stability testing](docs/stability.md) — `--runs N`, flip-source attribution
+- [Judge audit](docs/judge-audit.md) — is your LLM judge lying to you?
+- [Generate checks](docs/generate-checks.md) — mine KB facts into gated assertions
+- [Writing tests](docs/writing-tests.md) — the full spec reference
+- [Cost tracking](docs/cost-tracking.md) — budgets and spike detection
+- [Golden traces](docs/golden-traces.md) — record baselines, diff regressions
+- [CI/CD integration](docs/ci-cd.md) — GitHub Actions setup
+- [LangGraph](docs/langgraph.md) — graph-based agent support
+- [Metrics reference](docs/metrics_reference.md) — every metric, defined
+
+## Why not just an LLM judge?
+
+Judge-only evals are expensive, flaky, and blind to their own context. AgentCI is pytest-native regression testing: deterministic checks catch the factual failures, golden traces catch behavioral drift, cost budgets catch spend regressions — and the judge handles only what genuinely needs judgment. Mock mode (`agentci test --mock`) runs the whole suite with zero API keys and zero cost, so it can gate every PR.
+
+## Contributing
+
+[GitHub Issues](https://github.com/suniel12/AgentCI/issues) ·
+[DemoAgents](https://github.com/suniel12/DemoAgents) — working examples for OpenAI, Anthropic, and LangGraph agents
+
+Apache 2.0. If you build an agent and test it with AgentCI, I'd love to hear about it.
+
+---

{ciagent-0.6.0 → ciagent-0.7.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ciagent"
-version = "0.6.0"
+version = "0.7.0"
 description = "Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production."
 readme = "README.md"
 license = {text = "Apache-2.0"}