ciagent 0.6.0__tar.gz → 0.8.0__tar.gz

ciagent-0.8.0/CHANGELOG.md

@@ -0,0 +1,264 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.8.0] - 2026-07-06
+
+### Added — Claude Code plugin (F5: your coding agent onboards and operates AgentCI)
+- The repo is now a Claude Code plugin marketplace
+  (`/plugin marketplace add suniel12/AgentCI`, `/plugin install agentci@agentci`)
+  with two skills: **onboard** (write the runner, record goldens via
+  `bootstrap --yes`, generate + tighten the spec, verify with `test --runs 3`)
+  and **check** (run the right check after agent changes; route failures by
+  flip source; never weaken a correct check to go green)
+- `agentci bootstrap --yes`: fully non-interactive golden recording (requires
+  `--queries`); bootstrap now runs the runner through the same TraceContext
+  capture path as `agentci test`, so **runners returning plain strings work**
+  (previously crashed on anything but a Trace)
+- `--format json` output now includes the agent's `answer` text per query
+  (eng-review work item: JSON consumers need what the agent said, not just
+  verdicts; also unblocks JSON as a judge-audit answers source)
+- Tests: skills are lint-gated — every `agentci` command and flag a skill
+  teaches is asserted to exist in the CLI, so skill docs cannot rot silently
+
+### Fixed — from Codex cross-model review of this branch
+- **Nested `TraceContext` no longer double-records or loses capture**: a
+  Trace-returning runner that uses `TraceContext` internally, wrapped again by
+  the executor, used to stack SDK patches (every LLM call recorded once per
+  wrapper) and its exit cleared the outer context entirely. Patches now install
+  only in the outermost context, and exits restore the enclosing context via
+  contextvar tokens. Affected `agentci test` as well as the new bootstrap path
+- JSON `answer` field now uses the correctness layer's extractor (metadata →
+  last span output), so trace shapes the evaluator can grade also serialize
+  their answer instead of `null`
+
+### Added — zero-key demo (`uvx ciagent test --mock --runs 3`)
+- `agentci test --mock` with no `agentci_spec.yaml` in the working directory now
+  falls back to a bundled demo spec (8 synthetic support-agent queries), clearly
+  labeled as demo mode with synthetic data. An explicitly passed `--config` that
+  is missing remains an error — the fallback only applies to the default path
+- Demo multi-run sessions simulate a pseudo-flaky agent by default with the new
+  `"spread"` style (one query breaks per run), so the aggregate score stays
+  constant while individual verdicts flip — the report the demo exists to show.
+  `AGENTCI_MOCK_FLAKY=0` turns the simulation off; non-demo specs are unaffected
+  (flakiness stays env-var opt-in, `"alternate"` style unchanged)
+- Missing spec without `--mock` now exits with a hint pointing at both
+  `agentci init` and the zero-key demo command
+
+### Changed — stability report hardening (pre-launch fixes from eng review)
+- **Flip attribution now compares per-layer sub-verdicts first**: if every
+  deterministic check returned the same outcome across runs and only the LLM judge's
+  verdict changed, the flip is `judge-flake` even when the answer text was
+  paraphrased (previously mislabeled `agent-variance`)
+- **New flip source `infra-error`**: a judge API failure counted as a fail no longer
+  reads as `judge-flake` — one transient hiccup must not say "fix your rubric"
+- **Console shows observed facts only** (verdict history + pass rate); pass@k /
+  pass^k move to JSON output as clearly labeled estimates (at small k they restate
+  the pass rate)
+- Duplicate query texts in a spec are flagged (they merge into one stability
+  record); queries missing from some runs are marked `partial (k/N runs)`
+- `agentci run` (legacy suite runner) now prints a deprecation warning pointing at
+  `agentci test`; removal planned for 0.9.0
+- Removed stale `src/agentci/_version.py` (said 0.1.0; version comes from package
+  metadata)
+
+## [0.7.0] - 2026-07-05
+
+### Added
+
+#### KB-Derived Fact Checks — `agentci generate-checks`
+Mines the knowledge base for hard facts (prices, rates, SKUs, versions,
+explicit quantities) and proposes them as deterministic assertions on
+existing spec queries. One LLM call at authoring time; the checks run
+deterministically forever at zero cost.
+
+- **Brittleness gate**: every candidate is validated against recorded golden
+  answers before it is offered — a check that would fail a known-good answer
+  is rejected automatically with the failing answer shown
+- Only non-paraphrasable facts; prose facts become variant sets
+  (`any_expected_in_answer`), never single literal strings; only
+  `any_expected_in_answer` / `not_in_answer` / `regex_match` are proposed
+- Candidates without a golden answer are `unvalidated` and never
+  auto-accepted, even with `--yes`; interactive review for everything else
+- Merge never overwrites user-written assertions; `.bak` backup on write;
+  `--dry-run` mode
+- New module `agentci.engine.check_generator`; 21 new tests
+
+#### Judge Audit — `agentci judge-audit`
+Meta-evaluation of the LLM judge against ground truth you already have,
+re-scoring recorded golden baselines (the agent is never re-run):
+
+- **Mode 1 — judge vs. deterministic checks**: independent verdicts on the
+  same recorded answer; reports the disagreement matrix. The killer row:
+  answers the judge PASSED that a deterministic fact-check FAILED
+  (shared-context judge blindness, detected automatically)
+- **Mode 2 — retest stability** (`--repeats K`, default 3): same answer
+  judged K times; verdict flips on identical input are the judge's noise floor
+- **Mode 3 — hand labels** (`--labels FILE`): agreement + Cohen's κ against
+  human review, with standard trust thresholds
+- Scoped claim stated in the report itself: measured on fact-checkable
+  queries; one-directional (disqualifying) signal for judgment-only queries
+- Verdicts: TRUSTWORTHY / NEEDS CALIBRATION / UNRELIABLE (exit 1) /
+  ERROR when the judge never ran (exit 2 — a judge that couldn't run is
+  never scored)
+- `--sample N` cost cap; console + JSON output
+- New module `agentci.engine.judge_audit`; 27 new tests
+
+#### Stability Report — `agentci test --runs N`
+A stable suite score can hide per-query verdict flips: the aggregate holds
+because the errors move around. `--runs N` executes the whole suite N times
+and reports what a single run cannot show:
+
+- Per-query verdict history (✅❌✅), pass rate, pass@k and pass^k estimates
+- Suite score per run printed side by side with the queries that flipped
+- **Flip-source attribution** — every flip is labelled:
+  - `agent-variance`: the agent's output changed → fix the agent
+  - `judge-flake`: same output, the LLM judge changed its verdict → fix the eval
+  - `mixed`: near-identical paraphrase with a judge configured — not guessed
+  Attribution rests on a structural fact: deterministic checks cannot flip on
+  identical output, so identical answer + tools + flipped verdict = judge.
+- Exit semantics: flaky-but-passing exits 0 (warnings only); queries failing
+  in EVERY run exit 1; `--fail-on-flaky` escalates flips to exit 1
+- Works in every format: console section, GitHub `::warning` annotations,
+  `stability` block in JSON, stability card in the HTML report
+- Mock mode support: `AGENTCI_MOCK_FLAKY=1 agentci test --mock --runs 3`
+  demonstrates the report with zero API keys
+- New module `agentci.engine.stability` (`build_stability_report`,
+  `StabilityReport`, `QueryStability`, `FlipSource`); 21 new tests
+
+### Changed
+- **`expected_tools` now asserts by default**: a missing expected tool produces a WARN
+  (tool recall gates at 1.0 unless `min_tool_recall` explicitly loosens it). Previously,
+  without `min_tool_recall`, a recall of 0.0 displayed as PASS with a checkmark.
+- **`expected_tools: []` now asserts that no tools are called**: an explicit empty list
+  produces a WARN if the agent called any tool. Previously it was silently skipped.
+- `agentci doctor` no longer reports numpy as a required dependency (it was never used
+  by AgentCI, so every fresh install showed a false failure).
+
+### Docs
+- README: fixed the quickstart spec example (was missing the required `agent:` field),
+  added CI badge, "Check facts in code" section, docs index
+- Rewrote `docs/quickstart.md`, `docs/ci-cd.md`, `docs/cost-tracking.md`, and
+  `docs/golden-traces.md` to match the current `agentci test` workflow
+- Fixed dead clone URLs in CONTRIBUTING.md and quickstart
+- Removed unused demo GIF variants and cast recordings (6 files, ~2 MB)
+
+## [0.6.0] - 2026-03-05
+
+### Added
+
+#### `final_output` Auto-Capture
+- `TraceContext._auto_extract_final_output()` called in `__exit__()` — automatically extracts the agent's answer from traces
+- Extraction priority: LangGraph state messages > span `output_data` (string) > span `output_data` (dict with `content`/`message`/`text`/`output` keys) > last LLM call `output_text`
+- Manual `trace.metadata["final_output"]` still takes precedence (no overwrite)
+- LangGraph adapter: auto-sets `final_output` from last AI message in `parse_state()`
+- OpenAI Agents adapter: auto-sets `final_output` from last span output in `on_trace_end()`
+
+#### `agentci calibrate` Command
+- Runs N sample queries against the live agent, measures actual metrics, shows Rich comparison table
+- Updates spec budgets with headroom: +50% for LLM/tool calls, +100% for tokens/cost
+- Flags: `--samples N` (default 2), `--dry-run`, `--yes`, `--spec PATH`
+
+#### Strict Tool Sequence Assertions
+- `PathSpec.expected_tool_sequence: Optional[list[str]]` — strict ordered tool call check
+- Mismatch = WARN (soft warning) with position-level diff via `_format_sequence_diff()`
+
+#### HTML Trace Report
+- Self-contained `report.html.j2` Jinja2 template with dark theme
+- Summary dashboard (pass/fail/warn counts, total cost)
+- Per-query cards with status badges, answer preview, three-layer details
+- Collapsible trace tree with JS toggle
+- Available via `agentci test --format html --output report.html` or `agentci report -i results.json`
+
+### Changed
+- `max_llm_calls` default in spec generator raised from 8 to 10 (better headroom for real agents)
+- `max_llm_calls` fallback in mock runner raised from 3 to 10
+- Calibrate command floor raised from 8 to 10
+- `--format` choices in `test` and `eval` commands now include `html`
+- `--output / -o` option added to `test` and `eval` commands for HTML file path
+- `agentci report` command fully implemented (was stub) — converts JSON results to HTML
+- 22 new tests added (570 total, up from 548 in v0.5.1)
+
+## [0.5.0] - 2026-03-01
+
+### Added
+
+#### Three-Layer Evaluation Engine
+- **Correctness layer** (Layer 1 — hard fail): keyword matching, LLM-as-a-judge, safety checks, hallucination checks, regex/exact match, JSON schema validation
+- **Path layer** (Layer 2 — soft warn): tool trajectory validation, loop detection (default `max_loops=3`), routing assertions, handoff expectations
+- **Cost layer** (Layer 3 — soft warn): token budgets, cost caps, LLM call limits, latency thresholds
+- `runner.py` orchestrates all three layers per query
+- `parallel.py` for parallel query execution across specs
+
+#### OR-Logic Keywords
+- `any_expected_in_answer` field — at least one keyword must match (complementing `expected_in_answer` which requires all)
+
+#### LLM Judge Enhancements
+- `context_file` support in `JudgeRubric` — doc-grounded judging against reference documents
+- `refutes_premise` flag — injects built-in premise-correction rubric for trick questions
+
+#### Span Assertions
+- `SpanAssertionSpec` schema model for span-level assertions
+- `Span.attributes: dict[str, Any]` — OTel-style span-level data propagation
+- Span-level LLM judge support
+
+#### Mock Testing Mode
+- `agentci test --mock` — generates synthetic traces, zero API cost
+- `mock_runner.py` — synthetic trace generation from spec expectations
+- `--golden-file` flag — load Q&A pairs from JSON/CSV for mock mode
+
+#### Cost Estimator
+- `cost_estimator.py` — pre-execution cost estimates with pricing table
+- Cost estimate shown before live test runs; `--yes`/`-y` skips confirmation
+
+#### CLI Improvements
+- `agentci init --generate` — AI-assisted spec generation with guided interview
+- `agentci doctor` — health-check command (spec, runner, API keys, deps, CI)
+- Scan-first flow: auto-scan project before questions, show summary
+- Agent type auto-detected via `_detect_agent_type_from_code()`
+- Skeleton template generation with TODO placeholders for zero-API-key usage
+- Context-aware "Next Steps" based on mode (mock vs live)
+- Non-interactive flags: `--kb-path`, `--mode`
+
+#### Diff Engine
+- Three-tier diff engine for baseline comparison
+- 11 `DiffType` categories including `ROUTING_CHANGED`, `GUARDRAILS_CHANGED`
+- `agentci diff` CLI command
+
+#### Reporting
+- GitHub annotations with budget cap (`MAX_INLINE_ANNOTATIONS = 10` for warnings; errors uncapped)
+- JSON output format
+- Prometheus metrics export
+
+#### Trace Helpers
+- `Trace.called(tool)` / `never_called(tool)` / `loop_count(tool)` — readable assertion helpers
+- `Trace.cost_under(usd)` / `llm_calls_under(n)` — budget assertion helpers
+- `langgraph_trace(agent_name)` — context manager shortcut for LangGraph
+- `TraceContext.attach(state)` — alias for `attach_langgraph_state()`
+
+#### Adapters
+- OpenAI Agents SDK adapter (`openai_agents.py`)
+- LangGraph adapter (`langgraph.py`)
+
+#### Other
+- `python-dotenv` added as core dependency
+- Deep KB sampling: 2000 chars/file for spec generation
+- Progressive spec building: smoke queries (3) then full queries (10-12)
+- Pytest plugin entry point (`pytest11: agentci`)
+- GUARDRAIL span type, HANDOFF span type
+
+### Changed
+- `PathSpec.max_loops` now defaults to `3` (was `None`)
+- Development status upgraded from Alpha to Beta
+- Package version bumped to 0.5.0
+
+## [0.4.1] - 2026-02-20
+
+### Added
+- Initial project structure and core models
+- Basic trace capture and assertion framework
+- CLI scaffolding with `agentci init`
+- PyPI publishing as `ciagent`

ciagent-0.8.0/PKG-INFO

@@ -0,0 +1,307 @@
+Metadata-Version: 2.4
+Name: ciagent
+Version: 0.8.0
+Summary: Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production.
+Project-URL: Documentation, https://github.com/suniel12/AgentCI#readme
+Project-URL: Repository, https://github.com/suniel12/AgentCI.git
+Project-URL: Issues, https://github.com/suniel12/AgentCI/issues
+License: Apache-2.0
+License-File: LICENSE
+Keywords: agents,ai,ci-cd,llm,regression,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: click>=8.0
+Requires-Dist: jinja2>=3.0
+Requires-Dist: opentelemetry-api>=1.20
+Requires-Dist: opentelemetry-sdk>=1.20
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pytest>=7.0
+Requires-Dist: python-dotenv>=0.19
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Provides-Extra: all
+Requires-Dist: anthropic>=0.20; extra == 'all'
+Requires-Dist: jsonschema>=4.0; extra == 'all'
+Requires-Dist: langgraph>=0.1; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.20; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: jsonschema
+Requires-Dist: jsonschema>=4.0; extra == 'jsonschema'
+Provides-Extra: langgraph
+Requires-Dist: langgraph>=0.1; extra == 'langgraph'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# AgentCI
+
+**Pytest-native regression testing for AI agents.** Catch routing changes, tool call drift, and cost spikes before production.
+
+[![PyPI](https://img.shields.io/pypi/v/ciagent)](https://pypi.org/project/ciagent/)
+[![CI](https://github.com/suniel12/AgentCI/actions/workflows/ci.yml/badge.svg)](https://github.com/suniel12/AgentCI/actions/workflows/ci.yml)
+[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![AGENTS.md](https://img.shields.io/badge/AGENTS.md-supported-blue)](AGENTS.md)
+
+You changed a prompt. Your agent broke in production. Three days later, a user complained. You had no tests, no diff, no idea what went wrong.
+
+Works with OpenAI, Anthropic, and LangGraph. Runs inside pytest.
+
+## See It in 30 Seconds
+
+No install, no API keys, no config — one command runs a bundled demo suite three times on synthetic traces:
+
+```bash
+uvx ciagent test --mock --runs 3
+```
+
+```
+Run 1/3: 7/8 passed
+Run 2/3: 7/8 passed
+Run 3/3: 7/8 passed
+
+────────────────────────────────────────────────────────────
+Stability Report
+────────────────────────────────────────────────────────────
+Suite score across 3 runs: 88%  /  88%  /  88%     ← looks stable
+
+⚠️  FLAKY — 3/8 queries flipped verdicts across runs:      ← is not
+   "What's your return window?"    ❌✅✅  pass_rate=0.67  source: agent-variance
+   "Do you ship internationally?"  ✅❌✅  pass_rate=0.67  source: agent-variance
+   "How do I reset my password?"   ✅✅❌  pass_rate=0.67  source: agent-variance
+
+   Flip sources: 3 agent-variance (fix the agent) │ 0 judge-flake (fix the eval)
+```
+
+The aggregate score is identical every run. Three of the eight verdicts underneath it flipped. A single-run eval score would never tell you — the stability report does, and attributes every flip to the layer that caused it. (The demo simulates a flaky agent; point it at your own with the spec below.)
+
+## Add to Your Project
+
+```bash
+pip install ciagent
+```
+
+Write your golden queries — what should your agent handle, and what should it refuse?
+
+```yaml
+# agentci_spec.yaml
+agent: my-agent
+# runner: any function that takes a query string and returns a response
+runner: my_app.agent:run_for_agentci
+queries:
+  - query: "How do I install AgentCI?"
+    correctness:
+      any_expected_in_answer: ["pip install", "ciagent"]
+    path:
+      expected_tools: [retrieve_docs]
+    cost:
+      max_llm_calls: 8
+
+  - query: "What's the CEO's favorite restaurant?"
+    correctness:
+      not_in_answer: ["restaurant", "favorite"]
+    path:
+      expected_tools: []  # expect no tools called for out-of-scope queries
+```
+
+Run:
+
+```bash
+agentci test --mock       # start here: zero-cost with synthetic traces
+agentci test              # run live against your real agent
+```
+
+`agentci test` evaluates each query through 3 layers — correctness, path, and cost:
+
+```
+============================================================
+
+Query: How do I install AgentCI?
+Answer: To install AgentCI, you can use pip with the following command:
+        pip install ciagent. Make sure you have Python 3.10 or later.
+
+  ✅  CORRECTNESS: PASS
+       ✓ Found keywords: "pip install ciagent"
+       ✓ LLM judge passed (score: 5 ≥ 0.6)
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Tool precision: 0.500
+       ✓ No loops detected
+  💰  COST: PASS
+       ✓ LLM calls: 8 ≤ max 8
+
+============================================================
+
+Query: What Python version does AgentCI require and what frameworks does it support?
+Answer: AgentCI currently does not specify a required Python version
+        in the provided context, so I don't have that information...
+
+  ❌  CORRECTNESS: FAIL
+       • Expected '3.10' not found in answer
+  📈  PATH: PASS
+       ✓ Tool recall: 1.000 (expected: [retrieve_docs])
+       ✓ Loops: 1 ≤ max 3
+  💰  COST: PASS
+       ✓ LLM calls: 4 ≤ max 5
+
+============================================================
+```
+
+Don't have golden queries yet? `agentci init --generate` scans your code and generates a starter spec.
+
+## A stable score is not a stable system
+
+Run the identical eval three times and you can get 96% / 95% / 96% — rock solid — while
+individual queries flip verdicts every run. The aggregate holds because the errors move
+around. `--runs N` shows what a single run can't:
+
+```bash
+agentci test --runs 3
+```
+
+```
+Run 1/3: 18/19 passed
+Run 2/3: 18/19 passed
+Run 3/3: 18/19 passed
+
+────────────────────────────────────────────────────────────
+Stability Report
+────────────────────────────────────────────────────────────
+Suite score across 3 runs: 95%  /  95%  /  95%
+
+⚠️  FLAKY — 2/19 queries flipped verdicts across runs:
+   "What's your return window?"    ✅❌✅  pass_rate=0.67  source: agent-variance (answer changed)
+   "Do you sell gift cards?"       ❌✅✅  pass_rate=0.67  source: judge-flake (same answer, verdict flipped)
+
+   Flip sources: 1 agent-variance (fix the agent) │ 1 judge-flake (fix the eval) │ 0 infra-error (retry) │ 0 mixed
+
+Stability verdict: FLAKY
+```
+
+Every flip is attributed to its source, so it's a routed work item, not a scary number:
+**agent-variance** means the agent produced different output (fix the prompt, retrieval, or
+temperature); **judge-flake** means the output — or every deterministic check's outcome —
+was identical but the LLM judge changed its mind (fix the rubric, or replace the judge with
+a deterministic check); **infra-error** means a judge API call failed (retry, fix nothing).
+Attribution is structural, not guessed: deterministic checks cannot flip on identical
+output, and per-layer sub-verdicts are compared across runs. The console shows observed
+facts; pass@k/pass^k estimates live in the JSON output, labeled as estimates.
+
+Flaky-but-passing exits 0 so adoption won't break your CI; add `--fail-on-flaky` when
+you're ready to gate on it. Try it with zero API keys:
+`AGENTCI_MOCK_FLAKY=1 agentci test --mock --runs 3`. Details: [docs/stability.md](docs/stability.md).
+
+## Audit the judge itself
+
+An LLM judge that shares your agent's context inherits your agent's blind spots: when
+retrieval comes up empty, the agent answers from nothing — and the judge, reading the same
+nothing, passes it. `judge-audit` measures your judge against ground truth you already have,
+by re-scoring recorded baselines (the agent is never re-run):
+
+```bash
+agentci judge-audit
+```
+
+1. **Judge vs. deterministic checks** — the disagreement matrix. The row that matters:
+   answers the judge PASSED that a hard fact-check FAILED.
+2. **Retest stability** — the same answer judged `--repeats` times; flips on identical
+   input are the judge's own noise floor.
+3. **Hand labels** (`--labels`) — agreement + Cohen's κ against your own review.
+
+The claim is deliberately one-directional: a judge that fails where you *can* check it
+shouldn't be trusted where you can't. Verdict: `TRUSTWORTHY` / `NEEDS CALIBRATION` /
+`UNRELIABLE`. Details: [docs/judge-audit.md](docs/judge-audit.md).
+
+## Check facts in code. Save the judge for judgment.
+
+Most agent failures that matter involve a hard fact — a product name, a price, a version number. Those are checkable deterministically, for free. And an LLM judge grading against the same context as your agent inherits your agent's blind spots: when retrieval comes up empty, the agent answers from nothing and the judge — reading the same nothing — passes it.
+
+So AgentCI runs deterministic checks first and treats the judge as the last resort, not the default:
+
+1. **Fact checks in code** — `expected_in_answer`, `not_in_answer`, `regex_match`, `json_schema`. Zero LLM calls, zero flakiness, same verdict every run.
+2. **Path checks** — did the agent call the tools it should have? A missing expected tool warns; a forbidden tool fails.
+3. **Cost budgets** — LLM calls, tokens, dollars per query.
+4. **LLM judge** (`llm_judge` rubrics, optional) — only for answers that genuinely need judgment, evaluated after every deterministic check has run.
+
+Don't write the fact checks by hand — mine them from your knowledge base:
+
+```bash
+agentci generate-checks
+```
+
+It extracts hard facts (prices, rates, SKUs, "30 days") as variant-set assertions, and
+**validates every candidate against your recorded golden answers first** — a check that
+would fail a known-good answer is rejected before you ever see it. One LLM call at
+authoring time; the checks run free forever. Details: [docs/generate-checks.md](docs/generate-checks.md).
+
+## Let your coding agent set it up
+
+AgentCI ships as a Claude Code plugin. Two skills: **onboard** (writes the runner,
+records golden baselines, generates the spec, verifies it) and **check** (runs the
+right test after every change to your agent and routes failures by flip source).
+
+```
+/plugin marketplace add suniel12/AgentCI
+/plugin install agentci@agentci
+```
+
+Then ask your coding agent to "set up AgentCI for this repo." It records goldens with
+`agentci bootstrap --yes` and verifies with `agentci test --runs 3` — no human CLI use
+needed. The runner it writes is one function: `(query: str) -> str`; trace capture is
+automatic.
+
+## Demo
+
+Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. AgentCI catches the correctness regression:
+
+![AgentCI Demo](demo/agentci-rag-demo.gif)
+
+## CLI
+
+```bash
+agentci init --generate        # Scan project, generate test spec
+agentci init                   # Generate GitHub Actions workflow + pre-push hook
+agentci test --mock --yes      # Zero-cost synthetic traces, CI-friendly (no keys, no prompts)
+agentci test                   # Run 3-layer evaluation (correctness → path → cost)
+agentci test --runs 3          # Stability report: verdict flips + flip-source attribution
+agentci judge-audit            # Audit the LLM judge against checks, retests, hand labels
+agentci generate-checks        # Mine KB facts into deterministic assertions (gated)
+agentci test --format html -o report.html  # HTML report with per-query details
+agentci calibrate              # Measure real agent metrics, auto-tune spec budgets
+agentci doctor                 # Health check: spec, deps, API keys
+agentci record <test>          # Record golden baseline
+agentci diff                   # Diff against baseline
+agentci report -i results.json # Generate HTML report from JSON results
+```
+## Docs
+
+- [Quickstart](docs/quickstart.md) — install to first green run
+- [Stability testing](docs/stability.md) — `--runs N`, flip-source attribution
+- [Judge audit](docs/judge-audit.md) — is your LLM judge lying to you?
+- [Generate checks](docs/generate-checks.md) — mine KB facts into gated assertions
+- [Writing tests](docs/writing-tests.md) — the full spec reference
+- [Cost tracking](docs/cost-tracking.md) — budgets and spike detection
+- [Golden traces](docs/golden-traces.md) — record baselines, diff regressions
+- [CI/CD integration](docs/ci-cd.md) — GitHub Actions setup
+- [LangGraph](docs/langgraph.md) — graph-based agent support
+- [Metrics reference](docs/metrics_reference.md) — every metric, defined
+
+## Why not just an LLM judge?
+
+Judge-only evals are expensive, flaky, and blind to their own context. AgentCI is pytest-native regression testing: deterministic checks catch the factual failures, golden traces catch behavioral drift, cost budgets catch spend regressions — and the judge handles only what genuinely needs judgment. Mock mode (`agentci test --mock`) runs the whole suite with zero API keys and zero cost, so it can gate every PR.
+
+## Contributing
+
+[GitHub Issues](https://github.com/suniel12/AgentCI/issues) ·
+[DemoAgents](https://github.com/suniel12/DemoAgents) — working examples for OpenAI, Anthropic, and LangGraph agents
+
+Apache 2.0. If you build an agent and test it with AgentCI, I'd love to hear about it.
+
+---