ciagent 0.7.0__tar.gz → 0.8.0__tar.gz

{ciagent-0.7.0 → ciagent-0.8.0}/CHANGELOG.md

@@ -5,7 +5,48 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.8.0] - 2026-07-06
+
+### Added — Claude Code plugin (F5: your coding agent onboards and operates AgentCI)
+- The repo is now a Claude Code plugin marketplace
+  (`/plugin marketplace add suniel12/AgentCI`, `/plugin install agentci@agentci`)
+  with two skills: **onboard** (write the runner, record goldens via
+  `bootstrap --yes`, generate + tighten the spec, verify with `test --runs 3`)
+  and **check** (run the right check after agent changes; route failures by
+  flip source; never weaken a correct check to go green)
+- `agentci bootstrap --yes`: fully non-interactive golden recording (requires
+  `--queries`); bootstrap now runs the runner through the same TraceContext
+  capture path as `agentci test`, so **runners returning plain strings work**
+  (previously crashed on anything but a Trace)
+- `--format json` output now includes the agent's `answer` text per query
+  (eng-review work item: JSON consumers need what the agent said, not just
+  verdicts; also unblocks JSON as a judge-audit answers source)
+- Tests: skills are lint-gated — every `agentci` command and flag a skill
+  teaches is asserted to exist in the CLI, so skill docs cannot rot silently
+
+### Fixed — from Codex cross-model review of this branch
+- **Nested `TraceContext` no longer double-records or loses capture**: a
+  Trace-returning runner that uses `TraceContext` internally, wrapped again by
+  the executor, used to stack SDK patches (every LLM call recorded once per
+  wrapper) and its exit cleared the outer context entirely. Patches now install
+  only in the outermost context, and exits restore the enclosing context via
+  contextvar tokens. Affected `agentci test` as well as the new bootstrap path
+- JSON `answer` field now uses the correctness layer's extractor (metadata →
+  last span output), so trace shapes the evaluator can grade also serialize
+  their answer instead of `null`
+
+### Added — zero-key demo (`uvx ciagent test --mock --runs 3`)
+- `agentci test --mock` with no `agentci_spec.yaml` in the working directory now
+  falls back to a bundled demo spec (8 synthetic support-agent queries), clearly
+  labeled as demo mode with synthetic data. An explicitly passed `--config` that
+  is missing remains an error — the fallback only applies to the default path
+- Demo multi-run sessions simulate a pseudo-flaky agent by default with the new
+  `"spread"` style (one query breaks per run), so the aggregate score stays
+  constant while individual verdicts flip — the report the demo exists to show.
+  `AGENTCI_MOCK_FLAKY=0` turns the simulation off; non-demo specs are unaffected
+  (flakiness stays env-var opt-in, `"alternate"` style unchanged)
+- Missing spec without `--mock` now exits with a hint pointing at both
+  `agentci init` and the zero-key demo command
 
 ### Changed — stability report hardening (pre-launch fixes from eng review)
 - **Flip attribution now compares per-layer sub-verdicts first**: if every

{ciagent-0.7.0 → ciagent-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ciagent
-Version: 0.7.0
+Version: 0.8.0
 Summary: Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production.
 Project-URL: Documentation, https://github.com/suniel12/AgentCI#readme
 Project-URL: Repository, https://github.com/suniel12/AgentCI.git
@@ -54,6 +54,34 @@ You changed a prompt. Your agent broke in production. Three days later, a user c
 
 Works with OpenAI, Anthropic, and LangGraph. Runs inside pytest.
 
+## See It in 30 Seconds
+
+No install, no API keys, no config — one command runs a bundled demo suite three times on synthetic traces:
+
+```bash
+uvx ciagent test --mock --runs 3
+```
+
+```
+Run 1/3: 7/8 passed
+Run 2/3: 7/8 passed
+Run 3/3: 7/8 passed
+
+────────────────────────────────────────────────────────────
+Stability Report
+────────────────────────────────────────────────────────────
+Suite score across 3 runs: 88%  /  88%  /  88%     ← looks stable
+
+⚠️  FLAKY — 3/8 queries flipped verdicts across runs:      ← is not
+   "What's your return window?"    ❌✅✅  pass_rate=0.67  source: agent-variance
+   "Do you ship internationally?"  ✅❌✅  pass_rate=0.67  source: agent-variance
+   "How do I reset my password?"   ✅✅❌  pass_rate=0.67  source: agent-variance
+
+   Flip sources: 3 agent-variance (fix the agent) │ 0 judge-flake (fix the eval)
+```
+
+The aggregate score is identical every run. Three of the eight verdicts underneath it flipped. A single-run eval score would never tell you — the stability report does, and attributes every flip to the layer that caused it. (The demo simulates a flaky agent; point it at your own with the spec below.)
+
 ## Add to Your Project
 
 ```bash
@@ -213,6 +241,22 @@ It extracts hard facts (prices, rates, SKUs, "30 days") as variant-set assertion
 would fail a known-good answer is rejected before you ever see it. One LLM call at
 authoring time; the checks run free forever. Details: [docs/generate-checks.md](docs/generate-checks.md).
 
+## Let your coding agent set it up
+
+AgentCI ships as a Claude Code plugin. Two skills: **onboard** (writes the runner,
+records golden baselines, generates the spec, verifies it) and **check** (runs the
+right test after every change to your agent and routes failures by flip source).
+
+```
+/plugin marketplace add suniel12/AgentCI
+/plugin install agentci@agentci
+```
+
+Then ask your coding agent to "set up AgentCI for this repo." It records goldens with
+`agentci bootstrap --yes` and verifies with `agentci test --runs 3` — no human CLI use
+needed. The runner it writes is one function: `(query: str) -> str`; trace capture is
+automatic.
+
 ## Demo
 
 Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. AgentCI catches the correctness regression:

{ciagent-0.7.0 → ciagent-0.8.0}/README.md

@@ -11,6 +11,34 @@ You changed a prompt. Your agent broke in production. Three days later, a user c
 
 Works with OpenAI, Anthropic, and LangGraph. Runs inside pytest.
 
+## See It in 30 Seconds
+
+No install, no API keys, no config — one command runs a bundled demo suite three times on synthetic traces:
+
+```bash
+uvx ciagent test --mock --runs 3
+```
+
+```
+Run 1/3: 7/8 passed
+Run 2/3: 7/8 passed
+Run 3/3: 7/8 passed
+
+────────────────────────────────────────────────────────────
+Stability Report
+────────────────────────────────────────────────────────────
+Suite score across 3 runs: 88%  /  88%  /  88%     ← looks stable
+
+⚠️  FLAKY — 3/8 queries flipped verdicts across runs:      ← is not
+   "What's your return window?"    ❌✅✅  pass_rate=0.67  source: agent-variance
+   "Do you ship internationally?"  ✅❌✅  pass_rate=0.67  source: agent-variance
+   "How do I reset my password?"   ✅✅❌  pass_rate=0.67  source: agent-variance
+
+   Flip sources: 3 agent-variance (fix the agent) │ 0 judge-flake (fix the eval)
+```
+
+The aggregate score is identical every run. Three of the eight verdicts underneath it flipped. A single-run eval score would never tell you — the stability report does, and attributes every flip to the layer that caused it. (The demo simulates a flaky agent; point it at your own with the spec below.)
+
 ## Add to Your Project
 
 ```bash
@@ -170,6 +198,22 @@ It extracts hard facts (prices, rates, SKUs, "30 days") as variant-set assertion
 would fail a known-good answer is rejected before you ever see it. One LLM call at
 authoring time; the checks run free forever. Details: [docs/generate-checks.md](docs/generate-checks.md).
 
+## Let your coding agent set it up
+
+AgentCI ships as a Claude Code plugin. Two skills: **onboard** (writes the runner,
+records golden baselines, generates the spec, verifies it) and **check** (runs the
+right test after every change to your agent and routes failures by flip source).
+
+```
+/plugin marketplace add suniel12/AgentCI
+/plugin install agentci@agentci
+```
+
+Then ask your coding agent to "set up AgentCI for this repo." It records goldens with
+`agentci bootstrap --yes` and verifies with `agentci test --runs 3` — no human CLI use
+needed. The runner it writes is one function: `(query: str) -> str`; trace capture is
+automatic.
+
 ## Demo
 
 Here's a RAG agent demo where someone "optimizes for latency" by reducing retriever docs from 8 to 1. AgentCI catches the correctness regression:

{ciagent-0.7.0 → ciagent-0.8.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ciagent"
-version = "0.7.0"
+version = "0.8.0"
 description = "Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production."
 readme = "README.md"
 license = {text = "Apache-2.0"}
@@ -64,10 +64,11 @@ include = [
 [tool.hatch.build]
 include = [
   "src/agentci/templates/**/*",
+  "src/agentci/examples/**/*",
 ]
 
 [tool.setuptools.package-data]
-agentci = ["templates/*.j2"]
+agentci = ["templates/*.j2", "examples/*.yaml"]
 
 [tool.pytest.ini_options]
 pythonpath = ["src"]

{ciagent-0.7.0 → ciagent-0.8.0}/src/agentci/capture.py

@@ -25,6 +25,12 @@ _active_trace: contextvars.ContextVar[Trace | None] = contextvars.ContextVar(
 _active_span: contextvars.ContextVar[Span | None] = contextvars.ContextVar(
     '_active_span', default=None
 )
+# Nesting depth of active TraceContexts. SDK patches are installed only by the
+# outermost context — stacked patches would record every LLM call once per
+# wrapper into the same active span.
+_patch_depth: contextvars.ContextVar[int] = contextvars.ContextVar(
+    '_patch_depth', default=0
+)
 
 
 class TraceContext:
@@ -58,23 +64,31 @@ class TraceContext:
         self.trace = Trace(agent_name=agent_name, test_name=test_name)
         self._patches = []
         self._start_time = 0.0
-    
+        self._trace_token = None
+        self._span_token = None
+
     def __enter__(self):
         # Create root span
         root_span = Span(kind=SpanKind.AGENT, name=self.trace.agent_name)
         self.trace.spans.append(root_span)
-        
-        # Set context vars
-        _active_trace.set(self.trace)
-        _active_span.set(root_span)
-        
-        # Apply monkey patches
-        self._patch_openai()
-        self._patch_anthropic()
-        
+
+        # Set context vars, keeping reset tokens so a nested context restores
+        # the enclosing context on exit instead of clearing it
+        self._trace_token = _active_trace.set(self.trace)
+        self._span_token = _active_span.set(root_span)
+
+        # Apply monkey patches only in the outermost context: stacked patches
+        # would record every LLM call once per wrapper into the active span
+        # (e.g. a Trace-returning runner that uses TraceContext itself, wrapped
+        # again by _run_with_retry)
+        if _patch_depth.get() == 0:
+            self._patch_openai()
+            self._patch_anthropic()
+        _patch_depth.set(_patch_depth.get() + 1)
+
         self._start_time = time.perf_counter()
         return self
-    
+
     def __exit__(self, *args):
         # Compute duration
         self.trace.total_duration_ms = (time.perf_counter() - self._start_time) * 1000
@@ -85,13 +99,19 @@ class TraceContext:
         # Auto-extract final output if not manually set
         self._auto_extract_final_output()
 
-        # Remove patches
+        # Remove patches (only the outermost context installed any)
+        _patch_depth.set(max(_patch_depth.get() - 1, 0))
         for restore_fn in self._patches:
             restore_fn()
+        self._patches = []
 
-        # Clear context
-        _active_trace.set(None)
-        _active_span.set(None)
+        # Restore the enclosing context (None only when outermost)
+        if self._trace_token is not None:
+            _active_trace.reset(self._trace_token)
+            self._trace_token = None
+        if self._span_token is not None:
+            _active_span.reset(self._span_token)
+            self._span_token = None
 
     def _auto_extract_final_output(self) -> None:
         """Auto-extract the agent's final output from the trace.

{ciagent-0.7.0 → ciagent-0.8.0}/src/agentci/cli.py

@@ -1627,15 +1627,26 @@ queries:
 @click.option('--runner', required=True, help='Import path for runner function, e.g. myagent.run:run')
 @click.option('--output', default='agentci_spec.yaml', help='Output spec file')
 @click.option('--baseline-dir', default='./baselines', help='Directory for baselines')
-def bootstrap(queries, agent, runner, output, baseline_dir):
-    """Zero-to-Golden interactive bootstrapper."""
+@click.option('--yes', '-y', is_flag=True,
+              help='Accept every captured trace as golden without prompting '
+                   '(for CI and coding agents; requires --queries)')
+def bootstrap(queries, agent, runner, output, baseline_dir, yes):
+    """Zero-to-Golden bootstrapper: run queries, record golden baselines, write a spec.
+
+    The runner may return an agentci.models.Trace or a plain string — string
+    returns get automatic LLM/tool capture, same as `agentci test`.
+    """
     import yaml
     import json
     import re
     from datetime import datetime, timezone
     from pathlib import Path
     from rich.prompt import Prompt, Confirm
-    from .engine.parallel import resolve_runner
+    from .engine.parallel import resolve_runner, _run_with_retry
+
+    if yes and not queries:
+        console.print("[bold red]--yes requires --queries[/] (nothing to confirm interactively)")
+        sys.exit(2)
 
     try:
         runner_fn = resolve_runner(runner)
@@ -1678,8 +1689,13 @@ def bootstrap(queries, agent, runner, output, baseline_dir):
     for i, q in enumerate(query_list):
         console.print(f"\n[bold cyan]Running Query {i+1}/{len(query_list)}:[/] {q}")
         try:
-            trace = runner_fn(q)
-            
+            # Same execution path as `agentci test`: TraceContext capture plus
+            # coercion, so string-returning runners work here too.
+            trace = _run_with_retry(runner_fn, q, retry_count=0, backoff=1.0, agent_name=agent)
+            if trace is None:
+                console.print("  [yellow]Runner returned nothing — skipping this query.[/]")
+                continue
+
             # Print Tier 1 summary
             console.print(f"  Duration:   {trace.total_duration_ms:.1f}ms")
             console.print(f"  Cost:       ${trace.total_cost_usd:.4f}")
@@ -1687,8 +1703,11 @@ def bootstrap(queries, agent, runner, output, baseline_dir):
             console.print(f"  Tool Calls: {len(trace.tool_call_sequence)}")
             if trace.tool_call_sequence:
                 console.print(f"  Path:       {' → '.join(trace.tool_call_sequence)}")
-            
-            if Confirm.ask("Accept this trace as golden? [Y/n]", default=True):
+            answer_preview = (trace.metadata.get("final_output") or "")[:120]
+            if answer_preview:
+                console.print(f"  Answer:     {answer_preview}")
+
+            if yes or Confirm.ask("Accept this trace as golden? [Y/n]", default=True):
                 # Generates assertions
                 expected_tools = trace.tool_call_sequence
                 max_tool_calls = len(trace.tool_call_sequence) + 1
@@ -2779,6 +2798,8 @@ def test_cmd(config, tags, fmt, output, baseline_dir, workers, sample_ensemble,
     evaluates all three layers (Correctness / Path / Cost), and reports results.
 
     Use --mock to validate your spec with synthetic traces (zero API cost).
+    With no agentci_spec.yaml present, --mock runs a bundled demo spec —
+    try `agentci test --mock --runs 3` in an empty directory (zero API keys).
 
     Use --runs N to run the whole suite N times: a stable aggregate score can
     hide per-query verdict flips. Each flip is attributed to its source —
@@ -2806,6 +2827,25 @@ def test_cmd(config, tags, fmt, output, baseline_dir, workers, sample_ensemble,
     from .engine.runner import evaluate_spec
     from .exceptions import ConfigError
 
+    # ── Zero-key demo fallback ────────────────────────────────────────────────
+    # Only when the user did not pass --config themselves: an explicitly named
+    # spec that is missing must stay an error, never silently become the demo.
+    demo_mode = False
+    config_source = click.get_current_context().get_parameter_source("config")
+    if config_source == click.core.ParameterSource.DEFAULT and not Path(config).exists():
+        if mock:
+            from importlib.resources import files
+
+            config = str(files("agentci").joinpath("examples", "demo_spec.yaml"))
+            demo_mode = True
+        else:
+            console.print(
+                "[bold red]No agentci_spec.yaml found in this directory.[/]\n\n"
+                "  [cyan]agentci init[/]                     scaffold a spec for your own agent\n"
+                "  [cyan]agentci test --mock --runs 3[/]     try the bundled demo (synthetic traces, zero API keys)"
+            )
+            sys.exit(2)
+
     try:
         spec = load_spec(config)
     except ConfigError as e:
@@ -2849,14 +2889,33 @@ def test_cmd(config, tags, fmt, output, baseline_dir, workers, sample_ensemble,
         )
         console.print("[dim]Running with synthetic traces — zero API cost[/]\n")
 
+        if demo_mode:
+            console.print(
+                "[yellow]Demo mode:[/] no agentci_spec.yaml found — using the bundled "
+                "demo spec ([bold]synthetic data[/], simulated agent).\n"
+                "[dim]Test your own agent instead: agentci init[/]\n"
+            )
+
         # AGENTCI_MOCK_FLAKY=1 simulates agent-variance across runs so the
         # stability report (and its tests) can be exercised without API keys.
-        mock_flaky = os.environ.get("AGENTCI_MOCK_FLAKY", "").lower() in ("1", "true", "yes")
+        # Demo mode defaults the simulated flakiness ON for multi-run sessions —
+        # the whole point of the demo is showing a stable score hiding verdict
+        # flips — but an explicit env value always wins.
+        flaky_env = os.environ.get("AGENTCI_MOCK_FLAKY")
+        if flaky_env is None:
+            mock_flaky = demo_mode and runs > 1
+        else:
+            mock_flaky = flaky_env.lower() in ("1", "true", "yes")
 
         try:
             run_results = []
             for run_index in range(runs):
-                traces = run_mock_spec(spec, run_index=run_index, flaky=mock_flaky)
+                traces = run_mock_spec(
+                    spec,
+                    run_index=run_index,
+                    flaky=mock_flaky,
+                    flaky_style="spread" if demo_mode else "alternate",
+                )
                 results = evaluate_spec(spec, traces, None)
                 run_results.append(results)
                 if runs > 1 and fmt in ("console", "github"):
@@ -2875,6 +2934,11 @@ def test_cmd(config, tags, fmt, output, baseline_dir, workers, sample_ensemble,
             )
 
         exit_code = report_results(run_results[0], format=fmt, spec_file=config)
+        if demo_mode and fmt == "console":
+            console.print(
+                "\n[dim]Tip: agentci test --mock --runs 3 runs the demo suite three times "
+                "and shows the stability report with flip-source attribution.[/]"
+            )
         sys.exit(exit_code)
 
     # ── Check for runner ──────────────────────────────────────────────────────

{ciagent-0.7.0 → ciagent-0.8.0}/src/agentci/engine/mock_runner.py

@@ -82,7 +82,13 @@ def mock_run(query: str, query_spec: dict, flaky_break: bool = False) -> Trace:
     return trace
 
 
-def run_mock_spec(spec, run_index: int = 0, flaky: bool = False, **_kwargs) -> dict[str, Trace]:
+def run_mock_spec(
+    spec,
+    run_index: int = 0,
+    flaky: bool = False,
+    flaky_style: str = "alternate",
+    **_kwargs,
+) -> dict[str, Trace]:
     """Run all queries in a spec using mock traces.
 
     Parameters
@@ -92,11 +98,20 @@ def run_mock_spec(spec, run_index: int = 0, flaky: bool = False, **_kwargs) -> d
     run_index : int
         Which run this is (0-based) in a multi-run stability session.
     flaky : bool
-        Simulate deterministic pseudo-flakiness for stability testing:
-        even-indexed queries produce a broken answer on odd run indices
+        Simulate deterministic pseudo-flakiness for stability testing
         (agent-variance). Judge-flake cannot be simulated here — identical
         answers cannot flip deterministic checks by construction; that path
         is covered by unit tests constructing results directly.
+    flaky_style : str
+        How simulated flakiness is distributed across runs:
+
+        - ``"alternate"`` (default): even-indexed queries break on odd run
+          indices. Half the suite flips at once, so the aggregate score
+          visibly dips on flaky runs.
+        - ``"spread"``: exactly one of the first three queries breaks per
+          run (query ``run_index % 3``). The aggregate score stays constant
+          across runs while individual verdicts flip — the "stable score,
+          unstable system" shape the bundled demo exists to show.
 
     Returns
     -------
@@ -106,6 +121,9 @@ def run_mock_spec(spec, run_index: int = 0, flaky: bool = False, **_kwargs) -> d
     traces: dict[str, Trace] = {}
     for i, q in enumerate(spec.queries):
         query_dict = q.model_dump() if hasattr(q, "model_dump") else {}
-        flaky_break = flaky and (i % 2 == 0) and (run_index % 2 == 1)
+        if flaky_style == "spread":
+            flaky_break = flaky and i < 3 and run_index % 3 == i
+        else:
+            flaky_break = flaky and (i % 2 == 0) and (run_index % 2 == 1)
         traces[q.query] = mock_run(q.query, query_dict, flaky_break=flaky_break)
     return traces

{ciagent-0.7.0 → ciagent-0.8.0}/src/agentci/engine/reporter.py

@@ -528,8 +528,18 @@ def _serialize_stability(report: "StabilityReport") -> dict[str, Any]:
 
 
 def _serialize_result(r: QueryResult) -> dict[str, Any]:
+    # Answer text included so JSON consumers (coding agents, judge-audit
+    # answer sources) can see what the agent said, not just the verdicts.
+    # Same extractor the correctness layer uses, so any trace shape the
+    # evaluator can grade also serializes its answer.
+    answer = None
+    if r.trace is not None:
+        from .runner import _extract_answer
+
+        answer = _extract_answer(r.trace) or None
     return {
         "query": r.query,
+        "answer": answer,
         "hard_fail": r.hard_fail,
         "has_warnings": r.has_warnings,
         "correctness": {

ciagent-0.8.0/src/agentci/examples/__init__.py

@@ -0,0 +1,3 @@
+# Copyright 2025-2026 The AgentCI Authors
+# SPDX-License-Identifier: Apache-2.0
+"""Bundled example specs shipped with the package (used by the zero-key demo)."""

ciagent-0.8.0/src/agentci/examples/demo_spec.yaml

@@ -0,0 +1,73 @@
+# AgentCI bundled demo spec — SYNTHETIC DATA ONLY.
+#
+# This spec backs the zero-API-key demo:
+#
+#     uvx ciagent test --mock --runs 3
+#
+# When `agentci test --mock` runs with no agentci_spec.yaml in the working
+# directory, the CLI falls back to this file. Traces are synthesized by the
+# mock runner; in multi-run demo sessions a deterministic pseudo-flaky agent
+# is simulated so the stability report shows verdict flips with
+# flip-source attribution. Nothing here touches a real agent or API.
+#
+# To test your own agent instead: `agentci init`
+
+version: 1
+agent: demo-support-agent
+
+queries:
+  - query: "What's your return window?"
+    description: "Policy lookup — returns"
+    correctness:
+      expected_in_answer: ["30 days"]
+    path:
+      expected_tools: [search_kb]
+
+  - query: "Do you ship internationally?"
+    description: "Policy lookup — shipping"
+    correctness:
+      expected_in_answer: ["international shipping"]
+    path:
+      expected_tools: [search_kb]
+
+  - query: "How do I reset my password?"
+    description: "Account help — password reset"
+    correctness:
+      expected_in_answer: ["reset link"]
+    path:
+      expected_tools: [search_kb]
+
+  - query: "What payment methods do you accept?"
+    description: "Policy lookup — payments"
+    correctness:
+      expected_in_answer: ["credit card"]
+    path:
+      expected_tools: [search_kb]
+
+  - query: "Can I change my delivery address after ordering?"
+    description: "Order management — address change"
+    correctness:
+      expected_in_answer: ["before it ships"]
+    path:
+      expected_tools: [search_kb, lookup_order]
+
+  - query: "How do I cancel my subscription?"
+    description: "Account help — cancellation"
+    correctness:
+      expected_in_answer: ["account settings"]
+    path:
+      expected_tools: [search_kb]
+
+  - query: "Do you offer student discounts?"
+    description: "Policy lookup — discounts"
+    correctness:
+      expected_in_answer: ["10%"]
+    path:
+      expected_tools: [search_kb]
+
+  - query: "Where can I download my invoice?"
+    description: "Account help — invoices"
+    correctness:
+      expected_in_answer: ["billing page"]
+    path:
+      expected_tools: [search_kb]