ciagent 0.4.1__tar.gz → 0.5.0__tar.gz

ciagent-0.5.0/CHANGELOG.md

@@ -0,0 +1,87 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.5.0] - 2026-03-01
+
+### Added
+
+#### Three-Layer Evaluation Engine
+- **Correctness layer** (Layer 1 — hard fail): keyword matching, LLM-as-a-judge, safety checks, hallucination checks, regex/exact match, JSON schema validation
+- **Path layer** (Layer 2 — soft warn): tool trajectory validation, loop detection (default `max_loops=3`), routing assertions, handoff expectations
+- **Cost layer** (Layer 3 — soft warn): token budgets, cost caps, LLM call limits, latency thresholds
+- `runner.py` orchestrates all three layers per query
+- `parallel.py` for parallel query execution across specs
+
+#### OR-Logic Keywords
+- `any_expected_in_answer` field — at least one keyword must match (complementing `expected_in_answer` which requires all)
+
+#### LLM Judge Enhancements
+- `context_file` support in `JudgeRubric` — doc-grounded judging against reference documents
+- `refutes_premise` flag — injects built-in premise-correction rubric for trick questions
+
+#### Span Assertions
+- `SpanAssertionSpec` schema model for span-level assertions
+- `Span.attributes: dict[str, Any]` — OTel-style span-level data propagation
+- Span-level LLM judge support
+
+#### Mock Testing Mode
+- `agentci test --mock` — generates synthetic traces, zero API cost
+- `mock_runner.py` — synthetic trace generation from spec expectations
+- `--golden-file` flag — load Q&A pairs from JSON/CSV for mock mode
+
+#### Cost Estimator
+- `cost_estimator.py` — pre-execution cost estimates with pricing table
+- Cost estimate shown before live test runs; `--yes`/`-y` skips confirmation
+
+#### CLI Improvements
+- `agentci init --generate` — AI-assisted spec generation with guided interview
+- `agentci doctor` — health-check command (spec, runner, API keys, deps, CI)
+- Scan-first flow: auto-scan project before questions, show summary
+- Agent type auto-detected via `_detect_agent_type_from_code()`
+- Skeleton template generation with TODO placeholders for zero-API-key usage
+- Context-aware "Next Steps" based on mode (mock vs live)
+- Non-interactive flags: `--kb-path`, `--mode`
+
+#### Diff Engine
+- Three-tier diff engine for baseline comparison
+- 11 `DiffType` categories including `ROUTING_CHANGED`, `GUARDRAILS_CHANGED`
+- `agentci diff` CLI command
+
+#### Reporting
+- GitHub annotations with budget cap (`MAX_INLINE_ANNOTATIONS = 10` for warnings; errors uncapped)
+- JSON output format
+- Prometheus metrics export
+
+#### Trace Helpers
+- `Trace.called(tool)` / `never_called(tool)` / `loop_count(tool)` — readable assertion helpers
+- `Trace.cost_under(usd)` / `llm_calls_under(n)` — budget assertion helpers
+- `langgraph_trace(agent_name)` — context manager shortcut for LangGraph
+- `TraceContext.attach(state)` — alias for `attach_langgraph_state()`
+
+#### Adapters
+- OpenAI Agents SDK adapter (`openai_agents.py`)
+- LangGraph adapter (`langgraph.py`)
+
+#### Other
+- `python-dotenv` added as core dependency
+- Deep KB sampling: 2000 chars/file for spec generation
+- Progressive spec building: smoke queries (3) then full queries (10-12)
+- Pytest plugin entry point (`pytest11: agentci`)
+- GUARDRAIL span type, HANDOFF span type
+
+### Changed
+- `PathSpec.max_loops` now defaults to `3` (was `None`)
+- Development status upgraded from Alpha to Beta
+- Package version bumped to 0.5.0
+
+## [0.4.1] - 2026-02-20
+
+### Added
+- Initial project structure and core models
+- Basic trace capture and assertion framework
+- CLI scaffolding with `agentci init`
+- PyPI publishing as `ciagent`

{ciagent-0.4.1 → ciagent-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ciagent
-Version: 0.4.1
+Version: 0.5.0
 Summary: Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production.
 Project-URL: Documentation, https://github.com/agentci-org/agentci#readme
 Project-URL: Repository, https://github.com/agentci-org/agentci.git
@@ -8,7 +8,7 @@ Project-URL: Issues, https://github.com/agentci-org/agentci/issues
 License: Apache-2.0
 License-File: LICENSE
 Keywords: agents,ai,ci-cd,llm,regression,testing
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Framework :: Pytest
 Classifier: Intended Audience :: Developers
 Classifier: Topic :: Software Development :: Testing

{ciagent-0.4.1 → ciagent-0.5.0}/pyproject.toml

@@ -1,13 +1,13 @@
 [project]
 name = "ciagent"
-version = "0.4.1"
+version = "0.5.0"
 description = "Continuous Integration for AI Agents. Catch cost spikes and logic regressions before production."
 readme = "README.md"
 license = {text = "Apache-2.0"}
 requires-python = ">=3.10"
 keywords = ["ai", "agents", "testing", "llm", "ci-cd", "regression"]
 classifiers = [
-    "Development Status :: 3 - Alpha",
+    "Development Status :: 4 - Beta",
     "Intended Audience :: Developers",
     "Topic :: Software Development :: Testing",
     "Framework :: Pytest",

{ciagent-0.4.1 → ciagent-0.5.0}/src/agentci/cli.py

@@ -369,15 +369,21 @@ def _generate_skeleton_spec(
   - query: "TODO: Replace with an in-scope question your KB can answer"
     description: "Happy-path KB retrieval"
     correctness:
-      expected_in_answer: ["TODO: expected keyword"]
+      any_expected_in_answer: ["TODO: keyword1", "TODO: keyword2"]
+    cost:
+      max_llm_calls: 8
 
   - query: "What is the weather today?"
     description: "Out-of-scope question — agent should decline"
     path:
       max_tool_calls: 0
+    cost:
+      max_llm_calls: 2
 
   - query: "Hello"
     description: "Boundary — greeting / off-topic"
+    cost:
+      max_llm_calls: 2
 """
     elif agent_type == "tool":
         for tool in list(detected_tools)[:5]:
@@ -386,12 +392,16 @@ def _generate_skeleton_spec(
     description: "Tool usage test — {tool}"
     path:
       expected_tools: [{tool}]
+    cost:
+      max_llm_calls: 8
 """
         queries_yaml += """
   - query: "What is the weather today?"
     description: "Out-of-scope — should not use tools"
     path:
       max_tool_calls: 0
+    cost:
+      max_llm_calls: 2
 """
     else:  # conversational
         queries_yaml += """
@@ -407,8 +417,8 @@ runner: "{runner_path}"
 version: 1.0
 
 judge_config:
-  model: openai:gpt-4o-mini
-  temperature: 0.1
+  model: {_get_judge_model_for_spec()}
+  temperature: 0
 
 baseline_dir: ./baselines
 
@@ -450,6 +460,15 @@ def _build_next_steps(run_mode: str, created_workflow: bool, has_queries: bool)
     return steps
 
 
+def _get_judge_model_for_spec() -> str:
+    """Pick the best judge model for generated specs based on available API keys."""
+    if os.environ.get("ANTHROPIC_API_KEY"):
+        return "claude-sonnet-4-6"
+    if os.environ.get("OPENAI_API_KEY"):
+        return "gpt-4o"
+    return "claude-sonnet-4-6"
+
+
 def _generate_queries(context: dict, runner_path: str, interview: dict | None = None) -> list[dict]:
     """Call an LLM (Anthropic or OpenAI) to generate test queries from project context.
 
@@ -538,13 +557,75 @@ For each query, produce a YAML block with:
 - description: one sentence explaining what this tests
 - tags: list of tags (smoke, in-scope, out-of-scope, edge-case, etc.)
 - path: expected_tools list OR max_tool_calls: 0 for decline cases
-- correctness: expected_in_answer keywords if deterministic, or llm_judge rule
-- cost: max_llm_calls budget
-
-IMPORTANT: All expected_in_answer values MUST come directly from the knowledge base
-content or golden Q&A pairs above. Do NOT invent or hallucinate facts, numbers, or
-specific details. If a fact is not present in the provided content, use llm_judge
-with a descriptive rule instead of expected_in_answer.
+- correctness: use any_expected_in_answer (OR logic) for list-type answers,
+  expected_in_answer (AND logic) only when ALL terms are essential, or llm_judge rule
+- cost: max_llm_calls budget (default 8 for in-scope queries)
+
+JUDGE RULE GUIDELINES:
+Write judge rules that evaluate whether the response is HELPFUL and ACCURATE,
+not whether it follows a rigid script. The agent is a documentation assistant,
+not an exam candidate.
+
+1. THRESHOLD SELECTION:
+   - Use 0.7 for happy-path in-scope queries (agent should answer well)
+   - Use 0.8 for out-of-scope decline queries (agent must clearly decline)
+   - Use 0.7 for edge cases, compound, and mixed-intent queries (more tolerance)
+
+2. PARTIAL KNOWLEDGE:
+   For compound queries where one sub-question IS answerable from the KB and
+   another is NOT, the judge rule must accept:
+   - Answering the KB-covered part accurately
+   - Gracefully declining or saying "I don't have that information" for the rest
+   Do NOT require the agent to "address both parts" — require it to be accurate
+   on what it can answer and honest about what it cannot.
+
+3. RULE TONE:
+   - Use "should" instead of "must" for non-critical criteria
+   - Focus on what makes a GOOD response, not a checklist of requirements
+   - Avoid rules that penalize the agent for being honest about knowledge gaps
+
+4. KEYWORD CHECKS (expected_in_answer vs any_expected_in_answer):
+   All keyword values MUST come directly from the knowledge base content or
+   golden Q&A pairs above. Do NOT invent or hallucinate facts.
+   If a fact is not in the provided content, use llm_judge instead.
+
+   - Use `any_expected_in_answer` (OR logic) when the query expects a LIST or
+     enumeration (e.g., "What tools are available?" → any ONE tool name suffices).
+     This is the PREFERRED default for most keyword checks.
+   - Use `expected_in_answer` (AND logic) ONLY when ALL terms are essential to a
+     correct answer (e.g., "What is the install command?" → both "pip" and "install"
+     must appear).
+   - When in doubt, prefer `any_expected_in_answer` — it is more resilient to
+     agent paraphrasing and partial answers.
+
+5. COST BUDGET:
+   Set max_llm_calls to 8 for in-scope queries (RAG agents typically use 4-8
+   LLM calls per query). Use 2-3 for out-of-scope/greeting queries.
+
+EXAMPLES OF GOOD vs BAD JUDGE RULES:
+
+BAD (too strict, penalizes honest agents):
+  rule: "The agent must address both the refund question and the support
+        contact question with specific information from the documentation."
+  threshold: 0.8
+
+GOOD (accepts partial knowledge):
+  rule: "The agent should answer based on retrieved documentation. If the KB
+        covers the topic, the answer should be accurate. If the KB does not
+        cover a sub-question, the agent should honestly say it doesn't have
+        that information rather than inventing an answer."
+  threshold: 0.7
+
+BAD (requires specific phrasing):
+  rule: "The agent must state it is an AgentCI documentation assistant and
+        list its exact capabilities."
+  threshold: 0.85
+
+GOOD (evaluates helpfulness):
+  rule: "The agent should respond with a friendly greeting and indicate
+        readiness to help with AgentCI questions. It should not fabricate
+        any AgentCI facts unprompted."
+  threshold: 0.7
 
 Respond ONLY with valid YAML (a list of query objects, no surrounding text).
 """
@@ -627,6 +708,62 @@ def _generate_full_queries(
     return _generate_queries(context, runner_path, interview=full_interview)
 
 
+def _calibrate_spec_from_traces(
+    queries: list[dict],
+    runner_fn,
+    console,
+) -> list[dict]:
+    """Run 1-2 sample queries against the real agent to calibrate max_llm_calls.
+
+    Only called in live mode. Picks up to 2 in-scope queries, runs them,
+    measures actual LLM call counts, and sets max_llm_calls with 50% headroom.
+    Falls back gracefully if the runner fails.
+    """
+    candidates = [
+        q for q in queries
+        if "out-of-scope" not in (q.get("tags") or [])
+        and "greeting" not in (q.get("tags") or [])
+    ]
+    sample = candidates[:2] if len(candidates) >= 2 else candidates[:1]
+    if not sample:
+        return queries
+
+    console.print("\n[bold cyan]Calibrating cost budgets...[/]")
+    observed_llm_calls: list[int] = []
+
+    for q_spec in sample:
+        try:
+            trace = runner_fn(q_spec["query"])
+            llm_count = sum(len(s.llm_calls) for s in trace.spans)
+            observed_llm_calls.append(llm_count)
+            query_preview = q_spec["query"][:50]
+            console.print(f"  [dim]{query_preview}...[/] → {llm_count} LLM calls")
+        except Exception:
+            continue
+
+    if not observed_llm_calls:
+        console.print("[yellow]Calibration skipped:[/] no traces captured")
+        return queries
+
+    max_observed = max(observed_llm_calls)
+    calibrated_max = max(8, int(max_observed * 1.5))
+
+    for q in queries:
+        tags = q.get("tags") or []
+        cost = q.get("cost") or {}
+        if "out-of-scope" in tags or "greeting" in tags:
+            cost["max_llm_calls"] = max(2, min(observed_llm_calls))
+        else:
+            cost["max_llm_calls"] = calibrated_max
+        q["cost"] = cost
+
+    console.print(
+        f"[green]✓[/] Calibrated: max_llm_calls = {calibrated_max} "
+        f"(observed max: {max_observed})"
+    )
+    return queries
+
+
 @click.group()
 @click.version_option(version=__version__, prog_name="agentci")
 def cli():
@@ -673,6 +810,10 @@ def init(hook, force, example, generate, agent_description, kb_path, run_mode, g
             "Agent type is now auto-detected from code."
         )
 
+    # Track whether we generated real queries vs skeleton
+    has_queries = True
+    interview: dict = {}
+
     # 0. Generate agentci_spec.yaml
     spec_dest = Path("agentci_spec.yaml")
     if spec_dest.exists() and not force:
@@ -680,9 +821,7 @@ def init(hook, force, example, generate, agent_description, kb_path, run_mode, g
     else:
         console.print("\n[bold green]AgentCI Setup[/]")
 
-        # Track whether we generated real queries vs skeleton
         has_queries = False
-        interview: dict = {}
 
         if generate:
             import yaml as _yaml
@@ -903,6 +1042,19 @@ def init(hook, force, example, generate, agent_description, kb_path, run_mode, g
                     )
                     has_queries = False
 
+            # ── Calibrate cost budgets (live mode only) ─────────────
+            if (
+                queries
+                and isinstance(queries, list)
+                and interview.get("run_mode") == "live"
+            ):
+                try:
+                    from .engine.parallel import resolve_runner
+                    runner_fn = resolve_runner(runner_path)
+                    queries = _calibrate_spec_from_traces(queries, runner_fn, console)
+                except Exception as e:
+                    console.print(f"[yellow]Calibration skipped:[/] {e}")
+
             # ── Write spec ───────────────────────────────────────────
             if queries and isinstance(queries, list):
                 console.print(f"[green]✓[/] {len(queries)} test queries ready")
@@ -910,7 +1062,7 @@ def init(hook, force, example, generate, agent_description, kb_path, run_mode, g
                     "agent": "my-agent",
                     "runner": runner_path,
                     "version": 1.0,
-                    "judge_config": {"model": "openai:gpt-4o-mini", "temperature": 0.1},
+                    "judge_config": {"model": _get_judge_model_for_spec(), "temperature": 0},
                     "baseline_dir": "./baselines",
                     "queries": queries,
                 }
@@ -929,8 +1081,8 @@ runner: "{runner_path}"
 version: 1.0
 
 judge_config:
-  model: openai:gpt-4o-mini
-  temperature: 0.1
+  model: {_get_judge_model_for_spec()}
+  temperature: 0
 
 baseline_dir: ./baselines
 
@@ -954,7 +1106,8 @@ runner: "{runner_path}"
 version: 1.0
 
 judge_config:
-  model: openai:gpt-4o-mini
+  model: {_get_judge_model_for_spec()}
+  temperature: 0
 
 baseline_dir: ./baselines
 
@@ -1095,8 +1248,8 @@ def bootstrap(queries, agent, runner, output, baseline_dir):
         "runner": runner,
         "version": 1.0,
         "judge_config": {
-            "model": "openai:gpt-4o-mini",
-            "temperature": 0.1
+            "model": _get_judge_model_for_spec(),
+            "temperature": 0,
         },
         "baseline_dir": baseline_dir,
         "queries": []

{ciagent-0.4.1 → ciagent-0.5.0}/src/agentci/engine/correctness.py

@@ -47,6 +47,8 @@ def evaluate_correctness(
     spec: CorrectnessSpec,
     trace: Optional[Any] = None,
     judge_config: Optional[dict[str, Any]] = None,
+    query: Optional[str] = None,
+    spec_dir: Optional[str] = None,
 ) -> LayerResult:
     """Evaluate correctness assertions against an agent answer.
 
@@ -55,6 +57,8 @@ def evaluate_correctness(
         spec:         CorrectnessSpec defining what to check.
         trace:        Optional raw trace object (used for context in judge calls).
         judge_config: Global judge settings (model, temperature, ensemble).
+        query:        Optional original user query for judge evaluation context.
+        spec_dir:     Directory of the spec file (for context_file resolution).
 
     Returns:
         LayerResult with PASS or FAIL status.
@@ -74,11 +78,12 @@ def evaluate_correctness(
             effective_judges.extend(spec.llm_judge)
 
         for rubric in effective_judges:
-            result = _run_judge_safe(answer, rubric, judge_config, trace)
+            result = _run_judge_safe(answer, rubric, judge_config, trace, query, spec_dir)
             key = f"judge_{rubric.rule[:40]}"
             details[key] = result
             if not result.get("passed", False):
-                failures.append(f"Judge failed: {rubric.rule[:80]}")
+                score = result.get("score", "?")
+                failures.append(f"Judge failed (score: {score}): {rubric.rule[:80]}")
             else:
                 score = result.get("score", "")
                 threshold = getattr(rubric, "threshold", "")
@@ -86,7 +91,7 @@ def evaluate_correctness(
                 pass_messages.append(f"Premise correction verified by judge{score_str}")
 
         if spec.safety_check and not failures:
-            result = _run_judge_safe(answer, spec.safety_check, judge_config, trace)
+            result = _run_judge_safe(answer, spec.safety_check, judge_config, trace, query, spec_dir)
             details["safety"] = result
             if not result.get("passed", False):
                 failures.append(f"Safety check failed: {spec.safety_check.rule}")
@@ -112,12 +117,35 @@ def evaluate_correctness(
             "all_found": not missing,
         }
         if missing:
+            answer_preview = answer[:200] + ("..." if len(answer) > 200 else "")
             for term in missing:
-                failures.append(f"Expected '{term}' not found in answer")
+                failures.append(
+                    f"Expected '{term}' not found in answer"
+                    f"\n         Agent said: \"{answer_preview}\""
+                )
         else:
             found_str = ", ".join(f'"{t}"' for t in spec.expected_in_answer)
             pass_messages.append(f"Found keywords: {found_str}")
 
+    # ── 1b. any_expected_in_answer (case-insensitive, OR logic) ─────────────
+    if spec.any_expected_in_answer:
+        found_any = [t for t in spec.any_expected_in_answer if t.lower() in answer.lower()]
+        details["any_expected_in_answer"] = {
+            "checked": spec.any_expected_in_answer,
+            "found": found_any,
+            "any_found": bool(found_any),
+        }
+        if not found_any:
+            terms_str = ", ".join(f'"{t}"' for t in spec.any_expected_in_answer)
+            answer_preview = answer[:200] + ("..." if len(answer) > 200 else "")
+            failures.append(
+                f"None of [{terms_str}] found in answer"
+                f"\n         Agent said: \"{answer_preview}\""
+            )
+        else:
+            found_str = ", ".join(f'"{t}"' for t in found_any)
+            pass_messages.append(f"Found keyword (any-of): {found_str}")
+
     # ── 2. not_in_answer (case-insensitive exclusion) ───────────────────────
     if spec.not_in_answer:
         found = [t for t in spec.not_in_answer if t.lower() in answer.lower()]
@@ -164,11 +192,20 @@ def evaluate_correctness(
     if not failures:
         if spec.llm_judge:
             for rubric in spec.llm_judge:
-                result = _run_judge_safe(answer, rubric, judge_config, trace)
+                result = _run_judge_safe(answer, rubric, judge_config, trace, query, spec_dir)
                 key = f"judge_{rubric.rule[:40]}"
                 details[key] = result
                 if not result.get("passed", False):
-                    failures.append(f"Judge failed: {rubric.rule}")
+                    # Check if there was an actual API/execution error
+                    if "error" in result:
+                        failures.append(f"Judge error: {result['error']}")
+                    else:
+                        score = result.get("score", "?")
+                        rationale = result.get("rationale", "")
+                        msg = f"Judge failed (score: {score}): {rubric.rule}"
+                        if rationale:
+                            msg += f"\n         Rationale: {rationale}"
+                        failures.append(msg)
                 else:
                     score = result.get("score", "")
                     threshold = getattr(rubric, "threshold", "")
@@ -176,7 +213,7 @@ def evaluate_correctness(
                     pass_messages.append(f"LLM judge passed{score_str}")
 
         if spec.safety_check and not failures:
-            result = _run_judge_safe(answer, spec.safety_check, judge_config, trace)
+            result = _run_judge_safe(answer, spec.safety_check, judge_config, trace, query, spec_dir)
             details["safety"] = result
             if not result.get("passed", False):
                 failures.append(f"Safety check failed: {spec.safety_check.rule}")
@@ -184,7 +221,7 @@ def evaluate_correctness(
                 pass_messages.append("Safety check passed")
 
         if spec.hallucination_check and not failures:
-            result = _run_judge_safe(answer, spec.hallucination_check, judge_config, trace)
+            result = _run_judge_safe(answer, spec.hallucination_check, judge_config, trace, query, spec_dir)
             details["hallucination"] = result
             if not result.get("passed", False):
                 failures.append("Hallucination check failed")
@@ -226,11 +263,13 @@ def _run_judge_safe(
     rubric: Any,
     judge_config: Optional[dict[str, Any]],
     trace: Any,
+    query: Optional[str] = None,
+    spec_dir: Optional[str] = None,
 ) -> dict[str, Any]:
     """Run a judge call, catching errors and returning a failure dict if needed."""
     from agentci.engine.judge import run_judge
     try:
-        return run_judge(answer=answer, rubric=rubric, config=judge_config)
+        return run_judge(answer=answer, rubric=rubric, config=judge_config, query=query, spec_dir=spec_dir)
     except Exception as e:
         return {
             "passed": False,

{ciagent-0.4.1 → ciagent-0.5.0}/src/agentci/engine/diff.py

@@ -422,14 +422,24 @@ def _generate_legacy_diffs(baseline: "Trace", compare: "Trace") -> list[Any]:
 
 
 def _extract_answer(trace: "Trace") -> str:
-    """Extract final text answer from the last span's output."""
-    if not trace.spans:
-        return ""
-    last = trace.spans[-1]
-    out = last.output_data
-    if out is None:
-        return ""
-    return out if isinstance(out, str) else str(out)
+    """Extract the agent's final text answer from the trace.
+
+    Strategy:
+        1. trace.metadata["final_output"] (explicitly set by runner).
+        2. Last span's output_data (fallback for runners that don't set metadata).
+        3. Empty string if nothing found.
+    """
+    meta_output = trace.metadata.get("final_output")
+    if meta_output is not None:
+        return str(meta_output)
+
+    if trace.spans:
+        last = trace.spans[-1]
+        out = last.output_data
+        if out is not None:
+            return out if isinstance(out, str) else str(out)
+
+    return ""
 
 
 def _format_value(before: Any, after: Any, label: str) -> str:

{ciagent-0.4.1 → ciagent-0.5.0}/src/agentci/engine/judge.py

@@ -115,6 +115,7 @@ def run_judge(
     config: Optional[dict[str, Any]] = None,
     context: Optional[str] = None,
     spec_dir: Optional[str] = None,
+    query: Optional[str] = None,
 ) -> dict[str, Any]:
     """Execute an LLM-as-a-judge evaluation with safeguards.
 
@@ -124,6 +125,7 @@ def run_judge(
         config:   Optional judge config dict (model, temperature, ensemble).
         context:  Optional retrieved context for grounding checks.
         spec_dir: Directory of the spec file, used to resolve context_file paths.
+        query:    Optional original user query for evaluation context.
 
     Returns:
         dict with keys: passed, score, label, rationale, model
@@ -139,20 +141,15 @@ def run_judge(
         effective_context = _load_context_file(rubric.context_file, spec_dir)
 
     system_prompt = _build_judge_system_prompt(rubric)
-    user_prompt = _build_judge_user_prompt(answer, rubric, effective_context)
-
-    import sys
-    print("====== LLM JUDGE PROMPT ======", file=sys.stderr)
-    print("SYSTEM:", system_prompt, file=sys.stderr)
-    print("USER:", user_prompt, file=sys.stderr)
-    print("==============================", file=sys.stderr)
+    user_prompt = _build_judge_user_prompt(answer, rubric, effective_context, query)
 
     if ensemble_cfg.get("enabled", False):
         return _run_ensemble(system_prompt, user_prompt, ensemble_cfg, rubric)
 
     verdict = _call_judge(model, system_prompt, user_prompt, temperature)
     threshold_score = _score_threshold(rubric.threshold)
-    passed = verdict.score >= threshold_score
+    # Pass if score meets threshold OR judge explicitly labeled as "pass"
+    passed = verdict.score >= threshold_score or verdict.label == "pass"
 
     return {
         "passed": passed,
@@ -255,10 +252,14 @@ def _call_anthropic(model: str, system: str, user: str, temperature: float) -> s
             "Set it to use LLM-as-a-judge evaluation."
         )
 
+    # Strip prefix if present
+    if model.startswith("anthropic:"):
+        model = model.split(":", 1)[1]
+
     client = anthropic.Anthropic(api_key=api_key)
     response = client.messages.create(
         model=model,
-        max_tokens=256,
+        max_tokens=1024,
         temperature=temperature,
         system=system,
         messages=[{"role": "user", "content": user}],
@@ -281,11 +282,15 @@ def _call_openai(model: str, system: str, user: str, temperature: float) -> str:
             "OPENAI_API_KEY environment variable not set."
         )
 
+    # Strip prefix if present
+    if model.startswith("openai:"):
+        model = model.split(":", 1)[1]
+
     client = openai.OpenAI(api_key=api_key)
     response = client.chat.completions.create(
         model=model,
         temperature=temperature,
-        max_tokens=256,
+        max_tokens=1024,
         messages=[
             {"role": "system", "content": system},
             {"role": "user", "content": user},
@@ -295,25 +300,53 @@ def _call_openai(model: str, system: str, user: str, temperature: float) -> str:
 
 
 def _parse_verdict(raw: str) -> JudgeVerdict:
-    """Parse LLM response into a JudgeVerdict, with fallback extraction."""
+    """Parse LLM response into a JudgeVerdict, with robust fallback extraction."""
+    import re
+
     text = raw.strip()
-    # Try to extract JSON block if wrapped in markdown
+
+    # Strategy 1: Extract JSON from markdown code block
     if "```" in text:
-        import re
         match = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", text, re.DOTALL)
         if match:
             text = match.group(1)
 
+    # Strategy 2: Direct JSON parse
     try:
         data = json.loads(text)
         return JudgeVerdict(**data)
     except Exception:
-        # Fallback: treat as fail with explanation
-        return JudgeVerdict(
-            score=1,
-            label="fail",
-            rationale=f"Failed to parse judge response: {raw[:200]}",
-        )
+        pass
+
+    # Strategy 3: Find first JSON object containing "score" (handles preamble text)
+    match = re.search(r"\{[^{}]*\"score\"[^{}]*\}", text, re.DOTALL)
+    if match:
+        try:
+            data = json.loads(match.group(0))
+            return JudgeVerdict(**data)
+        except Exception:
+            pass
+
+    # Strategy 4: Extract fields via regex as last resort (handles truncated JSON)
+    score_match = re.search(r"\"score\"\s*:\s*(\d)", text)
+    label_match = re.search(r"\"label\"\s*:\s*\"(pass|fail|borderline)\"", text)
+    rationale_match = re.search(r"\"rationale\"\s*:\s*\"([^\"]{1,500})", text)
+    if score_match:
+        score = int(score_match.group(1))
+        if label_match:
+            label = label_match.group(1)
+        else:
+            # Label missing (likely truncated) — infer from score
+            label = "pass" if score >= 3 else "fail"
+        rationale = rationale_match.group(1) if rationale_match else "Extracted via regex fallback"
+        return JudgeVerdict(score=score, label=label, rationale=rationale)
+
+    # Final fallback: treat as fail with raw response for debugging
+    return JudgeVerdict(
+        score=1,
+        label="fail",
+        rationale=f"Failed to parse judge response: {raw[:300]}",
+    )
 
 
 # ── Prompt Builders ────────────────────────────────────────────────────────────
@@ -331,6 +364,15 @@ def _build_judge_system_prompt(rubric: JudgeRubric) -> str:
         prompt += "\nSCORING ANCHORS:\n"
         for anchor in rubric.scale:
             prompt += f"  {anchor}\n"
+    else:
+        prompt += (
+            "\nSCORING ANCHORS:\n"
+            "  1: Completely fails to address the rubric criteria\n"
+            "  2: Partially addresses the rubric but with significant issues\n"
+            "  3: Addresses the rubric with some gaps or minor inaccuracies\n"
+            "  4: Addresses the rubric well with only trivial omissions\n"
+            "  5: Perfectly addresses all rubric criteria\n"
+        )
     if rubric.few_shot_examples:
         prompt += "\nEXAMPLES:\n"
         for ex in rubric.few_shot_examples:
@@ -344,9 +386,12 @@ def _build_judge_user_prompt(
     answer: str,
     rubric: JudgeRubric,
     context: Optional[str],
+    query: Optional[str] = None,
 ) -> str:
     """Build the user-turn prompt for the judge."""
     parts = []
+    if query:
+        parts.append(f"USER QUERY:\n{query}\n")
     if context and rubric.context_file:
         # Doc-grounded judging: instruct judge to use ONLY the reference document
         parts.append(

{ciagent-0.4.1 → ciagent-0.5.0}/src/agentci/engine/reporter.py

@@ -150,6 +150,7 @@ def _emit_console(results: list[QueryResult]) -> None:
         _print_layer("COST", r.cost, fail_icon="⚠️", pass_icon="💰", warn_icon="⚠️")
 
         if r.hard_fail and getattr(r, "trace", None):
+            _print_answer_preview(r.trace)
             _print_trace_summary(r.trace)
 
     total = len(results)
@@ -159,6 +160,30 @@ def _emit_console(results: list[QueryResult]) -> None:
     print(f"\n{'=' * 60}")
     print(f"Results: {passed}/{total} passed  |  {warned} warnings  |  {failed} failures")
 
+def _print_answer_preview(trace: Any) -> None:
+    """Show a truncated preview of the extracted answer for failed queries."""
+    answer = ""
+    meta_output = getattr(trace, "metadata", {}).get("final_output")
+    if meta_output is not None:
+        answer = str(meta_output)
+    elif getattr(trace, "spans", None):
+        last_span = trace.spans[-1]
+        output = getattr(last_span, "output_data", None)
+        if output is not None:
+            answer = str(output)
+
+    if not answer:
+        print("  [ANSWER] (empty — no answer extracted from trace)")
+        return
+
+    # Collapse whitespace for compact display
+    preview = " ".join(answer.split())
+    max_len = 300
+    if len(preview) > max_len:
+        preview = preview[:max_len] + "..."
+    print(f"  [ANSWER] {preview}")
+
+
 def _print_trace_summary(trace: Any) -> None:
     from rich.console import Console
     from rich.panel import Panel

{ciagent-0.4.1 → ciagent-0.5.0}/src/agentci/engine/runner.py

@@ -52,6 +52,8 @@ def evaluate_query(
             spec=query.correctness,
             trace=trace,
             judge_config=judge_config,
+            query=query.query,
+            spec_dir=spec_dir,
         )
     else:
         correctness = LayerResult(
@@ -168,16 +170,18 @@ def _extract_answer(trace: "Trace") -> str:
     """Extract the agent's final text answer from the trace.
 
     Strategy:
-        1. Last span's output_data if it's a string.
-        2. str(output_data) for non-string types.
-        3. Empty string if no spans.
+        1. trace.metadata["final_output"] (explicitly set by runner).
+        2. Last span's output_data (fallback for runners that don't set metadata).
+        3. Empty string if nothing found.
     """
-    if not trace.spans:
-        return ""
-    last_span = trace.spans[-1]
-    output = last_span.output_data
-    if output is None:
-        return ""
-    if isinstance(output, str):
-        return output
-    return str(output)
+    meta_output = trace.metadata.get("final_output")
+    if meta_output is not None:
+        return str(meta_output)
+
+    if trace.spans:
+        last_span = trace.spans[-1]
+        output = last_span.output_data
+        if output is not None:
+            return output if isinstance(output, str) else str(output)
+
+    return ""

{ciagent-0.4.1 → ciagent-0.5.0}/src/agentci/schema/spec_models.py

@@ -147,7 +147,11 @@ class CorrectnessSpec(BaseModel):
     """Layer 1: Hard pass/fail. Any failure blocks the CI pipeline."""
     expected_in_answer: Optional[list[str]] = Field(
         None,
-        description="Strings that must appear in the answer (case-insensitive)",
+        description="Strings that must ALL appear in the answer (case-insensitive, AND logic)",
+    )
+    any_expected_in_answer: Optional[list[str]] = Field(
+        None,
+        description="At least ONE of these strings must appear in the answer (case-insensitive, OR logic)",
     )
     not_in_answer: Optional[list[str]] = Field(
         None,

ciagent-0.4.1/CHANGELOG.md

@@ -1,11 +0,0 @@
-# Changelog
-
-All notable changes to this project will be documented in this file.
-
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-
-## [Unreleased]
-
-### Added
-- Initial project structure and setup.