openapi-agent-ready 0.3.0__tar.gz → 0.4.0__tar.gz

{openapi_agent_ready-0.3.0 → openapi_agent_ready-0.4.0}/CHANGELOG.md

@@ -5,6 +5,21 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-08-08
+
+### Added
+- **Baseline mode** (`--write-baseline`, `--baseline`, `--max-new`). A mature
+  API audited for the first time produces hundreds of findings, so a score gate
+  is unusable on day one and gets switched off. Recording the current state lets
+  CI block only regressions while the backlog stays visible.
+
+  Findings are matched on category, endpoint, and severity rather than message
+  text — messages embed counts and parameter names ("3/3 parameters have no
+  description (a, b, c)") that shift when unrelated things change, which would
+  report unfixed problems as new. Severity is part of the identity so a warning
+  escalating to a fail is caught; message is excluded so rewording a finding in
+  a future release doesn't invalidate stored baselines.
+
 ## [0.3.0] - 2026-07-31
 
 ### Added

{openapi_agent_ready-0.3.0 → openapi_agent_ready-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: openapi-agent-ready
-Version: 0.3.0
+Version: 0.4.0
 Summary: Audit any OpenAPI spec for AI-agent and MCP readiness, and generate an MCP server scaffold.
 Project-URL: Homepage, https://github.com/sfaisal/agent-ready
 Project-URL: Issues, https://github.com/sfaisal/agent-ready/issues
@@ -111,11 +111,9 @@ directly onto the rubric:
 | **Frictionless authentication** | 4 (auth/scope clarity) |
 | **Structured error handling** | 3 (actionable errors) |
 
-Categories 2 (side effects) and 5 (ambiguity) have no counterpart in the
-article — they were added during implementation, once it became clear that
-unsignalled write operations and confusable endpoint descriptions cause agent
-failures as often as the original three. Category 8 came later still, from
-OpenAI's function-calling guidance.
+Categories 2 (side effects) and 5 (ambiguity) were added during implementation
+— they turned out to matter as much as the original three, which is the usual
+reward for actually building the thing you wrote about.
 
 The gap is measurable, and enormous. A 2026 study of 856 tools across 103 MCP
 servers found that **97.1% of tool descriptions contained at least one quality
@@ -185,9 +183,42 @@ Exit codes:
 | Code | Meaning |
 |---|---|
 | `0` | All configured gates passed |
-| `1` | A gate failed (`--min-score` / `--max-fails`) |
+| `1` | A gate failed (`--min-score` / `--max-fails` / `--max-new`) |
 | `2` | The spec could not be loaded or parsed |
 
+### Adopting on an existing API
+
+A mature API audited for the first time will produce hundreds of findings. No
+team can fix all of them before merging anything, so a score gate is unusable
+on day one and gets switched off within a week.
+
+Record the current state instead, and block only what gets worse:
+
+```bash
+# once, when you adopt the tool — commit this file
+agent-ready openapi.yaml --write-baseline .agent-ready-baseline.json
+
+# in CI, from then on
+agent-ready openapi.yaml --baseline .agent-ready-baseline.json --max-new 0
+```
+
+Existing findings stay visible in the report but don't fail the build. A pull
+request that introduces a new one does, and the output names exactly which
+findings it added:
+
+```
+Compared against .agent-ready-baseline.json: 5 new, 0 fixed, 11 unchanged
+  NEW FAIL  GET /invoices — Description is a bare label (12 chars) ...
+  NEW FAIL  GET /invoices — No error responses documented ...
+```
+
+Findings are matched on category, endpoint, and severity — not on message text,
+which contains counts and parameter names that shift when unrelated things
+change. A warning escalating to a fail counts as new; rewording a finding in a
+future release does not invalidate your baseline.
+
+As the team fixes the backlog, regenerate the baseline to lock in the progress.
+
 ## Real-world results
 
 Running it against Stripe's production spec (587 endpoints), as of v0.1.0.

{openapi_agent_ready-0.3.0 → openapi_agent_ready-0.4.0}/README.md

@@ -56,11 +56,9 @@ directly onto the rubric:
 | **Frictionless authentication** | 4 (auth/scope clarity) |
 | **Structured error handling** | 3 (actionable errors) |
 
-Categories 2 (side effects) and 5 (ambiguity) have no counterpart in the
-article — they were added during implementation, once it became clear that
-unsignalled write operations and confusable endpoint descriptions cause agent
-failures as often as the original three. Category 8 came later still, from
-OpenAI's function-calling guidance.
+Categories 2 (side effects) and 5 (ambiguity) were added during implementation
+— they turned out to matter as much as the original three, which is the usual
+reward for actually building the thing you wrote about.
 
 The gap is measurable, and enormous. A 2026 study of 856 tools across 103 MCP
 servers found that **97.1% of tool descriptions contained at least one quality
@@ -130,9 +128,42 @@ Exit codes:
 | Code | Meaning |
 |---|---|
 | `0` | All configured gates passed |
-| `1` | A gate failed (`--min-score` / `--max-fails`) |
+| `1` | A gate failed (`--min-score` / `--max-fails` / `--max-new`) |
 | `2` | The spec could not be loaded or parsed |
 
+### Adopting on an existing API
+
+A mature API audited for the first time will produce hundreds of findings. No
+team can fix all of them before merging anything, so a score gate is unusable
+on day one and gets switched off within a week.
+
+Record the current state instead, and block only what gets worse:
+
+```bash
+# once, when you adopt the tool — commit this file
+agent-ready openapi.yaml --write-baseline .agent-ready-baseline.json
+
+# in CI, from then on
+agent-ready openapi.yaml --baseline .agent-ready-baseline.json --max-new 0
+```
+
+Existing findings stay visible in the report but don't fail the build. A pull
+request that introduces a new one does, and the output names exactly which
+findings it added:
+
+```
+Compared against .agent-ready-baseline.json: 5 new, 0 fixed, 11 unchanged
+  NEW FAIL  GET /invoices — Description is a bare label (12 chars) ...
+  NEW FAIL  GET /invoices — No error responses documented ...
+```
+
+Findings are matched on category, endpoint, and severity — not on message text,
+which contains counts and parameter names that shift when unrelated things
+change. A warning escalating to a fail counts as new; rewording a finding in a
+future release does not invalidate your baseline.
+
+As the team fixes the backlog, regenerate the baseline to lock in the progress.
+
 ## Real-world results
 
 Running it against Stripe's production spec (587 endpoints), as of v0.1.0.

{openapi_agent_ready-0.3.0 → openapi_agent_ready-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "openapi-agent-ready"
-version = "0.3.0"
+version = "0.4.0"
 description = "Audit any OpenAPI spec for AI-agent and MCP readiness, and generate an MCP server scaffold."
 readme = "README.md"
 requires-python = ">=3.10"

openapi_agent_ready-0.4.0/src/agent_ready/baseline.py

@@ -0,0 +1,137 @@
+"""
+Baseline support: fail CI only on *new* findings.
+
+The problem this solves: a mature API audited for the first time produces
+hundreds of findings. A team cannot fix them all before merging anything, so
+`--min-score` and `--max-fails` are unusable as gates and get switched off
+within a week. A baseline records the current state and lets the gate fail only
+when something gets worse.
+
+The whole design rests on one question: what makes a finding "the same finding"
+across two runs? Message text is the obvious answer and the wrong one — it
+embeds counts and parameter names ("3/3 parameters have no description
+(date_from, date_to, location)") that change when an unrelated parameter is
+added, which would report a fixed finding as new.
+
+So findings are fingerprinted on (category, endpoint, severity) instead:
+
+- **category** — which rule fired.
+- **endpoint** — where. Stable across edits to the endpoint's content.
+- **severity** — a warning becoming a fail is a regression worth catching, so
+  it belongs in the identity rather than being treated as the same finding.
+
+Message is deliberately excluded. Rewording a finding's text in a future
+release must not invalidate every stored baseline.
+"""
+
+import json
+from collections import Counter
+from dataclasses import asdict
+from pathlib import Path
+
+BASELINE_VERSION = 1
+
+
+class BaselineError(Exception):
+    """Raised when a baseline file can't be read or is malformed."""
+
+
+def fingerprint(finding) -> str:
+    """A stable identity for a finding across runs. See module docstring."""
+    return f"{finding.category}|{finding.endpoint}|{finding.severity}"
+
+
+def _counted(findings) -> Counter:
+    """
+    Fingerprints with multiplicity.
+
+    A single endpoint can legitimately produce two findings in one category —
+    for example a parameter-explanation fail for undocumented parameters plus a
+    warning for a missing date format. Counting rather than set-membership means
+    going from one such finding to two is correctly reported as new.
+    """
+    return Counter(fingerprint(f) for f in findings)
+
+
+def write_baseline(path: str, spec: dict, result: dict) -> dict:
+    """Record the current findings so future runs can compare against them."""
+    info = spec.get("info") or {}
+    payload = {
+        "baseline_version": BASELINE_VERSION,
+        "api": {
+            "title": info.get("title", "Untitled API"),
+            "version": str(info.get("version", "unknown")),
+        },
+        "overall_score": result["overall_score"],
+        "endpoint_count": result["endpoint_count"],
+        "fingerprints": sorted(_counted(result["gaps"]).elements()),
+        # Stored for human readability when reviewing a baseline diff; never
+        # read back during comparison.
+        "findings": [asdict(f) for f in result["gaps"]],
+    }
+    Path(path).write_text(json.dumps(payload, indent=2), encoding="utf-8")
+    return payload
+
+
+def load_baseline(path: str) -> dict:
+    p = Path(path)
+    if not p.exists():
+        raise BaselineError(
+            f"Baseline file not found: {path}\n"
+            f"Create one from the current state with: --write-baseline {path}"
+        )
+    try:
+        data = json.loads(p.read_text(encoding="utf-8"))
+    except json.JSONDecodeError as exc:
+        raise BaselineError(f"Baseline file is not valid JSON: {exc}") from exc
+
+    if not isinstance(data, dict) or "fingerprints" not in data:
+        raise BaselineError(
+            f"{path} does not look like an agent-ready baseline "
+            "(no 'fingerprints' key)."
+        )
+
+    version = data.get("baseline_version", 0)
+    if version > BASELINE_VERSION:
+        raise BaselineError(
+            f"Baseline was written by a newer version of agent-ready "
+            f"(format v{version}, this build understands v{BASELINE_VERSION}). "
+            "Upgrade agent-ready or regenerate the baseline."
+        )
+    return data
+
+
+def compare_to_baseline(result: dict, baseline: dict) -> dict:
+    """
+    Classify current findings against a baseline.
+
+    Returns counts plus the actual new findings, so the CLI can show a team
+    exactly what their change introduced rather than the whole backlog.
+    """
+    baseline_counts = Counter(baseline.get("fingerprints", []))
+    current_counts = _counted(result["gaps"])
+
+    # A fingerprint appearing more often than the baseline recorded is new,
+    # by the excess amount.
+    new_counts = current_counts - baseline_counts
+    fixed_counts = baseline_counts - current_counts
+
+    # Map fingerprints back to findings so the report can name them. Consume
+    # the allowance so duplicates beyond it are still reported.
+    remaining = Counter(new_counts)
+    new_findings = []
+    for f in result["gaps"]:
+        fp = fingerprint(f)
+        if remaining.get(fp, 0) > 0:
+            new_findings.append(f)
+            remaining[fp] -= 1
+
+    return {
+        "new_findings": new_findings,
+        "new_count": sum(new_counts.values()),
+        "fixed_count": sum(fixed_counts.values()),
+        "unchanged_count": sum((current_counts & baseline_counts).values()),
+        "new_fails": sum(1 for f in new_findings if f.severity == "fail"),
+        "baseline_score": baseline.get("overall_score"),
+        "current_score": result["overall_score"],
+    }

{openapi_agent_ready-0.3.0 → openapi_agent_ready-0.4.0}/src/agent_ready/cli.py

@@ -21,6 +21,12 @@ from .auditor import (
     result_to_dict,
     run_audit,
 )
+from .baseline import (
+    BaselineError,
+    compare_to_baseline,
+    load_baseline,
+    write_baseline,
+)
 from .mcp_generator import generate_mcp_scaffold
 from .report_generator import generate_report
 
@@ -38,7 +44,12 @@ def build_parser() -> argparse.ArgumentParser:
         ),
         epilog=(
             "CI example:\n"
-            "  agent-ready openapi.yaml --min-score 70 --max-fails 0 --quiet\n"
+            "  agent-ready openapi.yaml --min-score 60 --max-fails 0 --quiet\n"
+            "\n"
+            "Adopting on an existing API, where fixing everything first is not\n"
+            "realistic — record today's findings, then block only regressions:\n"
+            "  agent-ready openapi.yaml --write-baseline .agent-ready-baseline.json\n"
+            "  agent-ready openapi.yaml --baseline .agent-ready-baseline.json --max-new 0\n"
         ),
         formatter_class=argparse.RawDescriptionHelpFormatter,
     )
@@ -60,6 +71,31 @@ def build_parser() -> argparse.ArgumentParser:
         type=int,
         help="Exit 1 if the number of FAIL findings exceeds this value (CI gate)",
     )
+    parser.add_argument(
+        "--write-baseline",
+        metavar="PATH",
+        help=(
+            "Record current findings to PATH as an accepted baseline. Run this "
+            "once when adopting the tool on an existing API."
+        ),
+    )
+    parser.add_argument(
+        "--baseline",
+        metavar="PATH",
+        help=(
+            "Compare against a baseline file and report what changed. Combine "
+            "with --max-new to fail only on regressions."
+        ),
+    )
+    parser.add_argument(
+        "--max-new",
+        type=int,
+        metavar="N",
+        help=(
+            "Exit 1 if more than N findings are new since --baseline. "
+            "Use --max-new 0 to block any regression."
+        ),
+    )
     parser.add_argument(
         "--generate-mcp",
         metavar="PATH",
@@ -185,6 +221,37 @@ def _run(argv=None) -> int:
                 f"({len(ready)}/{result['endpoint_count']} endpoints included)"
             )
 
+    # Baseline: write, or compare and gate on regressions only.
+    if args.write_baseline:
+        write_baseline(args.write_baseline, spec, result)
+        if not args.quiet:
+            print(
+                f"Baseline written to {args.write_baseline} "
+                f"({len(result['gaps'])} finding(s) recorded as accepted)"
+            )
+
+    comparison = None
+    if args.baseline:
+        try:
+            comparison = compare_to_baseline(result, load_baseline(args.baseline))
+        except BaselineError as exc:
+            print(f"error: {exc}", file=sys.stderr)
+            return EXIT_SPEC_ERROR
+
+        if not args.quiet:
+            print()
+            print(
+                f"Compared against {args.baseline}: "
+                f"{comparison['new_count']} new, "
+                f"{comparison['fixed_count']} fixed, "
+                f"{comparison['unchanged_count']} unchanged"
+            )
+            for f in comparison["new_findings"][:20]:
+                marker = "FAIL" if f.severity == "fail" else "WARN"
+                print(f"  NEW {marker}  {f.endpoint} — {f.message}")
+            if len(comparison["new_findings"]) > 20:
+                print(f"  ... and {len(comparison['new_findings']) - 20} more")
+
     # CI gates
     gate_failed = False
     if args.min_score is not None and result["overall_score"] < args.min_score:
@@ -202,6 +269,18 @@ def _run(argv=None) -> int:
         )
         gate_failed = True
 
+    if (
+        comparison is not None
+        and args.max_new is not None
+        and comparison["new_count"] > args.max_new
+    ):
+        print(
+            f"GATE FAILED: {comparison['new_count']} new finding(s) since "
+            f"baseline exceeds maximum {args.max_new}",
+            file=sys.stderr,
+        )
+        gate_failed = True
+
     return EXIT_GATE_FAILED if gate_failed else EXIT_OK
 
 

openapi_agent_ready-0.4.0/tests/test_baseline.py

@@ -0,0 +1,306 @@
+"""
+Tests for baseline mode.
+
+The behaviour that matters here is not "does it diff two lists" — it is whether
+the notion of finding identity survives the edits a real team makes. Each test
+below corresponds to an edit that would produce a false result under a naive
+implementation.
+"""
+
+import json
+from pathlib import Path
+
+import pytest
+
+from agent_ready.auditor import load_spec, run_audit
+from agent_ready.baseline import (
+    BaselineError,
+    compare_to_baseline,
+    fingerprint,
+    load_baseline,
+    write_baseline,
+)
+from agent_ready.cli import main
+
+
+def _spec(paths):
+    return {"openapi": "3.0.0", "info": {"title": "T", "version": "1"}, "paths": paths}
+
+
+WEAK_ENDPOINT = {
+    "get": {
+        "summary": "Get things",
+        "description": "",
+        "parameters": [{"name": "q", "in": "query", "schema": {"type": "string"}}],
+        "responses": {"200": {"description": "OK"}},
+    }
+}
+
+STRONG_ENDPOINT = {
+    "get": {
+        "summary": "List things",
+        "description": (
+            "Returns every thing in the account with its identifier and current "
+            "status. This is a read-only lookup and does not modify data. Use "
+            "this when you need to find a thing before acting on it."
+        ),
+        "parameters": [
+            {
+                "name": "q",
+                "in": "query",
+                "description": "Free-text search over thing names.",
+                "schema": {"type": "string"},
+            }
+        ],
+        "responses": {
+            "200": {"description": "A list of matching thing records."},
+            "400": {
+                "description": "The query was malformed; q must be non-empty.",
+                "content": {"application/json": {"schema": {"type": "object"}}},
+            },
+        },
+    }
+}
+
+
+def _baseline_for(spec, tmp_path, name="base.json"):
+    path = tmp_path / name
+    write_baseline(str(path), spec, run_audit(spec))
+    return str(path)
+
+
+# --------------------------------------------------------------------------
+# Core behaviour
+# --------------------------------------------------------------------------
+
+def test_unchanged_spec_reports_no_new_findings(tmp_path):
+    """The whole point: an existing backlog must not block every merge."""
+    spec = _spec({"/a": WEAK_ENDPOINT, "/b": WEAK_ENDPOINT})
+    path = _baseline_for(spec, tmp_path)
+
+    result = run_audit(spec)
+    assert result["fail_count"] > 0, "fixture should have pre-existing failures"
+
+    comparison = compare_to_baseline(result, load_baseline(path))
+    assert comparison["new_count"] == 0
+    assert comparison["unchanged_count"] == len(result["gaps"])
+
+
+def test_added_bad_endpoint_is_reported_as_new(tmp_path):
+    spec = _spec({"/a": WEAK_ENDPOINT})
+    path = _baseline_for(spec, tmp_path)
+
+    worse = _spec({"/a": WEAK_ENDPOINT, "/new": WEAK_ENDPOINT})
+    comparison = compare_to_baseline(run_audit(worse), load_baseline(path))
+
+    assert comparison["new_count"] > 0
+    assert all("/new" in f.endpoint for f in comparison["new_findings"]), (
+        "only the added endpoint should be reported as new"
+    )
+
+
+def test_fixed_findings_are_reported_and_do_not_fail(tmp_path):
+    spec = _spec({"/a": WEAK_ENDPOINT, "/b": WEAK_ENDPOINT})
+    path = _baseline_for(spec, tmp_path)
+
+    better = _spec({"/a": STRONG_ENDPOINT, "/b": WEAK_ENDPOINT})
+    comparison = compare_to_baseline(run_audit(better), load_baseline(path))
+
+    assert comparison["fixed_count"] > 0
+    assert comparison["new_count"] == 0, "improving a spec must never look like a regression"
+
+
+# --------------------------------------------------------------------------
+# Fingerprint stability — each of these breaks a naive implementation
+# --------------------------------------------------------------------------
+
+def test_message_text_changes_do_not_create_false_regressions(tmp_path):
+    """
+    Adding a second undocumented parameter changes the finding's message from
+    "1/1 parameter(s) ... (q)" to "2/2 parameter(s) ... (q, extra)". It is the
+    same unfixed problem. A message-based fingerprint would report it as new.
+    """
+    spec = _spec({"/a": WEAK_ENDPOINT})
+    path = _baseline_for(spec, tmp_path)
+
+    changed = json.loads(json.dumps(spec))
+    changed["paths"]["/a"]["get"]["parameters"].append(
+        {"name": "extra", "in": "query", "schema": {"type": "string"}}
+    )
+    comparison = compare_to_baseline(run_audit(changed), load_baseline(path))
+    assert comparison["new_count"] == 0, (
+        "a reworded message for the same unfixed problem must not count as new"
+    )
+
+
+def test_severity_escalation_is_caught(tmp_path):
+    """
+    A warning becoming a fail, in the same category on the same endpoint, is a
+    regression. Fingerprinting on category and endpoint alone would treat it as
+    unchanged and let it through.
+
+    The description lengths below are chosen deliberately: description_clarity
+    warns between 40 and 150 characters and fails below 40, so this shortening
+    changes severity without changing which categories fire. An earlier version
+    of this test degraded a description that had produced no finding at all, so
+    the new *category* made it pass and the severity component was never
+    exercised — it stayed green with severity removed from the fingerprint.
+    """
+    warning_level = _spec(
+        {
+            "/a": {
+                "get": {
+                    "summary": "",
+                    "description": "Returns the things in the account with status.",
+                    "parameters": [],
+                    "responses": {
+                        "200": {"description": "OK"},
+                        "default": {
+                            "description": "A structured error object.",
+                            "content": {"application/json": {"schema": {"type": "object"}}},
+                        },
+                    },
+                }
+            }
+        }
+    )
+    path = _baseline_for(warning_level, tmp_path)
+
+    before = [
+        f for f in run_audit(warning_level)["gaps"] if f.category == "description_clarity"
+    ]
+    assert [f.severity for f in before] == ["warning"], "fixture must start at warning"
+
+    fail_level = json.loads(json.dumps(warning_level))
+    fail_level["paths"]["/a"]["get"]["description"] = "Returns things."
+
+    after = [
+        f for f in run_audit(fail_level)["gaps"] if f.category == "description_clarity"
+    ]
+    assert [f.severity for f in after] == ["fail"], "fixture must degrade to fail"
+
+    comparison = compare_to_baseline(run_audit(fail_level), load_baseline(path))
+    assert comparison["new_fails"] > 0, (
+        "a warning escalating to a fail in the same category must be reported as new"
+    )
+
+
+def test_fingerprint_excludes_message():
+    """Rewording a finding in a future release must not invalidate baselines."""
+    from agent_ready.rubric import Finding
+
+    a = Finding("description_clarity", "fail", "GET /x", "original wording")
+    b = Finding("description_clarity", "fail", "GET /x", "completely different wording")
+    assert fingerprint(a) == fingerprint(b)
+
+
+def test_duplicate_findings_are_counted_not_deduplicated():
+    """
+    One endpoint can produce two findings in one category. Going from one to
+    two is a regression, so identity must carry multiplicity.
+    """
+    from agent_ready.rubric import Finding
+
+    f = Finding("parameter_explanation", "warning", "GET /x", "m")
+    baseline = {"fingerprints": [fingerprint(f)]}
+    result = {"gaps": [f, f], "overall_score": 50.0}
+
+    comparison = compare_to_baseline(result, baseline)
+    assert comparison["new_count"] == 1, "the second occurrence is new"
+
+
+# --------------------------------------------------------------------------
+# File handling
+# --------------------------------------------------------------------------
+
+def test_missing_baseline_gives_actionable_error(tmp_path):
+    with pytest.raises(BaselineError, match="--write-baseline"):
+        load_baseline(str(tmp_path / "nope.json"))
+
+
+def test_malformed_baseline_is_rejected(tmp_path):
+    p = tmp_path / "bad.json"
+    p.write_text("{not json")
+    with pytest.raises(BaselineError):
+        load_baseline(str(p))
+
+
+def test_wrong_shaped_json_is_rejected(tmp_path):
+    p = tmp_path / "other.json"
+    p.write_text(json.dumps({"some": "other file"}))
+    with pytest.raises(BaselineError, match="fingerprints"):
+        load_baseline(str(p))
+
+
+def test_newer_baseline_format_is_rejected(tmp_path):
+    p = tmp_path / "future.json"
+    p.write_text(json.dumps({"baseline_version": 999, "fingerprints": []}))
+    with pytest.raises(BaselineError, match="newer version"):
+        load_baseline(str(p))
+
+
+# --------------------------------------------------------------------------
+# CLI wiring and exit codes
+# --------------------------------------------------------------------------
+
+def _write_spec(tmp_path, spec, name="spec.json"):
+    p = tmp_path / name
+    p.write_text(json.dumps(spec))
+    return str(p)
+
+
+def test_cli_write_then_compare_passes(tmp_path):
+    spec_path = _write_spec(tmp_path, _spec({"/a": WEAK_ENDPOINT}))
+    base = str(tmp_path / "b.json")
+
+    assert main([spec_path, "--write-baseline", base, "--quiet"]) == 0
+    assert main([spec_path, "--baseline", base, "--max-new", "0", "--quiet"]) == 0
+
+
+def test_cli_fails_on_new_findings(tmp_path):
+    base_spec = _spec({"/a": WEAK_ENDPOINT})
+    base = str(tmp_path / "b.json")
+    main([_write_spec(tmp_path, base_spec), "--write-baseline", base, "--quiet"])
+
+    worse = _write_spec(tmp_path, _spec({"/a": WEAK_ENDPOINT, "/b": WEAK_ENDPOINT}), "worse.json")
+    assert main([worse, "--baseline", base, "--max-new", "0", "--quiet"]) == 1
+
+
+def test_cli_missing_baseline_exits_two(tmp_path):
+    spec_path = _write_spec(tmp_path, _spec({"/a": WEAK_ENDPOINT}))
+    assert main([spec_path, "--baseline", str(tmp_path / "absent.json"), "--quiet"]) == 2
+
+
+def test_baseline_alone_does_not_gate(tmp_path):
+    """--baseline reports; only --max-new turns it into a gate."""
+    base_spec = _spec({"/a": WEAK_ENDPOINT})
+    base = str(tmp_path / "b.json")
+    main([_write_spec(tmp_path, base_spec), "--write-baseline", base, "--quiet"])
+
+    worse = _write_spec(tmp_path, _spec({"/a": WEAK_ENDPOINT, "/b": WEAK_ENDPOINT}), "worse.json")
+    assert main([worse, "--baseline", base, "--quiet"]) == 0
+
+
+def test_baseline_file_is_valid_json_with_expected_keys(tmp_path):
+    spec = _spec({"/a": WEAK_ENDPOINT})
+    path = _baseline_for(spec, tmp_path)
+    data = json.loads(Path(path).read_text())
+
+    assert data["baseline_version"] >= 1
+    assert data["api"]["title"] == "T"
+    assert isinstance(data["fingerprints"], list)
+    assert data["fingerprints"] == sorted(data["fingerprints"]), (
+        "fingerprints must be sorted so the file is diff-friendly in git"
+    )
+
+
+def test_roundtrip_through_a_real_spec_file(tmp_path):
+    """End-to-end against the shipped example, not a synthetic fixture."""
+
+    examples = Path(__file__).parent.parent / "examples"
+    spec = load_spec(str(examples / "sample_booking_api.yaml"))
+    path = _baseline_for(spec, tmp_path, "real.json")
+
+    comparison = compare_to_baseline(run_audit(spec), load_baseline(path))
+    assert comparison["new_count"] == 0
+    assert comparison["fixed_count"] == 0