rosotacom-dev 2.5.dev15__tar.gz → 2.5.dev16__tar.gz

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev15
+Version: 2.5.dev16
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause
@@ -778,6 +778,26 @@ performance bands in `budgets.jsonl` via `rosotacom benchmark compare`, and
 bands move only through `rosotacom benchmark ratchet` — see
 [docs/performance-bands.md](docs/performance-bands.md) for the band schema,
 verdicts, and the ratchet workflow (RFC 0007).
+
+Two result sets taken in **different environments** — emulated versus a real
+link, or shaped versus unshaped — are compared with `rosotacom benchmark delta`:
+
+```bash
+rosotacom benchmark delta emulated/ ota/ \
+  --label-reference emulated --label-measured ota \
+  --out delta.json --markdown delta.md
+```
+
+It pairs runs on genre, profile, RMW and session name, so a row that ran in one
+environment and not the other is reported as unmatched rather than silently
+dropped — "it got faster" and "it did not run" must not look the same. Every
+report is labelled `monitor_only` and carries each side's command and profile
+file, so a table pasted into a paper or an issue can be traced back without the
+directory it came from.
+
+It exits 0 whatever the numbers say, and non-zero only when the two sets share
+no rows at all — a delta is an observation to explain, and an exit code would
+turn "explain this" into "re-run until it is quiet".
 For live benchmark probes, `--duration` is the shaped publish window; after that
 rosotacom stops synthetic publishers and waits `--drain-s` seconds before
 tearing down shaping, so delayed in-flight messages are not miscounted as loss.

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev16}/README.md

@@ -733,6 +733,26 @@ performance bands in `budgets.jsonl` via `rosotacom benchmark compare`, and
 bands move only through `rosotacom benchmark ratchet` — see
 [docs/performance-bands.md](docs/performance-bands.md) for the band schema,
 verdicts, and the ratchet workflow (RFC 0007).
+
+Two result sets taken in **different environments** — emulated versus a real
+link, or shaped versus unshaped — are compared with `rosotacom benchmark delta`:
+
+```bash
+rosotacom benchmark delta emulated/ ota/ \
+  --label-reference emulated --label-measured ota \
+  --out delta.json --markdown delta.md
+```
+
+It pairs runs on genre, profile, RMW and session name, so a row that ran in one
+environment and not the other is reported as unmatched rather than silently
+dropped — "it got faster" and "it did not run" must not look the same. Every
+report is labelled `monitor_only` and carries each side's command and profile
+file, so a table pasted into a paper or an issue can be traced back without the
+directory it came from.
+
+It exits 0 whatever the numbers say, and non-zero only when the two sets share
+no rows at all — a delta is an observation to explain, and an exit code would
+turn "explain this" into "re-run until it is quiet".
 For live benchmark probes, `--duration` is the shaped publish window; after that
 rosotacom stops synthetic publishers and waits `--drain-s` seconds before
 tearing down shaping, so delayed in-flight messages are not miscounted as loss.

rosotacom_dev-2.5.dev16/src/rosotacom/benchmark_delta.py

@@ -0,0 +1,279 @@
+"""Compare two benchmark result sets taken in different environments.
+
+The local emulated lanes encode most of the performance truth cheaply and
+deterministically. What they cannot show is **which effects exist only on the
+real link** — cellular variability, VPN/MTU quirks, radio behaviour, real
+cross-traffic — and which are emulation artifacts. Real links stay monitor-only
+by the determinism rule, so the missing piece was never another measurement: it
+was the *comparison* between two sets of the same rows.
+
+This is that comparison. It reads two directories (or lists) of `result.json`
+runs, pairs them by what makes a row a row, and reports per-row, per-metric
+deltas as JSON plus a short Markdown summary.
+
+Three properties are deliberate, because it is easy to produce numbers that lie:
+
+* **Pairing is explicit and refuses to guess.** Rows are matched on genre,
+  profile, RMW and session name. A row present on one side only is reported as
+  unmatched rather than silently dropped, because "it got faster" and "it did
+  not run" must not look the same.
+* **Every report says it is correlational.** The two sets differ in the
+  environment *and* in everything the environment drags along; a delta is an
+  observation, never a verdict. `monitor_only: true` is in the JSON and the
+  first line of the Markdown.
+* **It is self-describing.** Each side carries its rosotacom SHA, profile file,
+  and the command that produced it, so a report pasted into a paper or an issue
+  can be traced back without the directory it came from.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+SCHEMA_VERSION = 1
+
+#: Numeric leaves under `result` that are worth comparing across environments.
+#: A deny-list rather than an allow-list would drag in ids and counts whose
+#: "delta" means nothing.
+INTERESTING_SUFFIXES = (
+    "capacity",
+    "latency_ms",
+    "latency_p95_ms",
+    "loss_pct",
+    "rate_hz",
+    "mean_payload_bytes",
+    "offered_bandwidth_bps",
+    "jitter_ms",
+    "throughput_bps",
+    "delivered",
+    "received",
+    "published",
+)
+
+
+@dataclass(frozen=True)
+class RowKey:
+    """What makes two runs the same row in different environments."""
+
+    genre: str
+    profile: str
+    rmw: str
+    session: str
+
+    def label(self) -> str:
+        return f"{self.genre}/{self.profile}/{self.rmw}/{self.session}"
+
+
+@dataclass
+class Run:
+    key: RowKey
+    metrics: dict[str, float]
+    provenance: dict[str, Any] = field(default_factory=dict)
+    path: str = ""
+
+
+def _flatten_numbers(value: Any, prefix: str = "") -> dict[str, float]:
+    out: dict[str, float] = {}
+    if isinstance(value, dict):
+        for key, child in value.items():
+            out.update(_flatten_numbers(child, f"{prefix}.{key}" if prefix else str(key)))
+    elif isinstance(value, bool):
+        # Explicitly before the numeric branch: bool is an int in Python, and a
+        # "delta" between True and False is a category error dressed as -1.
+        return {}
+    elif isinstance(value, (int, float)):
+        return {prefix: float(value)}
+    return out
+
+
+def interesting_metrics(result: Any) -> dict[str, float]:
+    numbers = _flatten_numbers(result)
+    return {
+        name: value
+        for name, value in numbers.items()
+        if any(name == suffix or name.endswith("." + suffix) for suffix in INTERESTING_SUFFIXES)
+    }
+
+
+def load_run(path: Path) -> Run | None:
+    """Read one `result.json`. Returns None when it is not one."""
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        return None
+    if not isinstance(data, dict) or "genre" not in data:
+        return None
+
+    context = data.get("context") or {}
+    session = context.get("session") or {}
+    profile = context.get("profile") or {}
+    rmw = context.get("rmw") or {}
+
+    key = RowKey(
+        genre=str(data.get("genre") or ""),
+        profile=str(profile.get("name") or (data.get("configuration") or {}).get("profile") or ""),
+        rmw=str(rmw.get("requested") or session.get("rmw") or ""),
+        session=str(session.get("name") or ""),
+    )
+    return Run(
+        key=key,
+        metrics=interesting_metrics(data.get("result")),
+        provenance={
+            "created_at": data.get("created_at"),
+            "command": context.get("command"),
+            "profiles_file": (context.get("paths") or {}).get("profiles_file"),
+            "rosotacom_sha": (context.get("versions") or {}).get("rosotacom_sha"),
+            "verdict": (data.get("verdict") or {}).get("status"),
+        },
+        path=str(path),
+    )
+
+
+def collect(paths: list[Path]) -> dict[RowKey, Run]:
+    """Every `result.json` under the given files or directories, by row."""
+    found: dict[RowKey, Run] = {}
+    for path in paths:
+        candidates = [path] if path.is_file() else sorted(path.rglob("result.json"))
+        for candidate in candidates:
+            run = load_run(candidate)
+            if run is None:
+                continue
+            # Last one wins, and it is the newest by directory ordering — a
+            # re-run of a row supersedes it rather than doubling it.
+            found[run.key] = run
+    return found
+
+
+def compare(
+    reference: dict[RowKey, Run],
+    measured: dict[RowKey, Run],
+    *,
+    label_reference: str,
+    label_measured: str,
+) -> dict[str, Any]:
+    rows: list[dict[str, Any]] = []
+    for key in sorted(set(reference) & set(measured), key=lambda k: k.label()):
+        a, b = reference[key], measured[key]
+        metrics = []
+        for name in sorted(set(a.metrics) | set(b.metrics)):
+            left = a.metrics.get(name)
+            right = b.metrics.get(name)
+            if left is None or right is None:
+                metrics.append(
+                    {
+                        "metric": name,
+                        label_reference: left,
+                        label_measured: right,
+                        "delta": None,
+                        "relative": None,
+                        "note": "present on one side only",
+                    }
+                )
+                continue
+            delta = right - left
+            metrics.append(
+                {
+                    "metric": name,
+                    label_reference: left,
+                    label_measured: right,
+                    "delta": delta,
+                    "relative": (delta / left) if left else None,
+                }
+            )
+        rows.append(
+            {
+                "row": key.label(),
+                "genre": key.genre,
+                "profile": key.profile,
+                "rmw": key.rmw,
+                "session": key.session,
+                "metrics": metrics,
+                "provenance": {label_reference: a.provenance, label_measured: b.provenance},
+            }
+        )
+
+    return {
+        "schema_version": SCHEMA_VERSION,
+        # First key, and repeated in the Markdown: a delta between two
+        # environments is correlational. The two sets differ in the environment
+        # and in everything it drags along.
+        "monitor_only": True,
+        "labels": {"reference": label_reference, "measured": label_measured},
+        "rows": rows,
+        "unmatched": {
+            label_reference: sorted(k.label() for k in set(reference) - set(measured)),
+            label_measured: sorted(k.label() for k in set(measured) - set(reference)),
+        },
+        "counts": {
+            "matched": len(rows),
+            f"only_{label_reference}": len(set(reference) - set(measured)),
+            f"only_{label_measured}": len(set(measured) - set(reference)),
+        },
+    }
+
+
+def render_markdown(report: dict[str, Any]) -> str:
+    ref = report["labels"]["reference"]
+    mea = report["labels"]["measured"]
+    lines = [
+        f"# Benchmark delta: {ref} vs {mea}",
+        "",
+        "**Monitor-only and correlational.** The two sets differ in the environment "
+        "and in everything the environment drags along; a delta here is an "
+        "observation to explain, never a verdict and never a gate.",
+        "",
+        f"{report['counts']['matched']} row(s) matched, "
+        f"{report['counts'][f'only_{ref}']} only in {ref}, "
+        f"{report['counts'][f'only_{mea}']} only in {mea}.",
+        "",
+    ]
+
+    for row in report["rows"]:
+        lines.append(f"## `{row['row']}`")
+        lines.append("")
+        lines.append(f"| metric | {ref} | {mea} | Δ | Δ% |")
+        lines.append("|---|---:|---:|---:|---:|")
+        for metric in row["metrics"]:
+            left = metric[ref]
+            right = metric[mea]
+            delta = metric["delta"]
+            relative = metric["relative"]
+            lines.append(
+                f"| `{metric['metric']}` | {_fmt(left)} | {_fmt(right)} | {_fmt(delta, sign=True)} | {_pct(relative)} |"
+            )
+        lines.append("")
+        for side in (ref, mea):
+            provenance = row["provenance"][side]
+            command = provenance.get("command") or "-"
+            lines.append(f"- {side}: `{command}`")
+        lines.append("")
+
+    for side in (ref, mea):
+        missing = report["unmatched"][side]
+        if missing:
+            lines.append(f"## Only in {side}")
+            lines.append("")
+            lines.extend(f"- `{name}`" for name in missing)
+            lines.append("")
+            lines.append(
+                "A row on one side only is not a delta. It ran in one environment "
+                "and not the other, and treating that as an improvement is the "
+                "commonest way this kind of report lies."
+            )
+            lines.append("")
+
+    return "\n".join(lines)
+
+
+def _fmt(value: float | None, *, sign: bool = False) -> str:
+    if value is None:
+        return "—"
+    formatted = f"{value:+.4g}" if sign else f"{value:.4g}"
+    return formatted
+
+
+def _pct(value: float | None) -> str:
+    return "—" if value is None else f"{value * 100:+.1f}%"

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev16}/src/rosotacom/cli_benchmark.py

@@ -4843,6 +4843,56 @@ def _parse_values(raw: str) -> list[float]:
 # --------------------------------------------------------------------------- #
 
 
+def benchmark_delta(args: argparse.Namespace) -> int:
+    """`rosotacom benchmark delta` — the cross-environment comparison.
+
+    Exits 0 whenever the comparison itself succeeded, including when rows are
+    unmatched or every metric moved. This is monitor-only by construction
+    (`benchmark_delta.compare` puts `monitor_only: true` first in the report):
+    the number is the deliverable, and an exit code would turn "explain this"
+    into "re-run until it is quiet".
+
+    It exits non-zero for the one thing that is a failure — no rows in common,
+    which means the two sides were not the same rows and the report would be
+    an empty table with a confident heading.
+    """
+    from . import benchmark_delta as delta_module
+
+    reference = delta_module.collect([Path(args.reference)])
+    measured = delta_module.collect([Path(args.measured)])
+    report = delta_module.compare(
+        reference,
+        measured,
+        label_reference=args.label_reference,
+        label_measured=args.label_measured,
+    )
+
+    if args.out:
+        Path(args.out).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+    markdown = delta_module.render_markdown(report)
+    if args.markdown:
+        Path(args.markdown).write_text(markdown, encoding="utf-8")
+    else:
+        print(markdown)
+
+    counts = report["counts"]
+    print(
+        f"BENCHMARK DELTA: {counts['matched']} row(s) matched, "
+        f"{counts[f'only_{args.label_reference}']} only in {args.label_reference}, "
+        f"{counts[f'only_{args.label_measured}']} only in {args.label_measured} "
+        "(monitor-only)"
+    )
+    if not report["rows"]:
+        print(
+            "rosotacom: error: the two sets share no rows, so there is nothing to "
+            "compare. Check that both sides ran the same registry rows under the "
+            "same profile and RMW.",
+            file=sys.stderr,
+        )
+        return 1
+    return 0
+
+
 def _add_benchmark_common_args(parser: argparse.ArgumentParser, *, ota_benchmark: bool = False) -> None:
     from .cli import (
         OTA_DEFAULT_SUDO_MODE,
@@ -8068,6 +8118,18 @@ def _register_benchmark_band_parsers(benchmark_subparsers: Any) -> None:
     )
     compare_parser.set_defaults(func=benchmark_compare)
 
+    delta_parser = benchmark_subparsers.add_parser(
+        "delta",
+        help="Compare two result.json sets taken in different environments (monitor-only).",
+    )
+    delta_parser.add_argument("reference", help="Directory or result.json for the reference environment.")
+    delta_parser.add_argument("measured", help="Directory or result.json for the measured environment.")
+    delta_parser.add_argument("--label-reference", default="emulated", help="Name for the reference side.")
+    delta_parser.add_argument("--label-measured", default="ota", help="Name for the measured side.")
+    delta_parser.add_argument("--out", default=None, help="Write the JSON report here.")
+    delta_parser.add_argument("--markdown", default=None, help="Write the Markdown summary here.")
+    delta_parser.set_defaults(func=benchmark_delta)
+
     ratchet_parser = benchmark_subparsers.add_parser(
         "ratchet",
         help="Rewrite committed bands from result.json run(s) — bands are never hand-edited.",

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev16}/src/rosotacom_dev.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev15
+Version: 2.5.dev16
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause
@@ -778,6 +778,26 @@ performance bands in `budgets.jsonl` via `rosotacom benchmark compare`, and
 bands move only through `rosotacom benchmark ratchet` — see
 [docs/performance-bands.md](docs/performance-bands.md) for the band schema,
 verdicts, and the ratchet workflow (RFC 0007).
+
+Two result sets taken in **different environments** — emulated versus a real
+link, or shaped versus unshaped — are compared with `rosotacom benchmark delta`:
+
+```bash
+rosotacom benchmark delta emulated/ ota/ \
+  --label-reference emulated --label-measured ota \
+  --out delta.json --markdown delta.md
+```
+
+It pairs runs on genre, profile, RMW and session name, so a row that ran in one
+environment and not the other is reported as unmatched rather than silently
+dropped — "it got faster" and "it did not run" must not look the same. Every
+report is labelled `monitor_only` and carries each side's command and profile
+file, so a table pasted into a paper or an issue can be traced back without the
+directory it came from.
+
+It exits 0 whatever the numbers say, and non-zero only when the two sets share
+no rows at all — a delta is an observation to explain, and an exit code would
+turn "explain this" into "re-run until it is quiet".
 For live benchmark probes, `--duration` is the shaped publish window; after that
 rosotacom stops synthetic publishers and waits `--drain-s` seconds before
 tearing down shaping, so delayed in-flight messages are not miscounted as loss.

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev16}/src/rosotacom_dev.egg-info/SOURCES.txt

@@ -92,6 +92,7 @@ src/rosotacom/anonymize.py
 src/rosotacom/bag_ground_truth.py
 src/rosotacom/benched_set.py
 src/rosotacom/benchmark.py
+src/rosotacom/benchmark_delta.py
 src/rosotacom/bundle_check.py
 src/rosotacom/cli.py
 src/rosotacom/cli_benchmark.py
@@ -314,6 +315,7 @@ tests/unit/test_anonymize.py
 tests/unit/test_bag_ground_truth.py
 tests/unit/test_benched_set.py
 tests/unit/test_benchmark.py
+tests/unit/test_benchmark_delta.py
 tests/unit/test_bundle_check.py
 tests/unit/test_cli.py
 tests/unit/test_cli_benchmark.py

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev16}/src/rosotacom_dev.egg-info/scm_file_list.json

@@ -93,6 +93,7 @@
     "src/rosotacom/bag_ground_truth.py",
     "src/rosotacom/benched_set.py",
     "src/rosotacom/benchmark.py",
+    "src/rosotacom/benchmark_delta.py",
     "src/rosotacom/bundle_check.py",
     "src/rosotacom/cli.py",
     "src/rosotacom/cli_benchmark.py",
@@ -309,6 +310,7 @@
     "tests/unit/test_bag_ground_truth.py",
     "tests/unit/test_benched_set.py",
     "tests/unit/test_benchmark.py",
+    "tests/unit/test_benchmark_delta.py",
     "tests/unit/test_bundle_check.py",
     "tests/unit/test_cli.py",
     "tests/unit/test_cli_benchmark.py",

rosotacom_dev-2.5.dev16/src/rosotacom_dev.egg-info/scm_version.json

@@ -0,0 +1,8 @@
+{
+  "tag": "2.4",
+  "distance": 16,
+  "node": "ge8bcbe8f985ad79a3be256a4e360cdad4b9d13c0",
+  "dirty": false,
+  "branch": "HEAD",
+  "node_date": "2026-08-12"
+}

rosotacom_dev-2.5.dev16/tests/unit/test_benchmark_delta.py

@@ -0,0 +1,172 @@
+"""Unit tests for the cross-environment benchmark delta (harness #36).
+
+Fixtures, not runs. The point of the tool is what it says when two sets differ,
+and a real pair of environments is the one input that cannot be made to differ
+on purpose.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from rosotacom.benchmark_delta import collect, compare, interesting_metrics, render_markdown
+
+
+def write_result(
+    directory: Path,
+    *,
+    genre: str = "capacity",
+    profile: str = "cellular-4g-typical",
+    rmw: str = "cyclone",
+    session: str = "bench_1_1_capacity",
+    result: dict | None = None,
+    command: str = "rosotacom benchmark capacity",
+) -> Path:
+    directory.mkdir(parents=True, exist_ok=True)
+    payload = {
+        "schema_version": 1,
+        "genre": genre,
+        "created_at": "2026-08-12T09:00:00",
+        "context": {
+            "command": command,
+            "profile": {"name": profile},
+            "rmw": {"requested": rmw},
+            "session": {"name": session, "rmw": rmw},
+            "paths": {"profiles_file": "profiles/benchmark-profiles.yaml"},
+        },
+        "result": result if result is not None else {"capacity": 43000, "slice": {"loss_pct": 0.5}},
+        "verdict": {"passed": True, "status": "capacity_found"},
+    }
+    path = directory / "result.json"
+    path.write_text(json.dumps(payload), encoding="utf-8")
+    return path
+
+
+def test_metrics_are_selected_not_swept_up() -> None:
+    metrics = interesting_metrics(
+        {
+            "capacity": 43000,
+            "slice": {"loss_pct": 0.5, "knob": "size"},
+            "run_id": 17,
+            "passed": True,
+        }
+    )
+    assert metrics == {"capacity": 43000.0, "slice.loss_pct": 0.5}
+    assert "run_id" not in metrics, "an id's delta means nothing"
+    assert "passed" not in metrics, "a bool delta is a category error dressed as -1"
+
+
+def test_matching_rows_produce_a_delta(tmp_path: Path) -> None:
+    write_result(tmp_path / "emu" / "run1", result={"capacity": 43000, "slice": {"loss_pct": 0.5}})
+    write_result(tmp_path / "ota" / "run1", result={"capacity": 21000, "slice": {"loss_pct": 2.0}})
+
+    report = compare(
+        collect([tmp_path / "emu"]),
+        collect([tmp_path / "ota"]),
+        label_reference="emulated",
+        label_measured="ota",
+    )
+
+    assert report["counts"]["matched"] == 1
+    metrics = {m["metric"]: m for m in report["rows"][0]["metrics"]}
+    assert metrics["capacity"]["delta"] == -22000
+    assert metrics["capacity"]["relative"] < 0
+    assert metrics["slice.loss_pct"]["delta"] == 1.5
+
+
+def test_a_row_on_one_side_only_is_never_a_delta(tmp_path: Path) -> None:
+    """ "It got faster" and "it did not run" must not look the same."""
+    write_result(tmp_path / "emu" / "a", session="row_a")
+    write_result(tmp_path / "emu" / "b", session="row_b")
+    write_result(tmp_path / "ota" / "a", session="row_a")
+
+    report = compare(
+        collect([tmp_path / "emu"]),
+        collect([tmp_path / "ota"]),
+        label_reference="emulated",
+        label_measured="ota",
+    )
+
+    assert report["counts"]["matched"] == 1
+    assert report["counts"]["only_emulated"] == 1
+    assert any("row_b" in name for name in report["unmatched"]["emulated"])
+
+
+def test_rows_differing_only_in_profile_do_not_pair(tmp_path: Path) -> None:
+    """A tight profile is not a delta against a nominal one."""
+    write_result(tmp_path / "emu" / "a", profile="cellular-4g-typical")
+    write_result(tmp_path / "ota" / "a", profile="cellular-4g-degraded")
+
+    report = compare(
+        collect([tmp_path / "emu"]),
+        collect([tmp_path / "ota"]),
+        label_reference="emulated",
+        label_measured="ota",
+    )
+    assert report["counts"]["matched"] == 0
+
+
+def test_rows_differing_only_in_rmw_do_not_pair(tmp_path: Path) -> None:
+    write_result(tmp_path / "emu" / "a", rmw="cyclone")
+    write_result(tmp_path / "ota" / "a", rmw="fastdds")
+
+    report = compare(
+        collect([tmp_path / "emu"]),
+        collect([tmp_path / "ota"]),
+        label_reference="emulated",
+        label_measured="ota",
+    )
+    assert report["counts"]["matched"] == 0
+
+
+def test_the_report_says_it_is_monitor_only(tmp_path: Path) -> None:
+    write_result(tmp_path / "emu" / "a")
+    write_result(tmp_path / "ota" / "a")
+    report = compare(
+        collect([tmp_path / "emu"]),
+        collect([tmp_path / "ota"]),
+        label_reference="emulated",
+        label_measured="ota",
+    )
+
+    assert report["monitor_only"] is True
+    markdown = render_markdown(report)
+    assert "Monitor-only" in markdown.splitlines()[2]
+    assert "never a verdict" in markdown
+
+
+def test_the_report_carries_its_own_provenance(tmp_path: Path) -> None:
+    write_result(tmp_path / "emu" / "a", command="rosotacom benchmark capacity --profile nominal")
+    write_result(tmp_path / "ota" / "a", command="rosotacom ota-benchmark row --row s3")
+    report = compare(
+        collect([tmp_path / "emu"]),
+        collect([tmp_path / "ota"]),
+        label_reference="emulated",
+        label_measured="ota",
+    )
+    markdown = render_markdown(report)
+    assert "--profile nominal" in markdown
+    assert "ota-benchmark row --row s3" in markdown
+
+
+def test_a_metric_present_on_one_side_only_is_flagged(tmp_path: Path) -> None:
+    write_result(tmp_path / "emu" / "a", result={"capacity": 1000, "slice": {"loss_pct": 0.0}})
+    write_result(tmp_path / "ota" / "a", result={"capacity": 900})
+    report = compare(
+        collect([tmp_path / "emu"]),
+        collect([tmp_path / "ota"]),
+        label_reference="emulated",
+        label_measured="ota",
+    )
+    metrics = {m["metric"]: m for m in report["rows"][0]["metrics"]}
+    assert metrics["slice.loss_pct"]["delta"] is None
+    assert metrics["slice.loss_pct"]["note"] == "present on one side only"
+
+
+def test_non_result_json_files_are_ignored(tmp_path: Path) -> None:
+    write_result(tmp_path / "emu" / "a")
+    (tmp_path / "emu" / "noise").mkdir()
+    (tmp_path / "emu" / "noise" / "result.json").write_text('{"not": "a benchmark"}', encoding="utf-8")
+
+    assert len(collect([tmp_path / "emu"])) == 1

rosotacom_dev-2.5.dev15/src/rosotacom_dev.egg-info/scm_version.json

@@ -1,8 +0,0 @@
-{
-  "tag": "2.4",
-  "distance": 15,
-  "node": "g2001e7ec951a5e84a9457e2c47947db11655bb1b",
-  "dirty": false,
-  "branch": "HEAD",
-  "node_date": "2026-08-11"
-}