rosotacom-dev 2.5.dev15__tar.gz → 2.5.dev17__tar.gz

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev15
+Version: 2.5.dev17
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause
@@ -778,6 +778,26 @@ performance bands in `budgets.jsonl` via `rosotacom benchmark compare`, and
 bands move only through `rosotacom benchmark ratchet` — see
 [docs/performance-bands.md](docs/performance-bands.md) for the band schema,
 verdicts, and the ratchet workflow (RFC 0007).
+
+Two result sets taken in **different environments** — emulated versus a real
+link, or shaped versus unshaped — are compared with `rosotacom benchmark delta`:
+
+```bash
+rosotacom benchmark delta emulated/ ota/ \
+  --label-reference emulated --label-measured ota \
+  --out delta.json --markdown delta.md
+```
+
+It pairs runs on genre, profile, RMW and session name, so a row that ran in one
+environment and not the other is reported as unmatched rather than silently
+dropped — "it got faster" and "it did not run" must not look the same. Every
+report is labelled `monitor_only` and carries each side's command and profile
+file, so a table pasted into a paper or an issue can be traced back without the
+directory it came from.
+
+It exits 0 whatever the numbers say, and non-zero only when the two sets share
+no rows at all — a delta is an observation to explain, and an exit code would
+turn "explain this" into "re-run until it is quiet".
 For live benchmark probes, `--duration` is the shaped publish window; after that
 rosotacom stops synthetic publishers and waits `--drain-s` seconds before
 tearing down shaping, so delayed in-flight messages are not miscounted as loss.

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/README.md

@@ -733,6 +733,26 @@ performance bands in `budgets.jsonl` via `rosotacom benchmark compare`, and
 bands move only through `rosotacom benchmark ratchet` — see
 [docs/performance-bands.md](docs/performance-bands.md) for the band schema,
 verdicts, and the ratchet workflow (RFC 0007).
+
+Two result sets taken in **different environments** — emulated versus a real
+link, or shaped versus unshaped — are compared with `rosotacom benchmark delta`:
+
+```bash
+rosotacom benchmark delta emulated/ ota/ \
+  --label-reference emulated --label-measured ota \
+  --out delta.json --markdown delta.md
+```
+
+It pairs runs on genre, profile, RMW and session name, so a row that ran in one
+environment and not the other is reported as unmatched rather than silently
+dropped — "it got faster" and "it did not run" must not look the same. Every
+report is labelled `monitor_only` and carries each side's command and profile
+file, so a table pasted into a paper or an issue can be traced back without the
+directory it came from.
+
+It exits 0 whatever the numbers say, and non-zero only when the two sets share
+no rows at all — a delta is an observation to explain, and an exit code would
+turn "explain this" into "re-run until it is quiet".
 For live benchmark probes, `--duration` is the shaped publish window; after that
 rosotacom stops synthetic publishers and waits `--drain-s` seconds before
 tearing down shaping, so delayed in-flight messages are not miscounted as loss.

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/docs/rfcs/0002-expectation-concepts.md

@@ -290,6 +290,24 @@ once `optional` are all fixed:
       end-to-end by remote_assist (`/tf` local→global, `/move_base_free/goal`
       global→local, both delivered to `native_in`); ffmpeg video is n/a -- there is no
       video/ffmpeg feature anywhere in the codebase, so there is nothing to test.
+- [x] The third latched "not delivering" state is named (issue #231, Option A).
+      A latched topic that has a publisher but holds no value now reads distinctly
+      ("has a publisher but holds no retained latched value") from "no publisher"
+      and "stopped", so a broken/empty latch is no longer indistinguishable from a
+      silent producer (`classify_stage` carries a `latched` flag; `_stage_diagnosis`
+      names it). The transient_local read Option A asks for was **not** forced per
+      declared role: a first attempt did (`collect_latched_stage_topics` ->
+      forced QoS) and broke `e2e-remote-assist`, because a latched topic's *native
+      source* stage is published VOLATILE (the ROS 1 bridge offers no durability)
+      and a TRANSIENT_LOCAL reader is incompatible with a VOLATILE writer -- it
+      received nothing and reported a false STALLED. The correct, per-stage form
+      is the existing publisher-matched `_observation_qos`, which already requests
+      transient_local on the delivered stages that offer it (com_in/app_in); that
+      matched subscription is observable on the graph (it raises the subscriber
+      count). Validation:
+      `tests/unit/test_status_overview.py::test_observation_qos_does_not_force_transient_local_for_latched_roles`,
+      `::test_stage_diagnosis_names_the_three_latched_states`,
+      `::test_classify_stage_marks_latched_role`.
 
 ### Replay-only (ground-truth) assertions — from "Live vs replay" above
 

rosotacom_dev-2.5.dev17/src/rosotacom/benchmark_delta.py

@@ -0,0 +1,279 @@
+"""Compare two benchmark result sets taken in different environments.
+
+The local emulated lanes encode most of the performance truth cheaply and
+deterministically. What they cannot show is **which effects exist only on the
+real link** — cellular variability, VPN/MTU quirks, radio behaviour, real
+cross-traffic — and which are emulation artifacts. Real links stay monitor-only
+by the determinism rule, so the missing piece was never another measurement: it
+was the *comparison* between two sets of the same rows.
+
+This is that comparison. It reads two directories (or lists) of `result.json`
+runs, pairs them by what makes a row a row, and reports per-row, per-metric
+deltas as JSON plus a short Markdown summary.
+
+Three properties are deliberate, because it is easy to produce numbers that lie:
+
+* **Pairing is explicit and refuses to guess.** Rows are matched on genre,
+  profile, RMW and session name. A row present on one side only is reported as
+  unmatched rather than silently dropped, because "it got faster" and "it did
+  not run" must not look the same.
+* **Every report says it is correlational.** The two sets differ in the
+  environment *and* in everything the environment drags along; a delta is an
+  observation, never a verdict. `monitor_only: true` is in the JSON and the
+  first line of the Markdown.
+* **It is self-describing.** Each side carries its rosotacom SHA, profile file,
+  and the command that produced it, so a report pasted into a paper or an issue
+  can be traced back without the directory it came from.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+SCHEMA_VERSION = 1
+
+#: Numeric leaves under `result` that are worth comparing across environments.
+#: A deny-list rather than an allow-list would drag in ids and counts whose
+#: "delta" means nothing.
+INTERESTING_SUFFIXES = (
+    "capacity",
+    "latency_ms",
+    "latency_p95_ms",
+    "loss_pct",
+    "rate_hz",
+    "mean_payload_bytes",
+    "offered_bandwidth_bps",
+    "jitter_ms",
+    "throughput_bps",
+    "delivered",
+    "received",
+    "published",
+)
+
+
+@dataclass(frozen=True)
+class RowKey:
+    """What makes two runs the same row in different environments."""
+
+    genre: str
+    profile: str
+    rmw: str
+    session: str
+
+    def label(self) -> str:
+        return f"{self.genre}/{self.profile}/{self.rmw}/{self.session}"
+
+
+@dataclass
+class Run:
+    key: RowKey
+    metrics: dict[str, float]
+    provenance: dict[str, Any] = field(default_factory=dict)
+    path: str = ""
+
+
+def _flatten_numbers(value: Any, prefix: str = "") -> dict[str, float]:
+    out: dict[str, float] = {}
+    if isinstance(value, dict):
+        for key, child in value.items():
+            out.update(_flatten_numbers(child, f"{prefix}.{key}" if prefix else str(key)))
+    elif isinstance(value, bool):
+        # Explicitly before the numeric branch: bool is an int in Python, and a
+        # "delta" between True and False is a category error dressed as -1.
+        return {}
+    elif isinstance(value, (int, float)):
+        return {prefix: float(value)}
+    return out
+
+
+def interesting_metrics(result: Any) -> dict[str, float]:
+    numbers = _flatten_numbers(result)
+    return {
+        name: value
+        for name, value in numbers.items()
+        if any(name == suffix or name.endswith("." + suffix) for suffix in INTERESTING_SUFFIXES)
+    }
+
+
+def load_run(path: Path) -> Run | None:
+    """Read one `result.json`. Returns None when it is not one."""
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        return None
+    if not isinstance(data, dict) or "genre" not in data:
+        return None
+
+    context = data.get("context") or {}
+    session = context.get("session") or {}
+    profile = context.get("profile") or {}
+    rmw = context.get("rmw") or {}
+
+    key = RowKey(
+        genre=str(data.get("genre") or ""),
+        profile=str(profile.get("name") or (data.get("configuration") or {}).get("profile") or ""),
+        rmw=str(rmw.get("requested") or session.get("rmw") or ""),
+        session=str(session.get("name") or ""),
+    )
+    return Run(
+        key=key,
+        metrics=interesting_metrics(data.get("result")),
+        provenance={
+            "created_at": data.get("created_at"),
+            "command": context.get("command"),
+            "profiles_file": (context.get("paths") or {}).get("profiles_file"),
+            "rosotacom_sha": (context.get("versions") or {}).get("rosotacom_sha"),
+            "verdict": (data.get("verdict") or {}).get("status"),
+        },
+        path=str(path),
+    )
+
+
+def collect(paths: list[Path]) -> dict[RowKey, Run]:
+    """Every `result.json` under the given files or directories, by row."""
+    found: dict[RowKey, Run] = {}
+    for path in paths:
+        candidates = [path] if path.is_file() else sorted(path.rglob("result.json"))
+        for candidate in candidates:
+            run = load_run(candidate)
+            if run is None:
+                continue
+            # Last one wins, and it is the newest by directory ordering — a
+            # re-run of a row supersedes it rather than doubling it.
+            found[run.key] = run
+    return found
+
+
+def compare(
+    reference: dict[RowKey, Run],
+    measured: dict[RowKey, Run],
+    *,
+    label_reference: str,
+    label_measured: str,
+) -> dict[str, Any]:
+    rows: list[dict[str, Any]] = []
+    for key in sorted(set(reference) & set(measured), key=lambda k: k.label()):
+        a, b = reference[key], measured[key]
+        metrics = []
+        for name in sorted(set(a.metrics) | set(b.metrics)):
+            left = a.metrics.get(name)
+            right = b.metrics.get(name)
+            if left is None or right is None:
+                metrics.append(
+                    {
+                        "metric": name,
+                        label_reference: left,
+                        label_measured: right,
+                        "delta": None,
+                        "relative": None,
+                        "note": "present on one side only",
+                    }
+                )
+                continue
+            delta = right - left
+            metrics.append(
+                {
+                    "metric": name,
+                    label_reference: left,
+                    label_measured: right,
+                    "delta": delta,
+                    "relative": (delta / left) if left else None,
+                }
+            )
+        rows.append(
+            {
+                "row": key.label(),
+                "genre": key.genre,
+                "profile": key.profile,
+                "rmw": key.rmw,
+                "session": key.session,
+                "metrics": metrics,
+                "provenance": {label_reference: a.provenance, label_measured: b.provenance},
+            }
+        )
+
+    return {
+        "schema_version": SCHEMA_VERSION,
+        # First key, and repeated in the Markdown: a delta between two
+        # environments is correlational. The two sets differ in the environment
+        # and in everything it drags along.
+        "monitor_only": True,
+        "labels": {"reference": label_reference, "measured": label_measured},
+        "rows": rows,
+        "unmatched": {
+            label_reference: sorted(k.label() for k in set(reference) - set(measured)),
+            label_measured: sorted(k.label() for k in set(measured) - set(reference)),
+        },
+        "counts": {
+            "matched": len(rows),
+            f"only_{label_reference}": len(set(reference) - set(measured)),
+            f"only_{label_measured}": len(set(measured) - set(reference)),
+        },
+    }
+
+
+def render_markdown(report: dict[str, Any]) -> str:
+    ref = report["labels"]["reference"]
+    mea = report["labels"]["measured"]
+    lines = [
+        f"# Benchmark delta: {ref} vs {mea}",
+        "",
+        "**Monitor-only and correlational.** The two sets differ in the environment "
+        "and in everything the environment drags along; a delta here is an "
+        "observation to explain, never a verdict and never a gate.",
+        "",
+        f"{report['counts']['matched']} row(s) matched, "
+        f"{report['counts'][f'only_{ref}']} only in {ref}, "
+        f"{report['counts'][f'only_{mea}']} only in {mea}.",
+        "",
+    ]
+
+    for row in report["rows"]:
+        lines.append(f"## `{row['row']}`")
+        lines.append("")
+        lines.append(f"| metric | {ref} | {mea} | Δ | Δ% |")
+        lines.append("|---|---:|---:|---:|---:|")
+        for metric in row["metrics"]:
+            left = metric[ref]
+            right = metric[mea]
+            delta = metric["delta"]
+            relative = metric["relative"]
+            lines.append(
+                f"| `{metric['metric']}` | {_fmt(left)} | {_fmt(right)} | {_fmt(delta, sign=True)} | {_pct(relative)} |"
+            )
+        lines.append("")
+        for side in (ref, mea):
+            provenance = row["provenance"][side]
+            command = provenance.get("command") or "-"
+            lines.append(f"- {side}: `{command}`")
+        lines.append("")
+
+    for side in (ref, mea):
+        missing = report["unmatched"][side]
+        if missing:
+            lines.append(f"## Only in {side}")
+            lines.append("")
+            lines.extend(f"- `{name}`" for name in missing)
+            lines.append("")
+            lines.append(
+                "A row on one side only is not a delta. It ran in one environment "
+                "and not the other, and treating that as an improvement is the "
+                "commonest way this kind of report lies."
+            )
+            lines.append("")
+
+    return "\n".join(lines)
+
+
+def _fmt(value: float | None, *, sign: bool = False) -> str:
+    if value is None:
+        return "—"
+    formatted = f"{value:+.4g}" if sign else f"{value:.4g}"
+    return formatted
+
+
+def _pct(value: float | None) -> str:
+    return "—" if value is None else f"{value * 100:+.1f}%"

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/src/rosotacom/cli_benchmark.py

@@ -4843,6 +4843,56 @@ def _parse_values(raw: str) -> list[float]:
 # --------------------------------------------------------------------------- #
 
 
+def benchmark_delta(args: argparse.Namespace) -> int:
+    """`rosotacom benchmark delta` — the cross-environment comparison.
+
+    Exits 0 whenever the comparison itself succeeded, including when rows are
+    unmatched or every metric moved. This is monitor-only by construction
+    (`benchmark_delta.compare` puts `monitor_only: true` first in the report):
+    the number is the deliverable, and an exit code would turn "explain this"
+    into "re-run until it is quiet".
+
+    It exits non-zero for the one thing that is a failure — no rows in common,
+    which means the two sides were not the same rows and the report would be
+    an empty table with a confident heading.
+    """
+    from . import benchmark_delta as delta_module
+
+    reference = delta_module.collect([Path(args.reference)])
+    measured = delta_module.collect([Path(args.measured)])
+    report = delta_module.compare(
+        reference,
+        measured,
+        label_reference=args.label_reference,
+        label_measured=args.label_measured,
+    )
+
+    if args.out:
+        Path(args.out).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+    markdown = delta_module.render_markdown(report)
+    if args.markdown:
+        Path(args.markdown).write_text(markdown, encoding="utf-8")
+    else:
+        print(markdown)
+
+    counts = report["counts"]
+    print(
+        f"BENCHMARK DELTA: {counts['matched']} row(s) matched, "
+        f"{counts[f'only_{args.label_reference}']} only in {args.label_reference}, "
+        f"{counts[f'only_{args.label_measured}']} only in {args.label_measured} "
+        "(monitor-only)"
+    )
+    if not report["rows"]:
+        print(
+            "rosotacom: error: the two sets share no rows, so there is nothing to "
+            "compare. Check that both sides ran the same registry rows under the "
+            "same profile and RMW.",
+            file=sys.stderr,
+        )
+        return 1
+    return 0
+
+
 def _add_benchmark_common_args(parser: argparse.ArgumentParser, *, ota_benchmark: bool = False) -> None:
     from .cli import (
         OTA_DEFAULT_SUDO_MODE,
@@ -8068,6 +8118,18 @@ def _register_benchmark_band_parsers(benchmark_subparsers: Any) -> None:
     )
     compare_parser.set_defaults(func=benchmark_compare)
 
+    delta_parser = benchmark_subparsers.add_parser(
+        "delta",
+        help="Compare two result.json sets taken in different environments (monitor-only).",
+    )
+    delta_parser.add_argument("reference", help="Directory or result.json for the reference environment.")
+    delta_parser.add_argument("measured", help="Directory or result.json for the measured environment.")
+    delta_parser.add_argument("--label-reference", default="emulated", help="Name for the reference side.")
+    delta_parser.add_argument("--label-measured", default="ota", help="Name for the measured side.")
+    delta_parser.add_argument("--out", default=None, help="Write the JSON report here.")
+    delta_parser.add_argument("--markdown", default=None, help="Write the Markdown summary here.")
+    delta_parser.set_defaults(func=benchmark_delta)
+
     ratchet_parser = benchmark_subparsers.add_parser(
         "ratchet",
         help="Rewrite committed bands from result.json run(s) — bands are never hand-edited.",

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/src/rosotacom/resources/ws/ros2src/com_py/com_py/status_overview.py

@@ -161,6 +161,18 @@ class StageObserver(Node):
         value. Adopting the publisher's offered durability/reliability (request ==
         offered) is always compatible and replays the durable history. Falls back
         to best-effort/volatile when no publisher QoS is available.
+
+        Adopting the *publisher's* offered durability (rather than forcing
+        TRANSIENT_LOCAL for every stage of a declared latched topic) is
+        deliberate: a latched topic's own source stage -- the vehicle's native
+        publish -- is VOLATILE (the ROS 1 bridge offers no durability), and a
+        TRANSIENT_LOCAL reader is incompatible with a VOLATILE writer and would
+        receive nothing, turning a healthy native stage into a false STALLED.
+        Only the delivered stages (com_in/app_in) offer TRANSIENT_LOCAL, and the
+        per-publisher match already requests it there -- which is the
+        "subscribe transient_local on latched roles" of issue #231, made
+        per-stage-correct. The matched subscription is observable on the graph
+        (it raises the topic's subscriber count).
         """
         durability = DurabilityPolicy.VOLATILE
         reliability = ReliabilityPolicy.BEST_EFFORT

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/src/rosotacom/resources/ws/ros2src/com_py/com_py/status_overview_core.py

@@ -497,6 +497,7 @@ class StatusAggregator:
             "quality_reason": None,
             "observation": "graph" if obs is not None and obs.graph_only else "payload",
             "inferred_from": None,
+            "latched": str((expect or {}).get("mode", "")).strip().lower() == "latched",
         }
         if obs is None:
             return result
@@ -744,6 +745,16 @@ class StatusAggregator:
                     f"{topic} has a publisher; payload is intentionally not "
                     "subscribed and adjacent local flow is not observed"
                 )
+            if stage_result.get("latched"):
+                # The third state issue #231 names: distinct from "no publisher"
+                # (ABSENT) and "stopped" (STALE). The observation subscribes
+                # transient_local for latched roles, so a value published before
+                # it matched would have been replayed; holding none means nothing
+                # was ever latched (empty/broken latch), not a late-joining reader.
+                return (
+                    f"{topic} has a publisher but holds no retained latched "
+                    "value (nothing was latched, or the latch was lost)"
+                )
             return f"{topic} has a publisher but no messages observed yet"
         if st == STALE:
             return f"{topic} stopped receiving messages"

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/src/rosotacom_dev.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev15
+Version: 2.5.dev17
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause
@@ -778,6 +778,26 @@ performance bands in `budgets.jsonl` via `rosotacom benchmark compare`, and
 bands move only through `rosotacom benchmark ratchet` — see
 [docs/performance-bands.md](docs/performance-bands.md) for the band schema,
 verdicts, and the ratchet workflow (RFC 0007).
+
+Two result sets taken in **different environments** — emulated versus a real
+link, or shaped versus unshaped — are compared with `rosotacom benchmark delta`:
+
+```bash
+rosotacom benchmark delta emulated/ ota/ \
+  --label-reference emulated --label-measured ota \
+  --out delta.json --markdown delta.md
+```
+
+It pairs runs on genre, profile, RMW and session name, so a row that ran in one
+environment and not the other is reported as unmatched rather than silently
+dropped — "it got faster" and "it did not run" must not look the same. Every
+report is labelled `monitor_only` and carries each side's command and profile
+file, so a table pasted into a paper or an issue can be traced back without the
+directory it came from.
+
+It exits 0 whatever the numbers say, and non-zero only when the two sets share
+no rows at all — a delta is an observation to explain, and an exit code would
+turn "explain this" into "re-run until it is quiet".
 For live benchmark probes, `--duration` is the shaped publish window; after that
 rosotacom stops synthetic publishers and waits `--drain-s` seconds before
 tearing down shaping, so delayed in-flight messages are not miscounted as loss.

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/src/rosotacom_dev.egg-info/SOURCES.txt

@@ -92,6 +92,7 @@ src/rosotacom/anonymize.py
 src/rosotacom/bag_ground_truth.py
 src/rosotacom/benched_set.py
 src/rosotacom/benchmark.py
+src/rosotacom/benchmark_delta.py
 src/rosotacom/bundle_check.py
 src/rosotacom/cli.py
 src/rosotacom/cli_benchmark.py
@@ -314,6 +315,7 @@ tests/unit/test_anonymize.py
 tests/unit/test_bag_ground_truth.py
 tests/unit/test_benched_set.py
 tests/unit/test_benchmark.py
+tests/unit/test_benchmark_delta.py
 tests/unit/test_bundle_check.py
 tests/unit/test_cli.py
 tests/unit/test_cli_benchmark.py

{rosotacom_dev-2.5.dev15 → rosotacom_dev-2.5.dev17}/src/rosotacom_dev.egg-info/scm_file_list.json

@@ -93,6 +93,7 @@
     "src/rosotacom/bag_ground_truth.py",
     "src/rosotacom/benched_set.py",
     "src/rosotacom/benchmark.py",
+    "src/rosotacom/benchmark_delta.py",
     "src/rosotacom/bundle_check.py",
     "src/rosotacom/cli.py",
     "src/rosotacom/cli_benchmark.py",
@@ -309,6 +310,7 @@
     "tests/unit/test_bag_ground_truth.py",
     "tests/unit/test_benched_set.py",
     "tests/unit/test_benchmark.py",
+    "tests/unit/test_benchmark_delta.py",
     "tests/unit/test_bundle_check.py",
     "tests/unit/test_cli.py",
     "tests/unit/test_cli_benchmark.py",

rosotacom_dev-2.5.dev17/src/rosotacom_dev.egg-info/scm_version.json

@@ -0,0 +1,8 @@
+{
+  "tag": "2.4",
+  "distance": 17,
+  "node": "g1e061c3143329436d338320f8f58fd3d26affbed",
+  "dirty": false,
+  "branch": "HEAD",
+  "node_date": "2026-08-12"
+}