rosotacom-dev 2.5.dev50__tar.gz → 2.5.dev52__tar.gz

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev52}/.github/workflows/nightly-e2e.yml

@@ -108,6 +108,7 @@ jobs:
           - cyclone-local-zenoh-ros2dds-ota
           - cyclone-ota
           - fastdds
+          - fastdds-default
           - zen-endpoints
     steps:
       - name: Check out repository

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev52}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev50
+Version: 2.5.dev52
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause
@@ -580,6 +580,13 @@ the ssh client's own argv apart, and a transport that promises no more than
 `-t` and `-o BatchMode=yes` are dropped rather than forwarded. They are options
 of one client, and a transport that is not ssh never agreed to understand them.
 
+The same four options — `--peer-checkout`, `--peer-exec`, `--install-mode`,
+`--install-pin` — are on every `ota-benchmark` genre, not just `ota-smoke`. A
+measurement is the last thing that should require a shell on the peer: the
+whole point of a narrower transport is that the runs which touch a machine
+repeatedly can go through it, and a benchmark touches it far more often than a
+smoke test does.
+
 ### Live status / debugging overview
 
 Enable a continuously-updated, per-topic pipeline overview by setting

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev52}/README.md

@@ -535,6 +535,13 @@ the ssh client's own argv apart, and a transport that promises no more than
 `-t` and `-o BatchMode=yes` are dropped rather than forwarded. They are options
 of one client, and a transport that is not ssh never agreed to understand them.
 
+The same four options — `--peer-checkout`, `--peer-exec`, `--install-mode`,
+`--install-pin` — are on every `ota-benchmark` genre, not just `ota-smoke`. A
+measurement is the last thing that should require a shell on the peer: the
+whole point of a narrower transport is that the runs which touch a machine
+repeatedly can go through it, and a benchmark touches it far more often than a
+smoke test does.
+
 ### Live status / debugging overview
 
 Enable a continuously-updated, per-topic pipeline overview by setting

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev52}/docs/findings/README.md

@@ -28,6 +28,11 @@ living verification hook.
 
 ## Findings
 
+- [Sender-side transit rows made a dead link report zero loss](sender-rows-make-a-dead-link-look-loss-free.md) - `delivered = rows - lost` counted the sender's own `sent` rows as deliveries, so a topic the receiver never saw summarised as `delivered == expected` at 0.0% loss, with the publisher's timer reported as arrival spacing.
+- [A bridge that learns types from the graph cannot use a transport that does not carry the graph](graph-derived-types-block-a-non-graph-transport.md) - over `zenoh_ros2dds` the payload topic never arrived in either run: the bridge routes a topic once a local reader exists and the reader waited for the bridge, leaving only `Pending=1` in a log.
+- [A Fast DDS datagram cap stops every message larger than it](fastdds-datagram-cap-drops-large-samples.md) - copying CycloneDDS's 1200 B fragment size into the Fast DDS OTA profile makes every sample above the cap never arrive, at 1200 B and at 8192 B, sync and async — while the 84 B heartbeat keeps flowing at 10 Hz.
+- [A lost keyframe silences exactly the next gop_size−2 deltas](keyframe-loss-kills-the-next-deltas.md) - libavcodec emits nothing for them, the last delta decodes with artifacts, the next keyframe restores clean decode; a lost delta never blocks decode — so the display hole is gop_size × frame period, deterministically.
+- [Oversubscription queues — it does not lose until the queue itself dies](oversubscription-queues-not-losses.md) - 16% over an emulated cap climbed one-way delay 340 → 3250 ms across twelve bins with zero loss in every one; the losses arrived only when the queue died, all at once.
 - [Best-effort reordering becomes reader-side loss, independent of history depth](reorder-becomes-reader-loss.md) - under pure delay jitter a Cyclone best-effort reader discards every overtaken sample (57% at 100 Hz under 50+-45 ms), delivery stays strictly monotonic, and depth 50 loses the same as depth 1.
 - [Jitter causes loss while bandwidth shortage builds latency](jitter-loss-bandwidth-latency.md) - 18 KB at 20 Hz separates jitter-induced loss from bandwidth-induced queueing.
 - [Lighter alternating messages can lose more than steady messages](lighter-message-loss.md) - a 1x18KB+1x0KB pattern lost under a tight emulated profile while the steady 18 KB comparison did not.

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev52}/docs/findings/delay-alone-no-loss.md

@@ -54,13 +54,20 @@ rosotacom benchmark capacity --project src/rosotacom/resources/examples/rosotaco
 Migrated summary: 400 expected, 400 delivered, 0 lost, max p95 latency
 306.927 ms.
 
+Field recurrence (2026-08-17 CCNG drive): the negative control held on a real
+cellular link — five jam episodes ran one-way delay up to 401–459 ms with zero
+losses, and the drive's soggy last minute held p90 ≈ 138 ms at 0.04 losses/s.
+The positive counterpart (where the delay actually comes from when a link is
+too slow rather than too far) is
+[oversubscription-queues-not-losses.md](oversubscription-queues-not-losses.md).
+
 Verification: manual: run the commands above from a source checkout with Docker
 and `tc` privileges; this is a no-loss negative-control finding rather than a
 must-fail boundary row, so it remains outside the RFC 0007 boundary gate.
 
 ## Status
 
-confirmed, 2026-06-29.
+confirmed, 2026-08-18.
 
 ## Publication notes
 

rosotacom_dev-2.5.dev52/docs/findings/fastdds-datagram-cap-drops-large-samples.md

@@ -0,0 +1,89 @@
+# A Fast DDS Datagram Cap Stops Every Message Larger Than It
+
+## Claim
+
+Setting `<maxMessageSize>` on a Fast DDS UDPv4 transport to a value below a
+sample's serialized size makes that sample **never arrive** through the OTA
+pipeline — not "arrive fragmented", not "arrive with loss". Messages that fit
+inside the cap keep flowing at full rate on the same link, which is what makes
+this dangerous: an 84 B heartbeat crossing at 10 Hz with 0.0% loss is a link
+that looks alive from every direction except the one that matters.
+
+The obvious reason to set it is symmetry with CycloneDDS, whose OTA profile
+carries this link at `FragmentSize`/`MaxMessageSize` 1200 B and works. The
+symmetry does not transfer: on ROS 2 Kilted with Fast DDS 3.2.4 the cap is the
+single knob that breaks the link, at 1200 B and at 8192 B, with a synchronous
+writer and with an asynchronous one.
+
+## Setup
+
+- Host pair / topology: one host, the packaged local smoke rig — two
+  communication containers on their own Docker network, split ROS domains
+  (peer 50, peer 51, OTA 52) with the stock `domain_bridge` between them. The
+  same failure was first seen across two machines over a VPN; one host
+  reproduces it, so nothing private is needed.
+- Session: a two-peer session with a `com_msgs/msg/SizedPayload` stream at
+  12000 B and 10 Hz on the loaded direction plus the echo heartbeat, OTA QoS
+  best_effort / depth 1 / lifespan 0.7 (the deployed shape).
+- rosotacom SHA: measured at `6e4b748`; re-runs record theirs.
+- Profile: none for the bisect — the failure needs no impairment, which is what
+  makes it attributable to the configuration rather than to the link.
+- Seed policy: deterministic (no netem, no random load), `n=1` per variant; the
+  effect is present/absent, not a distribution.
+
+## Evidence
+
+Evidence grade: single-host reproduction on packaged material, one variant per
+run, read from the receiving peer's own `status.txt`.
+
+Each knob of the tuned profile was added to the working `fastdds_unicast.xml`
+one at a time (2026-08-19):
+
+| added to fastdds_unicast.xml | 12 kB stream at the receiver |
+|---|---|
+| interface allowlist (127.0.0.1 + OTA address) | FLOWING 10.0 Hz, 0.0% loss |
+| loopback locators (default / metatraffic / initial peers) | FLOWING 10.0 Hz |
+| `maxInitialPeersRange` 10 | FLOWING 10.0 Hz |
+| discovery lease / announcement | FLOWING 10.0 Hz |
+| socket buffer sizes (1 MB / 4 MB) | FLOWING 10.0 Hz |
+| `non_blocking_send` | FLOWING 10.0 Hz |
+| `publishMode` ASYNCHRONOUS | FLOWING, degraded: 9.3 Hz at 285 ms |
+| **`maxMessageSize` 1200** | **IDLE — nothing delivered** |
+| **`maxMessageSize` 8192** | **IDLE — nothing delivered** |
+| **`maxMessageSize` 1200 + ASYNCHRONOUS** | **IDLE — nothing delivered** |
+
+The heartbeat crossed at 10 Hz with 0.0% loss in every one of those runs,
+including the three that delivered no payload at all.
+
+Two nearby observations, both of which cost time before the bisect narrowed it:
+
+- Two Fast DDS participants in two containers, with the capped profile and
+  nothing else running, DO exchange 12 kB and 38 kB samples (57–59 of 60). The
+  cap only breaks the full pipeline, where the sample crosses a domain bridge
+  and an OTA bridge, so an isolated transport probe will report the cap as
+  harmless.
+- Asynchronous publication does not rescue it, which rules out the documented
+  "a synchronous writer cannot fragment" explanation as the whole story.
+
+Verification: manual: run the packaged smoke for a session whose OTA side names
+`fastdds_tuned.xml`, then again for one that adds `<maxMessageSize>1200` to it,
+and read `logs/<receiver>/status/status.txt`; the payload stage is FLOWING in
+the first and IDLE in the second while the heartbeat is FLOWING in both. The
+shipped profiles are pinned against the cap by
+`tests/unit/test_ota_link_configs.py::test_only_cyclone_caps_the_datagram_and_fast_dds_must_not`.
+
+## Status
+
+confirmed, 2026-08-19.
+
+## Publication notes
+
+This is why "the OTA middleware is a configuration choice" needs measuring
+rather than reasoning: the two profiles that look like translations of each
+other are not, and the failure mode of the wrong translation is a link that
+passes a heartbeat check. Consequence for the transport comparison: Fast DDS
+lets IP fragment a large sample here where CycloneDDS sends RTPS fragments, so
+the two differ in what a lost packet costs and in what the tunnel has to
+reassemble — the measured OpenVPN barrier on the operator's path is around
+110 kB per message. Report Fast DDS numbers with that difference stated, not as
+a like-for-like transport swap.

rosotacom_dev-2.5.dev52/docs/findings/graph-derived-types-block-a-non-graph-transport.md

@@ -0,0 +1,79 @@
+# A Bridge That Learns Types From The Graph Cannot Use A Transport That Does Not Carry The Graph
+
+## Claim
+
+`PubSubPair` resolved a stage's message type from
+`get_topic_names_and_types()`, so it could only create its endpoints once
+somebody else had created the matching one. On a DDS OTA link that is invisible
+— the remote writer is in the same domain, so the graph knows it. Over the
+zenoh DDS bridge (`rmw.ota: zenoh_ros2dds`) it is a deadlock: the bridge
+creates a subscriber route for a topic only once a **local DDS reader** exists,
+and the reader was waiting for the bridge to route a publisher into the graph.
+Neither ever happens, the payload topic never arrives, and the only trace is one
+word in a log line — `Initialized 1 pair(s). Pending=1`.
+
+The heartbeat escapes it because the receiving peer's bridge sees the remote
+liveliness token for it early enough to create the route before the reader is
+needed; the wrapped payload topic does not. So the observable is again a link
+that carries the small topic and silently drops the big one.
+
+## Setup
+
+- Host pair / topology: two machines over a VPN, `rmw.ota: zenoh_ros2dds`
+  (local cyclone), split ROS domains, one 12 kB `SizedPayload` stream at 10 Hz
+  plus the echo heartbeat.
+- rosotacom SHA: observed at `6e4b748` (before the fix), fixed in `48d84dc`.
+- Profile: the drive-derived loss process; the effect is not a loss effect and
+  reproduced identically in both runs under it.
+- Seed policy: not applicable — present/absent, `n=2` runs, both failing the
+  same way.
+
+## Evidence
+
+Evidence grade: two live two-machine runs, read from the receiving peer's node
+logs and from the zenoh bridge's own routing log.
+
+Receiving peer, `bridge_in` (2026-08-19):
+
+```
+[PubSubPair] Created: sub='/ota/b/heartbeat_b' ...
+[PubSubPair] Created: sub='/ota/b/cam_like/ota_stamped' ...
+[PubSubPair] Initialized: sub='/ota/b/heartbeat_b' type='com_msgs/msg/EchoHeartbeat'
+[bridge_in] Initialized 1 pair(s). Pending=1
+```
+
+Receiving peer, `zenoh_bridge_ros2dds`, for the whole run:
+
+```
+routes_mgr: Route Subscriber (Zenoh:ota/b/heartbeat_b -> ROS:/ota/b/heartbeat_b) created
+```
+
+— and no such line for `ota/b/cam_like/ota_stamped`, while the sending peer had
+created its publisher route for exactly that key. Delivered payload messages in
+both runs: 0 of ~810 sent.
+
+The fix removes the dependency rather than the symptom: every session now writes
+`<peer>/topic_types.yaml` (`<stage topic>: <message type>` for every stage that
+peer takes part in, derived from the same pipeline description the status node
+uses), the bridge and relay nodes read it, and the graph is consulted only for
+what the file does not name.
+
+Verification: `python -m pytest -q tests/unit/test_pub_sub_pair_types.py` — a
+pair with a declared type creates its endpoints against an empty graph, a pair
+without one still reads the graph, and a pair with neither stays pending.
+`tests/unit/test_ota_link_configs.py::test_every_peer_gets_the_stage_types_its_bridges_need`
+pins that the file is generated for every peer of every session.
+
+## Status
+
+confirmed, 2026-08-19.
+
+## Publication notes
+
+Worth stating in any "the transport is a configuration choice" claim: a
+pipeline can depend on a transport property nobody wrote down — here, that the
+transport propagates the ROS graph and not only the data. DDS does, the zenoh
+DDS bridge does not, and the dependency only shows up as a topic that never
+arrives. The general lesson is that a contract known at generation time should
+be handed to the runtime rather than rediscovered from the runtime, which is
+also what makes bring-up order-independent.

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev52}/docs/findings/head-of-line-irregularity.md

@@ -45,13 +45,23 @@ The mechanism is visible only per message: the stream carrying alternating
 heavy/light samples can lose more than the steady stream despite lower mean
 payload bandwidth.
 
+Field recurrence (2026-08-17 CCNG drive): the DELAY half of the mechanism
+recurred and was measured per message — a small message with more than 30 KB
+in flight ahead of it paid +17.4 ms p50 / +67 ms p90, a small message sent
+within 40 ms after a video keyframe paid p50 42.1 vs 28.4 ms, and 18.4% of
+cross-topic pairs sent within 2 ms flipped order after a >20 KB first message.
+The LOSS half (delayed-then-overwritten) did not trigger there: it needs a
+queue tight against the offered load, and that link had ~14 Mbit/s
+serialization headroom at 2.5 Mbit/s offered. Both halves stay reproducible
+with the tight profile above.
+
 Verification: manual: run the two commands above from a source checkout with
 Docker and `tc` privileges; automation waits for the RFC 0007 regression matrix
 row for patterned loads because the public row registry does not exist yet.
 
 ## Status
 
-confirmed, 2026-07-02.
+confirmed, 2026-08-18.
 
 ## Publication notes
 

rosotacom_dev-2.5.dev52/docs/findings/keyframe-loss-kills-the-next-deltas.md

@@ -0,0 +1,69 @@
+# A Lost Keyframe Silences Exactly The Next gop_size−2 Deltas
+
+## Claim
+
+In a best-effort libx264 stream (ABR, `tune: zerolatency`, fixed GOP), losing
+a KEYFRAME makes exactly the next `gop_size − 2` arrived deltas undecodable —
+libavcodec emits **no frame at all** for them — the last delta before the next
+keyframe decodes with concealment artifacts, and the next keyframe restores
+clean decode. Losing a DELTA never blocks decode: every following access unit
+still produces a frame (artifacts until the next keyframe, never a gap). The
+behaviour is deterministic — not timing, not decoder state noise — so the
+display hole a keyframe loss tears is `gop_size × frame_period`, which is what
+makes the GOP length the knob that decides whether one lost packet crosses an
+operator-visible threshold.
+
+## Setup
+
+- Host pair / topology: single host, single process, below rosotacom at the
+  codec layer — encode with PyAV/libx264 exactly as the ffmpeg transport does,
+  delete single access units explicitly, feed the survivors to a fresh
+  libavcodec h264 decoder (what a best-effort receiver does after a loss).
+- rosotacom SHA: measured at `73fa1de`; re-runs record theirs.
+- Profile: none — no network is involved; the deletion IS the loss, which is
+  what makes the rule attributable to the codec rather than the transport.
+- Seed policy: deterministic synthetic motion clip, `n=1` (the assertions are
+  exact-set comparisons, not statistics).
+
+## Evidence
+
+Evidence grade: scripted single-host reproduction, committed next to this file.
+
+```bash
+python3 docs/findings/repro/decode_chain_repro.py --gop 5   # needs: pip install av numpy
+```
+
+2026-08-18 runs (synthetic clip, libx264 1 Mbit/s ABR, zerolatency, 10 Hz):
+
+| gop | deleted | silent AUs after it | first output after the gap |
+|---|---|---|---|
+| 5 | keyframe | exactly 3 (= gop−2) | last delta of the GOP, with artifacts |
+| 3 | keyframe | exactly 1 (= gop−2) | last delta of the GOP, with artifacts |
+| 2 | keyframe | none (= gop−2 = 0) | the following delta, with artifacts |
+| any | one delta | none | every AU keeps decoding, artifacts to next KF |
+
+The same rule was first measured on real field material (2026-08-17 CCNG
+drive, camera FFMPEGPacket stream at gop 5: every lost keyframe silenced the
+next 3 arrived deltas, ~400 ms holes at 10 Hz; a `refs=1` encode does NOT
+shorten the chain — tested and rejected). The synthetic clip reproduces it
+exactly, so the public repro carries the claim without any private bag.
+
+Verification: manual: run the command above (per `--gop 5`, `--gop 3`,
+`--gop 2`); the script asserts the exact silent set and exits non-zero on any
+deviation.
+
+## Status
+
+confirmed, 2026-08-18.
+
+## Publication notes
+
+This is the quantitative bridge from "loss percent" to "operator-visible
+hole": camera loss numbers mean nothing next to costmap loss numbers until
+multiplied by the GOP geometry. It drove the 2026-08-17 field configuration
+change gop 5 → 3 (hole ≤ 300 ms at equal measured bitrate). For papers: plot
+hole duration vs GOP at fixed measured bitrate, and state that intra-refresh
+trades the hole for a several-frame artifact window instead (no silent AUs,
+lower clean-frame share). See also
+[docs/ffmpeg-keyframes.md](../ffmpeg-keyframes.md) for reading keyframe flags
+out of recorded transit records.

rosotacom_dev-2.5.dev52/docs/findings/oversubscription-queues-not-losses.md

@@ -0,0 +1,81 @@
+# Oversubscription Queues — It Does Not Lose Until The Queue Itself Dies
+
+## Claim
+
+Offered load slightly above a rate cap turns entirely into queueing delay, not
+loss: one-way delay climbs by hundreds of milliseconds per second of
+oversubscription — into the seconds — while per-bin loss stays exactly zero.
+Loss appears only when the queue itself is destroyed (in emulation: a
+shape-changing timeline step; in the field: a bearer/modem reset), and then it
+is the whole backlog at once. Consequences: loss statistics alone certify a
+badly broken link as healthy, delay-based failure precedes loss-based failure,
+and a recovering link replays a burst of stale messages unless a lifespan
+bounds them.
+
+## Setup
+
+- Host pair / topology: rosotacom local benchmark mode using the packaged
+  example project, synthetic `bench_1_1_capacity` session, one
+  `a->b:/bench_capacity` stream, uplink shaping only (tc inside the peer
+  containers' own netns — no host privileges).
+- rosotacom SHA: measured at `73fa1de`; public re-runs record the current
+  checkout SHA in `result.json.context`.
+- Profile: `field-20260817-case3-delay-jam` in
+  [`src/rosotacom/resources/examples/profiles.yaml`](../../src/rosotacom/resources/examples/profiles.yaml)
+  — 10 s clean, 15 s `rate: 900kbit` (the crunch), 20 s clean; fitted to the
+  2026-08-17 CCNG drive's case3 window (in-flight queueing to 1.4 s at almost
+  no loss).
+- Seed policy: no stochastic netem elements (rate/delay only), `n=1`.
+
+## Evidence
+
+Evidence grade: packaged-profile benchmark run plus per-bin timeline.
+
+```bash
+rosotacom benchmark probe --project src/rosotacom/resources/examples/rosotacom.yaml --profile field-20260817-case3-delay-jam --size-pattern 1x13KB --rate-hz 10 --duration 45 --repeats 1 --no-plot
+```
+
+2026-08-18 run (offered 1.04 Mbit/s against the 15 s 900 kbit crunch — 16%
+oversubscription): twelve consecutive 1 s bins during the crunch show p95
+one-way latency climbing 340 → 601 → 880 → 1117 → 1420 → 1684 → 1944 → 2208 →
+2469 → 2704 → 3009 → **3250 ms with `lost = 0` in every single bin** — the
+queue (netem child, default limit 1000 packets) absorbs the entire excess.
+All 27 lost messages (5.9% of the run) sit in the three bins at the
+crunch→clean boundary, where the timeline step replaces a tbf+netem tree with
+a netem-only tree and the queued ~2.7 s backlog dies with the old queue.
+
+Two readings beyond the headline: (a) a loss-based health verdict over the
+crunch window reads "0% loss" while the operator's picture is three seconds
+old and aging — delay-based failure criteria fire long before loss-based
+ones; (b) the backlog-drop at the boundary is the emulated twin of a bearer
+reset, and it is also the caveat's edge: `tc qdisc replace` steps are seamless
+only between same-shape trees — a step that adds or removes `rate` swaps the
+tree and drops its queue.
+
+Field recurrence (2026-08-17 CCNG drive): five jam episodes with one-way
+delay 401–459 ms at zero losses; the case3 window queued to 1.4 s in-flight
+with almost no loss; and the drive's last minute ran p90 ≈ 138 ms delay at
+0.04 losses/s — the same regime at lower intensity. A 2026-08-18 container
+replay of a 9% oversubscription reached 3.66 s delay at exactly 0% loss.
+
+Verification: manual: run the command above from a source checkout with
+Docker; read the per-bin `time-bins.jsonl` next to `result.json` — the crunch
+bins must show monotonically climbing latency at zero loss, with the losses
+(if any) confined to the boundary bins.
+
+## Status
+
+confirmed, 2026-08-18.
+
+## Publication notes
+
+This is the mechanism behind
+[delay-alone-no-loss.md](delay-alone-no-loss.md)'s negative control seen from
+the other side: delay alone does not lose, and oversubscription alone does not
+lose either — it defers. Pair with
+[reorder-becomes-reader-loss.md](reorder-becomes-reader-loss.md) for the two
+ways "the network was fine, loss says so" misleads. For papers: plot per-bin
+delay and loss on one timeline; the visual is a delay ramp under a flat zero
+loss line, ending in a loss spike exactly when the queue dies. Lifespan QoS
+(and its enforcement gap — writer-side only in CycloneDDS) decides what the
+recovery burst does to the application.

rosotacom_dev-2.5.dev52/docs/findings/repro/decode_chain_repro.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""Reproduce the keyframe decode-chain rule of a best-effort libx264 stream.
+
+Encodes a deterministic synthetic motion clip the way the ffmpeg transport
+does (libx264, ABR, ``tune: zerolatency``, fixed GOP), then deletes single
+access units and feeds the survivors to a fresh libavcodec h264 decoder --
+which is exactly what a best-effort receiver does after a network loss.
+
+Measured rule (docs/findings/keyframe-loss-kills-the-next-deltas.md):
+
+* delete a KEYFRAME  -> the next ``gop_size - 2`` arrived deltas produce NO
+  output at all, the last delta before the next keyframe decodes with
+  concealment artifacts, the next keyframe restores clean decode;
+* delete a DELTA     -> every following access unit still decodes (artifacts
+  until the next keyframe, never a gap).
+
+Usage (needs ``pip install av numpy``; no network, no privileges)::
+
+    python3 docs/findings/repro/decode_chain_repro.py [--gop 5] [--frames 40]
+
+Prints the per-AU decode table for both deletions and exits non-zero if the
+rule above does not hold.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from fractions import Fraction
+
+import av
+import numpy as np
+
+
+def synthetic_frames(count: int, width: int = 320, height: int = 240):
+    """Deterministic motion: drifting background plus a moving block."""
+    for index in range(count):
+        image = np.full((height, width, 3), (index * 3) % 200, dtype=np.uint8)
+        x = (index * 7) % (width - 40)
+        image[60:180, x : x + 40] = 255
+        frame = av.VideoFrame.from_ndarray(image, format="rgb24")
+        yield frame.reformat(format="yuv420p")
+
+
+def encode(gop: int, count: int) -> list[tuple[bytes, bool]]:
+    codec = av.CodecContext.create("libx264", "w")
+    codec.width, codec.height = 320, 240
+    codec.pix_fmt = "yuv420p"
+    codec.framerate = Fraction(10, 1)
+    codec.bit_rate = 1_000_000
+    codec.options = {"tune": "zerolatency", "g": str(gop)}
+    packets: list[tuple[bytes, bool]] = []
+    for frame in synthetic_frames(count):
+        for packet in codec.encode(frame):
+            packets.append((bytes(packet), bool(packet.is_keyframe)))
+    for packet in codec.encode(None):
+        packets.append((bytes(packet), bool(packet.is_keyframe)))
+    return packets
+
+
+def decode_survivors(packets: list[tuple[bytes, bool]], deleted: set[int]) -> list[tuple[int, int]]:
+    """Feed every surviving AU to a fresh decoder; return (au_index, frames_out)."""
+    decoder = av.CodecContext.create("h264", "r")
+    out: list[tuple[int, int]] = []
+    for index, (payload, _kf) in enumerate(packets):
+        if index in deleted:
+            continue
+        try:
+            frames = decoder.decode(av.Packet(payload))
+        except av.error.InvalidDataError:
+            frames = []
+        out.append((index, len(frames)))
+    return out
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--gop", type=int, default=5)
+    parser.add_argument("--frames", type=int, default=40)
+    args = parser.parse_args()
+
+    packets = encode(args.gop, args.frames)
+    kf_indices = [i for i, (_p, kf) in enumerate(packets) if kf]
+    if len(kf_indices) < 3:
+        print("clip too short to delete a mid-stream keyframe", file=sys.stderr)
+        return 2
+
+    baseline = decode_survivors(packets, set())
+    silent_baseline = [i for i, n in baseline if n == 0]
+
+    failures: list[str] = []
+
+    # --- delete the second keyframe -------------------------------------
+    kf = kf_indices[1]
+    next_kf = kf_indices[2]
+    rows = decode_survivors(packets, {kf})
+    silent = [i for i, n in rows if n == 0 and i not in silent_baseline]
+    deltas_in_gap = [i for i in range(kf + 1, next_kf)]
+    expected_silent = deltas_in_gap[: max(0, args.gop - 2)]
+    print(f"deleted keyframe AU {kf} (gop {args.gop}, next keyframe {next_kf})")
+    print(f"  silent AUs: {silent}")
+    print(f"  expected  : {expected_silent} (the next gop_size-2 deltas)")
+    if silent != expected_silent:
+        failures.append(f"keyframe deletion: silent AUs {silent} != expected {expected_silent}")
+    resumed = [i for i, n in rows if i >= next_kf and n == 0]
+    if resumed:
+        failures.append(f"decode did not resume at the next keyframe: silent {resumed}")
+
+    # --- delete one delta ------------------------------------------------
+    delta = kf_indices[1] + 1
+    rows = decode_survivors(packets, {delta})
+    silent = [i for i, n in rows if n == 0 and i not in silent_baseline]
+    print(f"deleted delta AU {delta}")
+    print(f"  silent AUs: {silent} (must be empty -- artifacts, never a gap)")
+    if silent:
+        failures.append(f"delta deletion silenced AUs {silent}; expected none")
+
+    if failures:
+        print("FINDING VIOLATED:", *failures, sep="\n  ", file=sys.stderr)
+        return 1
+    print("ok: keyframe loss silences exactly the next gop_size-2 deltas; delta loss silences nothing")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

rosotacom_dev-2.5.dev52/docs/findings/sender-rows-make-a-dead-link-look-loss-free.md

@@ -0,0 +1,59 @@
+# Sender-Side Transit Rows Made A Dead Link Report Zero Loss
+
+## Claim
+
+`summarize_transit_records` counted `delivered = rows − lost`. Sender-side
+transit rows carry the status `sent` and are written whether or not anything
+arrives, so a topic that the receiving peer never saw at all summarised as
+`delivered == expected`, `loss_pct 0.0` — with the sender's own publication
+cadence reported as the arrival spacing. The most dangerous shape a wrong number
+can take: a link that delivered nothing scored better than one that delivered
+96% of its messages.
+
+Delivery is a receiver fact. Rows still `sent` after the join are `unconfirmed`;
+`loss_pct` is `None` when no receiver accounted for the topic at all; and an
+oracle reads that `None` as the worst case rather than as zero.
+
+## Setup
+
+- Host pair / topology: not a transport property — the summary is a pure
+  function of collected records, so the reproduction is a unit test over
+  synthetic rows. It was found on a two-machine run whose OTA transport
+  delivered nothing.
+- rosotacom SHA: present up to `6e4b748`, fixed in `730ddf5`.
+- Profile: none — the defect is in the accounting, not in the link.
+- Seed policy: not applicable, deterministic.
+
+## Evidence
+
+Evidence grade: unit-level, deterministic, plus the live run that exposed it.
+
+Live (2026-08-19, `rmw.ota: zenoh_ros2dds` over two machines): the sending peer
+wrote 816 `outbound`/`sent` rows, the receiving peer wrote 5 rows and no
+delivery at all. The run's `result.json` reported `delivered 807, expected 807,
+loss_pct 0.0` for that topic, with inter-arrival p05/p95 of 102/104 ms — the
+publisher's own timer, not an arrival distribution. The receiver's `status.txt`
+for the same run said `ota_recv ABSENT/GRAPH pub=0` and `com_in IDLE`.
+
+The window mattered as much as the number: this only became reachable when
+sender-side transit rows landed (issue #294). Before that, a topic with no
+receiver rows had no rows at all and simply did not appear.
+
+Verification: `python -m pytest -q tests/unit/test_transit.py -k sender_rows`
+— sender rows alone report `delivered 0`, `unconfirmed 3` and `loss_pct None`,
+and a mixed set counts only the receiver's verdicts.
+`tests/unit/test_benchmark.py::test_an_unknown_loss_figure_is_read_as_the_worst_case`
+pins that an oracle does not pass on the `None`.
+
+## Status
+
+confirmed, 2026-08-19.
+
+## Publication notes
+
+A measurement pipeline that gains a second observer gains a way to be
+confidently wrong: the new rows were real, the join was correct, and the
+aggregate stopped meaning what its name said. Worth citing whenever a loss
+figure is quoted from an automated run — state which side observed it. The
+general rule the fix encodes: an aggregate over rows from two observers must say
+which observer each row came from before it counts anything.