rosotacom-dev 2.5.dev50__tar.gz → 2.5.dev51__tar.gz

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev51}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev50
+Version: 2.5.dev51
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev51}/docs/findings/README.md

@@ -28,6 +28,8 @@ living verification hook.
 
 ## Findings
 
+- [A lost keyframe silences exactly the next gop_size−2 deltas](keyframe-loss-kills-the-next-deltas.md) - libavcodec emits nothing for them, the last delta decodes with artifacts, the next keyframe restores clean decode; a lost delta never blocks decode — so the display hole is gop_size × frame period, deterministically.
+- [Oversubscription queues — it does not lose until the queue itself dies](oversubscription-queues-not-losses.md) - 16% over an emulated cap climbed one-way delay 340 → 3250 ms across twelve bins with zero loss in every one; the losses arrived only when the queue died, all at once.
 - [Best-effort reordering becomes reader-side loss, independent of history depth](reorder-becomes-reader-loss.md) - under pure delay jitter a Cyclone best-effort reader discards every overtaken sample (57% at 100 Hz under 50+-45 ms), delivery stays strictly monotonic, and depth 50 loses the same as depth 1.
 - [Jitter causes loss while bandwidth shortage builds latency](jitter-loss-bandwidth-latency.md) - 18 KB at 20 Hz separates jitter-induced loss from bandwidth-induced queueing.
 - [Lighter alternating messages can lose more than steady messages](lighter-message-loss.md) - a 1x18KB+1x0KB pattern lost under a tight emulated profile while the steady 18 KB comparison did not.

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev51}/docs/findings/delay-alone-no-loss.md

@@ -54,13 +54,20 @@ rosotacom benchmark capacity --project src/rosotacom/resources/examples/rosotaco
 Migrated summary: 400 expected, 400 delivered, 0 lost, max p95 latency
 306.927 ms.
 
+Field recurrence (2026-08-17 CCNG drive): the negative control held on a real
+cellular link — five jam episodes ran one-way delay up to 401–459 ms with zero
+losses, and the drive's soggy last minute held p90 ≈ 138 ms at 0.04 losses/s.
+The positive counterpart (where the delay actually comes from when a link is
+too slow rather than too far) is
+[oversubscription-queues-not-losses.md](oversubscription-queues-not-losses.md).
+
 Verification: manual: run the commands above from a source checkout with Docker
 and `tc` privileges; this is a no-loss negative-control finding rather than a
 must-fail boundary row, so it remains outside the RFC 0007 boundary gate.
 
 ## Status
 
-confirmed, 2026-06-29.
+confirmed, 2026-08-18.
 
 ## Publication notes
 

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev51}/docs/findings/head-of-line-irregularity.md

@@ -45,13 +45,23 @@ The mechanism is visible only per message: the stream carrying alternating
 heavy/light samples can lose more than the steady stream despite lower mean
 payload bandwidth.
 
+Field recurrence (2026-08-17 CCNG drive): the DELAY half of the mechanism
+recurred and was measured per message — a small message with more than 30 KB
+in flight ahead of it paid +17.4 ms p50 / +67 ms p90, a small message sent
+within 40 ms after a video keyframe paid p50 42.1 vs 28.4 ms, and 18.4% of
+cross-topic pairs sent within 2 ms flipped order after a >20 KB first message.
+The LOSS half (delayed-then-overwritten) did not trigger there: it needs a
+queue tight against the offered load, and that link had ~14 Mbit/s
+serialization headroom at 2.5 Mbit/s offered. Both halves stay reproducible
+with the tight profile above.
+
 Verification: manual: run the two commands above from a source checkout with
 Docker and `tc` privileges; automation waits for the RFC 0007 regression matrix
 row for patterned loads because the public row registry does not exist yet.
 
 ## Status
 
-confirmed, 2026-07-02.
+confirmed, 2026-08-18.
 
 ## Publication notes
 

rosotacom_dev-2.5.dev51/docs/findings/keyframe-loss-kills-the-next-deltas.md

@@ -0,0 +1,69 @@
+# A Lost Keyframe Silences Exactly The Next gop_size−2 Deltas
+
+## Claim
+
+In a best-effort libx264 stream (ABR, `tune: zerolatency`, fixed GOP), losing
+a KEYFRAME makes exactly the next `gop_size − 2` arrived deltas undecodable —
+libavcodec emits **no frame at all** for them — the last delta before the next
+keyframe decodes with concealment artifacts, and the next keyframe restores
+clean decode. Losing a DELTA never blocks decode: every following access unit
+still produces a frame (artifacts until the next keyframe, never a gap). The
+behaviour is deterministic — not timing, not decoder state noise — so the
+display hole a keyframe loss tears is `gop_size × frame_period`, which is what
+makes the GOP length the knob that decides whether one lost packet crosses an
+operator-visible threshold.
+
+## Setup
+
+- Host pair / topology: single host, single process, below rosotacom at the
+  codec layer — encode with PyAV/libx264 exactly as the ffmpeg transport does,
+  delete single access units explicitly, feed the survivors to a fresh
+  libavcodec h264 decoder (what a best-effort receiver does after a loss).
+- rosotacom SHA: measured at `73fa1de`; re-runs record theirs.
+- Profile: none — no network is involved; the deletion IS the loss, which is
+  what makes the rule attributable to the codec rather than the transport.
+- Seed policy: deterministic synthetic motion clip, `n=1` (the assertions are
+  exact-set comparisons, not statistics).
+
+## Evidence
+
+Evidence grade: scripted single-host reproduction, committed next to this file.
+
+```bash
+python3 docs/findings/repro/decode_chain_repro.py --gop 5   # needs: pip install av numpy
+```
+
+2026-08-18 runs (synthetic clip, libx264 1 Mbit/s ABR, zerolatency, 10 Hz):
+
+| gop | deleted | silent AUs after it | first output after the gap |
+|---|---|---|---|
+| 5 | keyframe | exactly 3 (= gop−2) | last delta of the GOP, with artifacts |
+| 3 | keyframe | exactly 1 (= gop−2) | last delta of the GOP, with artifacts |
+| 2 | keyframe | none (= gop−2 = 0) | the following delta, with artifacts |
+| any | one delta | none | every AU keeps decoding, artifacts to next KF |
+
+The same rule was first measured on real field material (2026-08-17 CCNG
+drive, camera FFMPEGPacket stream at gop 5: every lost keyframe silenced the
+next 3 arrived deltas, ~400 ms holes at 10 Hz; a `refs=1` encode does NOT
+shorten the chain — tested and rejected). The synthetic clip reproduces it
+exactly, so the public repro carries the claim without any private bag.
+
+Verification: manual: run the command above (per `--gop 5`, `--gop 3`,
+`--gop 2`); the script asserts the exact silent set and exits non-zero on any
+deviation.
+
+## Status
+
+confirmed, 2026-08-18.
+
+## Publication notes
+
+This is the quantitative bridge from "loss percent" to "operator-visible
+hole": camera loss numbers mean nothing next to costmap loss numbers until
+multiplied by the GOP geometry. It drove the 2026-08-17 field configuration
+change gop 5 → 3 (hole ≤ 300 ms at equal measured bitrate). For papers: plot
+hole duration vs GOP at fixed measured bitrate, and state that intra-refresh
+trades the hole for a several-frame artifact window instead (no silent AUs,
+lower clean-frame share). See also
+[docs/ffmpeg-keyframes.md](../ffmpeg-keyframes.md) for reading keyframe flags
+out of recorded transit records.

rosotacom_dev-2.5.dev51/docs/findings/oversubscription-queues-not-losses.md

@@ -0,0 +1,81 @@
+# Oversubscription Queues — It Does Not Lose Until The Queue Itself Dies
+
+## Claim
+
+Offered load slightly above a rate cap turns entirely into queueing delay, not
+loss: one-way delay climbs by hundreds of milliseconds per second of
+oversubscription — into the seconds — while per-bin loss stays exactly zero.
+Loss appears only when the queue itself is destroyed (in emulation: a
+shape-changing timeline step; in the field: a bearer/modem reset), and then it
+is the whole backlog at once. Consequences: loss statistics alone certify a
+badly broken link as healthy, delay-based failure precedes loss-based failure,
+and a recovering link replays a burst of stale messages unless a lifespan
+bounds them.
+
+## Setup
+
+- Host pair / topology: rosotacom local benchmark mode using the packaged
+  example project, synthetic `bench_1_1_capacity` session, one
+  `a->b:/bench_capacity` stream, uplink shaping only (tc inside the peer
+  containers' own netns — no host privileges).
+- rosotacom SHA: measured at `73fa1de`; public re-runs record the current
+  checkout SHA in `result.json.context`.
+- Profile: `field-20260817-case3-delay-jam` in
+  [`src/rosotacom/resources/examples/profiles.yaml`](../../src/rosotacom/resources/examples/profiles.yaml)
+  — 10 s clean, 15 s `rate: 900kbit` (the crunch), 20 s clean; fitted to the
+  2026-08-17 CCNG drive's case3 window (in-flight queueing to 1.4 s at almost
+  no loss).
+- Seed policy: no stochastic netem elements (rate/delay only), `n=1`.
+
+## Evidence
+
+Evidence grade: packaged-profile benchmark run plus per-bin timeline.
+
+```bash
+rosotacom benchmark probe --project src/rosotacom/resources/examples/rosotacom.yaml --profile field-20260817-case3-delay-jam --size-pattern 1x13KB --rate-hz 10 --duration 45 --repeats 1 --no-plot
+```
+
+2026-08-18 run (offered 1.04 Mbit/s against the 15 s 900 kbit crunch — 16%
+oversubscription): twelve consecutive 1 s bins during the crunch show p95
+one-way latency climbing 340 → 601 → 880 → 1117 → 1420 → 1684 → 1944 → 2208 →
+2469 → 2704 → 3009 → **3250 ms with `lost = 0` in every single bin** — the
+queue (netem child, default limit 1000 packets) absorbs the entire excess.
+All 27 lost messages (5.9% of the run) sit in the three bins at the
+crunch→clean boundary, where the timeline step replaces a tbf+netem tree with
+a netem-only tree and the queued ~2.7 s backlog dies with the old queue.
+
+Two readings beyond the headline: (a) a loss-based health verdict over the
+crunch window reads "0% loss" while the operator's picture is three seconds
+old and aging — delay-based failure criteria fire long before loss-based
+ones; (b) the backlog-drop at the boundary is the emulated twin of a bearer
+reset, and it is also the caveat's edge: `tc qdisc replace` steps are seamless
+only between same-shape trees — a step that adds or removes `rate` swaps the
+tree and drops its queue.
+
+Field recurrence (2026-08-17 CCNG drive): five jam episodes with one-way
+delay 401–459 ms at zero losses; the case3 window queued to 1.4 s in-flight
+with almost no loss; and the drive's last minute ran p90 ≈ 138 ms delay at
+0.04 losses/s — the same regime at lower intensity. A 2026-08-18 container
+replay of a 9% oversubscription reached 3.66 s delay at exactly 0% loss.
+
+Verification: manual: run the command above from a source checkout with
+Docker; read the per-bin `time-bins.jsonl` next to `result.json` — the crunch
+bins must show monotonically climbing latency at zero loss, with the losses
+(if any) confined to the boundary bins.
+
+## Status
+
+confirmed, 2026-08-18.
+
+## Publication notes
+
+This is the mechanism behind
+[delay-alone-no-loss.md](delay-alone-no-loss.md)'s negative control seen from
+the other side: delay alone does not lose, and oversubscription alone does not
+lose either — it defers. Pair with
+[reorder-becomes-reader-loss.md](reorder-becomes-reader-loss.md) for the two
+ways "the network was fine, loss says so" misleads. For papers: plot per-bin
+delay and loss on one timeline; the visual is a delay ramp under a flat zero
+loss line, ending in a loss spike exactly when the queue dies. Lifespan QoS
+(and its enforcement gap — writer-side only in CycloneDDS) decides what the
+recovery burst does to the application.

rosotacom_dev-2.5.dev51/docs/findings/repro/decode_chain_repro.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""Reproduce the keyframe decode-chain rule of a best-effort libx264 stream.
+
+Encodes a deterministic synthetic motion clip the way the ffmpeg transport
+does (libx264, ABR, ``tune: zerolatency``, fixed GOP), then deletes single
+access units and feeds the survivors to a fresh libavcodec h264 decoder --
+which is exactly what a best-effort receiver does after a network loss.
+
+Measured rule (docs/findings/keyframe-loss-kills-the-next-deltas.md):
+
+* delete a KEYFRAME  -> the next ``gop_size - 2`` arrived deltas produce NO
+  output at all, the last delta before the next keyframe decodes with
+  concealment artifacts, the next keyframe restores clean decode;
+* delete a DELTA     -> every following access unit still decodes (artifacts
+  until the next keyframe, never a gap).
+
+Usage (needs ``pip install av numpy``; no network, no privileges)::
+
+    python3 docs/findings/repro/decode_chain_repro.py [--gop 5] [--frames 40]
+
+Prints the per-AU decode table for both deletions and exits non-zero if the
+rule above does not hold.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from fractions import Fraction
+
+import av
+import numpy as np
+
+
+def synthetic_frames(count: int, width: int = 320, height: int = 240):
+    """Deterministic motion: drifting background plus a moving block."""
+    for index in range(count):
+        image = np.full((height, width, 3), (index * 3) % 200, dtype=np.uint8)
+        x = (index * 7) % (width - 40)
+        image[60:180, x : x + 40] = 255
+        frame = av.VideoFrame.from_ndarray(image, format="rgb24")
+        yield frame.reformat(format="yuv420p")
+
+
+def encode(gop: int, count: int) -> list[tuple[bytes, bool]]:
+    codec = av.CodecContext.create("libx264", "w")
+    codec.width, codec.height = 320, 240
+    codec.pix_fmt = "yuv420p"
+    codec.framerate = Fraction(10, 1)
+    codec.bit_rate = 1_000_000
+    codec.options = {"tune": "zerolatency", "g": str(gop)}
+    packets: list[tuple[bytes, bool]] = []
+    for frame in synthetic_frames(count):
+        for packet in codec.encode(frame):
+            packets.append((bytes(packet), bool(packet.is_keyframe)))
+    for packet in codec.encode(None):
+        packets.append((bytes(packet), bool(packet.is_keyframe)))
+    return packets
+
+
+def decode_survivors(packets: list[tuple[bytes, bool]], deleted: set[int]) -> list[tuple[int, int]]:
+    """Feed every surviving AU to a fresh decoder; return (au_index, frames_out)."""
+    decoder = av.CodecContext.create("h264", "r")
+    out: list[tuple[int, int]] = []
+    for index, (payload, _kf) in enumerate(packets):
+        if index in deleted:
+            continue
+        try:
+            frames = decoder.decode(av.Packet(payload))
+        except av.error.InvalidDataError:
+            frames = []
+        out.append((index, len(frames)))
+    return out
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--gop", type=int, default=5)
+    parser.add_argument("--frames", type=int, default=40)
+    args = parser.parse_args()
+
+    packets = encode(args.gop, args.frames)
+    kf_indices = [i for i, (_p, kf) in enumerate(packets) if kf]
+    if len(kf_indices) < 3:
+        print("clip too short to delete a mid-stream keyframe", file=sys.stderr)
+        return 2
+
+    baseline = decode_survivors(packets, set())
+    silent_baseline = [i for i, n in baseline if n == 0]
+
+    failures: list[str] = []
+
+    # --- delete the second keyframe -------------------------------------
+    kf = kf_indices[1]
+    next_kf = kf_indices[2]
+    rows = decode_survivors(packets, {kf})
+    silent = [i for i, n in rows if n == 0 and i not in silent_baseline]
+    deltas_in_gap = [i for i in range(kf + 1, next_kf)]
+    expected_silent = deltas_in_gap[: max(0, args.gop - 2)]
+    print(f"deleted keyframe AU {kf} (gop {args.gop}, next keyframe {next_kf})")
+    print(f"  silent AUs: {silent}")
+    print(f"  expected  : {expected_silent} (the next gop_size-2 deltas)")
+    if silent != expected_silent:
+        failures.append(f"keyframe deletion: silent AUs {silent} != expected {expected_silent}")
+    resumed = [i for i, n in rows if i >= next_kf and n == 0]
+    if resumed:
+        failures.append(f"decode did not resume at the next keyframe: silent {resumed}")
+
+    # --- delete one delta ------------------------------------------------
+    delta = kf_indices[1] + 1
+    rows = decode_survivors(packets, {delta})
+    silent = [i for i, n in rows if n == 0 and i not in silent_baseline]
+    print(f"deleted delta AU {delta}")
+    print(f"  silent AUs: {silent} (must be empty -- artifacts, never a gap)")
+    if silent:
+        failures.append(f"delta deletion silenced AUs {silent}; expected none")
+
+    if failures:
+        print("FINDING VIOLATED:", *failures, sep="\n  ", file=sys.stderr)
+        return 1
+    print("ok: keyframe loss silences exactly the next gop_size-2 deltas; delta loss silences nothing")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev51}/src/rosotacom_dev.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev50
+Version: 2.5.dev51
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev51}/src/rosotacom_dev.egg-info/SOURCES.txt

@@ -65,9 +65,12 @@ docs/findings/cyclone-spdp-discovery-bursts.md
 docs/findings/delay-alone-no-loss.md
 docs/findings/head-of-line-irregularity.md
 docs/findings/jitter-loss-bandwidth-latency.md
+docs/findings/keyframe-loss-kills-the-next-deltas.md
 docs/findings/lighter-message-loss.md
+docs/findings/oversubscription-queues-not-losses.md
 docs/findings/reorder-becomes-reader-loss.md
 docs/findings/zero-loss-boundaries-18kb-20hz.md
+docs/findings/repro/decode_chain_repro.py
 docs/findings/repro/reorder_drop_repro.py
 docs/playbooks/README.md
 docs/release-notes/TEMPLATE.md

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev51}/src/rosotacom_dev.egg-info/scm_file_list.json

@@ -50,8 +50,11 @@
     "docs/findings/delay-alone-no-loss.md",
     "docs/findings/head-of-line-irregularity.md",
     "docs/findings/jitter-loss-bandwidth-latency.md",
+    "docs/findings/keyframe-loss-kills-the-next-deltas.md",
     "docs/findings/lighter-message-loss.md",
+    "docs/findings/oversubscription-queues-not-losses.md",
     "docs/findings/reorder-becomes-reader-loss.md",
+    "docs/findings/repro/decode_chain_repro.py",
     "docs/findings/repro/reorder_drop_repro.py",
     "docs/findings/zero-loss-boundaries-18kb-20hz.md",
     "docs/forensics-report.md",

{rosotacom_dev-2.5.dev50 → rosotacom_dev-2.5.dev51}/src/rosotacom_dev.egg-info/scm_version.json

@@ -1,7 +1,7 @@
 {
   "tag": "2.4",
-  "distance": 50,
-  "node": "g88dd222e7957d3d9a25e6c03ae02cf474f4f8ddf",
+  "distance": 51,
+  "node": "gd9801cccdf6d474aa29dfa1804bcae322e4fe3cf",
   "dirty": false,
   "branch": "HEAD",
   "node_date": "2026-08-18"