rosotacom-dev 2.5.dev73__tar.gz → 2.5.dev74__tar.gz

{rosotacom_dev-2.5.dev73 → rosotacom_dev-2.5.dev74}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev73
+Version: 2.5.dev74
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause

{rosotacom_dev-2.5.dev73 → rosotacom_dev-2.5.dev74}/docs/findings/README.md

@@ -28,6 +28,7 @@ living verification hook.
 
 ## Findings
 
+- [A fitted delay table never reached the shaper](a-fitted-delay-table-never-reached-the-shaper.md) - every profile carrying a custom netem distribution failed at arming: the table was installed as `/usr/lib/tc/<name>.dist` while iproute2 on a multiarch image reads `/usr/lib/<triplet>/tc`, so the drive-fitted delay tail was never actually in use.
 - [Sender-side transit rows made a dead link report zero loss](sender-rows-make-a-dead-link-look-loss-free.md) - `delivered = rows - lost` counted the sender's own `sent` rows as deliveries, so a topic the receiver never saw summarised as `delivered == expected` at 0.0% loss, with the publisher's timer reported as arrival spacing.
 - [A bridge that learns types from the graph cannot use a transport that does not carry the graph](graph-derived-types-block-a-non-graph-transport.md) - over `zenoh_ros2dds` the payload topic never arrived in either run: the bridge routes a topic once a local reader exists and the reader waited for the bridge, leaving only `Pending=1` in a log.
 - [A Fast DDS datagram cap stops every message larger than it](fastdds-datagram-cap-drops-large-samples.md) - copying CycloneDDS's 1200 B fragment size into the Fast DDS OTA profile makes every sample above the cap never arrive, at 1200 B and at 8192 B, sync and async — while the 84 B heartbeat keeps flowing at 10 Hz.

rosotacom_dev-2.5.dev74/docs/findings/a-fitted-delay-table-never-reached-the-shaper.md

@@ -0,0 +1,121 @@
+# A Fitted Delay Table Never Reached The Shaper
+
+## Claim
+
+Every profile carrying a custom netem delay distribution failed at arming time,
+on every image this project ships, since the feature was added. The runner
+installed the table as `/usr/lib/tc/<name>.dist`; iproute2 compiles its library
+directory in, and on a multiarch Debian or Ubuntu build that directory is
+`/usr/lib/<triplet>/tc`. `/usr/lib/tc` does not exist there at all, so the copy
+created it, succeeded, and put the file where nothing reads it:
+
+```
+rosotacom: error: network shaping command failed in …_com_to_b:
+  tc qdisc replace dev eth0 root handle 10: netem delay 25ms 18ms
+    distribution field-20260817 loss gemodel 0.102% 2.5% 2.7% 0.098%
+  -> No distribution data for field-20260817
+     (/usr/lib/x86_64-linux-gnu/tc//field-20260817.dist: No such file or directory)
+```
+
+The error names the directory `tc` wanted, one line after the runner printed
+that it had installed the table somewhere else, and the two paths were never
+compared. What made it survive is that no committed finding needed the table:
+`field-20260817-case3-delay-jam`, the one field profile a published run used,
+carries `rate` and `delay` and no `distribution`. The three profiles that do
+carry one — `-steady`, `-events`, `-soggy` — are exactly the ones nothing had
+run end to end.
+
+The consequence is not a wrong number anywhere. It is that the drive-derived
+delay distribution, whose whole purpose is that a normal jitter model
+under-tails this link by about a factor of two, could not be used, so every
+replay of the 2026-08-17 link either omitted its tail or fell back to a builtin
+shape.
+
+## Setup
+
+- Host pair / topology: one host, the packaged local benchmark rig — two
+  communication containers on their own Docker network, `tc` applied inside
+  each container's own netns (`--sudo-mode container`), no host privileges.
+- Session: `bench_1_1_capacity` from the packaged example project, one
+  `a->b:/bench_capacity` stream at 12000 B and 10 Hz.
+- rosotacom SHA: present up to `25251cb`, fixed in this commit. The defect
+  reaches back to the commit that introduced custom distribution tables.
+- Profile: `field-20260817-events` from the packaged example set — the one that
+  names `distribution: field-20260817` with
+  `distribution_file: dist/field-20260817.dist`.
+- Seed policy: not applicable — the failure is present/absent at arming time,
+  before any traffic is offered.
+
+## Evidence
+
+Evidence grade: reproduction on packaged material, plus the image's own answer
+about where its `tc` looks.
+
+The image is asked rather than assumed:
+
+```bash
+docker run --rm --entrypoint sh <communication image> -c \
+  'tc -V; ls -d /usr/lib/tc /usr/lib/*-linux-gnu*/tc; strings $(command -v tc) | grep /tc/$'
+```
+
+```
+tc utility, iproute2-6.1.0, libbpf 1.3.0
+ls: cannot access '/usr/lib/tc': No such file or directory
+/usr/lib/x86_64-linux-gnu/tc
+/usr/lib/x86_64-linux-gnu/tc/
+```
+
+The directory the runner wrote to did not exist before the runner created it;
+the directory `tc` reads already held iproute2's own `normal.dist` and
+`pareto.dist`. That asymmetry is the fix: the directory holding those tables is
+proof, where a triplet derived from the host would be a guess — the containers
+need not share the host's architecture.
+
+Before and after, same command, same profile:
+
+| | arming | 30 s probe at 12 kB / 10 Hz |
+|---|---|---|
+| before | `No distribution data for field-20260817` | run aborted |
+| after | table installed in `/usr/lib/x86_64-linux-gnu/tc` | 300 of 306 delivered, 1.96 % loss, p50 55.4 ms, p95 102.5 ms |
+
+The delivered numbers are the second half of the evidence: 1.96 % against the
+fitted drive's own 1.9–2.7 %, and a p95 the builtin `normal` distribution does
+not produce at this mean and standard deviation.
+
+Verification: automated:
+`pytest tests/unit/test_network_profiles.py -k tc_lib_dir` builds the resolver's
+shell program against a temporary root and asserts all three rungs — a directory
+carrying iproute2's own tables wins over one that merely exists, and an image
+with neither gets `/usr/lib/tc` created. Manual, for the effect itself: run
+`rosotacom benchmark probe --profile field-20260817-events --size 12000
+--rate-hz 10 --duration 30 --sudo-mode container` from the packaged example
+project; the run must reach a delivery figure rather than fail at arming, and
+the line it prints must name the directory that already holds `normal.dist`.
+
+## Status
+
+confirmed, 2026-08-30.
+
+Fixed by resolving the directory inside the container instead of hard-coding
+it, in three rungs: a candidate that already carries one of iproute2's own
+tables, a candidate that merely exists, and finally `/usr/lib/tc`, created, for
+an image that has neither.
+
+## Publication notes
+
+Two things are worth reporting, and the smaller one is the bug.
+
+The first is about what a green run proves. Every layer here reported success —
+the copy succeeded, the directory was created, the runner printed the path it
+had written to — and the only component that knew the path was wrong was the
+one that read it, one line later, in a message nobody was comparing against the
+line above it. An installation step that does not verify against the consumer's
+own view of where it looks is not an installation step.
+
+The second is for anyone replaying a cellular trace through `netem`. The reason
+this defect matters is that the builtin `normal` distribution is the wrong
+shape for this link: at the 2026-08-17 drive's mean and standard deviation it
+predicts a p99 about half of what was measured, which is the half a deadline
+cares about. A trace-driven emulation that reports its delay model as
+"delay X ± Y, normal" is therefore not reproducing the tail it was fitted to,
+and the table that would have fixed it was, here, silently not in use.

{rosotacom_dev-2.5.dev73 → rosotacom_dev-2.5.dev74}/src/rosotacom/cli_benchmark.py

@@ -5614,22 +5614,40 @@ def _make_live_run_point(args: argparse.Namespace, session_name: str) -> RunPoin
                     dist_installs = required_dist_installs(profile_obj)
                     if dist_installs:
                         # Custom delay-distribution tables must exist inside the
-                        # shaping netns before netem references them by name.
+                        # shaping netns before netem references them by name, and
+                        # in the directory this image's `tc` was compiled to read
+                        # -- see tc_lib_dir_probe for why that is not a constant.
+                        from .network_profiles import tc_lib_dir_probe
+
                         for dist_container in (a_container, b_container):
                             if not dist_container:
                                 continue
+                            probe = subprocess.run(
+                                ["docker", "exec", "-u", "root", dist_container, "sh", "-c", tc_lib_dir_probe()],
+                                capture_output=True,
+                                text=True,
+                                check=False,
+                            )
+                            tc_lib_dir = probe.stdout.strip()
+                            if probe.returncode != 0 or not tc_lib_dir.startswith("/"):
+                                raise RuntimeError(
+                                    f"could not locate the tc library directory in {dist_container}: "
+                                    f"{probe.stderr or probe.stdout}"
+                                )
                             for table_name, file_path in dist_installs:
-                                dist_target = f"/usr/lib/tc/{table_name}.dist"
-                                for install_cmd in (
-                                    ["docker", "exec", "-u", "root", dist_container, "mkdir", "-p", "/usr/lib/tc"],
+                                dist_target = f"{tc_lib_dir}/{table_name}.dist"
+                                res = subprocess.run(
                                     ["docker", "cp", file_path, f"{dist_container}:{dist_target}"],
-                                ):
-                                    res = subprocess.run(install_cmd, capture_output=True, text=True, check=False)
-                                    if res.returncode != 0:
-                                        raise RuntimeError(
-                                            f"failed to install netem distribution table {table_name!r} in "
-                                            f"{dist_container}: {' '.join(install_cmd)} -> {res.stderr or res.stdout}"
-                                        )
+                                    capture_output=True,
+                                    text=True,
+                                    check=False,
+                                )
+                                if res.returncode != 0:
+                                    raise RuntimeError(
+                                        f"failed to install netem distribution table {table_name!r} in "
+                                        f"{dist_container}: {file_path} -> {dist_target}: "
+                                        f"{res.stderr or res.stdout}"
+                                    )
                                 print(
                                     f"  netem distribution {table_name!r} installed in {dist_container}: {dist_target}"
                                 )

{rosotacom_dev-2.5.dev73 → rosotacom_dev-2.5.dev74}/src/rosotacom/network_profiles.py

@@ -440,9 +440,46 @@ def _resolve_distribution_files(profile: Profile, base: Path) -> Profile:
     return replace(profile, uplink=fix(profile.uplink), downlink=fix(profile.downlink))
 
 
+#: Where `tc` looks for a delay-distribution table, in the order to try.
+#: iproute2 compiles the directory in (``LIBDIR "/tc"``), so on a multiarch
+#: Debian or Ubuntu build it is ``/usr/lib/<triplet>/tc`` and ``/usr/lib/tc``
+#: does not exist at all. A table written to the latter is then invisible with
+#: no error of its own: netem answers ``No distribution data for <name>`` and
+#: the run dies at arming time, which is what every profile carrying a fitted
+#: table did until this was found. The directory holding iproute2's own tables
+#: is the one it reads, so that is the evidence to look for rather than a
+#: guessed triplet.
+TC_LIB_DIR_CANDIDATES = ("/usr/lib/*-linux-gnu*/tc", "/usr/lib/tc")
+
+#: A table iproute2 ships, used as the marker for "this is the directory".
+TC_LIB_DIR_MARKERS = ("normal.dist", "pareto.dist")
+
+
+def tc_lib_dir_probe(root: str = "") -> str:
+    """A ``sh -c`` program that prints the directory ``tc`` reads tables from.
+
+    Three rungs, in order: a candidate that already holds one of iproute2's own
+    tables (proof, not inference); a candidate that merely exists; and finally
+    ``/usr/lib/tc``, created, for an image that has neither. `root` exists so
+    the rungs are testable without a container.
+    """
+    cands = " ".join(f"{root}{c}" for c in TC_LIB_DIR_CANDIDATES)
+    marker = " || ".join(f'[ -f "$d/{m}" ]' for m in TC_LIB_DIR_MARKERS)
+    fallback = f"{root}/usr/lib/tc"
+    return (
+        f"for d in {cands}; do "
+        f'  if {marker}; then printf %s "$d"; exit 0; fi; '
+        f"done; "
+        f"for d in {cands}; do "
+        f'  if [ -d "$d" ]; then printf %s "$d"; exit 0; fi; '
+        f"done; "
+        f"mkdir -p {fallback} && printf %s {fallback}"
+    )
+
+
 def required_dist_installs(profile: Profile) -> list[tuple[str, str]]:
     """``(table_name, file_path)`` pairs a runner must install as
-    ``/usr/lib/tc/<table_name>.dist`` inside the shaping netns before arming.
+    ``<tc lib dir>/<table_name>.dist`` inside the shaping netns before arming.
 
     Builtin distributions need nothing; duplicates collapse; conflicting files
     for one name are an error (two segments must mean the same table).

{rosotacom_dev-2.5.dev73 → rosotacom_dev-2.5.dev74}/src/rosotacom_dev.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rosotacom-dev
-Version: 2.5.dev73
+Version: 2.5.dev74
 Summary: Host CLI for ROS OTA communication sessions.
 Author: develNor
 License-Expression: BSD-3-Clause

{rosotacom_dev-2.5.dev73 → rosotacom_dev-2.5.dev74}/src/rosotacom_dev.egg-info/SOURCES.txt

@@ -62,6 +62,7 @@ docs/release.md
 docs/testing.md
 docs/work-items.md
 docs/findings/README.md
+docs/findings/a-fitted-delay-table-never-reached-the-shaper.md
 docs/findings/a-transport-element-doubles-every-fastdds-sample.md
 docs/findings/cyclone-fragment-equal-to-max-drops-fragmented-samples.md
 docs/findings/cyclone-spdp-discovery-bursts.md

{rosotacom_dev-2.5.dev73 → rosotacom_dev-2.5.dev74}/src/rosotacom_dev.egg-info/scm_file_list.json

@@ -46,6 +46,7 @@
     "docs/ci.md",
     "docs/ffmpeg-keyframes.md",
     "docs/findings/README.md",
+    "docs/findings/a-fitted-delay-table-never-reached-the-shaper.md",
     "docs/findings/a-transport-element-doubles-every-fastdds-sample.md",
     "docs/findings/cyclone-fragment-equal-to-max-drops-fragmented-samples.md",
     "docs/findings/cyclone-spdp-discovery-bursts.md",

rosotacom_dev-2.5.dev74/src/rosotacom_dev.egg-info/scm_version.json

@@ -0,0 +1,8 @@
+{
+  "tag": "2.4",
+  "distance": 74,
+  "node": "gb0fa038b84a614450fb24c09bd27fb8358d8b7ec",
+  "dirty": false,
+  "branch": "HEAD",
+  "node_date": "2026-08-30"
+}

{rosotacom_dev-2.5.dev73 → rosotacom_dev-2.5.dev74}/tests/unit/test_network_profiles.py

@@ -3,6 +3,8 @@
 
 from __future__ import annotations
 
+import subprocess
+
 import pytest
 
 from rosotacom.network_profiles import (
@@ -25,6 +27,7 @@ from rosotacom.network_profiles import (
     required_dist_installs,
     resolve_profile_selection,
     shaping_commands,
+    tc_lib_dir_probe,
     teardown_command,
 )
 
@@ -518,3 +521,43 @@ def test_load_profiles_file_resolves_distribution_files_and_lists_installs(tmp_p
         )
         == []
     )
+
+
+def _probe(root) -> str:
+    return subprocess.run(
+        ["sh", "-c", tc_lib_dir_probe(str(root))],
+        capture_output=True,
+        text=True,
+        check=True,
+    ).stdout
+
+
+def test_the_distribution_table_goes_where_tc_reads_it(tmp_path):
+    """A table installed into `/usr/lib/tc` on a multiarch image is invisible.
+
+    iproute2 compiles its library directory in, so a Debian or Ubuntu build
+    reads `/usr/lib/<triplet>/tc` and never looks at `/usr/lib/tc`. netem then
+    fails at arming time with "No distribution data", which reads like a broken
+    profile rather than a misplaced file -- so the directory is resolved from
+    the image instead of assumed.
+    """
+    multiarch = tmp_path / "usr/lib/x86_64-linux-gnu/tc"
+    multiarch.mkdir(parents=True)
+    (multiarch / "normal.dist").write_text("0\n", encoding="utf-8")
+    (tmp_path / "usr/lib/tc").mkdir(parents=True)
+
+    # Both exist; the one carrying iproute2's own table is the one tc reads.
+    assert _probe(tmp_path) == str(multiarch)
+
+
+def test_a_directory_that_merely_exists_is_the_second_choice(tmp_path):
+    multiarch = tmp_path / "usr/lib/aarch64-linux-gnu/tc"
+    multiarch.mkdir(parents=True)
+    assert _probe(tmp_path) == str(multiarch)
+
+
+def test_an_image_with_neither_gets_one_created(tmp_path):
+    (tmp_path / "usr/lib").mkdir(parents=True)
+    out = _probe(tmp_path)
+    assert out == str(tmp_path / "usr/lib/tc")
+    assert (tmp_path / "usr/lib/tc").is_dir()

rosotacom_dev-2.5.dev73/src/rosotacom_dev.egg-info/scm_version.json

@@ -1,8 +0,0 @@
-{
-  "tag": "2.4",
-  "distance": 73,
-  "node": "g25251cbc0967cd74a2417b707e9062c0b520491e",
-  "dirty": false,
-  "branch": "HEAD",
-  "node_date": "2026-08-22"
-}