pyModeS 3.3.0__tar.gz → 3.4.0__tar.gz

{pymodes-3.3.0 → pymodes-3.4.0}/CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to pyModeS are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/).
 
+## [3.4.0] — 2026-07-13
+
+### Fixed
+
+- `PipeDecoder` no longer scans every per-ICAO cache on every timestamped
+  message. Full TTL eviction is throttled to a configurable one-second
+  interval by default, capped at the TTL itself. A bounded per-ICAO check
+  still expires state exactly before it is used, so sweep throttling cannot
+  change BDS inference at the TTL boundary. Set `eviction_interval=0` for
+  the former sweep-per-message memory-reclamation behavior.
+- Timestamped messages that contain no fields used for stateful
+  disambiguation no longer create empty per-ICAO state entries. Existing
+  useful state is still refreshed when such a message arrives.
+- State updates skip field extraction for downlink formats that cannot
+  contribute to stateful inference.
+
+### Added
+
+- `scripts/stream_filtered.py`, a configurable live/replay Beast example
+  demonstrating high-volume DF/typecode prefiltering before `PipeDecoder`.
+- Reproducible version-isolated benchmarks for mixed live traffic and
+  high-cardinality streaming, including checked-in reports and the public
+  Beast capture used by the mixed-traffic benchmark.
+
 ## [3.1.0] — 2026-04-14
 
 Streaming-robustness release. Adds defences against phantom positions
@@ -98,7 +122,7 @@ the migration guide.
   `bin2hex`, `crc`, `df`, `icao`, `typecode`, `altcode`, `idcode`,
   `cprNL`. Thin wrappers over `_bits.py`, `_altcode.py`, `_idcode.py`
   and `position/_cpr.py` with no logic duplication.
-- `scripts/smoke_test.sh` installs the freshly built wheel
+- `scripts/smoke_test_wheel.sh` installs the freshly built wheel
   into a clean Python 3.12 venv and verifies the public API end to
   end (single-message decode, PipeDecoder baseline, batch CPR pair
   resolution, error-dict on malformed input).

{pymodes-3.3.0 → pymodes-3.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyModeS
-Version: 3.3.0
+Version: 3.4.0
 Summary: Python Mode-S and ADS-B Decoder
 Project-URL: homepage, https://mode-s.org
 Project-URL: repository, https://github.com/junzis/pyModeS
@@ -205,24 +205,24 @@ the full command reference.
 
 ## Performance
 
-Measured on jet1090's
-[`long_flight.csv`](https://github.com/xoolive/jet1090/blob/master/crates/rs1090/data/long_flight.csv)
-(172,432 Beast-format messages, 7 runs × 1 loop, mean timings,
-single-core only):
+The committed benchmarks run each released/working-tree version in an
+isolated environment and reject output mismatches before reporting timings.
+On the captured 176,612-frame mixed Beast feed, the header-prefiltered v3 path
+processes 85,963 incoming messages/s:
 
-| Decoder | Wall time | Throughput | vs pyModeS v3 |
-|---|---|---|---|
-| **pyModeS v3 (pure Python)** | **2.06s ± 0.01** | **83,549 msg/s** | **1.00×** |
-| pyModeS 2.21.1 (Python with compiled C) | 5.03s ± 0.01 | 34,303 msg/s | 0.41× |
-| rs1090 (Rust) | 5.60s ± 0.01 | 30,798 msg/s | 0.37× |
-| pyModeS 2.21.1 (Python) | 9.09s ± 0.02 | 18,959 msg/s | 0.23× |
+| Decoder path | Throughput |
+|---|---:|
+| pyModeS 2.21.1 selective helpers | 52,998 msg/s |
+| released v3.3.0 `PipeDecoder` | 3,897 msg/s |
+| v3.4.0 `PipeDecoder` | 49,243 msg/s |
+| v3.4.0 + header prefilter | **85,963 msg/s** |
 
-pyModeS v3 is **2.44× faster** than `pyModeS 2.21.1`'s compiled C
-extension, **4.41× faster** than `pyModeS 2.21.1` pure-Python, and
-**2.71× faster** than rs1090's single-core Rust — all while remaining
-pure Python with no C/Cython build.
+On the 2,000-aircraft synthetic PipeDecoder workload, v3.4.0 processes
+38,838 msg/s versus v2's 39,604 msg/s, with identical normalized output.
 
-Reproduce with `scripts/benchmark_decode.py`.
+See [`scripts/README.md`](./scripts/README.md) for the reproducible commands
+and [`scripts/benchmark_results/`](./scripts/benchmark_results/) for the full
+methodology, timings, and output digests.
 
 ## Supported messages
 

{pymodes-3.3.0 → pymodes-3.4.0}/README.md

@@ -178,24 +178,24 @@ the full command reference.
 
 ## Performance
 
-Measured on jet1090's
-[`long_flight.csv`](https://github.com/xoolive/jet1090/blob/master/crates/rs1090/data/long_flight.csv)
-(172,432 Beast-format messages, 7 runs × 1 loop, mean timings,
-single-core only):
-
-| Decoder | Wall time | Throughput | vs pyModeS v3 |
-|---|---|---|---|
-| **pyModeS v3 (pure Python)** | **2.06s ± 0.01** | **83,549 msg/s** | **1.00×** |
-| pyModeS 2.21.1 (Python with compiled C) | 5.03s ± 0.01 | 34,303 msg/s | 0.41× |
-| rs1090 (Rust) | 5.60s ± 0.01 | 30,798 msg/s | 0.37× |
-| pyModeS 2.21.1 (Python) | 9.09s ± 0.02 | 18,959 msg/s | 0.23× |
-
-pyModeS v3 is **2.44× faster** than `pyModeS 2.21.1`'s compiled C
-extension, **4.41× faster** than `pyModeS 2.21.1` pure-Python, and
-**2.71× faster** than rs1090's single-core Rust — all while remaining
-pure Python with no C/Cython build.
-
-Reproduce with `scripts/benchmark_decode.py`.
+The committed benchmarks run each released/working-tree version in an
+isolated environment and reject output mismatches before reporting timings.
+On the captured 176,612-frame mixed Beast feed, the header-prefiltered v3 path
+processes 85,963 incoming messages/s:
+
+| Decoder path | Throughput |
+|---|---:|
+| pyModeS 2.21.1 selective helpers | 52,998 msg/s |
+| released v3.3.0 `PipeDecoder` | 3,897 msg/s |
+| v3.4.0 `PipeDecoder` | 49,243 msg/s |
+| v3.4.0 + header prefilter | **85,963 msg/s** |
+
+On the 2,000-aircraft synthetic PipeDecoder workload, v3.4.0 processes
+38,838 msg/s versus v2's 39,604 msg/s, with identical normalized output.
+
+See [`scripts/README.md`](./scripts/README.md) for the reproducible commands
+and [`scripts/benchmark_results/`](./scripts/benchmark_results/) for the full
+methodology, timings, and output digests.
 
 ## Supported messages
 

{pymodes-3.3.0 → pymodes-3.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pyModeS"
-version = "3.3.0"
+version = "3.4.0"
 description = "Python Mode-S and ADS-B Decoder"
 authors = [{ name = "Junzi Sun", email = "git@junzis.com" }]
 license = { text = "GNU GPL v3" }
@@ -57,7 +57,6 @@ dev = [
 [tool.ruff]
 target-version = "py311"
 line-length = 88
-extend-exclude = ["scripts/bench_pms_v2.py"]
 
 [tool.ruff.lint]
 select = [

pymodes-3.4.0/scripts/README.md

@@ -0,0 +1,58 @@
+# Maintainer scripts
+
+Run these commands from the repository root. Scripts are grouped by purpose
+through their names; generated benchmark reports live in
+`scripts/benchmark_results/`, and committed input data lives in
+`scripts/data/`.
+
+## Streaming example
+
+- `stream_filtered.py` — configurable high-throughput example using a live
+  Mode-S Beast feed or the committed replay capture. Defaults to DF17/20/21
+  while excluding ADS-B typecodes 28/29/31.
+
+  ```bash
+  uv run python scripts/stream_filtered.py \
+    --network airsquitter.lr.tudelft.nl:10006 > decoded.jsonl
+  ```
+
+## Benchmarks
+
+- `benchmark_mixed_traffic.py` — isolated v2, released-v3, updated-v3, and
+  prefiltered-v3 comparison on mixed captured traffic. Writes
+  `benchmark_results/mixed_traffic.md` by default.
+- `benchmark_pipe.py` — synthetic high-cardinality PipeDecoder comparison.
+  Writes `benchmark_results/pipe.md` by default.
+- `benchmark_cpr.py` — focused CPR primitive comparison against the v2 NumPy
+  implementation.
+
+  ```bash
+  uv run python scripts/benchmark_mixed_traffic.py
+  uv run python scripts/benchmark_pipe.py
+  uv run --with numpy python scripts/benchmark_cpr.py
+  ```
+
+The version-comparison benchmarks build the working tree and resolve released
+versions from PyPI in isolated temporary environments. They require network
+access unless those packages are already cached.
+
+## Generated data
+
+- `generate_airport_db.py` — refreshes `src/pyModeS/data/airports.py` from
+  OurAirports.
+- `generate_v2_fixture.py` — regenerates the committed v2.21.1 golden oracle.
+
+  ```bash
+  uv run python scripts/generate_airport_db.py
+  uv run --no-project --with pyModeS==2.21.1 \
+    python scripts/generate_v2_fixture.py
+  ```
+
+## Release smoke tests
+
+- `smoke_test_wheel.sh [wheel]` — installs a built wheel in a clean virtual
+  environment and checks the public API.
+- `smoke_test_cli.sh` — checks CLI decoding and a short live Beast connection.
+
+The CLI smoke test requires network access. The wheel smoke test uses the most
+recent `dist/pymodes-*.whl` when no path is supplied.

pymodes-3.4.0/scripts/data/README.md

@@ -0,0 +1,37 @@
+# Benchmark capture data
+
+## `airsquitter_2026-07-13_120s.csv.gz`
+
+Two-minute capture from the public TU Delft AirSquitter Mode-S Beast feed:
+
+- Source: `airsquitter.lr.tudelft.nl:10006` (TCP, Beast binary)
+- Capture date: 2026-07-13
+- Frames: 176,612
+- Uncompressed format: `timestamp,hex_message`, one frame per line
+- Compressed size: approximately 2 MiB
+- SHA-256: `fa433eb8dc2ec9cf6ebd40c1bf0a095ae753960d5e8c42ae6da15cc28eb3baea`
+
+The timestamps are reconstructed by `pyModeS.cli._source.NetworkSource`
+from the Beast MLAT counter, anchored to wall-clock time when the TCP data
+is received. Mode A/C and Beast status frames are omitted because
+`NetworkSource` yields only Mode-S short and long frames.
+
+Capture and compact conversion:
+
+```bash
+timeout --signal=INT 120 \
+  modes live \
+  --network airsquitter.lr.tudelft.nl:10006 \
+  --quiet \
+  --dump-to /tmp/pymodes-live-mixed.jsonl
+
+jq -r '"\(.timestamp),\(.raw_msg)"' \
+  /tmp/pymodes-live-mixed.jsonl \
+  | gzip -9 > scripts/data/airsquitter_2026-07-13_120s.csv.gz
+```
+
+Run the associated isolated three-version benchmark with:
+
+```bash
+python scripts/benchmark_mixed_traffic.py
+```

{pymodes-3.3.0 → pymodes-3.4.0}/src/pyModeS/_pipe.py

@@ -82,6 +82,12 @@ _DECODED_TO_KNOWN: dict[str, str] = {
     "altitude": "altitude",  # informational; not in scoring tables
 }
 
+# Only these downlink formats can currently emit fields consumed by
+# stateful Comm-B inference. Keeping the set explicit lets the hot path
+# skip nine result-dict lookups for DF5/DF11 and unsupported DFs while
+# still refreshing useful state that already exists for the ICAO.
+_STATEFUL_DFS = frozenset((0, 4, 16, 17, 18, 20, 21))
+
 # BDS-payload fields cleared from a DF20 result that fails altitude
 # cross-check. The `altitude` field itself is preserved (it's the
 # 13-bit AC-code that triggered the mismatch — callers want it to
@@ -171,18 +177,25 @@ class PipeDecoder:
         pair_window: Maximum age difference (seconds) between an even
             and odd CPR frame for them to count as a pair. Default 10s.
         eviction_ttl: Per-ICAO state and pending CPR frames older than
-            this many seconds are dropped lazily on the next decode
-            call. Default 300s (5 minutes).
+            this many seconds are ignored immediately for the aircraft
+            being decoded and dropped globally by a periodic sweep.
+            Default 300s (5 minutes).
+        eviction_interval: Minimum timestamp gap between full cache
+            eviction sweeps. The effective interval is capped at
+            ``eviction_ttl``. Default 1s; set to 0 to sweep on every
+            timestamped decode call.
     """
 
     __slots__ = (
         "_adsb_altitude",
         "_adsb_velocity",
         "_bootstrap",
+        "_eviction_interval",
         "_eviction_ttl",
         "_full_dict",
         "_max_speed_kmps",
         "_motion_margin_km",
+        "_next_eviction_at",
         "_pair_window",
         "_pending_even",
         "_pending_odd",
@@ -200,13 +213,23 @@ class PipeDecoder:
         full_dict: bool = False,
         pair_window: float = 10.0,
         eviction_ttl: float = 300.0,
+        eviction_interval: float = 1.0,
         max_speed_kt: float = 1500.0,
         motion_margin_km: float = 2.0,
     ) -> None:
         self._surface_ref = surface_ref
         self._full_dict = full_dict
         self._pair_window = pair_window
+        if eviction_ttl < 0:
+            raise ValueError("eviction_ttl must be >= 0")
+        if eviction_interval < 0:
+            raise ValueError("eviction_interval must be >= 0")
         self._eviction_ttl = eviction_ttl
+        # Never delay a sweep by longer than the TTL itself. This keeps
+        # sub-second TTL configurations useful while avoiding an O(cache)
+        # walk for every message in normal high-rate streams.
+        self._eviction_interval = min(eviction_interval, eviction_ttl)
+        self._next_eviction_at = float("-inf")
         # 1500 kt is ~2x typical airliner cruise — loose enough not to
         # reject fast business jets or wind-boosted ground speed, tight
         # enough that a phantom position hundreds of km away cannot
@@ -284,7 +307,7 @@ class PipeDecoder:
         into state for future calls.
         """
         if timestamp is not None:
-            self._evict_expired(timestamp)
+            self._maybe_evict_expired(timestamp)
         self._stats["total"] += 1
 
         try:
@@ -293,13 +316,23 @@ class PipeDecoder:
             return Decoded({"error": str(e), "raw_msg": msg})
 
         # Look up prior state for this ICAO so the decoder can use it
-        # for Comm-B BDS 5,0/6,0 disambiguation. Filter out housekeeping
-        # keys (those starting with _) before passing as `known=`.
+        # for Comm-B BDS 5,0/6,0 disambiguation. Remove the internal
+        # timestamp before passing the state as `known=`.
         icao = message.icao
-        prior_state = self._state.get(icao)
+        if timestamp is not None:
+            # Full sweeps are deliberately throttled, but stale state must
+            # never influence the current aircraft's decode. This bounded
+            # per-ICAO prune preserves exact TTL semantics without bringing
+            # back an O(all active aircraft) operation on every message.
+            prior_state = self._evict_icao_expired(icao, message.df, timestamp)
+        else:
+            prior_state = self._state.get(icao)
         known: dict[str, Any] | None
         if prior_state:
-            known = {k: v for k, v in prior_state.items() if not k.startswith("_")}
+            # dict.copy() runs in C and is measurably cheaper in this hot
+            # path than filtering the small state dict with a Python loop.
+            known = prior_state.copy()
+            known.pop("_last_seen", None)
             # Derive the BDS 6,0 scoring fields (ias, mach) and the
             # BDS 5,0 tas slot from cached groundspeed + altitude
             # when the caller hasn't supplied observed values. Most
@@ -445,7 +478,7 @@ class PipeDecoder:
             return result
 
         self._handle_cpr_pair(result, icao, timestamp)
-        self._update_state(icao, result, timestamp)
+        self._update_state(icao, message.df, result, timestamp)
         return result
 
     def _reject_on_altitude_mismatch(
@@ -671,13 +704,28 @@ class PipeDecoder:
         self._stats["velocity_mismatch"] += 1
         return True
 
+    def _maybe_evict_expired(self, now: float) -> None:
+        """Run a full eviction sweep only when its interval is due.
+
+        A sweep touches every live per-ICAO cache, so running it for every
+        message makes decode cost proportional to the number of active
+        aircraft. At high message rates that dominates the actual decoder.
+        Timestamp throttling bounds stale memory retention by at most
+        ``eviction_interval`` while making sweep cost amortized. Exact
+        per-aircraft expiry happens separately before cached state is used.
+        """
+        if now < self._next_eviction_at:
+            return
+        self._evict_expired(now)
+        self._next_eviction_at = now + self._eviction_interval
+
     def _evict_expired(self, now: float) -> None:
         """Drop state and pending CPR entries older than eviction_ttl.
 
-        Runs lazily at the start of each decode() call when a timestamp
-        is provided. The trusted ICAO set is intentionally NOT evicted —
-        once a plain-text DF17/18 has been seen for an ICAO, it remains
-        trusted for the lifetime of the PipeDecoder (until reset()).
+        Called periodically by :meth:`_maybe_evict_expired`. The trusted
+        ICAO set is intentionally NOT evicted — once a plain-text DF17/18
+        has been seen for an ICAO, it remains trusted for the lifetime of
+        the PipeDecoder (until reset()).
         """
         cutoff = now - self._eviction_ttl
 
@@ -742,6 +790,76 @@ class PipeDecoder:
             else:
                 del self._bootstrap[icao]
 
+    def _evict_icao_expired(
+        self, icao: str, downlink_format: int, now: float
+    ) -> dict[str, Any] | None:
+        """Drop expired caches and return usable state for one ICAO.
+
+        Periodic full sweeps keep total memory bounded, but their interval
+        must not extend the semantic lifetime of state used for Comm-B
+        inference or validation. Every collection touched here has a small,
+        fixed per-ICAO bound, so this remains O(1) with respect to the number
+        of active aircraft.
+        """
+        cutoff = now - self._eviction_ttl
+
+        state = self._state.get(icao)
+        if state is not None and state.get("_last_seen", float("inf")) < cutoff:
+            self._state.pop(icao, None)
+            state = None
+
+        # Short surveillance/ACAS messages cannot consume validation or
+        # CPR caches. Their altitude can still refresh `_state` later, but
+        # the state expiry above is the only exact-TTL work they need here.
+        if downlink_format not in (17, 18, 20, 21):
+            return state
+
+        if downlink_format in (17, 18, 20):
+            altitude_anchor = self._adsb_altitude.get(icao)
+            if altitude_anchor is not None and altitude_anchor[0] < cutoff:
+                self._adsb_altitude.pop(icao, None)
+
+        velocity_anchor = self._adsb_velocity.get(icao)
+        if velocity_anchor is not None and velocity_anchor[0] < cutoff:
+            self._adsb_velocity.pop(icao, None)
+
+        # Only ADS-B frames participate in CPR pairing, bootstrap, and
+        # position-history validation.
+        if downlink_format not in (17, 18):
+            return state
+
+        for pending in (self._pending_even, self._pending_odd):
+            entries = pending.get(icao)
+            if entries is None:
+                continue
+            fresh_entries = [entry for entry in entries if entry[0] >= cutoff]
+            dropped = len(entries) - len(fresh_entries)
+            if fresh_entries:
+                pending[icao] = fresh_entries
+            else:
+                pending.pop(icao, None)
+            self._stats["pending_pairs"] = max(
+                0, self._stats["pending_pairs"] - dropped
+            )
+
+        history = self._position_history.get(icao)
+        if history is not None:
+            fresh_history = [entry for entry in history if entry[2] >= cutoff]
+            if fresh_history:
+                self._position_history[icao] = fresh_history
+            else:
+                self._position_history.pop(icao, None)
+
+        bootstrap = self._bootstrap.get(icao)
+        if bootstrap is not None:
+            fresh_bootstrap = [entry for entry in bootstrap if entry[2] >= cutoff]
+            if fresh_bootstrap:
+                self._bootstrap[icao] = fresh_bootstrap
+            else:
+                self._bootstrap.pop(icao, None)
+
+        return state
+
     def _motion_consistent(
         self,
         icao: str,
@@ -1074,10 +1192,17 @@ class PipeDecoder:
     def _update_state(
         self,
         icao: str,
+        downlink_format: int,
         result: Decoded,
         timestamp: float | None,
     ) -> None:
         """Merge tracked fields from the result into per-ICAO state."""
+        existing = self._state.get(icao)
+        if downlink_format not in _STATEFUL_DFS:
+            if existing is not None and timestamp is not None:
+                existing["_last_seen"] = timestamp
+            return
+
         new_fields: dict[str, Any] = {}
         for decoded_key, known_key in _DECODED_TO_KNOWN.items():
             val = result.get(decoded_key)
@@ -1095,10 +1220,18 @@ class PipeDecoder:
         elif airspeed is not None and airspeed_type == "TAS":
             new_fields["tas"] = airspeed
 
-        if not new_fields and timestamp is None:
+        if not new_fields:
+            # A timestamp alone is not useful state. Creating an entry here
+            # used to retain every parseable message's derived ICAO — even
+            # for untracked DFs and corrupt traffic — and made the periodic
+            # eviction scan much larger than the active-aircraft set.
+            if existing is not None and timestamp is not None:
+                existing["_last_seen"] = timestamp
             return
 
-        existing = self._state.setdefault(icao, {})
+        if existing is None:
+            existing = {}
+            self._state[icao] = existing
         existing.update(new_fields)
         if timestamp is not None:
             existing["_last_seen"] = timestamp
@@ -1118,5 +1251,6 @@ class PipeDecoder:
         self._adsb_velocity.clear()
         self._position_history.clear()
         self._bootstrap.clear()
+        self._next_eviction_at = float("-inf")
         for k in self._stats:
             self._stats[k] = 0

{pymodes-3.3.0 → pymodes-3.4.0}/src/pyModeS/data/airports.py

@@ -1,6 +1,6 @@
 """Curated airport database.
 
-Generated from OurAirports dataset by scripts/build_airport_db.py.
+Generated from OurAirports dataset by scripts/generate_airport_db.py.
 Source: https://davidmegginson.github.io/ourairports-data/airports.csv
 Filter: large_airport + medium_airport with an assigned 4-letter ICAO code.