pyModeS 3.2.0__tar.gz → 3.4.0__tar.gz

{pymodes-3.2.0 → pymodes-3.4.0}/CHANGELOG.md

@@ -5,6 +5,30 @@ All notable changes to pyModeS are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/).
 
+## [3.4.0] — 2026-07-13
+
+### Fixed
+
+- `PipeDecoder` no longer scans every per-ICAO cache on every timestamped
+  message. Full TTL eviction is throttled to a configurable one-second
+  interval by default, capped at the TTL itself. A bounded per-ICAO check
+  still expires state exactly before it is used, so sweep throttling cannot
+  change BDS inference at the TTL boundary. Set `eviction_interval=0` for
+  the former sweep-per-message memory-reclamation behavior.
+- Timestamped messages that contain no fields used for stateful
+  disambiguation no longer create empty per-ICAO state entries. Existing
+  useful state is still refreshed when such a message arrives.
+- State updates skip field extraction for downlink formats that cannot
+  contribute to stateful inference.
+
+### Added
+
+- `scripts/stream_filtered.py`, a configurable live/replay Beast example
+  demonstrating high-volume DF/typecode prefiltering before `PipeDecoder`.
+- Reproducible version-isolated benchmarks for mixed live traffic and
+  high-cardinality streaming, including checked-in reports and the public
+  Beast capture used by the mixed-traffic benchmark.
+
 ## [3.1.0] — 2026-04-14
 
 Streaming-robustness release. Adds defences against phantom positions
@@ -98,7 +122,7 @@ the migration guide.
   `bin2hex`, `crc`, `df`, `icao`, `typecode`, `altcode`, `idcode`,
   `cprNL`. Thin wrappers over `_bits.py`, `_altcode.py`, `_idcode.py`
   and `position/_cpr.py` with no logic duplication.
-- `scripts/smoke_test.sh` installs the freshly built wheel
+- `scripts/smoke_test_wheel.sh` installs the freshly built wheel
   into a clean Python 3.12 venv and verifies the public API end to
   end (single-message decode, PipeDecoder baseline, batch CPR pair
   resolution, error-dict on malformed input).

{pymodes-3.2.0 → pymodes-3.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyModeS
-Version: 3.2.0
+Version: 3.4.0
 Summary: Python Mode-S and ADS-B Decoder
 Project-URL: homepage, https://mode-s.org
 Project-URL: repository, https://github.com/junzis/pyModeS
@@ -205,24 +205,24 @@ the full command reference.
 
 ## Performance
 
-Measured on jet1090's
-[`long_flight.csv`](https://github.com/xoolive/jet1090/blob/master/crates/rs1090/data/long_flight.csv)
-(172,432 Beast-format messages, 7 runs × 1 loop, mean timings,
-single-core only):
+The committed benchmarks run each released/working-tree version in an
+isolated environment and reject output mismatches before reporting timings.
+On the captured 176,612-frame mixed Beast feed, the header-prefiltered v3 path
+processes 85,963 incoming messages/s:
 
-| Decoder | Wall time | Throughput | vs pyModeS v3 |
-|---|---|---|---|
-| **pyModeS v3 (pure Python)** | **2.06s ± 0.01** | **83,549 msg/s** | **1.00×** |
-| pyModeS 2.21.1 (Python with compiled C) | 5.03s ± 0.01 | 34,303 msg/s | 0.41× |
-| rs1090 (Rust) | 5.60s ± 0.01 | 30,798 msg/s | 0.37× |
-| pyModeS 2.21.1 (Python) | 9.09s ± 0.02 | 18,959 msg/s | 0.23× |
+| Decoder path | Throughput |
+|---|---:|
+| pyModeS 2.21.1 selective helpers | 52,998 msg/s |
+| released v3.3.0 `PipeDecoder` | 3,897 msg/s |
+| v3.4.0 `PipeDecoder` | 49,243 msg/s |
+| v3.4.0 + header prefilter | **85,963 msg/s** |
 
-pyModeS v3 is **2.44× faster** than `pyModeS 2.21.1`'s compiled C
-extension, **4.41× faster** than `pyModeS 2.21.1` pure-Python, and
-**2.71× faster** than rs1090's single-core Rust — all while remaining
-pure Python with no C/Cython build.
+On the 2,000-aircraft synthetic PipeDecoder workload, v3.4.0 processes
+38,838 msg/s versus v2's 39,604 msg/s, with identical normalized output.
 
-Reproduce with `scripts/benchmark_decode.py`.
+See [`scripts/README.md`](./scripts/README.md) for the reproducible commands
+and [`scripts/benchmark_results/`](./scripts/benchmark_results/) for the full
+methodology, timings, and output digests.
 
 ## Supported messages
 

{pymodes-3.2.0 → pymodes-3.4.0}/README.md

@@ -178,24 +178,24 @@ the full command reference.
 
 ## Performance
 
-Measured on jet1090's
-[`long_flight.csv`](https://github.com/xoolive/jet1090/blob/master/crates/rs1090/data/long_flight.csv)
-(172,432 Beast-format messages, 7 runs × 1 loop, mean timings,
-single-core only):
-
-| Decoder | Wall time | Throughput | vs pyModeS v3 |
-|---|---|---|---|
-| **pyModeS v3 (pure Python)** | **2.06s ± 0.01** | **83,549 msg/s** | **1.00×** |
-| pyModeS 2.21.1 (Python with compiled C) | 5.03s ± 0.01 | 34,303 msg/s | 0.41× |
-| rs1090 (Rust) | 5.60s ± 0.01 | 30,798 msg/s | 0.37× |
-| pyModeS 2.21.1 (Python) | 9.09s ± 0.02 | 18,959 msg/s | 0.23× |
-
-pyModeS v3 is **2.44× faster** than `pyModeS 2.21.1`'s compiled C
-extension, **4.41× faster** than `pyModeS 2.21.1` pure-Python, and
-**2.71× faster** than rs1090's single-core Rust — all while remaining
-pure Python with no C/Cython build.
-
-Reproduce with `scripts/benchmark_decode.py`.
+The committed benchmarks run each released/working-tree version in an
+isolated environment and reject output mismatches before reporting timings.
+On the captured 176,612-frame mixed Beast feed, the header-prefiltered v3 path
+processes 85,963 incoming messages/s:
+
+| Decoder path | Throughput |
+|---|---:|
+| pyModeS 2.21.1 selective helpers | 52,998 msg/s |
+| released v3.3.0 `PipeDecoder` | 3,897 msg/s |
+| v3.4.0 `PipeDecoder` | 49,243 msg/s |
+| v3.4.0 + header prefilter | **85,963 msg/s** |
+
+On the 2,000-aircraft synthetic PipeDecoder workload, v3.4.0 processes
+38,838 msg/s versus v2's 39,604 msg/s, with identical normalized output.
+
+See [`scripts/README.md`](./scripts/README.md) for the reproducible commands
+and [`scripts/benchmark_results/`](./scripts/benchmark_results/) for the full
+methodology, timings, and output digests.
 
 ## Supported messages
 

{pymodes-3.2.0 → pymodes-3.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pyModeS"
-version = "3.2.0"
+version = "3.4.0"
 description = "Python Mode-S and ADS-B Decoder"
 authors = [{ name = "Junzi Sun", email = "git@junzis.com" }]
 license = { text = "GNU GPL v3" }
@@ -44,6 +44,7 @@ issues = "https://github.com/junzis/pyModeS/issues"
 [dependency-groups]
 dev = [
     "mypy>=1.13",
+    "pre-commit>=4.5.1",
     "pytest>=7.2.0",
     "pytest-cov>=4.0.0",
     "ruff>=0.15.8",
@@ -56,7 +57,6 @@ dev = [
 [tool.ruff]
 target-version = "py311"
 line-length = 88
-extend-exclude = ["scripts/bench_pms_v2.py"]
 
 [tool.ruff.lint]
 select = [

pymodes-3.4.0/scripts/README.md

@@ -0,0 +1,58 @@
+# Maintainer scripts
+
+Run these commands from the repository root. Scripts are grouped by purpose
+through their names; generated benchmark reports live in
+`scripts/benchmark_results/`, and committed input data lives in
+`scripts/data/`.
+
+## Streaming example
+
+- `stream_filtered.py` — configurable high-throughput example using a live
+  Mode-S Beast feed or the committed replay capture. Defaults to DF17/20/21
+  while excluding ADS-B typecodes 28/29/31.
+
+  ```bash
+  uv run python scripts/stream_filtered.py \
+    --network airsquitter.lr.tudelft.nl:10006 > decoded.jsonl
+  ```
+
+## Benchmarks
+
+- `benchmark_mixed_traffic.py` — isolated v2, released-v3, updated-v3, and
+  prefiltered-v3 comparison on mixed captured traffic. Writes
+  `benchmark_results/mixed_traffic.md` by default.
+- `benchmark_pipe.py` — synthetic high-cardinality PipeDecoder comparison.
+  Writes `benchmark_results/pipe.md` by default.
+- `benchmark_cpr.py` — focused CPR primitive comparison against the v2 NumPy
+  implementation.
+
+  ```bash
+  uv run python scripts/benchmark_mixed_traffic.py
+  uv run python scripts/benchmark_pipe.py
+  uv run --with numpy python scripts/benchmark_cpr.py
+  ```
+
+The version-comparison benchmarks build the working tree and resolve released
+versions from PyPI in isolated temporary environments. They require network
+access unless those packages are already cached.
+
+## Generated data
+
+- `generate_airport_db.py` — refreshes `src/pyModeS/data/airports.py` from
+  OurAirports.
+- `generate_v2_fixture.py` — regenerates the committed v2.21.1 golden oracle.
+
+  ```bash
+  uv run python scripts/generate_airport_db.py
+  uv run --no-project --with pyModeS==2.21.1 \
+    python scripts/generate_v2_fixture.py
+  ```
+
+## Release smoke tests
+
+- `smoke_test_wheel.sh [wheel]` — installs a built wheel in a clean virtual
+  environment and checks the public API.
+- `smoke_test_cli.sh` — checks CLI decoding and a short live Beast connection.
+
+The CLI smoke test requires network access. The wheel smoke test uses the most
+recent `dist/pymodes-*.whl` when no path is supplied.

pymodes-3.4.0/scripts/data/README.md

@@ -0,0 +1,37 @@
+# Benchmark capture data
+
+## `airsquitter_2026-07-13_120s.csv.gz`
+
+Two-minute capture from the public TU Delft AirSquitter Mode-S Beast feed:
+
+- Source: `airsquitter.lr.tudelft.nl:10006` (TCP, Beast binary)
+- Capture date: 2026-07-13
+- Frames: 176,612
+- Uncompressed format: `timestamp,hex_message`, one frame per line
+- Compressed size: approximately 2 MiB
+- SHA-256: `fa433eb8dc2ec9cf6ebd40c1bf0a095ae753960d5e8c42ae6da15cc28eb3baea`
+
+The timestamps are reconstructed by `pyModeS.cli._source.NetworkSource`
+from the Beast MLAT counter, anchored to wall-clock time when the TCP data
+is received. Mode A/C and Beast status frames are omitted because
+`NetworkSource` yields only Mode-S short and long frames.
+
+Capture and compact conversion:
+
+```bash
+timeout --signal=INT 120 \
+  modes live \
+  --network airsquitter.lr.tudelft.nl:10006 \
+  --quiet \
+  --dump-to /tmp/pymodes-live-mixed.jsonl
+
+jq -r '"\(.timestamp),\(.raw_msg)"' \
+  /tmp/pymodes-live-mixed.jsonl \
+  | gzip -9 > scripts/data/airsquitter_2026-07-13_120s.csv.gz
+```
+
+Run the associated isolated three-version benchmark with:
+
+```bash
+python scripts/benchmark_mixed_traffic.py
+```