pyModeS 3.3.0__tar.gz → 3.5.0__tar.gz

{pymodes-3.3.0 → pymodes-3.5.0}/CHANGELOG.md

@@ -5,6 +5,93 @@ All notable changes to pyModeS are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/).
 
+## [3.5.0] — 2026-07-15
+
+### Added
+
+- `PipeDecoder` now uses the last validated airborne position as a local CPR
+  reference for subsequent BDS 0,5 frames. This makes a position available on
+  the current streaming `decode()` call when no fresh global even/odd pair is
+  available, instead of relying on retroactive mutation of an earlier result.
+- `local_ref_window` configures the maximum reference age (30 seconds by
+  default); set it to zero to disable local airborne decoding.
+  `PipeDecoder.stats["local_positions"]` reports positions produced by this
+  path.
+- Position bootstrap now locks as soon as three candidates are pairwise
+  motion-consistent instead of always waiting for all five candidate slots.
+  The five-candidate limit still allows two unrelated outliers to be ignored.
+- The mixed-traffic benchmark now reports immediate streaming position yield
+  and frame-by-frame coordinate agreement for v2.21.1, released v3.4.0, and
+  v3.5.0 in addition to throughput.
+
+### Changed
+
+- `crc_valid` is now `None` when an address-parity message cannot be checked
+  independently. It remains `True` or `False` for frames whose parity can be
+  validated without assuming the address derived from that same parity.
+- `PipeDecoder` uses each message's source timestamp for state validity while
+  retaining a monotonic high-water mark for eviction. Reordered capture data
+  therefore cannot revive expired observations or prematurely evict newer
+  state.
+- Plausibility and position validation are isolated in a dedicated internal
+  module, keeping the streaming pipeline and its decision comments easier to
+  follow without changing the public API.
+- Release publishing now runs only after the full test matrix succeeds, checks
+  that the Git tag matches the package version, and smoke-tests the exact wheel
+  artifact that will be uploaded.
+
+### Fixed
+
+- Surface references remain optional and are used only for BDS 0,6 surface
+  positions. Positions derived from `surface_ref` can never seed local
+  airborne CPR decoding.
+- Rejected global position candidates cannot become local CPR references; a
+  separate last-accepted airborne reference is maintained and TTL-evicted.
+- Bootstrap promotion, including `flush()`, initializes the latest accepted
+  airborne reference while keeping surface-derived positions separate.
+- Globally decoded surface CPR positions now select the correct longitude zone
+  across the antimeridian and other CPR zone boundaries.
+- Beast stream input resynchronizes after malformed or truncated frames instead
+  of losing subsequent valid messages.
+- Beast receiver timestamps are back-projected from the current wall clock with
+  rollover-safe receiver-tick deltas, avoiding dates near the Unix epoch and
+  preserving correct timing across the 12 MHz counter rollover.
+- BDS inference compares headings using circular angular distance, so values
+  around 0/360 degrees are scored correctly.
+- Trusted ICAO observations expire according to the configured state TTL and
+  cannot be refreshed by older, reordered messages.
+- Integer messages outside the valid 56-bit and 112-bit Mode S frame ranges are
+  rejected instead of being silently truncated or misclassified.
+- Network endpoint and surface-reference CLI arguments are validated before a
+  stream starts, and network sources are closed deterministically on normal
+  exit, errors, and interruption.
+- The live terminal UI synchronizes access to shared aircraft state, preventing
+  concurrent decoder and renderer updates from racing.
+
+## [3.4.0] — 2026-07-13
+
+### Fixed
+
+- `PipeDecoder` no longer scans every per-ICAO cache on every timestamped
+  message. Full TTL eviction is throttled to a configurable one-second
+  interval by default, capped at the TTL itself. A bounded per-ICAO check
+  still expires state exactly before it is used, so sweep throttling cannot
+  change BDS inference at the TTL boundary. Set `eviction_interval=0` for
+  the former sweep-per-message memory-reclamation behavior.
+- Timestamped messages that contain no fields used for stateful
+  disambiguation no longer create empty per-ICAO state entries. Existing
+  useful state is still refreshed when such a message arrives.
+- State updates skip field extraction for downlink formats that cannot
+  contribute to stateful inference.
+
+### Added
+
+- `scripts/stream_filtered.py`, a configurable live/replay Beast example
+  demonstrating high-volume DF/typecode prefiltering before `PipeDecoder`.
+- Reproducible version-isolated benchmarks for mixed live traffic and
+  high-cardinality streaming, including checked-in reports and the public
+  Beast capture used by the mixed-traffic benchmark.
+
 ## [3.1.0] — 2026-04-14
 
 Streaming-robustness release. Adds defences against phantom positions
@@ -14,8 +101,8 @@ Comm-B reply to the wrong aircraft.
 ### Added
 
 - `PipeDecoder` bootstrap cluster analysis for the first position per
-  ICAO: lat/lon emission is held until `_BOOTSTRAP_K` (=5) candidate
-  positions agree under a motion-consistency check. Prevents an
+  ICAO: lat/lon emission is held until five candidate positions agree
+  under a motion-consistency check. Prevents an
   initial phantom frame from anchoring the rolling position history.
   On each decode the held result dicts are retro-filled once the
   cluster locks. Scattered buffers reset and accumulate a fresh K.
@@ -98,7 +185,7 @@ the migration guide.
   `bin2hex`, `crc`, `df`, `icao`, `typecode`, `altcode`, `idcode`,
   `cprNL`. Thin wrappers over `_bits.py`, `_altcode.py`, `_idcode.py`
   and `position/_cpr.py` with no logic duplication.
-- `scripts/smoke_test.sh` installs the freshly built wheel
+- `scripts/smoke_test_wheel.sh` installs the freshly built wheel
   into a clean Python 3.12 venv and verifies the public API end to
   end (single-message decode, PipeDecoder baseline, batch CPR pair
   resolution, error-dict on malformed input).

{pymodes-3.3.0 → pymodes-3.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyModeS
-Version: 3.3.0
+Version: 3.5.0
 Summary: Python Mode-S and ADS-B Decoder
 Project-URL: homepage, https://mode-s.org
 Project-URL: repository, https://github.com/junzis/pyModeS
@@ -109,7 +109,7 @@ print(r["latitude"], r["longitude"])  # 43.6264..., 1.3747...
 `PipeDecoder` is stateful — it holds per-ICAO state across calls,
 matches CPR pairs automatically, evicts stale aircraft after a
 TTL, and flags DF20/21 messages as `icao_verified` when their
-CRC-derived ICAO was already seen in a clean DF17/18 plaintext.
+CRC-derived ICAO was seen recently in clean DF17/18 plaintext.
 
 ```python
 from pyModeS import PipeDecoder
@@ -117,7 +117,7 @@ from pyModeS import PipeDecoder
 pipe = PipeDecoder(surface_ref="EHAM")
 for msg, timestamp in stream:
     decoded = pipe.decode(msg, timestamp=timestamp)
-    if "latitude" in decoded:
+    if decoded.get("latitude") is not None:
         print(decoded["icao"], decoded["latitude"], decoded["longitude"])
 ```
 
@@ -194,8 +194,8 @@ the full command reference.
 - Unified `decode()` returns every decodable field in one dict
 - Batch mode preserves list length (errors become error-dicts, not
   exceptions)
-- `PipeDecoder` for streams: per-ICAO state, CPR pair accumulation,
-  TTL eviction, DF20/21 ICAO verification via trusted-set promotion
+- `PipeDecoder` for streams: per-ICAO state, global and local CPR position
+  decoding, TTL eviction, DF20/21 ICAO verification via trusted-set promotion
 - `full_dict=True` populates every key in the canonical 123-field
   schema for pandas / parquet workflows
 - `known=` aircraft state disambiguates Comm-B BDS 5,0/6,0 ambiguity
@@ -203,27 +203,6 @@ the full command reference.
 - Type-checked under mypy strict across all source files
 - Golden-file oracle regression test against `pyModeS 2.21.1`
 
-## Performance
-
-Measured on jet1090's
-[`long_flight.csv`](https://github.com/xoolive/jet1090/blob/master/crates/rs1090/data/long_flight.csv)
-(172,432 Beast-format messages, 7 runs × 1 loop, mean timings,
-single-core only):
-
-| Decoder | Wall time | Throughput | vs pyModeS v3 |
-|---|---|---|---|
-| **pyModeS v3 (pure Python)** | **2.06s ± 0.01** | **83,549 msg/s** | **1.00×** |
-| pyModeS 2.21.1 (Python with compiled C) | 5.03s ± 0.01 | 34,303 msg/s | 0.41× |
-| rs1090 (Rust) | 5.60s ± 0.01 | 30,798 msg/s | 0.37× |
-| pyModeS 2.21.1 (Python) | 9.09s ± 0.02 | 18,959 msg/s | 0.23× |
-
-pyModeS v3 is **2.44× faster** than `pyModeS 2.21.1`'s compiled C
-extension, **4.41× faster** than `pyModeS 2.21.1` pure-Python, and
-**2.71× faster** than rs1090's single-core Rust — all while remaining
-pure Python with no C/Cython build.
-
-Reproduce with `scripts/benchmark_decode.py`.
-
 ## Supported messages
 
 - **DF4 / DF20:** altitude code (surveillance altitude reply)

{pymodes-3.3.0 → pymodes-3.5.0}/README.md

@@ -82,7 +82,7 @@ print(r["latitude"], r["longitude"])  # 43.6264..., 1.3747...
 `PipeDecoder` is stateful — it holds per-ICAO state across calls,
 matches CPR pairs automatically, evicts stale aircraft after a
 TTL, and flags DF20/21 messages as `icao_verified` when their
-CRC-derived ICAO was already seen in a clean DF17/18 plaintext.
+CRC-derived ICAO was seen recently in clean DF17/18 plaintext.
 
 ```python
 from pyModeS import PipeDecoder
@@ -90,7 +90,7 @@ from pyModeS import PipeDecoder
 pipe = PipeDecoder(surface_ref="EHAM")
 for msg, timestamp in stream:
     decoded = pipe.decode(msg, timestamp=timestamp)
-    if "latitude" in decoded:
+    if decoded.get("latitude") is not None:
         print(decoded["icao"], decoded["latitude"], decoded["longitude"])
 ```
 
@@ -167,8 +167,8 @@ the full command reference.
 - Unified `decode()` returns every decodable field in one dict
 - Batch mode preserves list length (errors become error-dicts, not
   exceptions)
-- `PipeDecoder` for streams: per-ICAO state, CPR pair accumulation,
-  TTL eviction, DF20/21 ICAO verification via trusted-set promotion
+- `PipeDecoder` for streams: per-ICAO state, global and local CPR position
+  decoding, TTL eviction, DF20/21 ICAO verification via trusted-set promotion
 - `full_dict=True` populates every key in the canonical 123-field
   schema for pandas / parquet workflows
 - `known=` aircraft state disambiguates Comm-B BDS 5,0/6,0 ambiguity
@@ -176,27 +176,6 @@ the full command reference.
 - Type-checked under mypy strict across all source files
 - Golden-file oracle regression test against `pyModeS 2.21.1`
 
-## Performance
-
-Measured on jet1090's
-[`long_flight.csv`](https://github.com/xoolive/jet1090/blob/master/crates/rs1090/data/long_flight.csv)
-(172,432 Beast-format messages, 7 runs × 1 loop, mean timings,
-single-core only):
-
-| Decoder | Wall time | Throughput | vs pyModeS v3 |
-|---|---|---|---|
-| **pyModeS v3 (pure Python)** | **2.06s ± 0.01** | **83,549 msg/s** | **1.00×** |
-| pyModeS 2.21.1 (Python with compiled C) | 5.03s ± 0.01 | 34,303 msg/s | 0.41× |
-| rs1090 (Rust) | 5.60s ± 0.01 | 30,798 msg/s | 0.37× |
-| pyModeS 2.21.1 (Python) | 9.09s ± 0.02 | 18,959 msg/s | 0.23× |
-
-pyModeS v3 is **2.44× faster** than `pyModeS 2.21.1`'s compiled C
-extension, **4.41× faster** than `pyModeS 2.21.1` pure-Python, and
-**2.71× faster** than rs1090's single-core Rust — all while remaining
-pure Python with no C/Cython build.
-
-Reproduce with `scripts/benchmark_decode.py`.
-
 ## Supported messages
 
 - **DF4 / DF20:** altitude code (surveillance altitude reply)

{pymodes-3.3.0 → pymodes-3.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pyModeS"
-version = "3.3.0"
+version = "3.5.0"
 description = "Python Mode-S and ADS-B Decoder"
 authors = [{ name = "Junzi Sun", email = "git@junzis.com" }]
 license = { text = "GNU GPL v3" }
@@ -57,7 +57,6 @@ dev = [
 [tool.ruff]
 target-version = "py311"
 line-length = 88
-extend-exclude = ["scripts/bench_pms_v2.py"]
 
 [tool.ruff.lint]
 select = [

pymodes-3.5.0/scripts/README.md

@@ -0,0 +1,63 @@
+# Maintainer scripts
+
+Run these commands from the repository root. Scripts are grouped by purpose
+through their names; generated benchmark reports live in
+`scripts/benchmark_results/`, and committed input data lives in
+`scripts/data/`.
+
+## Streaming example
+
+- `stream_filtered.py` — configurable high-throughput example using a live
+  Mode-S Beast feed or the committed replay capture. Defaults to DF17/20/21
+  while excluding ADS-B typecodes 28/29/31.
+
+  ```bash
+  uv run python scripts/stream_filtered.py \
+    --network airsquitter.lr.tudelft.nl:10006 > decoded.jsonl
+  ```
+
+## Benchmarks
+
+- `benchmark_mixed_traffic.py` — isolated v2.21.1, released v3.4.0, v3.5.0,
+  and prefiltered v3.5.0 comparison on mixed captured traffic. It measures
+  throughput, immediate streaming position yield, and coordinate agreement.
+  By default it fails below 80% of v2 position yield, below 99% agreement
+  within 100 m, or when v3.5.0 yields fewer positions than v3.4.0. The two
+  percentage floors are configurable through command-line options. Writes
+  `benchmark_results/mixed_traffic.md` by default.
+- `benchmark_pipe.py` — synthetic high-cardinality PipeDecoder comparison.
+  Writes `benchmark_results/pipe.md` by default.
+- `benchmark_cpr.py` — focused CPR primitive comparison against the v2 NumPy
+  implementation.
+
+  ```bash
+  uv run python scripts/benchmark_mixed_traffic.py
+  uv run python scripts/benchmark_pipe.py
+  uv run --with numpy python scripts/benchmark_cpr.py
+  ```
+
+The version-comparison benchmarks build the working tree and resolve released
+versions from PyPI in isolated temporary environments. They require network
+access unless those packages are already cached.
+
+## Generated data
+
+- `generate_airport_db.py` — refreshes `src/pyModeS/data/airports.py` from
+  OurAirports.
+- `generate_v2_fixture.py` — regenerates the committed v2.21.1 golden oracle.
+
+  ```bash
+  uv run python scripts/generate_airport_db.py
+  uv run --no-project --with pyModeS==2.21.1 \
+    python scripts/generate_v2_fixture.py
+  ```
+
+## Release smoke tests
+
+- `smoke_test_wheel.sh [wheel]` — installs a built wheel in a clean virtual
+  environment and checks the public API.
+- `smoke_test_cli.sh` — checks CLI decoding and a short live Beast connection.
+
+The CLI smoke test requires network access. The wheel smoke test accepts an
+explicit wheel path; without one, it builds a fresh working-tree wheel in a
+temporary directory so stale files under `dist/` cannot be selected.

pymodes-3.5.0/scripts/data/README.md

@@ -0,0 +1,37 @@
+# Benchmark capture data
+
+## `airsquitter_2026-07-13_120s.csv.gz`
+
+Two-minute capture from the public TU Delft AirSquitter Mode-S Beast feed:
+
+- Source: `airsquitter.lr.tudelft.nl:10006` (TCP, Beast binary)
+- Capture date: 2026-07-13
+- Frames: 176,612
+- Uncompressed format: `timestamp,hex_message`, one frame per line
+- Compressed size: approximately 2 MiB
+- SHA-256: `fa433eb8dc2ec9cf6ebd40c1bf0a095ae753960d5e8c42ae6da15cc28eb3baea`
+
+The timestamps are reconstructed by `pyModeS.cli._source.NetworkSource`
+from the Beast MLAT counter, anchored to wall-clock time when the TCP data
+is received. Mode A/C and Beast status frames are omitted because
+`NetworkSource` yields only Mode-S short and long frames.
+
+Capture and compact conversion:
+
+```bash
+timeout --signal=INT 120 \
+  modes live \
+  --network airsquitter.lr.tudelft.nl:10006 \
+  --quiet \
+  --dump-to /tmp/pymodes-live-mixed.jsonl
+
+jq -r '"\(.timestamp),\(.raw_msg)"' \
+  /tmp/pymodes-live-mixed.jsonl \
+  | gzip -9 > scripts/data/airsquitter_2026-07-13_120s.csv.gz
+```
+
+Run the associated isolated three-version benchmark with:
+
+```bash
+python scripts/benchmark_mixed_traffic.py
+```

{pymodes-3.3.0 → pymodes-3.5.0}/src/pyModeS/__init__.py

@@ -18,9 +18,9 @@ and are NOT interchangeable:
   45 NM of the true position — tighter than the airborne tolerance.
 
 For streaming workflows where references aren't known ahead of time,
-use :class:`PipeDecoder` which resolves airborne positions from
-even/odd CPR pairs without any reference and still accepts
-``surface_ref`` for surface messages.
+use :class:`PipeDecoder`. It establishes airborne positions from even/odd CPR
+pairs, then resolves later frames locally from the last validated airborne
+position. The optional ``surface_ref`` applies only to surface messages.
 """
 
 from importlib.metadata import version as _version