pyModeS 2.22.0__tar.gz → 3.1.0__tar.gz

{pymodes-2.22.0 → pymodes-3.1.0}/.gitignore

@@ -6,8 +6,12 @@ tmp/
 __pycache__/
 *.py[cod]
 .pytest_cache/
+.ipynb_checkpoints/
+.mypy_cache/
 
-# C extensions
+# Compiled extensions (none shipped in v3 — pure Python — but
+# left in place to keep `.so` build artifacts out of accidental
+# commits if a user experiments locally).
 *.so
 
 # Distribution / packaging
@@ -55,6 +59,9 @@ coverage.xml
 # Sphinx documentation
 docs/_build/
 
+# MkDocs build output
+site/
+
 # Local planning / brainstorming artifacts (not for publication)
 docs/superpowers/
 

pymodes-3.1.0/CHANGELOG.md

@@ -0,0 +1,182 @@
+# Changelog
+
+All notable changes to pyModeS are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/).
+
+## [3.1.0] — 2026-04-14
+
+Streaming-robustness release. Adds defences against phantom positions
+at stream start and against DF20 CRC-collisions that attribute a
+Comm-B reply to the wrong aircraft.
+
+### Added
+
+- `PipeDecoder` bootstrap cluster analysis for the first position per
+  ICAO: lat/lon emission is held until `_BOOTSTRAP_K` (=5) candidate
+  positions agree under a motion-consistency check. Prevents an
+  initial phantom frame from anchoring the rolling position history.
+  On each decode the held result dicts are retro-filled once the
+  cluster locks. Scattered buffers reset and accumulate a fresh K.
+- `PipeDecoder` altitude cross-check on DF20 Comm-B replies: if the
+  message's 13-bit AC-code disagrees with the ICAO's most recent
+  ADS-B-derived altitude by more than `_altitude_tolerance(dt)` (a
+  linearly growing, floored-and-capped window), the BDS payload
+  fields are stripped from the result and `altitude_mismatch=True`
+  is set. The AC-code altitude itself is preserved so callers see
+  why the frame was flagged.
+- `PipeDecoder.flush()` method that finalizes any still-bootstrapping
+  ICAOs, running cluster analysis on whatever candidates are
+  buffered (even if fewer than K) and retro-filling the held result
+  dicts in place. Automatically called by the batch `decode(list,
+  timestamps=list)` code path so batch callers get positions on
+  every resolvable frame without having to reach into `PipeDecoder`.
+- Four new per-stream stat counters exposed via `PipeDecoder.stats`:
+  `altitude_mismatch`, `position_rejected`, `bootstrap_held`,
+  `bootstrap_reset`.
+
+### Fixed
+
+- `airborne_position_pair` now rejects CPR pairs that resolve to
+  physically impossible latitudes (|lat| > 90). Root cause: real
+  DF17 pairs that straddle a `cprNL` zone can slip past the
+  pre-pair `NL(lat_even) == NL(lat_odd)` check and produce
+  latitudes in [90, 270) — e.g. an aircraft climbing out of
+  Amsterdam resolving to lat ≈ 113° over the Bering Sea.
+  Observed on ICAO 485A33 (2025-04-13 19:34:57 UTC, OpenSky feed)
+  — now rejected at the pair resolver rather than poisoning the
+  receiver's position history.
+
+## [3.0.0] — 2026-04-13
+
+First release of the v3 ground-up rewrite. **Not backwards-compatible
+with pyModeS 2.x.** See [docs/migration.md](./docs/migration.md) for
+the migration guide.
+
+### Added
+
+- Unified `pyModeS.decode(msg)` returning a `Decoded` dict with every
+  decodable field populated in one call
+- Batch mode: `pyModeS.decode(list_of_msgs, timestamps=...)` preserves
+  list length (errors become error-dicts, no exceptions)
+- `PipeDecoder` stateful streaming decoder — per-ICAO state, CPR
+  pair matching, TTL eviction, DF20/21 `icao_verified` promotion
+  via a trusted-ICAO set populated from clean DF17/18 plain-text
+  addresses
+- `full_dict=True` mode populates every key in the canonical
+  `_FULL_SCHEMA` (123 fields) for pandas / parquet workflows
+- Phase 3 BDS 5,0/6,0 disambiguation via a `known=` aircraft-state
+  kwarg — scores candidate registers against recent observations
+  of groundspeed / track / heading / mach
+- Airport database (ICAO 4-letter codes only) for surface CPR
+  resolution: `surface_ref="EHAM"` or `surface_ref=(lat, lon)`
+- Golden-file oracle test asserting v3 decode output matches
+  `pyModeS 2.21.1` on the deduplicated `tests/data/` corpus
+- MkDocs + Material documentation site (docs added in a later
+  commit in this release)
+- `modes` command-line tool with two subcommands:
+  - `modes decode MESSAGE` for one-shot decoding (pretty or compact JSON)
+  - `modes decode HEX1,HEX2,HEX3` inline-batch mode with shared
+    `PipeDecoder` state so CPR pairs resolve across the batch
+  - `modes decode --file PATH` for batch decoding a file of hex messages
+  - `modes live --network HOST:PORT` for streaming TCP decode from a
+    Mode-S Beast binary feed
+  - `modes live --dump-to FILE` to tee JSON lines to a file. Every
+    record carries both `raw_msg` (source hex) and `timestamp` so
+    captures are self-contained for offline re-decode
+  - `modes live --tui` for an interactive textual-based aircraft
+    table: keyboard navigation (j/k/g/G), live incremental search
+    (/) that filters as you type, sort cycling (s/r), responsive
+    column set (7/10/18 cols by terminal width). Requires
+    `pyModeS[tui]` extra
+  - Graceful SIGINT/SIGTERM shutdown with final stats line
+- `pyModeS[tui]` optional dependency: pulls in `textual>=0.50` for
+  the `modes live --tui` interactive display
+- `pyModeS.util` public helpers for low-level message inspection
+  without going through `decode()`: `hex2bin`, `bin2int`, `hex2int`,
+  `bin2hex`, `crc`, `df`, `icao`, `typecode`, `altcode`, `idcode`,
+  `cprNL`. Thin wrappers over `_bits.py`, `_altcode.py`, `_idcode.py`
+  and `position/_cpr.py` with no logic duplication.
+- `scripts/smoke_test.sh` installs the freshly built wheel
+  into a clean Python 3.12 venv and verifies the public API end to
+  end (single-message decode, PipeDecoder baseline, batch CPR pair
+  resolution, error-dict on malformed input).
+
+### Fixed
+
+- BDS 5,0 / 6,0 Phase 3 disambiguation now works cold-start
+  (before any ADS-B frames have been cached). Root cause: the
+  BDS 5,0 validator accepted roll angles up to ±50°, the full
+  wire-format range, which let garbage payloads pass as track-
+  and-turn reports with physically implausible banks (observed
+  -44.5° on the TU Delft live feed). Tightened the gate to ±35°
+  to match the commercial-airliner envelope (30° bank ≈ 2 G in
+  a level turn).
+- BDS 6,0 scorer now has reference fields to compare against in
+  streaming mode. PipeDecoder derives approximate IAS and mach
+  from cached groundspeed + altitude using a minimal ISA
+  calculator in `pyModeS._aero` (zero-wind assumption, low-Mach
+  approximation). Previously the scorer's `ias`/`mach` slots
+  were always empty because ADS-B airborne-velocity BDS 0,9
+  subtypes 1/2 only emit groundspeed; the scorer then returned
+  +inf for every BDS 6,0 candidate and BDS 5,0 won every tie by
+  default.
+- `modes live --tui` no longer freezes after the "detected beast
+  format, resyncing" line. Two root causes: the sink context
+  manager wrap was lost during a prior refactor, and the
+  on-detect stderr print corrupted rich's alt-screen buffer.
+  Restored the context manager and silenced stderr writes when
+  `--tui` is active.
+
+### Changed
+
+- Python 3.11+ minimum (was 3.9+)
+- Internal message representation: Python int + bit-shift primitives
+  (replaces numpy arrays and the Cython `c_common` extension)
+- Every decoder API → single `decode()` returning a dict
+- CPR resolution API: `reference=(lat, lon)` for airborne (180 NM
+  tolerance), `surface_ref="EHAM"` or `surface_ref=(lat, lon)` for
+  surface (45 NM tolerance)
+- Header fields (df, icao, crc, crc_valid) are computed eagerly in
+  `Message.__init__` as plain attributes instead of lazy
+  `cached_property` descriptors (measurably faster on streaming
+  decode loads, and the decode path always needs all four)
+- `modes live` stream timestamps are now derived from the Beast
+  48-bit MLAT counter, with tick rate auto-calibrated per
+  connection from the delta between consecutive `recv()` bursts.
+  Works on both dump1090 12 MHz counters and radarcape / GNS
+  AirSquitter 1 GHz nanosecond counters without configuration.
+  Gives sub-microsecond per-frame precision across a TCP burst,
+  replacing the previous coarse one-`time.time()`-per-batch
+  behaviour.
+- `modes live --tui` rewritten from `rich.live.Live` to a full
+  `textual.App` with a `DataTable` widget, interactive row
+  cursor, and per-tick diff-update of cells (only cells whose
+  values actually change are redrawn). The result: no more
+  whole-table flashing on each message, cursor and scroll
+  position survive refreshes, and the default sort is now by
+  ICAO address (stable) instead of last-seen (reshuffles on
+  every message). Dropped the `rich>=13.0` dependency in favour
+  of `textual>=0.50`.
+
+### Removed
+
+- Cython extension (`c_common`) and its build dependency
+- Legacy function-per-field API (`pms.adsb.callsign`, `pms.commb.bds`,
+  `pms.common.hex2bin`, etc.). The former submodule paths
+  (`pyModeS.adsb`, `pyModeS.commb`, `pyModeS.ehs`, `pyModeS.els`,
+  `pyModeS.common`, `pyModeS.bds`, `pyModeS.streamer`,
+  `pyModeS.extra`) are intercepted by a meta-path finder in
+  `pyModeS._v2_removed` which raises `V2APIRemovedError` (an
+  `ImportError` subclass) with a migration hint pointing at
+  `pyModeS.decode()` and — for `pyModeS.common` specifically —
+  at the restored helpers in `pyModeS.util`. Both `from pyModeS.X
+  import Y` and bare attribute access (`pms.X`) hit the same
+  error message.
+- numpy hard dependency.
+- `modeslive` CLI entry point: replaced by `modes live` / `modes
+  decode`. The v3 package still registers `modeslive` as a console
+  script — running it prints a migration hint to stderr and exits
+  with code 2, rather than vanishing into a bare `command not
+  found` after a `pip install -U pyModeS`.

pymodes-3.1.0/PKG-INFO

@@ -0,0 +1,309 @@
+Metadata-Version: 2.4
+Name: pyModeS
+Version: 3.1.0
+Summary: Python Mode-S and ADS-B Decoder
+Project-URL: homepage, https://mode-s.org
+Project-URL: repository, https://github.com/junzis/pyModeS
+Project-URL: issues, https://github.com/junzis/pyModeS/issues
+Author-email: Junzi Sun <git@junzis.com>
+License: GNU GPL v3
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Provides-Extra: tui
+Requires-Dist: textual>=0.50; extra == 'tui'
+Description-Content-Type: text/markdown
+
+# pyModeS
+
+Fast decoder for Mode-S and ADS-B messages in Python. Ground-up v3 rewrite of `pyModeS` with a single unified `decode()` function.
+
+[![license](https://img.shields.io/badge/license-GPL--3.0-blue)](https://github.com/junzis/pyModeS/blob/main/LICENSE)
+
+## Install
+
+```sh
+pip install "pyModeS>=3"
+```
+
+Python 3.11+ required.
+
+## Usage examples
+
+### Single-message decode
+
+The `decode()` function returns a `Decoded` dict with every decodable
+field populated in one pass.
+
+```python
+from pyModeS import decode
+
+result = decode("8D406B902015A678D4D220AA4BDA")
+print(result)
+# {
+#     'df': 17,
+#     'icao': '406B90',
+#     'crc_valid': True,
+#     'typecode': 4,
+#     'bds': '0,8',
+#     'callsign': 'EZY85MH',
+#     'category': 0,
+#     'wake_vortex': 'No category information',
+# }
+```
+
+### Batch decode (mixed message types)
+
+Pass a list of hex strings and parallel timestamps. Any mix of
+downlink formats, typecodes, and Comm-B registers is fine —
+the dispatcher routes each message to the right decoder and
+uses the timestamps to resolve CPR pairs and disambiguate
+ambiguous Comm-B registers.
+
+```python
+from pyModeS import decode
+
+results = decode(
+    [
+        "8D406B902015A678D4D220AA4BDA",  # DF17 BDS 0,8 identification
+        "8D485020994409940838175B284F",  # DF17 BDS 0,9 airborne velocity
+        "8D40058B58C901375147EFD09357",  # DF17 BDS 0,5 airborne pos (even)
+        "8D40058B58C904A87F402D3B8C59",  # DF17 BDS 0,5 airborne pos (odd)
+        "A000178D10010080F50000D5893C",  # DF20 BDS 1,0 data link capability
+        "A8000D9FA55A032DBFFC000D8123",  # DF21 BDS 6,0 heading & speed
+    ],
+    timestamps=[1.0, 2.0, 3.0, 4.0, 5.0, 6.0],
+)
+assert results[0]["callsign"] == "EZY85MH"
+assert results[1]["groundspeed"] == 159
+assert results[3]["latitude"] is not None  # CPR pair resolved
+```
+
+### Surface position with airport reference
+
+Surface CPR needs a reference within ~45 NM. Pass an ICAO airport
+code (looked up in the shipped airport database) or an explicit
+`(lat, lon)` tuple.
+
+```python
+from pyModeS import decode
+
+# Real DF18 surface movement on LFBO (Toulouse-Blagnac).
+r = decode("903a23ff426a4e65f7487a775d17", surface_ref="LFBO")
+print(r["latitude"], r["longitude"])  # 43.6264..., 1.3747...
+```
+
+### Streaming decoder
+
+`PipeDecoder` is stateful — it holds per-ICAO state across calls,
+matches CPR pairs automatically, evicts stale aircraft after a
+TTL, and flags DF20/21 messages as `icao_verified` when their
+CRC-derived ICAO was already seen in a clean DF17/18 plaintext.
+
+```python
+from pyModeS import PipeDecoder
+
+pipe = PipeDecoder(surface_ref="EHAM")
+for msg, timestamp in stream:
+    decoded = pipe.decode(msg, timestamp=timestamp)
+    if "latitude" in decoded:
+        print(decoded["icao"], decoded["latitude"], decoded["longitude"])
+```
+
+See [`docs/quickstart.md`](./docs/quickstart.md) for the full tour
+(full-dict mode, error handling, attribute access).
+
+### Low-level helpers
+
+For ad-hoc message inspection that doesn't need a full decode,
+`pyModeS.util` exposes thin wrappers around the bit/hex/CRC
+primitives: `hex2bin`, `bin2int`, `hex2int`, `bin2hex`, `crc`,
+`df`, `icao`, `typecode`, `altcode`, `idcode`, `cprNL`.
+
+```python
+from pyModeS.util import hex2bin, crc, icao, typecode
+
+msg = "8D406B902015A678D4D220AA4BDA"
+hex2bin(msg)[:16]   # '1000110101000000'
+crc(msg)            # 0 — valid DF17
+icao(msg)           # '406B90'
+typecode(msg)       # 4 — ADS-B identification
+```
+
+## CLI
+
+pyModeS ships with a `modes` command-line tool for ad-hoc decoding
+and live streaming.
+
+### `modes decode` — one-shot and file mode
+
+```sh
+# Decode one hex message (pretty-printed JSON)
+modes decode 8D406B902015A678D4D220AA4BDA
+
+# Decode several messages inline — comma-separated, emits JSON lines
+modes decode 8D40058B58C901375147EFD09357,8D40058B58C904A87F402D3B8C59,8D406B902015A678D4D220AA4BDA
+
+# With airborne CPR reference (single message only)
+modes decode 8D40058B58C901375147EFD09357 --reference 49.0 6.0
+
+# Compact JSON piped to jq
+modes decode 8D406B902015A678D4D220AA4BDA --compact | jq .
+
+# Decode a file of hex messages (one per line OR timestamp,hex CSV)
+modes decode --file captures/flight.log
+
+# Stdin + surface CPR
+cat taxi.log | modes decode --file - --surface-ref LFBO
+```
+
+### `modes live` — streaming TCP source
+
+```sh
+# Stream decoded JSON lines from a dump1090-style beast feed
+modes live --network localhost:30005
+
+# Tee output to a file
+modes live --network host:30005 --dump-to flight.jsonl
+
+# Stream from the TU Delft public feed (live aircraft over Europe)
+modes live --network airsquitter.lr.tudelft.nl:10006
+
+# Interactive live aircraft table (requires pyModeS[tui] extra)
+pip install "pyModeS[tui]"
+modes live --network host:30005 --tui
+```
+
+Mode-S Beast binary format is supported (dump1090 port 30005 and
+equivalents). See [`docs/quickstart.md`](./docs/quickstart.md) for
+the full command reference.
+
+## Features
+
+- Unified `decode()` returns every decodable field in one dict
+- Batch mode preserves list length (errors become error-dicts, not
+  exceptions)
+- `PipeDecoder` for streams: per-ICAO state, CPR pair accumulation,
+  TTL eviction, DF20/21 ICAO verification via trusted-set promotion
+- `full_dict=True` populates every key in the canonical 123-field
+  schema for pandas / parquet workflows
+- `known=` aircraft state disambiguates Comm-B BDS 5,0/6,0 ambiguity
+- Airport ICAO database for surface CPR resolution (`surface_ref="EHAM"`)
+- Type-checked under mypy strict across all source files
+- Golden-file oracle regression test against `pyModeS 2.21.1`
+
+## Performance
+
+Measured on jet1090's
+[`long_flight.csv`](https://github.com/xoolive/jet1090/blob/master/crates/rs1090/data/long_flight.csv)
+(172,432 Beast-format messages, 7 runs × 1 loop, mean timings,
+single-core only):
+
+| Decoder | Wall time | Throughput | vs pyModeS v3 |
+|---|---|---|---|
+| **pyModeS v3 (pure Python)** | **2.06s ± 0.01** | **83,549 msg/s** | **1.00×** |
+| pyModeS 2.21.1 (Python with compiled C) | 5.03s ± 0.01 | 34,303 msg/s | 0.41× |
+| rs1090 (Rust) | 5.60s ± 0.01 | 30,798 msg/s | 0.37× |
+| pyModeS 2.21.1 (Python) | 9.09s ± 0.02 | 18,959 msg/s | 0.23× |
+
+pyModeS v3 is **2.44× faster** than `pyModeS 2.21.1`'s compiled C
+extension, **4.41× faster** than `pyModeS 2.21.1` pure-Python, and
+**2.71× faster** than rs1090's single-core Rust — all while remaining
+pure Python with no C/Cython build.
+
+Reproduce with `scripts/benchmark_decode.py`.
+
+## Supported messages
+
+- **DF4 / DF20:** altitude code (surveillance altitude reply)
+- **DF5 / DF21:** identity code (squawk)
+- **DF11:** all-call reply (partial — II/SI decoding deferred)
+- **DF17 / DF18 ADS-B:**
+  - TC 1-4 (BDS 0,8): identification + category
+  - TC 5-8 (BDS 0,6): surface position
+  - TC 9-18 (BDS 0,5): airborne position (barometric altitude)
+  - TC 19 (BDS 0,9): airborne velocity (all 4 subtypes)
+  - TC 20-22 (BDS 0,5): airborne position (GNSS altitude)
+  - TC 28 (BDS 6,1): aircraft status
+  - TC 29 (BDS 6,2): target state and status
+  - TC 31 (BDS 6,5): operational status
+- **DF20 / DF21 Comm-B:**
+  - BDS 1,0: data link capability
+  - BDS 1,7: common-usage GICB capability
+  - BDS 2,0: aircraft identification
+  - BDS 3,0: ACAS active resolution advisory
+  - BDS 4,0: selected vertical intention
+  - BDS 4,4: meteorological routine air report
+  - BDS 4,5: meteorological hazard report
+  - BDS 5,0: track and turn report
+  - BDS 6,0: heading and speed report
+
+## Migrating from pyModeS 2.x
+
+pyModeS 3 is **not backwards-compatible** with pyModeS 2.x. The
+function-per-field API (`pms.adsb.callsign(msg)`, ...) is replaced
+by a single `decode()` returning a dict. See the [migration
+guide](https://github.com/junzis/pyModeS/blob/main/docs/migration.md) for the full equivalence table.
+
+If you aren't ready to migrate:
+
+```sh
+pip install "pyModeS<3"
+```
+
+v3 replaces v2 on the same PyPI slot and import name, so
+`pip install -U pyModeS` upgrades existing v2 users to v3
+(and triggers the v2-API removal shims — see the migration
+guide). Pinning `<3` keeps you on the 2.x line.
+
+## Documentation
+
+The full docs live under `docs/` and are published via MkDocs + Material.
+To rebuild locally:
+
+```sh
+# One-shot build (strict mode fails on warnings). Output → site/
+uv run --with mkdocs-material --with mkdocs-include-markdown-plugin --with "mkdocstrings[python]" mkdocs build --clean --strict
+
+# Live-reload dev server at http://127.0.0.1:8000
+uv run --with mkdocs-material --with mkdocs-include-markdown-plugin --with "mkdocstrings[python]" mkdocs serve
+```
+
+## Links
+
+- Source: https://github.com/junzis/pyModeS
+- Changelog: [CHANGELOG.md](https://github.com/junzis/pyModeS/blob/main/CHANGELOG.md)
+- Issues: https://github.com/junzis/pyModeS/issues
+- License: GPL-3.0 (see `LICENSE`)
+
+## Attribution
+
+pyModeS is a project created by Junzi Sun, who works at TU Delft,
+[Aerospace Engineering Faculty](https://tudelft.nl/en/ae/). It is
+supported by many
+[contributors](https://github.com/junzis/pyModeS/graphs/contributors)
+from different institutions.
+
+If you use pyModeS in academic work, please cite:
+
+```bibtex
+@article{sun2019pyModeS,
+    author={J. {Sun} and H. {V\^u} and J. {Ellerbroek} and J. M. {Hoekstra}},
+    journal={IEEE Transactions on Intelligent Transportation Systems},
+    title={pyModeS: Decoding Mode-S Surveillance Data for Open Air Transportation Research},
+    year={2019},
+    doi={10.1109/TITS.2019.2914770},
+    ISSN={1524-9050},
+}
+```