telemetry-cli 1.0.0__tar.gz

telemetry_cli-1.0.0/.gitignore

@@ -0,0 +1,160 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/

telemetry_cli-1.0.0/CHANGELOG.md

@@ -0,0 +1,156 @@
+# Changelog
+
+## 1.0.0 (2026-09-28)
+
+The first release.
+
+`telemetry-cli` brings four Perl tools into one Python package: `pick`, `recs`,
+`recl` and `tgen`. The originals, with their git history, are in `legacy/`.
+
+The command-line interfaces are unchanged. Output is identical to the
+originals' except for the fixes listed here. Each fix is named by the golden
+test cases that check it (`fix = "..."` in `tests/golden/*/cases.*`).
+
+### pick
+
+pick is ported. It passes about 2,200 golden cases that compare its output
+byte for byte with the Perl original's.
+
+**Fixes**
+
+- **Byte order** (`pick byte order`). pick was written on big-endian machines.
+  On little-endian machines, which is nearly all of them today, the Perl
+  version:
+  - got bit fields wrong (`u:0:8d` printed the wrong byte);
+  - printed `x`, `o` and `b` output, and `c`/`S`/`A` output of multi-byte
+    types, in memory order rather than value order (`0x03915ed3` as `ux`
+    printed `d3 5e 91 03`).
+
+  pick now behaves the same on every machine: it prints each value as the
+  Perl did on a big-endian machine. `-r` and the new `order=` parameter say
+  how the data are stored.
+- **`-r` on complex numbers** (`pick complex -r`) reversed all 8 (or 16) bytes
+  at once, which also swapped the real and imaginary parts. Now each part is
+  reversed on its own.
+- **`g` on complex types** (`pick complex g`) is documented as meaning `RI`,
+  but the Perl rejected it. It now prints the real and imaginary parts.
+- **`-l` and `-n`** did the reverse of what the documentation says: `-l`
+  printed the record number and `-n` the line count. They now match the
+  documentation: `-l` prints the line number and `-n` the record number.
+- **Errors go to stderr, with a nonzero exit status.** The Perl printed "Matched
+  neither a move nor a data request" to stdout and exited 0. It also dumped
+  the help text to stdout before other errors, and ignored a `file=` that
+  couldn't be opened (`pick missing file`).
+- **Endless loops are errors** (`pick endless loops`): `every=0`, a record
+  size of 0, a symbol defined in terms of itself, and some malformed groups
+  made the Perl run forever.
+- **Help text** (`pick help text`):
+  - The format list said `'x' octal`; it is `'o'`.
+  - The example using awk lost its `$1` and `$2` (Perl expanded them as
+    variables) and its line continuations.
+  - The version text is updated, and the pointer to `perldoc` is replaced
+    by a link to the manual.
+  - Several examples miscounted items, which count from 0: `z60-2` is the
+    61st complex number, not the 60th; `w5` and `b5` are the sixth short and
+    char, not the fifth; `Z20` is the 21st double complex; `f500` is the
+    501st float.
+  - The group example `20[ f +96f i -100f ]` was off by three floats: it
+    doesn't pair each float with the int 400 bytes on. That's
+    `20[ f +99f i -100f ]`.
+- **`U` output of bit fields** writes the masked value's bytes. The Perl wrote
+  the wrong end of them.
+
+**New**
+
+- `order=big|little|native` gives the data's byte order. The default,
+  `native`, is the machine's order, as before.
+- `pick --help [topic]` works like `pick ?topic`, which is handy in zsh, where
+  an unquoted `?p` is a filename pattern.
+- The symbol tables from the original distribution (`moc`, `aux`, `corr`,
+  `subcom`, `hw`, `airmoc`, ...) are bundled. `sym=moc` finds them when the
+  file isn't in the current directory or in `$PICKPATH`.
+
+### recs
+
+recs is ported. It passes about 700 golden cases (every way of finding
+records, crossed with its options, on text, binary and 40 kB inputs) and the
+Perl script's own `-test` suite, ported to pytest.
+
+**Fixes**
+
+- **Padding** (`recs null fill`) uses null bytes by default, as documented.
+  The Perl padded records with the character `0`.
+- **Warnings go to stderr.** The Perl wrote them to stdout, in the middle of
+  the extracted records, with its own source line numbers.
+- **Command-line errors** (`recs options`, `recs arguments`) stop recs with a
+  message and a nonzero status. The Perl warned about a bad option and
+  carried on without it, and with a missing marker or a non-numeric length it
+  quietly wrote nothing.
+- **Endless loops are errors** (`recs endless loops`): padding with an empty
+  fill string, and markers that match without consuming anything (such as a
+  record length of 0 without `-x`), made the Perl run forever.
+- `-test` ran the Perl script's built-in tests; it now says they have moved
+  into the test suite.
+
+Regexes (`-r`) are now Python's rather than Perl's. They agree on the kinds of
+patterns recs is used with.
+
+### tgen
+
+tgen is ported. Its command files are now Python, and `tgen --convert`
+translates Perl command files. The eight example command files from the
+original, and further test files covering the rest of the language, give the
+same output byte for byte as the Perl tgen did with the originals. That
+includes random output: both versions use Perl's random number generator
+(drand48), so a seeded run matches the Perl run with the same `srand` seed.
+Random tests generate thousands of Perl command files, convert them, and
+compare the two versions' output.
+
+**Changes**
+
+- **Command files are Python.** The layout is the same (five `#`-separated
+  fields, comments, continuation lines); the code in the fields is Python.
+  `$::I` is `I`, `$P{name}{value}` is `P.name.value`, and helpers give Perl's
+  `%`, bit operators and `.` where the behavior differs from Python's.
+- **`tgen --convert`** translates Perl command files, and marks lines it can't
+  translate with `# TODO(convert)`.
+- **Errors in a field stop tgen** with the file, line and field. The Perl
+  silently used the field's source text as its value, which usually packed as
+  0.
+- **`-seed=n`** (new) makes random output repeatable.
+- The file helpers open each file once, rather than once per frame.
+- A bad option is an error; the Perl warned and carried on.
+
+### recl
+
+recl is ported, and its measurement is fixed, so its results differ from the
+Perl version's (which were usually wrong). The command line and output lines
+are the same; about 100 golden cases compare everything but the scores with
+the Perl. Whether recl finds the right length is tested on files of known
+record length made with tgen, including random lengths up to 700 bytes,
+headers, partial last records and `-fact`.
+
+**Fixes**
+
+- **The measurement** counted agreeing bits in only the first eighth of each
+  buffer. On a file of 48-byte records, the Perl answered 3. recl now counts
+  the bytes equal to the byte one record length on, over the whole buffer,
+  and reports the smallest of a length and its multiples when they score
+  alike (they always nearly do). Counting bytes rather than bits needs much
+  less data, and isn't pulled toward multiples of the length by counters. So
+  `CORR` is now a percentage of equal bytes (about 0.4 for random data),
+  not of agreeing bits (about 50).
+- **`-full`, `-limit` and `-reduce`** (`recl options`) were accepted but
+  ignored. They now work.
+- **`-only`** sized buffers from the last length listed rather than the
+  longest, so longer lengths were never scored.
+- **The `-verbose` table** showed fractions labeled as percentages.
+- **stdin** (`recl stdin`) required `-max`, which was then taken as the file
+  size. stdin now works like a file.
+- **Short data** (`recl short data`): data shorter than one buffer made the
+  Perl divide by zero; it is now compared as one piece.
+- **No lengths to check** (`recl endless loops`), e.g. `-min=90 -max=80`,
+  made the Perl loop forever; it is now an error. So are a bad option and an
+  unreadable `-only` list.
+
+It is also much faster.

telemetry_cli-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2000-2026 David A. Imel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

telemetry_cli-1.0.0/PKG-INFO

@@ -0,0 +1,106 @@
+Metadata-Version: 2.5
+Name: telemetry-cli
+Version: 1.0.0
+Summary: Command-line tools for binary telemetry: decode, extract, measure and generate records
+Project-URL: Repository, https://github.com/donalgrant/telemetry-cli
+Project-URL: Documentation, https://github.com/donalgrant/telemetry-cli/tree/main/docs
+Project-URL: Changelog, https://github.com/donalgrant/telemetry-cli/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/donalgrant/telemetry-cli/issues
+Author: David A. Imel
+License-Expression: MIT
+License-File: LICENSE
+Keywords: binary,cli,decoding,pcm,records,telemetry
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.23
+Provides-Extra: dev
+Requires-Dist: hypothesis>=6; extra == 'dev'
+Requires-Dist: pytest-cov>=4; extra == 'dev'
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: tomli>=2; (python_version < '3.11') and extra == 'dev'
+Description-Content-Type: text/markdown
+
+# telemetry-cli
+
+Command-line tools for binary telemetry, built for Unix pipelines:
+
+- **pick** decodes fields from fixed-length binary records: integers, floats,
+  complex numbers, characters and bit fields, in any byte order
+  ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/pick.md)).
+- **recs** extracts records from a byte stream by their start (and end)
+  markers ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/recs.md)).
+- **recl** estimates the record length of a binary file
+  ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/recl.md)).
+- **tgen** generates synthetic telemetry from a command file describing each
+  record ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/tgen.md)).
+
+```console
+$ printf '\x00\x00\x00\x01\xff\xfeHi\x00\x00\x00\x02\x00\x07ok' > demo.bin
+$ pick demo.bin 8 -q order=big u s 2S
+1 -2 Hi
+2 7 ok
+$ pick demo.bin 8 -q order=big -n u0x w2b
+0 00 00 00 01  1111 1111 1111 1110
+1 00 00 00 02  0000 0000 0000 0111
+```
+
+Together: tgen makes a stream, recs pulls the frames out of it, recl finds
+their length, and pick decodes them. See
+[Using the tools together](https://github.com/donalgrant/telemetry-cli/blob/main/docs/pipelines.md).
+
+## Install
+
+Requires Python 3.10 or later. The package is `telemetry-cli` on PyPI, and is
+best installed as an isolated command-line tool:
+
+```text
+pipx install telemetry-cli
+# or
+uv tool install telemetry-cli
+```
+
+Either installs the commands `pick`, `recs`, `recl` and `tgen`. You can also
+`pip install telemetry-cli` in a virtual environment.
+
+## History
+
+These are Python ports of tools written at JPL. pick and recl began together
+as C++ tools, for a project whose telemetry arrived with no documentation of
+its structure: recl to find the length of the records, and pick to take them
+apart and work out what was in them. (A bit reminiscent of the film
+*Contact*.) Both were later rewritten in Perl (pick by 2000), and tgen (2003)
+and recs joined them.
+
+The command lines are the same, and so is the output, apart from fixes listed
+in the [CHANGELOG](https://github.com/donalgrant/telemetry-cli/blob/main/CHANGELOG.md):
+chiefly pick's byte-order handling on little-endian machines, and recl's
+measurement. tgen's command files are now Python; `tgen --convert` translates
+the old Perl ones. The Perl originals, with their history, are in
+[legacy/](https://github.com/donalgrant/telemetry-cli/tree/main/legacy).
+
+## Development
+
+```text
+python -m venv .venv && .venv/bin/pip install -e '.[dev]'
+.venv/bin/pytest
+```
+
+The tests compare each tool with its Perl original. `tests/golden/` holds the
+Perl's output for large tables of command lines (about 2,200 for pick, 700
+for recs), and the tests check that the Python version reproduces it byte for
+byte. Where it deliberately differs, the case names the CHANGELOG entry that
+explains why. With perl installed, further tests run random command lines
+through both versions and compare them (`HYPOTHESIS_PROFILE=deep pytest` for
+a longer search), and `python tools/make_golden.py --check` confirms the saved
+output is current. The examples in the documentation run as tests too.

telemetry_cli-1.0.0/README.md

@@ -0,0 +1,73 @@
+# telemetry-cli
+
+Command-line tools for binary telemetry, built for Unix pipelines:
+
+- **pick** decodes fields from fixed-length binary records: integers, floats,
+  complex numbers, characters and bit fields, in any byte order
+  ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/pick.md)).
+- **recs** extracts records from a byte stream by their start (and end)
+  markers ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/recs.md)).
+- **recl** estimates the record length of a binary file
+  ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/recl.md)).
+- **tgen** generates synthetic telemetry from a command file describing each
+  record ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/tgen.md)).
+
+```console
+$ printf '\x00\x00\x00\x01\xff\xfeHi\x00\x00\x00\x02\x00\x07ok' > demo.bin
+$ pick demo.bin 8 -q order=big u s 2S
+1 -2 Hi
+2 7 ok
+$ pick demo.bin 8 -q order=big -n u0x w2b
+0 00 00 00 01  1111 1111 1111 1110
+1 00 00 00 02  0000 0000 0000 0111
+```
+
+Together: tgen makes a stream, recs pulls the frames out of it, recl finds
+their length, and pick decodes them. See
+[Using the tools together](https://github.com/donalgrant/telemetry-cli/blob/main/docs/pipelines.md).
+
+## Install
+
+Requires Python 3.10 or later. The package is `telemetry-cli` on PyPI, and is
+best installed as an isolated command-line tool:
+
+```text
+pipx install telemetry-cli
+# or
+uv tool install telemetry-cli
+```
+
+Either installs the commands `pick`, `recs`, `recl` and `tgen`. You can also
+`pip install telemetry-cli` in a virtual environment.
+
+## History
+
+These are Python ports of tools written at JPL. pick and recl began together
+as C++ tools, for a project whose telemetry arrived with no documentation of
+its structure: recl to find the length of the records, and pick to take them
+apart and work out what was in them. (A bit reminiscent of the film
+*Contact*.) Both were later rewritten in Perl (pick by 2000), and tgen (2003)
+and recs joined them.
+
+The command lines are the same, and so is the output, apart from fixes listed
+in the [CHANGELOG](https://github.com/donalgrant/telemetry-cli/blob/main/CHANGELOG.md):
+chiefly pick's byte-order handling on little-endian machines, and recl's
+measurement. tgen's command files are now Python; `tgen --convert` translates
+the old Perl ones. The Perl originals, with their history, are in
+[legacy/](https://github.com/donalgrant/telemetry-cli/tree/main/legacy).
+
+## Development
+
+```text
+python -m venv .venv && .venv/bin/pip install -e '.[dev]'
+.venv/bin/pytest
+```
+
+The tests compare each tool with its Perl original. `tests/golden/` holds the
+Perl's output for large tables of command lines (about 2,200 for pick, 700
+for recs), and the tests check that the Python version reproduces it byte for
+byte. Where it deliberately differs, the case names the CHANGELOG entry that
+explains why. With perl installed, further tests run random command lines
+through both versions and compare them (`HYPOTHESIS_PROFILE=deep pytest` for
+a longer search), and `python tools/make_golden.py --check` confirms the saved
+output is current. The examples in the documentation run as tests too.