telemetry-cli 1.0.0__tar.gz → 1.1.0__tar.gz

{telemetry_cli-1.0.0 → telemetry_cli-1.1.0}/CHANGELOG.md

@@ -1,5 +1,47 @@
 # Changelog
 
+## 1.1.0 (2026-09-28)
+
+Bit-level telemetry: recl finds frame lengths in bits and where frames start,
+a new command, realign, puts frames that start at any bit on byte boundaries
+for pick, and tgen can write bit strings.
+
+### recl
+
+- **`-bits`** finds record lengths in bits, for frames that aren't a whole
+  number of bytes, such as PCM frames of 10- or 12-bit words in a raw bit
+  stream. It compares 8-bit windows at every bit position, and by default
+  checks the lengths suggested by repeating bit patterns (sync words); with
+  `-min`/`-max` or `-only` it checks the lengths given, in bits. The result
+  is shown in bits and bytes: `RESULT 250 bits (31 bytes + 2 bits)`.
+- **`-lsb`** reads bits least significant first (with `-bits`).
+- **`-sync`** finds the bits that are the same in every record (usually a
+  sync word, with any fixed bits beside it) and reports where they are
+  (`SYNC offset length bits`), with suggested `pick head=` and `recs`
+  arguments. The record length then comes from the repeats of the pattern
+  when there are any, which works for records of one continuously sampled
+  signal, where comparing bytes doesn't. If the pattern repeats at varying
+  intervals, recl says the records may vary in length. Records are lined up
+  on the pattern's repeats, so bit slips don't hide it; recl reports them.
+
+### realign (new)
+
+- **realign** cuts a bit stream into frames of any number of bits, starting
+  at any bit, and writes each on byte boundaries, padded to whole bytes, so
+  pick can decode them. With `-sync`, it finds frames by their sync pattern
+  (optionally with bit errors) instead, which survives bit slips.
+  `recl -bits -sync` reports the frame length and offset, and suggests the
+  realign command. Without a file name, realign reads stdin.
+
+### tgen
+
+- **The `B` and `b` pack types** pack a string of `0`s and `1`s as bits, most
+  or least significant first in each byte, as Perl's `pack` does. The Perl
+  tgen had them through Perl; the Python version's first release didn't.
+- **`bitstring(values, width)`** turns words of any width into such a
+  string, for telemetry words of 10 or 12 bits: `bitstring([3, 5], 4)` is
+  `'00110101'`.
+
 ## 1.0.0 (2026-09-28)
 
 The first release.

{telemetry_cli-1.0.0 → telemetry_cli-1.1.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.5
 Name: telemetry-cli
-Version: 1.0.0
-Summary: Command-line tools for binary telemetry: decode, extract, measure and generate records
+Version: 1.1.0
+Summary: Command-line tools for binary telemetry: decode, extract, measure, realign and generate records
 Project-URL: Repository, https://github.com/donalgrant/telemetry-cli
 Project-URL: Documentation, https://github.com/donalgrant/telemetry-cli/tree/main/docs
 Project-URL: Changelog, https://github.com/donalgrant/telemetry-cli/blob/main/CHANGELOG.md
@@ -44,6 +44,9 @@ Command-line tools for binary telemetry, built for Unix pipelines:
   ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/recl.md)).
 - **tgen** generates synthetic telemetry from a command file describing each
   record ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/tgen.md)).
+- **realign** puts frames that start at any bit (in a raw bit stream) on byte
+  boundaries, so pick can read them
+  ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/realign.md)).
 
 ```console
 $ printf '\x00\x00\x00\x01\xff\xfeHi\x00\x00\x00\x02\x00\x07ok' > demo.bin
@@ -70,7 +73,7 @@ pipx install telemetry-cli
 uv tool install telemetry-cli
 ```
 
-Either installs the commands `pick`, `recs`, `recl` and `tgen`. You can also
+Either installs the commands `pick`, `recs`, `recl`, `tgen` and `realign`. You can also
 `pip install telemetry-cli` in a virtual environment.
 
 ## History

{telemetry_cli-1.0.0 → telemetry_cli-1.1.0}/README.md

@@ -11,6 +11,9 @@ Command-line tools for binary telemetry, built for Unix pipelines:
   ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/recl.md)).
 - **tgen** generates synthetic telemetry from a command file describing each
   record ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/tgen.md)).
+- **realign** puts frames that start at any bit (in a raw bit stream) on byte
+  boundaries, so pick can read them
+  ([manual](https://github.com/donalgrant/telemetry-cli/blob/main/docs/realign.md)).
 
 ```console
 $ printf '\x00\x00\x00\x01\xff\xfeHi\x00\x00\x00\x02\x00\x07ok' > demo.bin
@@ -37,7 +40,7 @@ pipx install telemetry-cli
 uv tool install telemetry-cli
 ```
 
-Either installs the commands `pick`, `recs`, `recl` and `tgen`. You can also
+Either installs the commands `pick`, `recs`, `recl`, `tgen` and `realign`. You can also
 `pip install telemetry-cli` in a virtual environment.
 
 ## History

telemetry_cli-1.1.0/docs/realign.md

@@ -0,0 +1,128 @@
+# realign
+
+Put frames that start at any bit on byte boundaries.
+
+```text
+realign [options] file length_bits [offset_bits] > frames.dat
+cat file | realign [options] length_bits [offset_bits] > frames.dat
+```
+
+Without a file name (or with `-`), realign reads stdin. It reads all of its
+input before writing, so it suits files and finite pipes rather than a live
+feed.
+
+In a raw bit stream, such as PCM telemetry straight off a bit synchronizer,
+frames needn't start at the start of a byte, nor be a whole number of bytes
+long. [pick](pick.md) reads records byte by byte, so it can't decode them as
+they are. realign cuts the stream into frames and writes each one starting on
+a byte boundary, padded with 0 bits to a whole number of bytes: a 250-bit
+frame becomes 32 bytes, with the frame's bit 0 as the first byte's top bit.
+
+## Example
+
+A stream of 250-bit frames (25 words of 10 bits), recorded starting 3 bits
+before a frame, made with [tgen](tgen.md):
+
+<!-- file: pcm.tgen -->
+```text
+W = 10
+words = lambda n: [0b1111101011, 0b1100110011, 0b0100000111, n % 1024] \
+    + [(97 * k + n // 4) % 1024 for k in range(21)]
+stream = '101' + ''.join(bitstring(words(n), W) for n in range(400))
+chunk # 0 # 1 # 'B1000' # stream[1000 * I : 1000 * (I + 1)]
+```
+
+[recl](recl.md) finds the frame length and where the frames start, and
+suggests the realign command:
+
+```console
+$ tgen -q pcm.tgen 100 > pcm.dat
+$ recl pcm.dat -bits -sync -q
+NOTE Record length from repeats of a bit pattern: every 250 bits (4345 of 4469 repeats, 97%)
+RESULT 250 bits (31 bytes + 2 bits)
+SYNC 3 31 1111101011110011001101000001110
+```
+
+realign then puts each frame on byte boundaries, and pick can read it. The
+counter is the fourth word, bits 30 to 39 of each frame: bits 16 to 25 of the
+32-bit value that starts at byte 3.
+
+```console
+$ realign pcm.dat 250 3 2>&1 > frames.dat
+realign: 399 frames of 250 bits from bit 3; 247 bits left over
+realign: each written as 32 bytes (6 padding bits)
+$ pick frames.dat 32 -q order=big -n u0+3:16:10d | head -3
+0 0
+1 1
+2 2
+```
+
+Or all in one pipeline:
+
+```console
+$ tgen -q pcm.tgen 100 | realign -q 250 3 | pick 32 -q order=big -n u0+3:16:10d | tail -1
+398 398
+```
+
+## Bit slips
+
+Serial streams sometimes lose or gain a bit (a bit slip), and every frame
+after it is then out of step. Here, the stream loses one bit in frame 150:
+
+<!-- file: slip.tgen -->
+```text
+W = 10
+words = lambda n: [0b1111101011, 0b1100110011, 0b0100000111, n % 1024] \
+    + [(97 * k + n // 4) % 1024 for k in range(21)]
+whole = '101' + ''.join(bitstring(words(n), W) for n in range(400))
+stream = whole[:37580] + whole[37581:] + '0'
+chunk # 0 # 1 # 'B1000' # stream[1000 * I : 1000 * (I + 1)]
+```
+
+Stepping 250 bits at a time goes wrong from frame 151 on:
+
+```console
+$ tgen -q slip.tgen 100 > slip.dat
+$ realign slip.dat 250 3 -q | pick 32 -q order=big u0+3:16:10d | sed -n '150,152p'
+149
+150
+302
+```
+
+With `-sync`, realign finds each frame by its sync word instead, and reports
+the frames that weren't the usual distance apart:
+
+```console
+$ realign slip.dat 250 -sync=1111101011110011001101000001110 2>&1 > frames.dat
+realign: 399 frames found by the sync pattern
+realign: frames other than 250 bits apart (bit slips, or lost frames): 249 bits (1x)
+realign: each written as 32 bytes (6 padding bits)
+$ pick frames.dat 32 -q order=big u0+3:16:10d | sed -n '150,152p'
+149
+150
+151
+```
+
+A match of the pattern up to 8 bits before the end of the previous frame
+(`-slip` changes this) starts a new frame: bits were lost. An earlier match
+is taken to be the pattern turning up by chance in the data. `-errors=n`
+lets the pattern match with up to n bits wrong, for streams with bit errors.
+`recl -bits -sync` notices slips too, and suggests the `-sync` form.
+
+## Options
+
+Options can be abbreviated, and take their values after `=` or as the next
+argument.
+
+| Option | Meaning |
+|---|---|
+| `-sync=bits` | find frames by this pattern of 0s and 1s (up to 56 bits), rather than every `length_bits` |
+| `-errors=n` | with `-sync`: allow up to n wrong bits in the pattern |
+| `-slip=n` | with `-sync`: a match up to n bits early starts a new frame (default 8) |
+| `-lsb` | the input's bits are least significant first in each byte (frames are always written most significant bit first) |
+| `-left` | pad at the start of each frame rather than the end |
+| `-fill=1` | pad with 1 bits rather than 0 |
+| `-max=n` | write at most n frames |
+| `-offset=n` | the same as giving `offset_bits` |
+| `-quiet` | no summary on stderr |
+| `-help` | show help |

{telemetry_cli-1.0.0 → telemetry_cli-1.1.0}/docs/recl.md

@@ -73,6 +73,9 @@ argument.
 | `-limit=n` | use only the first n bytes of data |
 | `-reduce=f` | use only the first 1/f of the data (f ≥ 1) |
 | `-full` | compare the data as one piece, not buffer by buffer |
+| `-sync` | find the pattern that starts each record, and where records start (see below) |
+| `-bits` | record lengths in bits (see below); all lengths are then in bits |
+| `-lsb` | with `-bits`: bits are least significant first in each byte |
 | `-verbose` | show every length's score in every buffer, and a sorted table |
 | `-quiet` | show only the result |
 | `-help` | show help |
@@ -91,6 +94,102 @@ NOTE Checking Record Lengths 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
 RESULT 48
 ```
 
+## Finding the sync word, and where records start
+
+A recording may start partway through a record, or after some junk. With
+`-sync`, recl also finds the bits that are the same in every record, usually
+a sync word, and says where they are. Here the example's records follow 5
+bytes of junk:
+
+```console
+$ (printf 'JUNK!'; cat frames.dat) > capture.dat
+$ recl capture.dat -sync
+NOTE Record length from repeats of a bit pattern: every 48 bytes (7984 of 7984 repeats, 100%)
+RESULT 48
+SYNC 40 39 000000111001000101011110110100110000000
+NOTE Bits 40-78 of each record (0x01c8af6980) are the same in 100.0% of 500 records (100.0% exactly): a sync word, perhaps with fixed bits beside it
+NOTE Records start at byte 5, if they start with the pattern: pick capture.dat 48 head=5 ...
+NOTE To extract records by the pattern: recs capture.dat "$(printf '\x03\x91\x5e\xd3')" 48
+```
+
+`SYNC` gives the pattern's position in bits from the start of the data
+(modulo the record length), its length in bits, and its bits. The 39 bits
+here are the 32-bit sync word `03915ed3` and the top 7 bits of the counter
+after it, which are 0 in all 500 records: from the data alone, recl can't
+tell fixed bits beside a sync word from the sync word itself. It prefers a
+pattern with both 0s and 1s to one of all 0s (unused bits, padding), and
+says so when a pattern starts or ends with a long run of 0s or 1s, which may
+be padding next to the sync word. It allows some bit errors: a bit position
+counts as fixed if 85% of the records agree. Records are lined up on the
+repeats of the most common pattern that recurs exactly once per record, so a
+bit slip (a bit lost or added, moving every later record) doesn't hide the
+sync word; recl reports the slips.
+
+The suggestions assume that records start with the pattern. They work:
+
+```console
+$ recs capture.dat "$(printf '\x03\x91\x5e\xd3')" 48 | pick 48 -q order=big -n ux wd | sed -n '1p;500p'
+0 03 91 5e d3  0
+499 03 91 5e d3  499
+```
+
+With `-sync`, the record length comes from the pattern's repeats when there
+are any: the gap between repeats of a sync word is the record length. That
+works even where comparing bytes doesn't, as for records of one continuously
+sampled signal. If the pattern repeats at varying intervals, recl says so:
+the records may vary in length, which is a job for [recs](recs.md).
+`-sync` works with `-bits` too, giving the position in bits.
+
+## Records that aren't whole bytes
+
+In a raw bit stream, such as PCM telemetry straight off a bit synchronizer,
+frames needn't be a whole number of bytes: 25 words of 10 bits make a 250-bit
+frame. This [tgen](tgen.md) command file makes such a stream. tgen writes
+whole bytes, so each of its records holds four 250-bit frames, 1000 bits or
+125 bytes, written as a bit string with the `B` pack type:
+
+<!-- file: pcm.tgen -->
+```text
+# PCM frames of 25 10-bit words: a 3-word sync, a counter, 21 samples
+words = lambda n: [0b1111101011, 0b1100110011, 0b0100000111, n % 1024] \
+    + [(n * 7 + 31 * k) % 1024 for k in range(21)]
+frames # 0 # 1 # 'B1000' # bitstring([w for n in range(4*I, 4*I + 4) for w in words(n)], 10)
+```
+
+Comparing bytes finds only the stretch after which frames line up with
+bytes again, four frames at a time:
+
+```console
+$ tgen -q pcm.tgen 100 > pcm.dat
+$ recl pcm.dat -q -max=200
+NOTE Checking Record Lengths 1, 2, 4, 5, 10, 20, 25, 50, 100, 125 bytes
+RESULT 125
+```
+
+With `-bits`, recl compares the data with itself shifted by a number of
+*bits*: for a length of L bits, the fraction of bit positions at which the
+8 bits starting there equal the 8 bits L bits on. It finds the frames:
+
+```console
+$ recl pcm.dat -bits
+NOTE Checking Record Lengths 1, 2, 4, 5, 8, 10, 20, 25, 40, 50, 100, 125, 200, 250, 500, 1000 bits
+RESULT 250 bits (31 bytes + 2 bits)
+CORR 12.4070100860219
+NOTE Based on 99992 bit positions (most significant bit first)
+```
+
+Checking every length at every bit position would be slow, so unless you
+give `-min` and `-max` (a range to check, in bits) or `-only`, recl checks
+the lengths the data suggest: a sync word makes the same 24-bit pattern
+appear once per frame, so the gaps between repeats of the most common 24-bit
+patterns are the likely lengths. It checks those and their divisors. With
+no pattern that repeats, give a range.
+
+Bits are read most significant first in each byte, as a serial stream is
+usually packed; `-lsb` reads them least significant first. The frames may
+start anywhere, not just at the start of a byte: `-bits -sync` says where,
+and [realign](realign.md) puts each frame on byte boundaries for pick.
+
 ## How much data recl needs
 
 Not much, if something in each record repeats. The fraction of equal bytes
@@ -111,6 +210,13 @@ bytes of values that change slowly still do, but more records are needed.
 about the format helps too: `-fact`, `-min` and `-max` rule out lengths, and
 `-only` checks just a few.
 
+One kind of record is hard for recl: one signal sampled continuously, record
+after record, with only a sync word marking where records start. There, the
+data are more alike one sample apart than one record apart. (Waveform records
+that each start afresh, like radar echoes, pulse after pulse, are fine: there
+the data repeat record to record.) `-sync` takes the length from the
+repeats of a sync word, and finds it there.
+
 ## Differences from the Perl version
 
 recl began as a C++ tool at JPL, written with the first, C++ version of pick

{telemetry_cli-1.0.0 → telemetry_cli-1.1.0}/docs/tgen.md

@@ -93,10 +93,11 @@ semicolons; its value is the last one's.
   `'s'` (16 bits), `'L'`, `'l'` (32 bits), `'n'`, `'N'` (big-endian 16 and 32
   bits), `'v'`, `'V'` (little-endian), `'f'`, `'d'` (floats), `'a10'` (a
   null-padded 10-byte string), `'A10'` (space-padded), `'Z10'`
-  (null-terminated). Integer and float types can take `<` or `>` for the byte
-  order; without one, it is this machine's. A count takes a list:
-  `f"C{n}"` with the value `[1, 2, 3]`. The type isn't evaluated for items
-  with a negative offset.
+  (null-terminated), `'B16'` (16 bits, from a string of `0`s and `1`s,
+  most significant bit first; `'b16'` for least significant first). Integer
+  and float types can take `<` or `>` for the byte order; without one, it is
+  this machine's. A count takes a list: `f"C{n}"` with the value `[1, 2, 3]`.
+  The type isn't evaluated for items with a negative offset.
 - **value** is the value to write.
 
 Frames are as long as their last item. Bytes that no item sets are filled
@@ -115,6 +116,7 @@ with the fill byte.
 | `rand(n)` | a random float in [0, n), 1 by default (Perl's generator, so `-seed` runs repeat) |
 | `pmod`, `pand`, `por`, `pxor`, `pshl`, `pshr` | Perl's `%`, `&`, `\|`, `^`, `<<`, `>>`, on integers |
 | `cat(a, b, ...)` | Perl's `.`: join values as text |
+| `bitstring(values, width)` | values as a string of bits for the `B` type: `bitstring([3, 5], 4)` is `'00110101'`; `width` may be a list, one per value |
 | `sprintf` | Perl's sprintf |
 | `pi`, `sin`, `cos`, `tan`, `atan2`, `sqrt`, `exp`, `log`, `floor`, `ceil` | from Python's `math` |
 | `stderr` | for diagnostics: `print(..., file=stderr)`. Don't print to stdout, which is the data |
@@ -230,6 +232,26 @@ $ tgen -q -seed=9 value.tgen 4 | pick 8 -q f 2wd
 0.150752 2 1
 ```
 
+### Words that aren't whole bytes
+
+Telemetry words are often 10 or 12 bits. `bitstring` and the `B` type pack
+them: two 12-bit words, then a 16-bit count, in 5 bytes:
+
+<!-- file: words.tgen -->
+```text
+packed # 0 # 1 # 'B40' # bitstring([0xABC, I, 1000 + I], [12, 12, 16])
+```
+
+```console
+$ tgen -q words.tgen 3 | pick 5 -q 5bx
+ab  c0  00  03  e8
+ab  c0  01  03  e9
+ab  c0  02  03  ea
+```
+
+Frames whose length isn't a whole number of bytes go several to a record: see
+[recl's `-bits`](recl.md#records-that-arent-whole-bytes).
+
 ### A caltone sweep
 
 <!-- file: calsweep.tgen -->

{telemetry_cli-1.0.0 → telemetry_cli-1.1.0}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "hatchling.build"
 
 [project]
 name = "telemetry-cli"
-version = "1.0.0"
-description = "Command-line tools for binary telemetry: decode, extract, measure and generate records"
+version = "1.1.0"
+description = "Command-line tools for binary telemetry: decode, extract, measure, realign and generate records"
 readme = "README.md"
 license = "MIT"
 license-files = ["LICENSE"]
@@ -40,6 +40,7 @@ dev = [
 pick = "telemetry_cli.pick.cli:main"
 recs = "telemetry_cli.recs.cli:main"
 recl = "telemetry_cli.recl.cli:main"
+realign = "telemetry_cli.realign.cli:main"
 tgen = "telemetry_cli.tgen.cli:main"
 
 [project.urls]

{telemetry_cli-1.0.0 → telemetry_cli-1.1.0}/src/telemetry_cli/_pack.py

@@ -12,6 +12,8 @@ reproduces Perl's behavior for the letters those tools use:
     v V       16 / 32-bit unsigned, little-endian
     f d       single / double float
     a A Z     byte strings: null-padded, space-padded, null-terminated
+    B b       bit strings ("1011..."), most / least significant bit first in
+              each byte; the count is in bits
     x         a null byte
 
 Each letter can take a repeat count (``C4``) or ``*``, and the integer and
@@ -53,7 +55,8 @@ _FIXED_ORDER = {"n": ">", "N": ">", "v": "<", "V": "<"}
 _FLOATS = {"f": "f", "d": "d"}
 _STRINGS = "aAZ"
 
-_ITEM = re.compile(r"\s*([cCsSlLiIqQnNvVfdaAZx])([<>])?(\*|\d+)?\s*")
+_ITEM = re.compile(r"\s*([cCsSlLiIqQnNvVfdaAZxBb])([<>])?(\*|\d+)?\s*")
+_BITS = "Bb"
 NATIVE = "<" if sys.byteorder == "little" else ">"
 
 
@@ -75,7 +78,7 @@ def parse(template: str) -> list[tuple[str, str, int | None]]:
                 break
             raise PackError(f"unsupported pack template {template!r} at {template[pos:]!r}")
         letter, order, count = m.groups()
-        if order and (letter in _STRINGS or letter in _FIXED_ORDER or letter in "cCx"):
+        if order and (letter in _STRINGS or letter in _FIXED_ORDER or letter in "cCxBb"):
             raise PackError(f"'{order}' is not allowed after '{letter}' in {template!r}")
         order = _FIXED_ORDER.get(letter, order or NATIVE)
         n = None if count is None else (-1 if count == "*" else int(count))
@@ -120,6 +123,8 @@ def pack(template: str, *values) -> bytes:
     for letter, order, count in parse(template):
         if letter == "x":
             out += b"\0" * (1 if count is None else max(count, 0))
+        elif letter in _BITS:
+            out += _pack_bits(letter, _to_bytes(take()), count)
         elif letter in _STRINGS:
             s = _to_bytes(take())
             if count == -1:
@@ -154,6 +159,13 @@ def unpack(template: str, data: bytes) -> tuple:
     for letter, order, count in parse(template):
         if letter == "x":
             pos += 1 if count is None else max(count, 0)
+        elif letter in _BITS:
+            avail = 8 * (len(data) - pos)
+            n = avail if count == -1 else min(1 if count is None else count, avail)
+            chunk = data[pos : pos + (n + 7) // 8]
+            pos += len(chunk)
+            bits = _bit_string(chunk, letter)[:n]
+            out.append(bits)
         elif letter in _STRINGS:
             n = len(data) - pos if count == -1 else (1 if count is None else count)
             s = data[pos : pos + n]
@@ -178,6 +190,25 @@ def unpack(template: str, data: bytes) -> tuple:
     return tuple(out)
 
 
+def _pack_bits(letter: str, s: bytes, count: int | None) -> bytes:
+    """Perl's B/b: each character's lowest bit, padded or cut to count bits."""
+    n = len(s) if count == -1 else (1 if count is None else count)
+    bits = [c & 1 for c in s[:n]] + [0] * max(n - len(s), 0)
+    out = bytearray()
+    for i in range(0, n, 8):
+        chunk = bits[i : i + 8] + [0] * (8 - len(bits[i : i + 8]))
+        if letter == "b":
+            chunk.reverse()
+        out.append(int("".join(map(str, chunk)), 2))
+    return bytes(out)
+
+
+def _bit_string(data: bytes, letter: str) -> str:
+    """The bits of data as a string of 0s and 1s (B: most significant first)."""
+    order = slice(None) if letter == "B" else slice(None, None, -1)
+    return "".join(format(byte, "08b")[order] for byte in data)
+
+
 def calcsize(template: str) -> int:
     """Bytes produced by a template with fixed counts (no ``*``)."""
     return len(pack(template))

telemetry_cli-1.1.0/src/telemetry_cli/realign/__init__.py

@@ -0,0 +1 @@
+"""realign: put frames that start at any bit on byte boundaries."""

telemetry_cli-1.1.0/src/telemetry_cli/realign/__main__.py

@@ -0,0 +1,7 @@
+"""Allow ``python -m telemetry_cli.realign``."""
+
+import sys
+
+from .cli import main
+
+sys.exit(main())

telemetry_cli-1.1.0/src/telemetry_cli/realign/cli.py

@@ -0,0 +1,158 @@
+"""The realign command: put bit-offset frames on byte boundaries.
+
+    realign [options] file length_bits [offset_bits] > frames.dat
+    cat file | realign [options] length_bits [offset_bits] > frames.dat
+
+Run ``realign -help`` for help.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from typing import BinaryIO, TextIO
+
+from .._cli import broken_pipe
+from .._getopt import OptionError, getoptions
+from . import frames as fr
+
+SPECS = ["sync=s", "errors=i", "slip=i", "lsb", "left", "fill=i", "max=i", "offset=i", "q|quiet",
+         "h|help"]  # fmt: skip
+
+HELP = """\
+realign - put frames that start at any bit on byte boundaries
+
+SYNOPSIS
+    realign [options] file length_bits [offset_bits] > frames.dat
+    cat file | realign [options] length_bits [offset_bits] > frames.dat
+
+    Without a file (or with -), realign reads stdin.
+
+DESCRIPTION
+    In a raw bit stream, frames needn't start at the start of a byte, nor be
+    a whole number of bytes long. realign cuts the stream into frames of
+    length_bits bits, starting offset_bits bits in (default 0), and writes
+    each frame starting on a byte boundary, padded with 0 bits to whole bytes.
+    A 250-bit frame becomes 32 bytes, which pick can then decode. recl -bits
+    -sync reports the frame length and offset.
+
+    With -sync, realign finds each frame by its sync pattern instead of
+    stepping a fixed number of bits, which copes with bit slips (a bit lost
+    or added in the stream).
+
+OPTIONS
+    -sync=bits
+        Find frames where this pattern of 0s and 1s (up to 56 bits) starts.
+    -errors=n
+        With -sync: allow up to n wrong bits in the pattern (default 0).
+    -slip=n
+        With -sync: a pattern up to n bits before the end of the previous
+        frame starts a new frame (bits were lost); default 8. Matches
+        earlier than that are taken to be false ones in the data.
+    -lsb
+        The input's bits are least significant first in each byte. (Frames
+        are always written most significant bit first.)
+    -left
+        Pad at the start of each frame rather than the end.
+    -fill=1
+        Pad with 1 bits rather than 0.
+    -max=n
+        Write at most n frames.
+    -offset=n
+        The same as giving offset_bits.
+    -quiet
+        No summary on stderr.
+    -help
+        Show this help.
+
+The full manual is at
+    https://github.com/donalgrant/telemetry-cli/blob/main/docs/realign.md
+"""
+
+
+def main(
+    argv: list[str] | None = None,
+    *,
+    stdin: BinaryIO | None = None,
+    stdout: BinaryIO | None = None,
+    stderr: TextIO | None = None,
+) -> int:
+    argv = sys.argv[1:] if argv is None else list(argv)
+    stdin = stdin if stdin is not None else sys.stdin.buffer
+    stdout = stdout if stdout is not None else sys.stdout.buffer
+    stderr = stderr if stderr is not None else sys.stderr
+
+    def error(message: str, status: int = 1) -> int:
+        print(f"realign: {message}", file=stderr)
+        return status
+
+    try:
+        o, args = getoptions(argv, SPECS)
+    except OptionError as e:
+        return error(f"{e} (see 'realign -help')", 2)
+    if o.get("h"):
+        stdout.write(HELP.encode())
+        return 0
+    # The file is optional: a first argument that's a number, and not the name
+    # of a file, is the frame length, and the input is stdin (as with pick).
+    if args and args[0] != "-" and args[0].isdigit() and not os.path.exists(args[0]):
+        args = ["-", *args]
+    if len(args) < 2:
+        return error("give a frame length in bits; see 'realign -help'", 2)
+    try:
+        length = int(args[1])
+        offset = int(args[2]) if len(args) > 2 else o.get("offset", 0)
+    except ValueError:
+        return error("the frame length and offset must be numbers of bits", 2)
+    fill = o.get("fill", 0)
+    if fill not in (0, 1):
+        return error("-fill must be 0 or 1", 2)
+    try:
+        if args[0] == "-":
+            data = stdin.read()
+        else:
+            with open(args[0], "rb") as f:
+                data = f.read()
+    except OSError as e:
+        return error(f"Can't open {args[0]} for input: {e.strerror}")
+
+    lsb = bool(o.get("lsb"))
+    try:
+        if "sync" in o:
+            found = fr.search(data, o["sync"], length, lsb=lsb, errors=o.get("errors", 0),
+                              slip=o.get("slip", 8), limit=o.get("max"))  # fmt: skip
+            starts = [p for p in found.starts if p >= offset]
+            frames = fr.gather(fr.stream_bits(data, lsb=lsb), starts, length)
+            summary = _sync_summary(found, starts, length)
+        else:
+            frames = fr.fixed(fr.stream_bits(data, lsb=lsb), length, offset, o.get("max"))
+            left_over = 8 * len(data) - offset - len(frames) * length
+            summary = [f"{len(frames)} frames of {length} bits from bit {offset}; "
+                       f"{max(left_over, 0)} bits left over"]  # fmt: skip
+        stdout.write(fr.pack_frames(frames, left=bool(o.get("left")), fill=fill))
+        stdout.flush()
+    except fr.RealignError as e:
+        return error(str(e))
+    except BrokenPipeError:
+        return broken_pipe(stdout)
+    if not o.get("q"):
+        nbytes = (length + 7) // 8
+        pad = 8 * nbytes - length
+        summary.append(f"each written as {nbytes} bytes ({pad} padding bits)")
+        for line in summary:
+            print(f"realign: {line}", file=stderr)
+    return 0
+
+
+def _sync_summary(found: fr.Found, starts: list[int], length: int) -> list[str]:
+    lines = [f"{len(starts)} frames found by the sync pattern"]
+    with_errors = sum(1 for e in found.errors if e)
+    if with_errors:
+        lines[0] += f" ({with_errors} with bit errors in it)"
+    others = {g: n for g, n in found.gaps.items() if g != length}
+    if others:
+        slips = ", ".join(
+            f"{g} bits ({n}x)" for g, n in sorted(others.items(), key=lambda x: -x[1])[:5]
+        )
+        lines.append(f"frames other than {length} bits apart (bit slips, or lost frames): {slips}")
+    return lines