meshtastic 0.0.184 → 0.0.186

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 763ef1042cfde175868eab3896f7b53eaa75daf8984e64295b1b6db22a39a31e
-  data.tar.gz: eb3ba83188a23a19013c769fd8181962c8d8e0ca9e35c98a8c87efa600bef281
+  metadata.gz: 3e07fc03dd4ac92690b8209db83c3878cedaeeccae799f64d5a66ebcd77c661f
+  data.tar.gz: b084165dc768b1446b4977a1ab8be30657773b2549c6ef42f699ecb336516890
 SHA512:
-  metadata.gz: 03e7f64573c6baaa1a52624a184dd6c119f3e56fa0dd39dae82ed3540451fd835e6783fc59a6dd7cc2af2a3b0fe965c70c64a6049e0626f225762e6185141765
-  data.tar.gz: 394b62b327b1ca44b89d961f2896d9e1c1389a677353314cd7319a589e009372ab56868fb5ece662d84d6f2e0a1a3e29d9b13f622e89d9c85c417acbb7f2d19c
+  metadata.gz: 981f4b515e1820458ddfa5655aeb9d568e72fa92e05b7458b9069dc2b3baeb6bd7ab6c7a61c91e6d74fdd6f34e77c3aa4ae25f8b47c4f202667a8d0aaa147bc3
+  data.tar.gz: 54c9de46786ccf0052ede5e441cc526a3294eabb098a8c717d477edd46cf28a47d4e4cfa67349a97f067780ff7a056e3d80a6be598495754d967a69729b49135

data/.rubocop_todo.yml

@@ -94,6 +94,7 @@ Style/RedundantStringEscape:
   Exclude:
     - 'lib/meshtastic/admin_pb.rb'
     - 'lib/meshtastic/atak_pb.rb'
+    - 'lib/meshtastic/config_pb.rb'
     - 'lib/meshtastic/deviceonly_pb.rb'
     - 'lib/meshtastic/mesh_pb.rb'
     - 'lib/meshtastic/module_config_pb.rb'

data/Gemfile

@@ -9,9 +9,10 @@ gemspec
 gem 'bundler', '>=4.0.21'
 gem 'bundler-audit', '>=0.9.3'
 gem 'executable-hooks', '1.7.1'
+gem 'fiddle', '1.1.8'
 gem 'gem-wrappers', '1.4.0'
 gem 'geocoder', '1.8.6'
-gem 'google-protobuf', '4.36.1'
+gem 'google-protobuf', '4.36.2'
 gem 'grpc-tools', '1.84.0'
 gem 'mqtt', '0.7.0'
 gem 'rake', '13.4.2'

data/documentation/README.md

@@ -23,6 +23,8 @@ Examples use placeholder addresses only:
 
 Feature modules send protobufs over Serial, Bluetooth, or TCP via `Meshtastic.deliver_data`. Pass `serial_obj:`, `bluetooth_obj:`, or `tcp_obj:`. MQTT encrypts on the host; the radio encrypts on the other three.
 
+All four subscriptions share [payload decoding and the complete port inventory](mesh-interface.md#receive-payload-coverage), including empty proto3 messages and explicit raw/unsupported codec boundaries.
+
 Do not open Serial and Bluetooth to the same radio at once. Always `disconnect` in `ensure`.
 
 ## Feature modules
@@ -43,6 +45,9 @@ Do not open Serial and Bluetooth to the same radio at once. Always `disconnect`
 - [Meshtastic::RemoteHardware](remote-hardware.md)
 - [Meshtastic::Storeforward](storeforward.md)
 - [Meshtastic::ATAK](atak.md)
+- [Meshtastic::Forwarder](forwarder.md)
+- [Meshtastic::Reticulum](reticulum.md)
+- [Meshtastic::PayloadFormats](payload-formats.md)
 - [Meshtastic::Paxcount](paxcount.md)
 - [Meshtastic::Cannedmessages](cannedmessages.md)
 - [Meshtastic::RTTTL](rtttl.md)

data/documentation/forwarder.md

@@ -0,0 +1,149 @@
+# ATAK Forwarder (port 257)
+
+`require 'meshtastic/forwarder'` loads a Ruby decoder independent of ATAK V1/V2.
+The caller must preserve the original payload when decoding fails.
+
+```ruby
+# One actual port-257 packet, including the chunk header:
+result = Meshtastic::Forwarder.decode_packet(payload: bytes)
+
+# A complete explicitly grouped set of header-bearing chunks (any order):
+result = Meshtastic::Forwarder.decode_chunks(chunks: chunks)
+
+# Already reassembled libcotshrink bytes, without the chunk header:
+result = Meshtastic::Forwarder.decode(payload: bytes)
+```
+
+## Implemented support and precise limits
+
+* `:libcotshrink_protobuf` results contain `:protobuf`, a schema-level hash
+  covering the complete 43-file upstream CoT descriptor graph, including all
+  22 Detail fields, nested shapes, routes, GeoChat, video, sensors and medevac.
+  This is **not** Meshtastic TAKPacket or standard TAK Protocol CotEvent.
+* `:event` additionally interprets UID, omitted PLI type, scaled latitude and
+  longitude, CE/LE, packed HAE, time offset and stale interval. `:extensions`
+  interprets all 16 customBytesExt fields, including nullable values and mapped
+  strings. These conversions follow libcotshrink's own converters.
+* Nested Detail values remain **schema-level encoded values**: this module does
+  not implement all per-detail scaling, string substitutions, defaults or CoT
+  XML reconstruction. A decoded protobuf hash is not a full reconstructed CoT
+  event. The packed source values remain in `:protobuf`.
+* Optional GZIP wrapping is detected by its magic bytes. Both compressed input
+  and decompressed output are bounded by `max_bytes` (default and hard ceiling
+  1,048,576). Malformed/truncated GZIP and trailing bytes are rejected. Plain
+  zlib/DEFLATE is not the libcotshrink wrapper.
+* Year and timezone are **not transmitted**. Upstream derives its epoch using
+  local `Calendar`, January 1 of its startup year. Without context the decoder
+  returns `time_offset_seconds` and `stale_after_seconds`; it does not guess.
+  Supply `start_of_year: Time.new(2025, 1, 1, 0, 0, 0, '+00:00')` only when the
+  sender's epoch is known. Then `time`, `start`, and `stale` are emitted as UTC
+  ISO 8601 strings. This matters for archived messages and New Year boundaries.
+* Invalid mapped values, malformed protobuf, and size violations raise
+  `ArgumentError`. EXI raises `Meshtastic::Forwarder::UnsupportedFormat`, a
+  subclass of `ArgumentError`. Empty reassembled bytes remain a valid default
+  proto3 message; an empty radio packet lacks its mandatory chunk header.
+
+### Framing is not optional
+
+Upstream MeshSender prepends **one byte**: high nibble is zero-based chunk
+index, low nibble is total chunk count (1–15). Its normal chunk body is at most
+200 bytes. Even an unfragmented message begins with `0x01`, **not `0x00`**.
+`decode_packet` strips that byte and decodes a single-chunk message. For a
+multi-chunk message it returns `format: :forwarder_fragment`, `index`, `count`,
+and binary `body`; it never treats a fragment as a complete protobuf.
+
+`decode_chunks` requires every index exactly once with a consistent count,
+rejects missing/duplicate chunks, orders the chunks, bounds aggregate bytes,
+and decodes only after reassembly. It is deliberately stateless. Integration
+must group by connection/channel/sender and impose a short lifetime and a
+bounded queue. The upstream header has **no message ID**, so concurrent
+same-sender messages cannot be disambiguated reliably from payload bytes.
+Do not silently merge unrelated generations. A Meshtastic packet ID identifies
+one radio packet, not all chunks of an application message.
+
+`ATAKBCAST,mesh-id,uid,callsign,initial` discovery broadcasts are returned as
+`:forwarder_discovery`, not sent through a CoT decoder.
+
+## EXI blocker — full Forwarder decoding is not complete
+
+libcotshrink's lossless mode uses EXIficient `DefaultEXIFactory.newInstance()`;
+its lossy mode falls back to that mode for unsupported CoT details. Either can
+be GZIP wrapped. No XSD is supplied: the required format is schema-less EXI,
+not a fixed CoT field table. Real support needs EXI header/options handling,
+bit-level event codes, dynamically learned XML grammars, QName/string-table
+partitions and EXI datatype decoding. A zlib inflater or protobuf schema cannot
+replace this engine.
+
+This implementation recognizes the EXI distinguishing bits/cookie and fails
+explicitly. No working Ruby EXI engine or captured EXI wire fixture was found
+in this research. That is an implementation/dependency gap, not a claim that
+EXI is mathematically impossible in Ruby. The superficially named `xi_parser`
+gem is a personal-wiki-to-HTML parser, not EXI. No Java or Python subprocess is
+used; full EXI support remains unimplemented.
+
+## Provenance and verification
+
+The generated `forwarder_pb.rb` embeds FileDescriptorProto bytes from
+`paulmandal/libcotshrink` revision
+`7d818b0d119a8df291770923b463fbb734b7e1b0`. They were produced with `protoc
+--include_imports --descriptor_set_out=... cotevent.proto` and registered in a
+private Ruby DescriptorPool in dependency order. The upstream MIT license is
+included in that generated file. Runtime requires neither protoc nor the
+Android libraries.
+
+Specs include an independent `protoc --encode` wire fixture, direct
+hand-authored wire, packed-bit/null handling, nested schema decoding, actual
+nibble framing, shuffled reassembly, missing/duplicate fragments, discovery,
+GZIP and output limits. These are schema-conformance fixtures, **not captured
+radio packets**. Upstream `HackyTests.java` supplies CoT XML test inputs and
+Android runtime round-trip tests, not checked-in binary fixtures. The Android
+implementation was inspected but not executed, and interoperability with live
+hardware is not claimed.
+
+Primary implementation sources:
+
+* [CotShrinker format selection and GZIP](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/pub/api/CotShrinker.java)
+* [CoT schema](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/proto/cotevent.proto)
+* [Detail schema graph](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/proto/detail/detail.proto)
+* [Packed time/height](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/protobuf/CustomBytesConverter.java)
+* [Packed extensions/mappings](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/protobuf/CustomBytesExtConverter.java)
+* [Nullable bit semantics](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/protobuf/utils/BitUtils.java)
+* [Local-year epoch](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/protobuf/cotevent/CotEventProtobufConverterFactory.java)
+* [EXI configuration](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/exi/ExiConverterFactory.java), [EXI standard](https://www.w3.org/TR/exi/)
+* [Forwarder chunk construction](https://github.com/paulmandal/atak-forwarder/blob/ad47f2c69e45785eb6029945be232bc52acd695c/app/src/main/java/com/paulmandal/atak/forwarder/comm/meshtastic/MeshSender.java)
+* [Forwarder chunk reception](https://github.com/paulmandal/atak-forwarder/blob/ad47f2c69e45785eb6029945be232bc52acd695c/app/src/main/java/com/paulmandal/atak/forwarder/comm/meshtastic/InboundMeshMessageHandler.java)
+* [Upstream XML tests](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/hackytests/HackyTests.java)
+
+## Read-only fragmentation assessment: ports 76 and 75
+
+### Reticulum tunnel (76)
+
+The inspected [RNS Meshtastic interface, revision e5eb5d23](https://github.com/landandair/RNS_Over_Meshtastic/blob/e5eb5d23a619958ae0556c88fcbc4bf1d867ab0a/Interface/Meshtastic_Interface.py)
+uses a two-byte `Bb` header: unsigned message index and signed fragment
+position. Positions start at 1; a negative position marks the final fragment
+and its absolute value is the total count. `REQ` followed by the same two-byte
+metadata requests retransmission; it is not RNS packet data. Reassembly must
+key by sender and message index (plus connection/channel isolation), wait for
+every position through the known last index, strip headers, and only then
+parse the RNS packet. Out-of-order final-first delivery, duplicate conflicts,
+index wrap/reuse, timeouts, maximum packet bytes and bounded state all need
+coverage. An encrypted RNS inner payload cannot be made plaintext by tunnel
+reassembly. This assessment does not assert every third-party port-76 sender
+uses this identical protocol. [Reticulum](reticulum.md) implements this pinned
+tunnel framing and explicitly grouped, bounded reassembly; the receive dispatcher
+decodes individual frames only. Inner RNS parsing and decryption remain unsupported.
+
+### LoRaWAN bridge (75)
+
+The [pinned authoritative bridge protobuf](https://github.com/meshtastic/protobufs/blob/51028ca5a6945c76d3977c2bb803f9947d319ac5/meshtastic/lorawan_bridge.proto)
+explicitly defines fragmentation: Uplink/Downlink `chunk_count == 0` is a full
+PHY frame; `chunk_count == 2` means `payload` is its first part and `payload_id`
+is 1–255. A separate `PayloadChunk` has the same ID, `chunk_index == 1` and the
+remaining `payload_chunk`. Reassembly keys on sender and payload ID; retain
+head RF metadata, append exactly one continuation and reject other counts or
+indexes. Isolate connections/channels/directions, expire incomplete entries,
+handle continuation-before-head, and reject conflicting duplicates. This is
+not the RNS signed-position format. Only the reassembled LoRaWAN PHYPayload
+should reach a LoRaWAN frame parser; application plaintext also needs the
+correct LoRaWAN security context. TxResult correlates by `request_id`, not
+`tmst`. No LoRaWAN transport/reassembly code was changed.

data/documentation/mesh-interface.md

@@ -13,7 +13,8 @@ Packet builder used by Serial, Bluetooth, TCP, and MQTT. Instantiated internally
 - `send_packet` — encrypt when `psks` present; `via: :radio` → `ToRadio`, `via: :mqtt` → `ServiceEnvelope`
 - `send_data` / `send_text`
 - `send_to_radio` / `send_to_mqtt` — serialize only
-- `decode_payload` — TEXT_MESSAGE_APP as UTF-8; other portnums as nested protobufs when known
+- `decode_payload` — shared protobuf/text/binary decoding; [complete port inventory](#receive-payload-coverage)
+- `decrypt_packet` — selected channel AES-CTR only; ciphertext preserved on failure
 - `help` / `authors`
 
 On Serial/Bluetooth/TCP, transports pass `psks: nil` so the radio owns channel crypto. MQTT must pass `psks`.
@@ -52,6 +53,161 @@ envelope = mesh.send_text(
 )
 ```
 
+## Receive payload coverage
+
+All four subscriptions use the same `MeshInterface#decode_payload` registry:
+
+- Serial reads `0x94 0xC3` length-prefixed `FromRadio`, then `Serial.enrich_packet`.
+- TCP delegates its framed stream and subscription to Serial.
+- Bluetooth reads raw `FromRadio` from GATT, then calls `Serial.enrich_packet` (no UART framing).
+- MQTT reads `ServiceEnvelope`, decrypts only if needed, then calls `decode_payload`.
+
+`recv_from_radio`, `drain_from_radio`, and raw buffer APIs intentionally return the
+original protobufs rather than enriched subscription hashes. Non-packet `FromRadio`
+variants remain decoded by their generated outer schema.
+
+Known protobuf payloads return a Hash, including `{}` for valid empty/default
+messages. This applies when `Data#to_h` omits the payload entirely; nil is **not** a
+reason to skip a known schema. Both numeric and symbolic portnums work. Unknown or
+opaque bytes remain a binary String (an omitted unknown payload stays nil). Text
+is UTF-8 with invalid sequences scrubbed for display: that returned text is not
+lossless, but decoding does not mutate the input bytes. A malformed known protobuf returns its
+original bytes rather than killing the subscription; a String at a protobuf port
+therefore means parsing failed. Existing position/MAC/public-key/time enrichment
+remains. Use `include_raw: true` when exact original wire bytes are needed, including
+unknown protobuf fields, which `to_h` does not expose.
+
+The inventory below covers **all 41 bundled enum entries**, verified against
+[protobufs at 51028ca5](https://github.com/meshtastic/protobufs/blob/51028ca5a6945c76d3977c2bb803f9947d319ac5/meshtastic/portnums.proto)
+and [firmware at 8ef996f5](https://github.com/meshtastic/firmware/blob/8ef996f5b474300c9f06128beff5ce36369f812e/src/mesh/generated/meshtastic/portnums.pb.h).
+Schema selection was also checked against firmware module handlers (including
+KeyVerification, StatusMessage, PowerStress and the simulator `Compressed` wrapper).
+Several existing bundled application schemas were missing from the receive decoder.
+The separate Forwarder decoder adds the pinned libcotshrink protobuf graph rather
+than guessing that protocol from Meshtastic's similarly named TAK messages.
+
+| ID | Port | Subscription payload / limitation |
+|---:|---|---|
+| 0 | UNKNOWN_APP | Raw, undefined binary; **not** `Data` |
+| 1 | TEXT_MESSAGE_APP | UTF-8 text |
+| 2 | REMOTE_HARDWARE_APP | `HardwareMessage` |
+| 3 | POSITION_APP | `Position` |
+| 4 | NODEINFO_APP | `User` |
+| 5 | ROUTING_APP | `Routing` (empty/default ACK supported) |
+| 6 | ADMIN_APP | `AdminMessage` |
+| 7 | TEXT_MESSAGE_COMPRESSED_APP | Unishox2 default-preset text; malformed bytes preserved |
+| 8 | WAYPOINT_APP | `Waypoint` |
+| 9 | AUDIO_APP | Codec2 mode/header/frame Hash; PCM is opt-in via `PayloadFormats` |
+| 10 | DETECTION_SENSOR_APP | UTF-8 text; **not** `DeviceState` |
+| 11 | ALERT_APP | UTF-8 text |
+| 12 | KEY_VERIFICATION_APP | `KeyVerification` |
+| 13 | REMOTE_SHELL_APP | `RemoteShell`; stream body remains bytes |
+| 32 | REPLY_APP | Text (ASCII wire convention) |
+| 33 | IP_TUNNEL_APP | IPv4/IPv6 base headers and raw body; no network injection |
+| 34 | PAXCOUNTER_APP | `Paxcount` |
+| 35 | STORE_FORWARD_PLUSPLUS_APP | `StoreForwardPlusPlus`; encrypted/fragmented body remains bytes |
+| 36 | NODE_STATUS_APP | `StatusMessage` |
+| 37 | MESH_BEACON_APP | `MeshBeacon` |
+| 38 | PAGING_APP | Raw: upstream enum names `PagingPacket`, but neither audited source tree defines that schema |
+| 64 | SERIAL_APP | Raw serial bytes; **not** `SerialConnectionStatus` |
+| 65 | STORE_FORWARD_APP | `StoreAndForward` |
+| 66 | RANGE_TEST_APP | Text (ASCII wire convention); **not** `FromRadio` |
+| 67 | TELEMETRY_APP | `Telemetry` and its nested metrics |
+| 68 | ZPS_APP | Unsupported status + raw bytes by default; explicit legacy profile via `PayloadFormats` |
+| 69 | SIMULATOR_APP | `Compressed` wrapper; `data` recursively decoded by its `portnum`, up to eight wrappers |
+| 70 | TRACEROUTE_APP | `RouteDiscovery` |
+| 71 | NEIGHBORINFO_APP | `NeighborInfo` |
+| 72 | ATAK_PLUGIN | `TAKPacket`; Unishox2 Contact/GeoChat fields decompressed before UTF-8 parsing; detail remains bytes |
+| 73 | MAP_REPORT_APP | `MapReport` |
+| 74 | POWERSTRESS_APP | `PowerStressMessage` |
+| 75 | LORAWAN_BRIDGE | `LoRaWANBridge`; embedded radio frames remain bytes |
+| 76 | RETICULUM_TUNNEL_APP | Reticulum tunnel fragment/REQ metadata and raw bytes; opaque body, explicit reassembly only |
+| 77 | CAYENNE_APP | Cayenne LPP typed records; unknown type stops parsing and retains remainder |
+| 78 | ATAK_PLUGIN_V2 | Flags-prefixed `TAKPacketV2`; raw `0xFF` or official zstd dictionaries 0/1 (requires libzstd) |
+| 79 | LORA_OTA_APP | ota-common header, START/LOAD bodies; no signature verification or firmware installation |
+| 112 | GROUPALARM_APP | Raw external protocol; no bundled schema |
+| 256 | PRIVATE_APP | Raw application-owned bytes |
+| 257 | ATAK_FORWARDER | Chunk header, discovery, protobuf/GZIP libcotshrink; EXI unsupported; no automatic grouping |
+| 511 | MAX | Enum limit, not an application schema; raw |
+
+Other numeric/private ports preserve bytes without attempting arbitrary protobuf
+detection. Unsupported private/unknown payloads are returned without warnings
+or payload dumps. Decoding does not implement application state machines, automatic
+fragment grouping, shell execution, paging acknowledgement, or inner decryption.
+
+### Binary codecs and limits
+
+`PayloadFormats` ports return Hashes with `:raw`, `:status` (`:decoded`,
+`:unsupported`, or `:malformed`) and available format fields. A malformed packet
+never discards its original bytes. Subscriptions use `pcm: false` and no ZPS
+profile: call `Meshtastic::PayloadFormats.decode(portnum:, payload:, pcm: true)`
+for optional libcodec2 PCM, or pass `zps_profile: :esp32_legacy` only for that
+experimental sender dialect. Neither native audio playback nor packet-stream
+synthesis is automatic. See [PayloadFormats](payload-formats.md).
+
+Unishox2 and ATAK decoding are bounded to 4096 decoded bytes. ATAK V2 uses
+flags-prefixed wire frames, not naked protobuf bytes; `0xFF` denotes uncompressed
+protobuf. For compatibility, absent/empty V2 application payloads still yield
+`{}`; direct `ATAK.decode_v2` requires a flags byte. Invalid compressed bytes or
+unavailable libzstd return original wire bytes in subscriptions. The `fiddle` gem
+is a packaged runtime dependency, not assumed bundled with Ruby.
+
+[Forwarder](forwarder.md) consumes the port-257 chunk header, including `0x01`
+for single-packet events. Multi-chunk packets yield fragment metadata; callers
+must explicitly group a complete message before `Forwarder.decode_chunks`.
+There is no application message ID and no safe automatic concurrent grouping.
+GZIP protobuf events return schema-level detail, not reconstructed CoT XML;
+EXI stays raw because its grammar decoder is unsupported. Packed timestamps
+retain sender-relative offsets unless explicit year/timezone context is supplied
+to the direct Forwarder API. Legacy `ATAK.compress_cot`/`decompress_cot` zlib
+helpers are not the upstream Forwarder framing protocol.
+
+### Simulator wrapper decoding
+
+[Reticulum](reticulum.md) decodes the pinned port-76 tunnel header and `REQ`
+control frames on all four receive paths, including inside simulator wrappers.
+Malformed frames retain their original bytes. Fragment bodies remain opaque;
+call `Reticulum.decode_chunks` only with an explicitly isolated complete group.
+There is no automatic grouping, retransmission, inner RNS decoding or decryption.
+
+`SIMULATOR_APP` retains the `Compressed` wrapper Hash and decodes its `:data`
+through the same application dispatcher using the contained `:portnum`. Omitted
+portnums mean zero (`UNKNOWN_APP`), so their data stays raw. Omitted data for a
+known protobuf port decodes as the default message; an entirely empty wrapper
+remains `{}`. Existing enrichment and `gps_metadata` also apply to inner messages.
+
+At most eight simulator wrappers are decoded per call. A further simulator
+wrapper remains its original binary bytes; a non-simulator leaf at that boundary
+still decodes normally. Malformed outer wrappers remain raw, and malformed inner
+messages remain raw in `:data`, without discarding successfully decoded wrappers.
+This is schema dispatch, not decompression of arbitrary bytes.
+
+### Encryption boundary
+
+`decrypt_packet(message:, psks:, channel:)` preserves already-decoded packets,
+even if PKI metadata is present. MQTT selects the exact `ServiceEnvelope.channel_id`
+(string or symbol PSK key), using the topic channel only when the envelope omits it.
+It never falls back from an unknown channel to LongFast. Radio ciphertext carries
+an eight-bit XOR hash of channel name and key, not a local channel slot; only a
+unique matching configured name/key is selected. Hash collisions are refused.
+Supply the actual firmware channel name and full Base64 16- or 32-byte key.
+Subscription wrappers also accept the existing `LongFast: 'AQ=='` shorthand.
+
+PKI ciphertext is never fed to an AES channel cipher. Missing/ambiguous/invalid keys
+and malformed decrypted `Data` preserve ciphertext with `:decryption_error`.
+Successful channel decoding retains the existing `encrypted: :decrypted` marker.
+AES-CTR is unauthenticated: valid protobuf parsing is not proof the key was correct
+or the sender authentic. Device-owned private-key decryption remains the radio's
+responsibility. No private-key discovery or downgrade fallback is performed.
+
+### Verification
+
+Shared independent fixtures exercise every enum through real PTY UART framing,
+TCP stream framing, fake GATT reads, and both already-decoded and AES-encrypted
+MQTT envelopes. Every mapped protobuf is tested with populated, empty/default,
+and malformed payloads followed by further packets. These are hardware-free host
+protocol tests, not claims of live radio/broker interoperability.
+
 ## Related
 
 - [Meshtastic::Serial](serial.md)

data/documentation/payload-formats.md

@@ -0,0 +1,187 @@
+# Binary application payload formats
+
+`Meshtastic::PayloadFormats` is a Ruby, transport-independent decoder. It does not
+open radios, send packets, install firmware, or alter the original payload.
+
+```ruby
+require 'meshtastic/payload_formats'
+
+result = Meshtastic::PayloadFormats.decode(
+  portnum: :CAYENNE_APP, # alternatively integer 77
+  payload: ['03670110056700ff'].pack('H*')
+)
+# result[:records] contains channel 3 / 27.2 C and channel 5 / 25.5 C.
+```
+
+## Integration contract
+
+Call `decode(portnum:, payload:, pcm: false, zps_profile: nil)` using a Ruby options
+Hash. `payload` must be a String of actual binary bytes, **not base64 or hex**.
+Numeric ports, enum symbols, and enum-name strings are supported. `PORTS` maps
+these enum names to numbers. No protobuf or shared dispatcher changes are required
+inside this module; callers must explicitly require and invoke it.
+
+Every result retains `:raw` (binary String) and `:portnum`. `:status` is:
+
+- `:decoded`: the documented format layer was decoded, not necessarily its nested
+  application content, cryptographic identity, or audio samples.
+- `:unsupported`: an unknown schema/version/type remains opaque; `:error` explains
+  why. Known partial fields may be present.
+- `:malformed`: a known format failed bounds/framing validation; original bytes
+  and an error remain available. No partial success is claimed.
+
+Invalid Ruby input types raise `ArgumentError`. Malformed on-wire input returns a
+Hash instead of disrupting a receive callback. A dispatcher should preserve these
+statuses and raw bytes rather than displaying unsupported bytes as text. This
+module does not decrypt encrypted MeshPackets.
+
+| Port | Name | Decoded layer |
+|---|---|---|
+| 77 | `CAYENNE_APP` | Original myDevices Cayenne LPP sensor records |
+| 33 | `IP_TUNNEL_APP` | IPv4 header/options or IPv6 base header, remaining bytes |
+| 9 | `AUDIO_APP` | Firmware Codec2 magic/mode/framing; optional native PCM |
+| 79 | `LORA_OTA_APP` | Pinned ota-common transport header and verified body layouts |
+| 68 | `ZPS_APP` | Opaque by default; explicit experimental ESP32 dialect available |
+
+Other ports remain opaque. Do not apply the ZPS dialect to arbitrary private-port
+traffic simply because the historical implementation used `PRIVATE_APP`.
+
+## Cayenne LPP
+
+Records preserve wire order and repeated channels. Each contains `channel`,
+`type`, `name`, `value`, and `unit`. Supported original type IDs are 0, 1, 2, 3,
+101, 102, 103, 104, 113, 115, 134, 136. Signed big-endian values use the source
+resolutions, including 24-bit signed GPS latitude/longitude/altitude. Acceleration
+and gyrometer values use `{x:, y:, z:}`; GPS uses
+`{latitude:, longitude:, altitude:}` in degrees/degrees/metres.
+
+An unknown type stops decoding at its channel byte: because LPP has no generic
+length field, skipping it would invent a layout. Earlier valid records,
+`undecoded_offset`, and `remainder` are returned. Fork-specific extended LPP types
+are not inferred. An empty LPP payload is an empty record list.
+
+## IP tunnel
+
+The official tunnel sends raw IP datagrams, without an Ethernet or TUN prefix.
+IPv4 decoding validates IHL and exact total length, extracts addresses, protocol,
+TTL, DSCP/ECN byte, identification, flags, fragment offset **in bytes**, checksum,
+raw options and `body`. `header_checksum_valid` reports the checksum separately:
+a bad checksum does not erase an otherwise readable header.
+
+IPv6 decoding validates its fixed header and exact payload length, extracts
+addresses, traffic class, flow label, next-header and hop-limit, and preserves
+`body`. IPv6 jumbograms remain unsupported. The Python tunnel implementation is
+IPv4-oriented; parsing an IPv6 header here does not claim IPv6 forwarding support
+in that client. Neither parser reassembles fragments, decodes upper-layer
+TCP/UDP/ICMP, walks IPv6 extension headers, or validates transport checksums.
+
+## Audio and optional native Codec2
+
+The firmware payload begins `C0 DE C2`, followed by a **Codec2 mode ID**, not a
+literal bitrate. IDs 0–7 correspond to 3200, 2400, 1600, 1400, 1300, 1200, 700,
+700B. Mode 8 is the documented modern Codec2 700C extension. Each compressed
+frame occupies `ceil(bits_per_frame / 8)` bytes: 8, 6, 8, 7, 7, 6, 4, 4, 4.
+A final partial frame is malformed. Padding bits in 52-/28-bit modes are not
+reinterpreted as another frame.
+
+By default, `frames` and `encoded` contain compressed bytes. `pcm_decoded: false`
+means **no waveform has been produced**, even though framing is decoded.
+
+```ruby
+result = Meshtastic::PayloadFormats.decode(
+  portnum: 9,
+  payload: audio_payload,
+  pcm: true
+)
+# On success: result[:pcm] is mono signed 16-bit little-endian, 8000 Hz.
+# Check result[:pcm_decoded], not merely result[:status].
+```
+
+Optional PCM conversion uses Ruby Fiddle calling an installed `libcodec2.so`;
+there are no shell commands, Python dependencies, playback or file writes.
+Frame geometry is checked against the native library before entering its decoder.
+Unavailable Fiddle/library/mode produces `pcm_error` without losing framing or
+raw bytes. Modern libcodec2 often lacks the historical 700/700B modes. The binding
+creates a fresh decoder per packet and retains state **within** its frames, not
+across successive mesh packets. Therefore this is packet-local waveform decoding,
+not a continuous-stream audio player or a guarantee of bit-identical firmware
+post-filter output. Library loading currently uses the Linux shared-object name;
+other platforms can still use pure-Ruby framing.
+
+## OTA: decoding is not signature verification
+
+This implementation targets `caveman99/ota-common` at
+`e0d0e37d23a21df093981b40a09b454b9ce1c327`, not a fictional protobuf or a different
+ESP32 Wi-Fi/BLE loader protocol. Frames contain an 8-byte header:
+`type:u8, session:u8, index:u16le, offset:u16le, total:u16le`; maximum frame size is
+233 bytes. `index == 65535` denotes the manifest unit where that frame type uses
+an index.
+
+- START (1): 12-byte geometry body, exposed as `start` with `block_size`,
+  `block_count`, `payload_length`, `manifest_length`, `signature_length`.
+- MANIFEST (2), BLOCK (3), PROOF (4): `body` is a fragment of the unit. Bounds are
+  checked against `offset`/`total`. Empty single-leaf proofs are valid.
+- REQUEST (5), ACK (6), DONE (7), ABORT (8), LOAD_COMMIT (10): header-only control
+  frames according to the verified sender conventions.
+- LOAD (9): 8-byte little-endian `total_length:u32, offset:u32` prefix followed by
+  `chunk`, exposed as `load`; package bounds are checked.
+- ANNOUNCE (11): its enum/header is known, but its body layout is not defined in
+  the pinned transport codec. Its body remains opaque with `:unsupported`.
+  Unknown future frame types receive the same treatment.
+
+`signature_verified` is always false. These are **source-verified layouts**, not
+cryptographically verified messages. No cross-frame reassembly, manifest parsing,
+Merkle-proof checking, XEdDSA signature verification, flash state or OTA session
+completion is claimed. START geometry is an unsigned hint; even an ACK/DONE
+packet is not independent evidence that an image was installed.
+
+## ZPS: explicit experimental profile only
+
+The registry says “arrays of int64 fields,” which is not enough to establish a
+portable schema. The linked original project has a concrete ESP32 implementation
+but still sends on `PRIVATE_APP`; it uses native-memory `uint64_t` copies, has no
+version marker, and documents unfinished behavior. Accordingly port 68 remains
+opaque unless the caller knows the sender and explicitly supplies
+`zps_profile: :esp32_legacy`.
+
+That profile follows commit `8d56d8e29f24e6f5eeeae20e9daa04d1d5cd3fc3`:
+little-endian timestamp/header word, position/reserved word, followed by up to
+20 packed scan words. It returns the timestamp, raw header words, Wi-Fi/BLE
+records (address, channel marker and negative RSSI), and optional raw signed
+latitude/longitude integers plus PDOP when bit 47 indicates a position. No
+geolocation service, units normalization for PDOP, or universal port-68
+compatibility is implied.
+
+## Primary sources and fixture provenance
+
+- [myDevices CayenneLPP README](https://github.com/myDevicesIoT/CayenneLPP/blob/master/README.md):
+  original type table and literal temperature/acceleration wire examples used in
+  specs. Additional edge vectors follow that table.
+- [Meshtastic port registry](https://github.com/meshtastic/protobufs/blob/master/meshtastic/portnums.proto):
+  port assignments and encoding descriptions; the registry alone is not treated
+  as a complete schema.
+- [Official tunnel implementation](https://github.com/meshtastic/python/blob/master/meshtastic/tunnel.py),
+  [RFC 791](https://www.rfc-editor.org/rfc/rfc791),
+  [RFC 8200](https://www.rfc-editor.org/rfc/rfc8200): raw IP payload and headers.
+  IPv4 fixture instantiates RFC 791 figure 5 with test addresses/data and a
+  computed valid checksum; IPv6 fixture is a constructed RFC-format datagram.
+- [Firmware AudioModule](https://github.com/meshtastic/firmware/blob/6d41e279f1f51bd59f687b9d441c1bf47b1594fc/src/modules/esp32/AudioModule.cpp),
+  [historical Codec2 header/source](https://github.com/deulis/ESP32_Codec2/tree/a2bb5afb0c3f28f49bb77bcde65abbf9416be99d/codec2),
+  [modern Codec2 API](https://github.com/drowe67/codec2/blob/main/src/codec2.h):
+  magic, mode IDs, frame sizes and native calls. Specs exercise native
+  encoder-produced bytes, not a mocked PCM decoder. Pattern-byte framing vectors
+  test boundaries, not speech fidelity.
+- [ota-common transport header](https://github.com/caveman99/ota-common/blob/e0d0e37d23a21df093981b40a09b454b9ce1c327/include/ota_common/transport.h),
+  [codec/senders](https://github.com/caveman99/ota-common/blob/e0d0e37d23a21df093981b40a09b454b9ce1c327/src/transport.cpp),
+  [upstream tests](https://github.com/caveman99/ota-common/blob/e0d0e37d23a21df093981b40a09b454b9ce1c327/test/test_transport/test_transport.cpp):
+  header vector uses upstream `{Block,7,1234,200,1024}`; START geometry uses its
+  `{1024,50,50000,192,64}` example, serialized to literal little-endian bytes.
+- [ZPS original source](https://github.com/a-f-G-U-C/Meshtastic-ZPS/tree/8d56d8e29f24e6f5eeeae20e9daa04d1d5cd3fc3):
+  `outBufAdd`, `encodeBSS`, `encodeBLE`, `allocReply`, and `handleReceived` define
+  the explicitly selected dialect. Tests instantiate those layouts with dummy
+  addresses; they are not captured port-68 packets.
+
+Tests do not require hardware or network access. They distinguish published
+vectors, independently produced native Codec2 output, and constructed
+source-conformant boundary fixtures. No claim of live transport/firmware testing
+is made by this module's isolated specs.

data/documentation/reticulum.md

@@ -0,0 +1,89 @@
+# Reticulum tunnel (port 76)
+
+```ruby
+require 'meshtastic'
+frame = Meshtastic::Reticulum.decode_packet(payload: bytes)
+packet = Meshtastic::Reticulum.decode_chunks(chunks: explicitly_grouped_wire_strings)
+```
+
+This is a **tunnel framing decoder**, not a full Reticulum packet decoder.
+No transport is opened, no retransmission is sent, and no state is retained.
+The shared receive dispatcher calls `decode_packet` for port 76 on Serial, TCP,
+Bluetooth and MQTT, including recursively within simulator wrappers. Malformed
+frames preserve their original payload bytes; the direct API raises ArgumentError.
+
+## Verified wire dialect
+
+Primary source: [landandair/RNS_Over_Meshtastic Meshtastic_Interface.py](https://github.com/landandair/RNS_Over_Meshtastic/blob/e5eb5d23a619958ae0556c88fcbc4bf1d867ab0a/Interface/Meshtastic_Interface.py),
+revision `e5eb5d23a619958ae0556c88fcbc4bf1d867ab0a`.
+
+* `PacketHandler.struct_format` (line 333) is Python `Bb`: **one unsigned byte
+  message index followed by one signed byte position**, exactly two bytes.
+  Python's default native struct mode adds no padding between these byte fields;
+  byte order has no effect on either single-byte field. Ruby uses `Cc`.
+* `split_data` (345–358) starts positions at 1 and negates the last position.
+  `-1` is a complete single-fragment message; `-3` is fragment 3 of 3.
+  A positive position provides no total count. Zero is invalid.
+* `process_message` (239–243, 257 onward) reserves the prefix `REQ`, followed
+  by those same two metadata bytes, for retransmission control. Exactly five
+  bytes are accepted here. Signed request positions are preserved, with `index`
+  exposing their absolute value; upstream lookup accepts either sign.
+* The message index wraps modulo 256 (`calc_index`, 419–420).
+* Supported bounds follow this pinned interface's default 200-byte fragment
+  body and 564-byte hardware MTU (line 102). Wire data is 3–202 bytes: a header
+  plus a nonempty body. Reassembly accepts 1–128 fragments and at most 564 body
+  bytes. `max_bytes:` on `decode_chunks` can lower, not increase, that ceiling.
+  Customized upstream fragment sizes and other port-76 dialects are not claimed.
+
+## Returned values
+
+`decode_packet` returns `format: :reticulum_fragment`, `message_index`, signed
+`position`, one-based absolute `index`, `final`, `count` (nil until final),
+`complete` (true only for `-1`), `body`, and original binary `raw`.
+Even a complete single fragment remains a tunnel frame; body bytes are opaque.
+`REQ` returns `format: :reticulum_request`, `message_index`, `position`, `index`,
+`complete: false`, and `raw`; it is never treated as packet data.
+
+`decode_chunks` takes original header-bearing String payloads in any order,
+including final-first order. It returns `format: :reticulum_packet`,
+`complete: true`, `message_index`, `count`, concatenated binary `body`, and
+position-sorted `fragments` plus position-sorted original `raw_chunks`.
+It rejects missing positions/final markers, multiple final markers, out-of-range
+positions, mixed message indexes, requests, all duplicates (including byte-identical
+retransmissions), conflicting duplicates, invalid types and size violations with
+`ArgumentError`. The caller can explicitly deduplicate identical retransmissions
+before submission; this decoder never silently resolves conflicts.
+
+## Grouping and security limits
+
+The caller must isolate connection, channel, sender, message index **and message
+generation**, impose expiration and bound its pending queue. Index reuse/wrap is
+not detectable from these bytes; same-index fragments from different generations
+can be indistinguishable. There is deliberately no global auto-reassembler and no
+claim that a consistent index authenticates a group. A missing final fragment
+cannot be inferred from arrival order. The reserved `REQ` prefix is ambiguous
+with data whose header/body happen to start with those bytes; this implementation
+follows upstream's control-prefix precedence and rejects malformed control lengths.
+
+No inner RNS header parsing, decryption, signature verification, destination
+interpretation or plaintext claims are implemented. Reassembled bytes are not
+necessarily a valid RNS packet. No hardware or installed-gem validation was done.
+
+## Fixtures and tests
+
+`spec/support/reticulum_fixtures.json` contains actual output from executing only
+`PacketHandler` extracted via Python AST from the pinned source (no imports of
+RNS/meshtastic, hardware access or rewritten splitting algorithm). It records the
+source SHA-256. Inputs were opaque `bytes(range(256))*2 + bytes(range(52))` with
+index 255, and `b'\x00\xffRNS opaque'` with index 0. Request bytes were generated
+with the upstream `struct_format` and `b'REQ' + struct.pack(..., 255, 2)`.
+These are real upstream-encoder framing fixtures, **not captured radio traffic or
+valid-RNS-packet fixtures**. Python is not a runtime dependency of the Ruby module.
+
+Specs exercise these fixtures, signed extrema, strict malformed inputs,
+final-first grouping, duplicate/conflict/missing/control rejection, and aggregate
+limits. Shared transport fixtures cover an upstream single fragment, REQ control,
+malformed byte preservation and a simulator-wrapped fragment over all four receive
+boundaries (MQTT both decoded and AES-encrypted). The JSON vectors are included in
+the gem manifest even without a Git file listing. New behavior was exercised with
+failing tests before implementation.