meshtastic 0.0.183 → 0.0.185

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. checksums.yaml +4 -4
  2. data/.rubocop_todo.yml +1 -0
  3. data/Gemfile +2 -1
  4. data/documentation/README.md +6 -0
  5. data/documentation/admin-backup.md +369 -0
  6. data/documentation/admin.md +4 -0
  7. data/documentation/forwarder.md +149 -0
  8. data/documentation/mesh-interface.md +157 -1
  9. data/documentation/payload-formats.md +187 -0
  10. data/documentation/reticulum.md +89 -0
  11. data/lib/meshtastic/admin/backup.rb +560 -0
  12. data/lib/meshtastic/admin.rb +2 -0
  13. data/lib/meshtastic/atak.rb +98 -13
  14. data/lib/meshtastic/atak_pb.rb +3 -1
  15. data/lib/meshtastic/config_pb.rb +1 -1
  16. data/lib/meshtastic/field_metadata_pb.rb +1 -1
  17. data/lib/meshtastic/forwarder.rb +190 -0
  18. data/lib/meshtastic/forwarder_pb.rb +77 -0
  19. data/lib/meshtastic/mesh_beacon_pb.rb +1 -1
  20. data/lib/meshtastic/mesh_interface.rb +115 -65
  21. data/lib/meshtastic/mesh_pb.rb +2 -1
  22. data/lib/meshtastic/module_config_pb.rb +2 -1
  23. data/lib/meshtastic/mqtt.rb +4 -34
  24. data/lib/meshtastic/payload_compression.rb +90 -0
  25. data/lib/meshtastic/payload_formats.rb +248 -0
  26. data/lib/meshtastic/reticulum.rb +71 -0
  27. data/lib/meshtastic/serial.rb +1 -19
  28. data/lib/meshtastic/telemetry_pb.rb +2 -2
  29. data/lib/meshtastic/unishox2.rb +467 -0
  30. data/lib/meshtastic/version.rb +1 -1
  31. data/lib/meshtastic.rb +4 -0
  32. data/meshtastic.gemspec +2 -0
  33. data/spec/lib/meshtastic/admin/backup_spec.rb +704 -0
  34. data/spec/lib/meshtastic/atak_spec.rb +50 -0
  35. data/spec/lib/meshtastic/bluetooth_spec.rb +14 -0
  36. data/spec/lib/meshtastic/forwarder_pb_spec.rb +11 -0
  37. data/spec/lib/meshtastic/forwarder_spec.rb +97 -0
  38. data/spec/lib/meshtastic/mesh_interface_spec.rb +169 -0
  39. data/spec/lib/meshtastic/mqtt_spec.rb +56 -0
  40. data/spec/lib/meshtastic/payload_compression_spec.rb +43 -0
  41. data/spec/lib/meshtastic/payload_formats_spec.rb +160 -0
  42. data/spec/lib/meshtastic/reticulum_spec.rb +91 -0
  43. data/spec/lib/meshtastic/serial_spec.rb +46 -0
  44. data/spec/lib/meshtastic/tcp_spec.rb +15 -0
  45. data/spec/lib/meshtastic/unishox2_spec.rb +34 -0
  46. data/spec/support/payload_fixtures.rb +124 -0
  47. data/spec/support/reticulum_fixtures.json +12 -0
  48. data/spec/support/tak_codec_fixtures.rb +4 -0
  49. data/spec/support/unishox_fixtures.rb +4 -0
  50. metadata +39 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: eea2644a6ee8e957223fd803b7b2e944bf0ceb1ca43704da2a58ebdfd6354aff
4
- data.tar.gz: cf26d06b47b3bf0f7ec8d006dd791c77ce077b9b75e51e23a5b5b447d3906472
3
+ metadata.gz: f78bc7d067491d26e216b42e75192c1609e8e9c70cd7193bb615a9f934052936
4
+ data.tar.gz: e61f34b1e8834fd08c17911b8223a36fced64f0cc9548dfbd2a61ca494ce614d
5
5
  SHA512:
6
- metadata.gz: d1b64a05eba845736dd161a666e98464315fecfa242ccff79bf915711dd88a453cfe438788738070202f5e8d2c07f9e3657e102bfd42a020881c449e2a71ff63
7
- data.tar.gz: cbd371281dd057d34c6e2d9c8bb697996b0c8e646ab6b475da637c579009cfc7095f6122e5a4a223c17ed358259d3bb701b1ae8bc70e98f7bffa18eb812cb75e
6
+ metadata.gz: e7d9948952297de8db885f77df10dd5ab1b3a4db37d020ba4f43ad1ee73c68c5b13bbecc0eab0c8a35490d6c3107a04f53069ce9db3b9cd9db945bfc2fdbc074
7
+ data.tar.gz: 00c3afd0cd032883f2ca8d075d8e4ec7f8347b2cfa31b9e66a66ef5c7120af1ec21d8e354a69dd10e9427e92f7db0675a4b7a2fca21ecb79a48ded40186ee613
data/.rubocop_todo.yml CHANGED
@@ -94,6 +94,7 @@ Style/RedundantStringEscape:
94
94
  Exclude:
95
95
  - 'lib/meshtastic/admin_pb.rb'
96
96
  - 'lib/meshtastic/atak_pb.rb'
97
+ - 'lib/meshtastic/config_pb.rb'
97
98
  - 'lib/meshtastic/deviceonly_pb.rb'
98
99
  - 'lib/meshtastic/mesh_pb.rb'
99
100
  - 'lib/meshtastic/module_config_pb.rb'
data/Gemfile CHANGED
@@ -9,9 +9,10 @@ gemspec
9
9
  gem 'bundler', '>=4.0.21'
10
10
  gem 'bundler-audit', '>=0.9.3'
11
11
  gem 'executable-hooks', '1.7.1'
12
+ gem 'fiddle', '1.1.8'
12
13
  gem 'gem-wrappers', '1.4.0'
13
14
  gem 'geocoder', '1.8.6'
14
- gem 'google-protobuf', '4.36.1'
15
+ gem 'google-protobuf', '4.36.2'
15
16
  gem 'grpc-tools', '1.84.0'
16
17
  gem 'mqtt', '0.7.0'
17
18
  gem 'rake', '13.4.2'
@@ -23,11 +23,14 @@ Examples use placeholder addresses only:
23
23
 
24
24
  Feature modules send protobufs over Serial, Bluetooth, or TCP via `Meshtastic.deliver_data`. Pass `serial_obj:`, `bluetooth_obj:`, or `tcp_obj:`. MQTT encrypts on the host; the radio encrypts on the other three.
25
25
 
26
+ All four subscriptions share [payload decoding and the complete port inventory](mesh-interface.md#receive-payload-coverage), including empty proto3 messages and explicit raw/unsupported codec boundaries.
27
+
26
28
  Do not open Serial and Bluetooth to the same radio at once. Always `disconnect` in `ensure`.
27
29
 
28
30
  ## Feature modules
29
31
 
30
32
  - [Meshtastic::Admin](admin.md)
33
+ - [Meshtastic::Admin::Backup](admin-backup.md)
31
34
  - [Meshtastic::Admin::Channel](admin-channel.md)
32
35
  - [Meshtastic::Admin::Config](admin-config.md)
33
36
  - [Meshtastic::Admin::Firmware](admin-firmware.md)
@@ -42,6 +45,9 @@ Do not open Serial and Bluetooth to the same radio at once. Always `disconnect`
42
45
  - [Meshtastic::RemoteHardware](remote-hardware.md)
43
46
  - [Meshtastic::Storeforward](storeforward.md)
44
47
  - [Meshtastic::ATAK](atak.md)
48
+ - [Meshtastic::Forwarder](forwarder.md)
49
+ - [Meshtastic::Reticulum](reticulum.md)
50
+ - [Meshtastic::PayloadFormats](payload-formats.md)
45
51
  - [Meshtastic::Paxcount](paxcount.md)
46
52
  - [Meshtastic::Cannedmessages](cannedmessages.md)
47
53
  - [Meshtastic::RTTTL](rtttl.md)
@@ -0,0 +1,369 @@
1
+ # Admin::Backup — native host-side configuration snapshots
2
+
3
+ `Meshtastic::Admin::Backup` exports a versioned JSON document (default) or binary
4
+ Meshtastic `DeviceProfile` (`format: :device_profile`) using fresh,
5
+ request-ID/source-correlated Admin requests. It imports validated protobuf
6
+ sections with bounded routing ACK waits. It does not invoke the firmware's
7
+ filesystem backup/restore commands or shell out to another client.
8
+
9
+ **Backups contain secrets by default**, including channel PSKs, Wi-Fi/MQTT
10
+ credentials, private security keys and UI PINs when firmware returns them in
11
+ selected sections. Keep the file and returned Hash private; do not print either
12
+ in logs. Firmware can redact secrets or return default/empty values. Such a
13
+ snapshot cannot recover values the device did not disclose. Importing those
14
+ values can clear existing settings. Review the selection before exporting and
15
+ before restoring, especially across devices or firmware versions.
16
+
17
+ ## Export
18
+
19
+ ```ruby
20
+ require 'meshtastic'
21
+ require 'meshtastic/admin/backup'
22
+
23
+ snapshot = Meshtastic::Admin::Backup.export(
24
+ transport_obj: connection,
25
+ path: '/private/directory/node-backup.json', # optional; MUST NOT exist
26
+ config_types: %i[DEVICE_CONFIG POSITION_CONFIG POWER_CONFIG DISPLAY_CONFIG LORA_CONFIG],
27
+ module_config_types: %i[MQTT_CONFIG TELEMETRY_CONFIG],
28
+ channel_indexes: [0, 1],
29
+ include_owner: true,
30
+ include_ui: false,
31
+ timeout: 10
32
+ )
33
+ # Safe to display status/count; NOT snapshot[:backup].
34
+ snapshot.slice(:status, :count)
35
+ ```
36
+
37
+ `transport_obj:` is a connected Serial, Bluetooth or TCP handle with a receive
38
+ queue. **MQTT synchronous export/import is explicitly unsupported**, including
39
+ dry runs. `to:` optionally selects a unicast target; otherwise the connected
40
+ local node is used. `channel:` and `hop_limit:` are Admin delivery options.
41
+ Pause external queue consumers and do not interleave other Admin edits with a
42
+ backup/restore. The underlying Admin layer serializes individual requests,
43
+ not an entire multi-request snapshot or transaction.
44
+
45
+ JSON selection defaults (binary-specific differences are listed below):
46
+
47
+ - `include_owner: true`: only `long_name`, `short_name`, `is_licensed`.
48
+ Never clone User ID, MAC, hardware model, public key, role or messaging
49
+ capability metadata. Role belongs to device configuration; security keys
50
+ belong to explicitly selected security configuration.
51
+ - `config_types:` defaults to `DEVICE_CONFIG`, `POSITION_CONFIG`, `POWER_CONFIG`,
52
+ `NETWORK_CONFIG`, `DISPLAY_CONFIG`, `LORA_CONFIG`, `BLUETOOTH_CONFIG`,
53
+ `SECURITY_CONFIG`. Use a narrower list for firmware that lacks sections.
54
+ - `module_config_types: []`: opt in to supported module ConfigType symbols.
55
+ All 17 currently generated ModuleConfig sections are supported, including
56
+ `EXTNOTIF_CONFIG`, `STOREFORWARD_CONFIG`, `CANNEDMSG_CONFIG`,
57
+ `STATUSMESSAGE_CONFIG`, `TRAFFICMANAGEMENT_CONFIG`, `TAK_CONFIG`,
58
+ `MESHBEACON_CONFIG`. `Backup::MODULE_TYPES` lists the complete set.
59
+ - `channel_indexes: [0, 1, 2, 3, 4, 5, 6, 7]`: explicit zero-based slots,
60
+ including disabled channels. Wire getters use one-based indexes.
61
+ - `include_ui: false`: opt in to dedicated get/store UI operations.
62
+ - `timeout: 10`: positive finite seconds **per Admin request**, not a deadline
63
+ for the entire backup. Remote writes can first acquire an Admin session.
64
+
65
+ Unsupported selected sections fail the export; they are not silently skipped.
66
+ No file is created until every selected response has been collected and the
67
+ document validated. Empty JSON selections are allowed. A successful JSON return contains
68
+ `:status => :exported`, `:count`, `:backup` and `:warnings`. A read error raises.
69
+ Snapshots are sequential fresh reads, not an atomic device-wide snapshot.
70
+
71
+ Files are exclusively created with mode `0600`, `O_EXCL` and `O_NOFOLLOW`, then
72
+ flushed and fsynced. There is no overwrite flag. Existing files and final-path
73
+ symlinks fail. Use a trusted parent directory: parent-directory symlinks are not
74
+ resolved or pinned. A disk write/sync failure raises and can leave a partial
75
+ 0600 file; it is never reported as a successful backup. File mode protection is
76
+ not encryption and does not protect against privileged host access.
77
+
78
+ ## Validate, plan and import
79
+
80
+ ```ruby
81
+ plan = Meshtastic::Admin::Backup.import(
82
+ transport_obj: connection,
83
+ path: '/private/directory/node-backup.json',
84
+ dry_run: true
85
+ )
86
+
87
+ result = Meshtastic::Admin::Backup.import(
88
+ transport_obj: connection,
89
+ backup: snapshot[:backup], # use exactly one of backup: or path:
90
+ edit_transaction: true, # ONLY when target firmware supports it
91
+ verify: true,
92
+ timeout: 10
93
+ )
94
+ ```
95
+
96
+ `dry_run: true` performs **no radio requests or writes**, including no session
97
+ acquisition, edit begin, getters or readback. It validates the entire document
98
+ first. `dry_run`, `edit_transaction` and `verify` default to `false` and must be
99
+ booleans. Unknown options are rejected. Import paths must be regular files and
100
+ must not be final-path symlinks.
101
+
102
+ The importer rejects incompatible format/version, unknown document/record/
103
+ protobuf fields, unknown section selectors, duplicate section/slot records,
104
+ invalid channel indexes, section/slot mismatches, conflicting oneofs, malformed
105
+ protobuf scalar values, null values and noncanonical bytes/base64 **before any
106
+ radio request**. A version-1 document is a Hash with string keys:
107
+
108
+ ```json
109
+ {
110
+ "format": "meshtastic-admin-backup",
111
+ "version": 1,
112
+ "warning": "A human-readable limitations notice",
113
+ "records": [
114
+ {
115
+ "section": "config",
116
+ "slot": "DEVICE_CONFIG",
117
+ "value": { "device": { "serial_enabled": false } }
118
+ }
119
+ ]
120
+ }
121
+ ```
122
+
123
+ Sections are `owner`, `config`, `module_config`, `channel`, `ui`. Owner/UI slots
124
+ are JSON null; config/module slots are uppercase enum names; channel slots are
125
+ integers. `value` is protobuf JSON with **snake_case protobuf field names**,
126
+ canonical base64 bytes, protobuf enum representations and emitted scalar
127
+ defaults. Known fields and protobuf presence/default semantics round-trip;
128
+ unknown binary wire fields are not archived. This is not an opaque protobuf
129
+ wire backup. Missing fields in a manually edited document mean protobuf
130
+ defaults, not a patch/merge. YAML and arbitrary object deserialization are never
131
+ used. Session envelope passkeys and `SESSIONKEY_CONFIG` are never exported or
132
+ accepted; automatic session state stays inside the existing Admin connection.
133
+
134
+ Writes preserve input order within priority groups: ordinary owner/module/UI/
135
+ core settings first, channels next, then LoRa, Bluetooth, network and security.
136
+ These latter settings can disrupt the connection. Other firmware-specific
137
+ settings can also disrupt it; ordering cannot guarantee connectivity.
138
+
139
+ ## Binary DeviceProfile (`.cfg`) export and import
140
+
141
+ Select binary output explicitly; export accepts only `format: :json` (default)
142
+ or `format: :device_profile`, not `:auto`. The path extension does not choose
143
+ export format. JSON version-1 output and return keys are unchanged.
144
+
145
+ ```ruby
146
+ snapshot = Meshtastic::Admin::Backup.export(
147
+ transport_obj: connection,
148
+ format: :device_profile,
149
+ path: '/private/directory/nodeConfig.cfg', # optional; MUST NOT exist
150
+ config_types: %i[DEVICE_CONFIG POSITION_CONFIG LORA_CONFIG],
151
+ module_config_types: %i[MQTT_CONFIG TELEMETRY_CONFIG],
152
+ channel_indexes: [0, 1], # actual contiguous PRIMARY/SECONDARY slots only
153
+ include_owner: true,
154
+ include_ui: false,
155
+ include_ringtone: true,
156
+ include_canned_messages: true,
157
+ timeout: 10
158
+ )
159
+ # Safe status fields only. NEVER print backup bytes or decoded profile.
160
+ snapshot.slice(:status, :format, :count)
161
+ # snapshot[:backup] is a raw ASCII-8BIT String, not JSON/base64 or a protobuf object.
162
+ # Pass it directly to Backup.import(transport_obj: connection,
163
+ # backup: snapshot[:backup], dry_run: true).
164
+ ```
165
+
166
+ Binary returns `{ status: :exported, format: :device_profile, count: Integer,
167
+ backup: binary_string, warnings: [...] }`. `count` is the number of restorable
168
+ sections (owner counts once, each channel counts once, duplicate URL/config
169
+ LoRa counts once), not the byte count or number of protobuf fields. The `.cfg`
170
+ file contains exactly `snapshot[:backup]`. File protections are the same
171
+ exclusive/no-follow 0600 creation, flush and fsync as JSON. No file is created
172
+ until reads and strict binary validation succeed.
173
+
174
+ Binary selection uses the same core/module/owner/channel defaults as JSON,
175
+ with these format-specific rules:
176
+
177
+ - All eight core and 17 module selections are supported. Present empty nested
178
+ messages, byte fields, repeated values and scalar defaults round-trip. Local
179
+ container storage versions are not invented.
180
+ - Owner names and license flag are explicitly present, including empty names
181
+ and `false`. Optional `is_unmessagable` is copied only when present in the
182
+ fresh User response, including explicitly present `false`. ID, MAC, hardware,
183
+ public key and envelope session passkeys are not copied.
184
+ - `include_ui: true` is **rejected before any radio requests**: DeviceProfile
185
+ has no UI field. Use JSON for UI snapshots.
186
+ - ChannelSet URLs cannot preserve arbitrary indexes or disabled slots. Select
187
+ `channel_indexes: []` to omit channels, or an ordered contiguous prefix such
188
+ as `[0]` or `[0, 1]`. Sparse/reordered selections are rejected before requests.
189
+ Returned slot 0 must be PRIMARY, later selected slots SECONDARY, each with
190
+ settings; mismatched indexes, disabled slots and other roles fail export
191
+ without creating a file. No selected channel is silently skipped or moved.
192
+ **The default all-eight selection therefore fails if any slot is disabled**;
193
+ choose the actual enabled prefix or use JSON to preserve disabled/sparse slots.
194
+ Selected LoRa is included identically in both `config.lora` and the URL;
195
+ if LoRa is not selected, the URL does not invent or fetch it. Unselected
196
+ trailing channel slots are not disabled by a later binary import.
197
+ - `include_ringtone: true` and `include_canned_messages: true` request fresh
198
+ strings and preserve present empty strings. Both default to false and are
199
+ binary-only options; JSON rejects these keys rather than ignoring them.
200
+ - `fixed_position:` optionally accepts an explicit `Meshtastic::Position` or
201
+ field Hash, e.g. `{ latitude_i: 0, longitude_i: 0, altitude: 0 }`. It is
202
+ validated before any requests and copied with protobuf presence/defaults.
203
+ This is caller-supplied input, **not a fresh radio read**: no dedicated Admin
204
+ getter exists. It defaults to absent and is binary-only. No position is
205
+ inferred from cached GPS data or the position configuration.
206
+ - An entirely empty binary selection is rejected before requests because it
207
+ would not produce an importable profile. JSON still permits empty records.
208
+
209
+ The importer accepts the same binary `meshtastic.DeviceProfile` used by
210
+ Meshtastic clients, entirely in Ruby.
211
+ Use exactly one source and select `format: :device_profile` explicitly, or use
212
+ `format: :auto` (the default for import):
213
+
214
+ ```ruby
215
+ plan = Meshtastic::Admin::Backup.import(
216
+ transport_obj: connection,
217
+ path: '/private/directory/nodeConfig.cfg',
218
+ format: :device_profile,
219
+ dry_run: true
220
+ )
221
+ # Only section/slot identities and counts are returned; no profile contents.
222
+ plan.slice(:status, :planned, :plan)
223
+
224
+ # Alternative in-memory source, still offline validation with no radio requests:
225
+ plan = Meshtastic::Admin::Backup.import(
226
+ transport_obj: connection,
227
+ backup: binary_profile_string,
228
+ format: :device_profile,
229
+ dry_run: true
230
+ )
231
+ ```
232
+
233
+ `format:` accepts only `:auto`, `:json`, or `:device_profile`. In auto mode a
234
+ `.cfg` path (case-insensitive) selects binary, `.json` selects JSON, and a Hash
235
+ selects the existing JSON document API. Other paths and Strings are validated
236
+ as binary first, then as JSON when JSON-looking content fails binary validation.
237
+ This avoids confusing a valid binary name field beginning with a newline and
238
+ `{` with JSON. Explicit format overrides the extension. JSON Strings are also
239
+ accepted; versioned JSON Hash behavior and JSON export selections are unchanged.
240
+ No YAML, executable serialization, or implicit protobuf object input is used.
241
+
242
+ Presence-aware mapping:
243
+
244
+ - Present `long_name`, `short_name`, `is_licensed`, and `is_unmessagable` become
245
+ one `owner`/`set_owner` operation. Explicit `false` and empty names are not
246
+ treated as absent. User identity/MAC/hardware/public-key fields are never
247
+ synthesized. The Admin User schema has no presence for names/license, so
248
+ omitted members of a sent owner message carry protobuf defaults; this is not
249
+ a read/merge patch API. Firmware determines how those defaults are applied.
250
+ - Every present `LocalConfig` and `LocalModuleConfig` message becomes its own
251
+ writable Config/ModuleConfig oneof, including present empty submessages.
252
+ All eight core and 17 module sections are supported. Nested bytes, repeated
253
+ values, defaults and optional presence are retained. The local containers'
254
+ `version` fields are storage-schema metadata, not Admin settings; they are
255
+ validated but not written. A metadata-only/empty profile is rejected.
256
+ - `channel_url` must be an HTTPS `meshtastic.org/e/` or `/d/` URL with a valid
257
+ base64url ChannelSet and one through eight settings, no query or userinfo.
258
+ Settings map in order to primary slot 0 and secondary slots 1–7; unlisted
259
+ trailing slots are **not** disabled. PSK lengths and channel-name bounds are
260
+ checked. Embedded LoRa configuration is restored too. If both the profile and
261
+ URL carry LoRa, equal messages are deduplicated; conflicting messages reject
262
+ the entire profile rather than silently choosing one.
263
+ - Present `fixed_position`, `ringtone`, and `canned_messages` map respectively
264
+ to `set_fixed_position`, `set_ringtone_message`, and
265
+ `set_canned_message_module_messages`, with nil slots and matching section
266
+ names in the plan. Zero coordinates/altitude and present empty strings are
267
+ preserved. Absent fields are not turned into clears or removal operations.
268
+
269
+ The **entire** binary profile and nested ChannelSet are validated before any
270
+ Admin request, including session acquisition or begin-edit. A strict recursive
271
+ wire check rejects unknown tags (including sessionkey additions), unknown enum
272
+ values, duplicate singular fields/conflicting oneofs, invalid wire types,
273
+ truncation, oversized/noncanonical varints, invalid booleans and nonfinite
274
+ floats. Binary messages are limited to 1 MiB and 32 nested levels. Empty or
275
+ arbitrary bytes are not accepted as an empty restore. Strictness intentionally
276
+ rejects unsupported future schemas rather than dropping fields. This is format
277
+ validation, not authentication: protobuf has no file magic or signature, so a
278
+ valid nonempty DeviceProfile cannot be distinguished from another byte stream
279
+ that happens to encode the same schema. Session passkeys are never imported.
280
+
281
+ `verify: true` compares fresh owner/config/module/channel/ringtone/canned-message
282
+ responses. Fixed position has **no dedicated Admin getter**: its record reports
283
+ `readback: :unsupported`, making the overall status `:readback_incomplete` even
284
+ when all available comparisons match. No cached position or config flag is
285
+ misrepresented as fixed-position verification. All existing ACK, transaction,
286
+ no-replay, secret-handling and persistence limitations apply.
287
+
288
+ ## Transactions and failure reporting
289
+
290
+ `edit_transaction: true` explicitly sends `begin_edit_settings`, waits for its
291
+ ACK, sends each section once with a correlated ACK wait (or matching timeout
292
+ readback as described below), then sends
293
+ `commit_edit_settings` and waits again. Current upstream
294
+ [AdminModule.cpp](https://github.com/meshtastic/firmware/blob/master/src/modules/AdminModule.cpp)
295
+ implements a begin flag and commit save of configuration/module/device/channel/
296
+ node database segments. Commit also disables Bluetooth. **This is not an atomic
297
+ rollback facility**: firmware may apply values before commit, UI has its own
298
+ storage operation, and the protocol has no supported cancel/rollback here.
299
+ There is no reliable capability probe, so enable this option only with firmware
300
+ you have verified supports it. Default `false` avoids claiming universal edit
301
+ transaction support.
302
+
303
+ Result fields:
304
+
305
+ - `status`: `dry_run`, `acknowledged`, `applied`, `partial_failure`,
306
+ `readback_matched` or `readback_incomplete`. `applied` means all sections
307
+ completed, but at least one required timeout readback instead of an ACK;
308
+ it does not claim every mutation was acknowledged or persisted.
309
+ - `planned`, `attempted`, `acknowledged`, `readback_confirmed`: section counts, excluding transaction
310
+ commands and automatic session requests. Attempted is an API attempt and
311
+ does not prove the write reached the wire.
312
+ - `plan`: ordered section/slot identities, including in dry runs; no secret values.
313
+ - `acknowledged` counts only actual successful setter ACKs. `readback_confirmed`
314
+ counts only setters whose ACK timed out but whose fresh getter matched.
315
+ These counts are separate, never double-counted, and retained on later failure.
316
+ - `records`: completed section/slot statuses (`acknowledged` or
317
+ `readback_confirmed`), without configuration values.
318
+ - `transaction`: `not_requested`, `begin_uncertain`, `open`, `commit_uncertain`
319
+ or `commit_acknowledged`.
320
+ - `failure`: on write/begin/commit failure, the operation, section/slot where
321
+ applicable, exception class and routing reason where available. Exception
322
+ contents and secrets are not included. Failed timeout recovery also includes
323
+ `readback: mismatch|unsupported|failed`; getter exceptions retain their class
324
+ and routing reason where available.
325
+ - `warnings`: secret/redaction/persistence limitations.
326
+ - `persistence_verified`: always `false`.
327
+
328
+ Only a section setter's `Timeout::Error` triggers recovery: if a supported
329
+ getter exists, import issues one **new**, request-ID/source-correlated Admin GET
330
+ with its own per-request timeout. This happens even with `verify: false`.
331
+ It continues only if the entire expected protobuf setting matches, using the
332
+ same presence-aware comparison as optional verification. Portable owner fields
333
+ are compared while generated owner identity/hardware metadata is excluded.
334
+ No cached value, unrelated response, late setter ACK, or transport submission
335
+ is substituted for that fresh read. The mutation is **never resent**.
336
+
337
+ Mismatch (including redacted/empty secrets in place of expected nonempty
338
+ secrets), absent getter, getter timeout/error, and correlated routing rejection
339
+ remain partial failures. Secret fields are not ignored or treated as wildcards;
340
+ the report never includes expected/actual values. Firmware normalization can
341
+ also prevent exact matching. A matching empty/default value cannot establish
342
+ that firmware disclosed a secret; existing backup completeness limitations
343
+ still apply. Fixed-position writes have no supported getter. Begin and commit
344
+ timeouts do not use this recovery, and routing errors do not trigger it.
345
+
346
+ After an unresolved failed write, no later sections, commit, rollback or replay are sent.
347
+ A transaction may remain open. A missing commit ACK is uncertain even when all
348
+ section ACKs arrived; the node may have committed and rebooted. Inspect the
349
+ node and establish a fresh connection before deciding on any manual recovery.
350
+ Do not automatically rerun the import.
351
+
352
+ `verify: true` performs another round of fresh correlated reads **after** all
353
+ sections complete and any requested commit ACK arrives. Timeout-recovery reads
354
+ do not replace this optional post-commit check.
355
+ It compares protobuf values (only portable owner fields), adds per-record
356
+ `readback: matched|mismatch|failed` and `readback_matched` count. Redaction,
357
+ normalization, link loss or reboot can prevent a match. Readback failures never
358
+ replay writes. Even a match proves only current reported state, not persistence
359
+ across reboot or successful firmware-side storage.
360
+
361
+ ## Verification scope
362
+
363
+ The automated peer decodes production Serial framing/ToRadio/Admin protobufs,
364
+ maintains configuration state, and returns encoded correlated FromRadio Admin
365
+ or routing packets through the real Admin queue. Tests cover round-trip bytes
366
+ and defaults, dry runs, validation, secret handling, file permissions/symlinks,
367
+ ACK failures, timeouts, commit uncertainty, optional readback and help
368
+ conventions. No hardware, firmware reboot/persistence, radio-link reliability,
369
+ Bluetooth GATT or live TCP endpoint was exercised for this feature.
@@ -24,6 +24,10 @@ Every generated payload field is available through `encode` and `send`, includin
24
24
 
25
25
  ## Operation catalog
26
26
 
27
+ For host-side configuration files, use [Admin::Backup](admin-backup.md).
28
+ The `backup_preferences` / `restore_preferences` operations below instead manage
29
+ the device's own FLASH/SD preference backup; they do not export a file to the host.
30
+
27
31
  | Group | Methods and payload arguments |
28
32
  | --- | --- |
29
33
  | Identity | `get_owner`; `set_owner(owner:)` or `set_owner(long_name:, short_name:)`; `set_ham_mode(ham:)` or callsign/frequency/power/name fields |
@@ -0,0 +1,149 @@
1
+ # ATAK Forwarder (port 257)
2
+
3
+ `require 'meshtastic/forwarder'` loads a Ruby decoder independent of ATAK V1/V2.
4
+ The caller must preserve the original payload when decoding fails.
5
+
6
+ ```ruby
7
+ # One actual port-257 packet, including the chunk header:
8
+ result = Meshtastic::Forwarder.decode_packet(payload: bytes)
9
+
10
+ # A complete explicitly grouped set of header-bearing chunks (any order):
11
+ result = Meshtastic::Forwarder.decode_chunks(chunks: chunks)
12
+
13
+ # Already reassembled libcotshrink bytes, without the chunk header:
14
+ result = Meshtastic::Forwarder.decode(payload: bytes)
15
+ ```
16
+
17
+ ## Implemented support and precise limits
18
+
19
+ * `:libcotshrink_protobuf` results contain `:protobuf`, a schema-level hash
20
+ covering the complete 43-file upstream CoT descriptor graph, including all
21
+ 22 Detail fields, nested shapes, routes, GeoChat, video, sensors and medevac.
22
+ This is **not** Meshtastic TAKPacket or standard TAK Protocol CotEvent.
23
+ * `:event` additionally interprets UID, omitted PLI type, scaled latitude and
24
+ longitude, CE/LE, packed HAE, time offset and stale interval. `:extensions`
25
+ interprets all 16 customBytesExt fields, including nullable values and mapped
26
+ strings. These conversions follow libcotshrink's own converters.
27
+ * Nested Detail values remain **schema-level encoded values**: this module does
28
+ not implement all per-detail scaling, string substitutions, defaults or CoT
29
+ XML reconstruction. A decoded protobuf hash is not a full reconstructed CoT
30
+ event. The packed source values remain in `:protobuf`.
31
+ * Optional GZIP wrapping is detected by its magic bytes. Both compressed input
32
+ and decompressed output are bounded by `max_bytes` (default and hard ceiling
33
+ 1,048,576). Malformed/truncated GZIP and trailing bytes are rejected. Plain
34
+ zlib/DEFLATE is not the libcotshrink wrapper.
35
+ * Year and timezone are **not transmitted**. Upstream derives its epoch using
36
+ local `Calendar`, January 1 of its startup year. Without context the decoder
37
+ returns `time_offset_seconds` and `stale_after_seconds`; it does not guess.
38
+ Supply `start_of_year: Time.new(2025, 1, 1, 0, 0, 0, '+00:00')` only when the
39
+ sender's epoch is known. Then `time`, `start`, and `stale` are emitted as UTC
40
+ ISO 8601 strings. This matters for archived messages and New Year boundaries.
41
+ * Invalid mapped values, malformed protobuf, and size violations raise
42
+ `ArgumentError`. EXI raises `Meshtastic::Forwarder::UnsupportedFormat`, a
43
+ subclass of `ArgumentError`. Empty reassembled bytes remain a valid default
44
+ proto3 message; an empty radio packet lacks its mandatory chunk header.
45
+
46
+ ### Framing is not optional
47
+
48
+ Upstream MeshSender prepends **one byte**: high nibble is zero-based chunk
49
+ index, low nibble is total chunk count (1–15). Its normal chunk body is at most
50
+ 200 bytes. Even an unfragmented message begins with `0x01`, **not `0x00`**.
51
+ `decode_packet` strips that byte and decodes a single-chunk message. For a
52
+ multi-chunk message it returns `format: :forwarder_fragment`, `index`, `count`,
53
+ and binary `body`; it never treats a fragment as a complete protobuf.
54
+
55
+ `decode_chunks` requires every index exactly once with a consistent count,
56
+ rejects missing/duplicate chunks, orders the chunks, bounds aggregate bytes,
57
+ and decodes only after reassembly. It is deliberately stateless. Integration
58
+ must group by connection/channel/sender and impose a short lifetime and a
59
+ bounded queue. The upstream header has **no message ID**, so concurrent
60
+ same-sender messages cannot be disambiguated reliably from payload bytes.
61
+ Do not silently merge unrelated generations. A Meshtastic packet ID identifies
62
+ one radio packet, not all chunks of an application message.
63
+
64
+ `ATAKBCAST,mesh-id,uid,callsign,initial` discovery broadcasts are returned as
65
+ `:forwarder_discovery`, not sent through a CoT decoder.
66
+
67
+ ## EXI blocker — full Forwarder decoding is not complete
68
+
69
+ libcotshrink's lossless mode uses EXIficient `DefaultEXIFactory.newInstance()`;
70
+ its lossy mode falls back to that mode for unsupported CoT details. Either can
71
+ be GZIP wrapped. No XSD is supplied: the required format is schema-less EXI,
72
+ not a fixed CoT field table. Real support needs EXI header/options handling,
73
+ bit-level event codes, dynamically learned XML grammars, QName/string-table
74
+ partitions and EXI datatype decoding. A zlib inflater or protobuf schema cannot
75
+ replace this engine.
76
+
77
+ This implementation recognizes the EXI distinguishing bits/cookie and fails
78
+ explicitly. No working Ruby EXI engine or captured EXI wire fixture was found
79
+ in this research. That is an implementation/dependency gap, not a claim that
80
+ EXI is mathematically impossible in Ruby. The superficially named `xi_parser`
81
+ gem is a personal-wiki-to-HTML parser, not EXI. No Java or Python subprocess is
82
+ used; full EXI support remains unimplemented.
83
+
84
+ ## Provenance and verification
85
+
86
+ The generated `forwarder_pb.rb` embeds FileDescriptorProto bytes from
87
+ `paulmandal/libcotshrink` revision
88
+ `7d818b0d119a8df291770923b463fbb734b7e1b0`. They were produced with `protoc
89
+ --include_imports --descriptor_set_out=... cotevent.proto` and registered in a
90
+ private Ruby DescriptorPool in dependency order. The upstream MIT license is
91
+ included in that generated file. Runtime requires neither protoc nor the
92
+ Android libraries.
93
+
94
+ Specs include an independent `protoc --encode` wire fixture, direct
95
+ hand-authored wire, packed-bit/null handling, nested schema decoding, actual
96
+ nibble framing, shuffled reassembly, missing/duplicate fragments, discovery,
97
+ GZIP and output limits. These are schema-conformance fixtures, **not captured
98
+ radio packets**. Upstream `HackyTests.java` supplies CoT XML test inputs and
99
+ Android runtime round-trip tests, not checked-in binary fixtures. The Android
100
+ implementation was inspected but not executed, and interoperability with live
101
+ hardware is not claimed.
102
+
103
+ Primary implementation sources:
104
+
105
+ * [CotShrinker format selection and GZIP](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/pub/api/CotShrinker.java)
106
+ * [CoT schema](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/proto/cotevent.proto)
107
+ * [Detail schema graph](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/proto/detail/detail.proto)
108
+ * [Packed time/height](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/protobuf/CustomBytesConverter.java)
109
+ * [Packed extensions/mappings](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/protobuf/CustomBytesExtConverter.java)
110
+ * [Nullable bit semantics](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/protobuf/utils/BitUtils.java)
111
+ * [Local-year epoch](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/protobuf/cotevent/CotEventProtobufConverterFactory.java)
112
+ * [EXI configuration](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/exi/ExiConverterFactory.java), [EXI standard](https://www.w3.org/TR/exi/)
113
+ * [Forwarder chunk construction](https://github.com/paulmandal/atak-forwarder/blob/ad47f2c69e45785eb6029945be232bc52acd695c/app/src/main/java/com/paulmandal/atak/forwarder/comm/meshtastic/MeshSender.java)
114
+ * [Forwarder chunk reception](https://github.com/paulmandal/atak-forwarder/blob/ad47f2c69e45785eb6029945be232bc52acd695c/app/src/main/java/com/paulmandal/atak/forwarder/comm/meshtastic/InboundMeshMessageHandler.java)
115
+ * [Upstream XML tests](https://github.com/paulmandal/libcotshrink/blob/7d818b0d119a8df291770923b463fbb734b7e1b0/src/main/java/com/paulmandal/atak/libcotshrink/hackytests/HackyTests.java)
116
+
117
+ ## Read-only fragmentation assessment: ports 76 and 75
118
+
119
+ ### Reticulum tunnel (76)
120
+
121
+ The inspected [RNS Meshtastic interface, revision e5eb5d23](https://github.com/landandair/RNS_Over_Meshtastic/blob/e5eb5d23a619958ae0556c88fcbc4bf1d867ab0a/Interface/Meshtastic_Interface.py)
122
+ uses a two-byte `Bb` header: unsigned message index and signed fragment
123
+ position. Positions start at 1; a negative position marks the final fragment
124
+ and its absolute value is the total count. `REQ` followed by the same two-byte
125
+ metadata requests retransmission; it is not RNS packet data. Reassembly must
126
+ key by sender and message index (plus connection/channel isolation), wait for
127
+ every position through the known last index, strip headers, and only then
128
+ parse the RNS packet. Out-of-order final-first delivery, duplicate conflicts,
129
+ index wrap/reuse, timeouts, maximum packet bytes and bounded state all need
130
+ coverage. An encrypted RNS inner payload cannot be made plaintext by tunnel
131
+ reassembly. This assessment does not assert every third-party port-76 sender
132
+ uses this identical protocol. [Reticulum](reticulum.md) implements this pinned
133
+ tunnel framing and explicitly grouped, bounded reassembly; the receive dispatcher
134
+ decodes individual frames only. Inner RNS parsing and decryption remain unsupported.
135
+
136
+ ### LoRaWAN bridge (75)
137
+
138
+ The [pinned authoritative bridge protobuf](https://github.com/meshtastic/protobufs/blob/51028ca5a6945c76d3977c2bb803f9947d319ac5/meshtastic/lorawan_bridge.proto)
139
+ explicitly defines fragmentation: Uplink/Downlink `chunk_count == 0` is a full
140
+ PHY frame; `chunk_count == 2` means `payload` is its first part and `payload_id`
141
+ is 1–255. A separate `PayloadChunk` has the same ID, `chunk_index == 1` and the
142
+ remaining `payload_chunk`. Reassembly keys on sender and payload ID; retain
143
+ head RF metadata, append exactly one continuation and reject other counts or
144
+ indexes. Isolate connections/channels/directions, expire incomplete entries,
145
+ handle continuation-before-head, and reject conflicting duplicates. This is
146
+ not the RNS signed-position format. Only the reassembled LoRaWAN PHYPayload
147
+ should reach a LoRaWAN frame parser; application plaintext also needs the
148
+ correct LoRaWAN security context. TxResult correlates by `request_id`, not
149
+ `tmst`. No LoRaWAN transport/reassembly code was changed.