@moqtap/trace 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (12) hide show
  1. package/README.md +95 -0
  2. package/SPEC.md +378 -111
  3. package/dist/index.cjs +461 -54
  4. package/dist/index.d.cts +251 -5
  5. package/dist/index.d.ts +251 -5
  6. package/dist/index.js +460 -53
  7. package/package.json +2 -2
  8. package/src/binary.ts +467 -57
  9. package/src/cbor-scan.ts +185 -0
  10. package/src/index.ts +18 -1
  11. package/src/recorder.ts +10 -0
  12. package/src/types.ts +198 -2
package/SPEC.md CHANGED
@@ -1,78 +1,32 @@
1
- # `.moqtrace` File Format Specification
1
+ # `.moqtrace` Specification
2
2
 
3
- > **Version:** 1
3
+ > **Version:** 2
4
4
  > **Status:** Draft
5
5
 
6
6
  ## Overview
7
7
 
8
- `.moqtrace` is a binary file format for recording MoQT (Media over QUIC Transport) sessions. It captures protocol-level events at configurable detail levels, from lightweight control-message-only traces to full wire-level recordings with payload bytes.
8
+ This document defines two layered artifacts:
9
9
 
10
- The format is designed to be:
10
+ 1. **The MoQT trace event model** (Part 1, normative) — the set of event types, their fields, and their semantics. This model is transport-independent; it can be serialized to a file, streamed over MoQT, shipped as JSON to a collector, or embedded in another container.
11
+ 2. **The `.moqtrace` binary file format** (Part 2, normative) — a CBOR-based container for the event model above. Streamable, segmentable, self-describing.
12
+
13
+ A third artifact, the MoQT carriage convention for live trace transport, is **Appendix A (informative)**. It is transport behavior rather than file-format core and is expected to migrate to a separate moqtap protocol draft; it is included here only so early implementers have a single reference.
14
+
15
+ Design goals of the `.moqtrace` container:
11
16
 
12
17
  - **Streamable** — events can be appended as they occur; no backpatching required
18
+ - **Segmentable** — a stream can be carved into self-contained segments, each independently parseable, enabling live carriage and capture rotation
13
19
  - **Compact** — CBOR encoding handles binary data natively (no base64 overhead)
14
20
  - **Self-describing** — the header declares the detail level, protocol version, and recording context so readers know what to expect without scanning events
15
21
  - **Cross-language** — readable by any language with a CBOR library (JS, Rust, Go, Python)
16
22
 
17
- ## File Layout
23
+ ---
18
24
 
19
- ```
20
- Offset Length Content
21
- ------ ------ -------
22
- 0 8 Magic bytes: "MOQTRACE" (0x4d4f515452414345)
23
- 8 4 Format version (uint32 LE) — currently 1
24
- 12 4 Header length (uint32 LE) — byte length of the CBOR header blob
25
- 16 N Header (CBOR map) — session metadata and recording configuration
26
- 16+N ... Event stream (CBOR sequence) — concatenated CBOR items, one per event
27
- ```
25
+ # Part 1: Event Model (normative)
28
26
 
29
- ### Magic Bytes
30
-
31
- The 8-byte ASCII string `MOQTRACE` identifies the file format. Any tool can check these bytes before attempting to parse.
32
-
33
- ### Format Version
34
-
35
- uint32 little-endian. Currently `1`. Readers MUST reject files with a version they don't support.
27
+ ## Detail Levels
36
28
 
37
- ### Header Length
38
-
39
- uint32 little-endian. The byte length of the CBOR-encoded header that immediately follows. This allows readers to extract metadata without scanning the event stream — useful for file browsers, search indexing, and quick filtering ("show me all draft-14 client traces").
40
-
41
- ## Header
42
-
43
- A single CBOR map with the following keys:
44
-
45
- | Key | CBOR Type | Required | Description |
46
- | --------------- | ----------- | -------- | --------------------------------------------------------------------------------------------- |
47
- | `"protocol"` | text string | Yes | MoQT version identifier (e.g., `"draft-14"`, `"rfc9999"`) |
48
- | `"perspective"` | text string | Yes | Recording viewpoint: `"client"`, `"server"`, or `"observer"` |
49
- | `"detail"` | text string | Yes | Detail level (see below) |
50
- | `"startTime"` | integer | Yes | Recording start time (Unix epoch milliseconds) |
51
- | `"transport"` | text string | No | Transport type (e.g., `"webtransport"`, `"raw-quic"`) |
52
- | `"endTime"` | integer | No | Recording end time (Unix epoch milliseconds). Written when trace is finalized. |
53
- | `"source"` | text string | No | Software that produced the trace (e.g., `"moqtap-devtools/0.1.0"`, `"my-relay/2.3.1"`) |
54
- | `"endpoint"` | text string | No | Remote peer URI (e.g., `"https://relay.example.com/moq"`) |
55
- | `"sessionId"` | text string | No | Opaque identifier for correlating traces from the same session across multiple capture points |
56
- | `"custom"` | map | No | User-defined metadata (arbitrary key-value pairs) |
57
-
58
- ### Protocol Identifier
59
-
60
- The `"protocol"` field identifies the MoQT version without assuming lifecycle stage. Use the IETF document name minus the `draft-ietf-` prefix for drafts, and the RFC number for published standards:
61
-
62
- - Draft phase: `"moq-transport-07"`, `"moq-transport-14"`
63
- - RFC phase: `"moq-transport-rfc9999"`
64
-
65
- ### Perspective
66
-
67
- The `"perspective"` field describes the vantage point of the recording:
68
-
69
- - `"client"` — captured at the MoQT client (initiator of the QUIC connection)
70
- - `"server"` — captured at the MoQT server or relay
71
- - `"observer"` — passive capture (e.g., DevTools extension, network tap) that intercepts traffic without participating in the session
72
-
73
- ### Detail Levels
74
-
75
- The `"detail"` field declares what was recorded. Each level is a strict superset of the one above it. Readers can handle any level — they just find more or fewer fields populated per event.
29
+ The `"detail"` field (see Part 2 header) declares what was recorded. Each level is a strict superset of the one above it. Readers can handle any level — they just find more or fewer fields populated per event.
76
30
 
77
31
  | Detail Level | What's Recorded | Typical Use Case |
78
32
  | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- |
@@ -82,48 +36,69 @@ The `"detail"` field declares what was recorded. Each level is a strict superset
82
36
  | `"headers+data"` | Everything in `"headers"` + full payload bytes for each object. | Full session replay, debugging media corruption. |
83
37
  | `"full"` | Everything above + raw wire bytes for every message (pre-decode). | Wire-level debugging, compliance testing. |
84
38
 
85
- At any detail level, the recorder MAY be configured to **mask payloads** — replacing payload bytes with zeroes before writing. This preserves payload sizes for bandwidth analysis while stripping media content. The header SHOULD include `"payloadMasked": true` in the `"custom"` map when this is active.
39
+ Levels `"headers+data"` and `"full"`, and the `"raw"` field on `"control"`-level events, are **payload-bearing** — see [Privacy Considerations](#privacy-considerations).
86
40
 
87
- ## Event Stream
41
+ At any detail level, the recorder MAY be configured to **mask payloads** — replacing payload bytes with zeroes before writing. This preserves payload sizes for bandwidth analysis while stripping media content. Sources MUST advertise masking via `"payloadMasked": true` in the header's `"custom"` map when active.
42
+
43
+ ## Event Types
44
+
45
+ | `"e"` Value | Name | Description |
46
+ | ----------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------ |
47
+ | `0` | Control message | A control-stream message was sent or received |
48
+ | `1` | Stream opened | A unidirectional or bidirectional QUIC stream was opened |
49
+ | `2` | Stream closed | A QUIC stream was closed |
50
+ | `3` | Object header | An object header was parsed from a data stream |
51
+ | `4` | Object payload | Object payload bytes were received/sent |
52
+ | `5` | State change | Session FSM phase transition |
53
+ | `6` | Error | Protocol error or transport error |
54
+ | `7` | Annotation | User-defined event (custom label + data) |
55
+ | `8` | Peer connected | A new peer session was established (relay-tap) |
56
+ | `9` | Peer disconnected | A peer session ended (relay-tap) |
57
+ | `10` | Subscription derivation | A subscription was created or extended in causal response to other subscriptions (relay-tap; multi-hop correlation) |
88
58
 
89
- After the header, the remainder of the file is a sequence of concatenated CBOR items (one per event). This is a standard CBOR sequence (RFC 8742) — no array wrapper, no separators.
59
+ Event types `8`, `9` and `10` are defined by this version but are emitted only by `"relay-tap"` sources. A reader MUST be able to parse them; a recorder that is not a relay tap never produces them.
90
60
 
91
- ### Common Event Fields
61
+ ## Common Event Fields
92
62
 
93
63
  Every event is a CBOR map with at minimum:
94
64
 
95
65
  | Key | CBOR Type | Description |
96
66
  | ----- | ---------------- | --------------------------------------------------------------------------------------------------- |
97
- | `"n"` | unsigned integer | Monotonically increasing sequence number (0-based). Disambiguates events with identical timestamps. |
98
- | `"t"` | integer | Timestamp in microseconds since `startTime` in the header |
99
- | `"e"` | integer | Event type (see table below) |
67
+ | `"n"` | unsigned integer | Monotonically increasing sequence number (0-based). Disambiguates events with identical timestamps. Segment-local in segmented traces. |
68
+ | `"t"` | integer | Timestamp in microseconds since the containing segment's `startTime`. |
69
+ | `"e"` | integer | Event type (see table above). |
70
+ | `"p"` | text string | **Optional.** Peer identifier. REQUIRED when `"perspective": "relay-tap"`; identifies which connected peer the event pertains to. Source-local scope — see [Identifier Scoping](#identifier-scoping). |
100
71
 
101
72
  Short key names are used intentionally — traces can contain hundreds of thousands of events, and CBOR encodes short strings more compactly.
102
73
 
103
- ### Event Types
74
+ ## Identifier Scoping
104
75
 
105
- | `"e"` Value | Name | Description |
106
- | ----------- | --------------- | -------------------------------------------------------- |
107
- | `0` | Control message | A control-stream message was sent or received |
108
- | `1` | Stream opened | A unidirectional or bidirectional QUIC stream was opened |
109
- | `2` | Stream closed | A QUIC stream was closed |
110
- | `3` | Object header | An object header was parsed from a data stream |
111
- | `4` | Object payload | Object payload bytes were received/sent |
112
- | `5` | State change | Session FSM phase transition |
113
- | `6` | Error | Protocol error or transport error |
114
- | `7` | Annotation | User-defined event (custom label + data) |
76
+ The event model uses three distinct identifiers with different scopes. Tooling MUST NOT conflate them.
115
77
 
116
- ### Event-Specific Fields
78
+ | Identifier | Scope | Assigned by | Purpose |
79
+ | -------------- | ---------------------------------------------------- | ------------------------------------------------- | -------------------------------------------------------------- |
80
+ | `"p"` | **Source-local** — unique only within one trace source (one recorder, one `sessionId`) | The recording source; format is source-chosen (UUID, numeric ID, `version@addr`, etc.) | Demultiplexing events from multiple peers within a relay-tap trace. |
81
+ | `"sessionId"` | **Capture-correlation** — groups traces recorded from a single logical session by different capture points (e.g., client + server + observer) | Chosen out-of-band by the operator, or propagated via session setup | Aligning multiple viewpoints on the **same** session. |
82
+ | `"traceId"` | **End-to-end / federated** — globally unique across operator boundaries | The originating endpoint, propagated on-wire via `MOQTAP_TRACE_ID` | Stitching multi-hop subscription chains into a single tree, even across organizations. |
117
83
 
118
- #### Event 0: Control Message
84
+ Because `"p"` is source-local, a collector correlating traces from multiple sources MUST NOT assume that the same `"p"` value in two different traces refers to the same peer. To cross-reference peers across sources, use `"traceId"` (for subscription-linked events) or compose `(source, p)` as the effective key (using the header's `"source"` field).
119
85
 
120
- | Key | Type | Detail Level | Description |
121
- | ------- | ----------- | ------------ | ---------------------------------------------------------------- |
122
- | `"d"` | integer | `control`+ | Direction: `0` = sent (tx), `1` = received (rx) |
123
- | `"mt"` | integer | `control`+ | Wire message type ID (e.g., `0x03` for SUBSCRIBE) |
124
- | `"msg"` | map | `control`+ | Decoded message fields (draft-specific structure) |
86
+ `"sessionId"` and `"traceId"` are **orthogonal**: a single session may carry many subscriptions each with a distinct trace ID, and a single end-to-end trace may span many sessions on many hosts. Presence rules:
87
+
88
+ - `"sessionId"` SHOULD be set whenever the recording party has a stable session handle and more than one vantage point is likely to be captured (e.g., client-side and server-side of the same WebTransport session). It MAY be absent in single-vantage captures.
89
+ - `"traceId"` appears **only on events derived from a SUBSCRIBE carrying `MOQTAP_TRACE_ID`**. It is never required on the header. An event with no triggering subscription (e.g., Event 8 Peer Connected) has no trace ID.
90
+
91
+ ## Event-Specific Fields
92
+
93
+ ### Event 0: Control Message
94
+
95
+ | Key | Type | Detail Level | Description |
96
+ | ------- | ----------- | ------------ | ------------------------------------------------- |
97
+ | `"d"` | integer | `control`+ | Direction: `0` = sent (tx), `1` = received (rx) |
98
+ | `"mt"` | integer | `control`+ | Wire message type ID (e.g., `0x03` for SUBSCRIBE) |
99
+ | `"msg"` | map | `control`+ | Decoded message fields (draft-specific structure) |
125
100
  | `"sid"` | integer | `control`+ | QUIC stream ID the message travelled on. Optional (see below). |
126
- | `"raw"` | byte string | `full` only | Raw wire bytes (including type and length prefix) |
101
+ | `"raw"` | byte string | `full` only | Raw wire bytes (including type and length prefix). Payload-bearing — see [Privacy Considerations](#privacy-considerations). |
127
102
 
128
103
  `"sid"` is optional: a recorder that sits at the session level rather than the
129
104
  stream level has no stream to report, and omitting the key says so. Readers
@@ -139,7 +114,7 @@ track names, aliases, request outcomes — is lost to the reader. Through
139
114
  draft-16 all control messages share one stream and the field is merely
140
115
  informative.
141
116
 
142
- #### Event 1: Stream Opened
117
+ ### Event 1: Stream Opened
143
118
 
144
119
  | Key | Type | Detail Level | Description |
145
120
  | ------- | ------- | ------------ | -------------------------------------------------------- |
@@ -147,14 +122,14 @@ informative.
147
122
  | `"d"` | integer | `headers`+ | Direction: `0` = outgoing, `1` = incoming |
148
123
  | `"st"` | integer | `headers`+ | Stream type: `0` = subgroup, `1` = datagram, `2` = fetch |
149
124
 
150
- #### Event 2: Stream Closed
125
+ ### Event 2: Stream Closed
151
126
 
152
127
  | Key | Type | Detail Level | Description |
153
128
  | ------- | ------- | ------------ | ---------------------------- |
154
129
  | `"sid"` | integer | `headers`+ | QUIC stream ID |
155
130
  | `"ec"` | integer | `headers`+ | Error code (0 = clean close) |
156
131
 
157
- #### Event 3: Object Header
132
+ ### Event 3: Object Header
158
133
 
159
134
  Object header events and object payload events (event 4) for the same object share `(sid, g, o)` as a composite key. An object header event is always emitted before its corresponding payload event.
160
135
 
@@ -166,37 +141,249 @@ Object header events and object payload events (event 4) for the same object sha
166
141
  | `"pp"` | integer | `headers`+ | Publisher priority |
167
142
  | `"os"` | integer | `headers`+ | Object status (0=normal, 1=end-of-group, 2=end-of-track, 3=does-not-exist) |
168
143
 
169
- #### Event 4: Object Payload
144
+ ### Event 4: Object Payload
145
+
146
+ **Payload-bearing event** — see [Privacy Considerations](#privacy-considerations).
170
147
 
171
- | Key | Type | Detail Level | Description |
172
- | ------- | ----------- | ---------------- | ----------------------------------- |
173
- | `"sid"` | integer | `headers+sizes`+ | Stream ID |
174
- | `"g"` | integer | `headers+sizes`+ | Group ID |
175
- | `"o"` | integer | `headers+sizes`+ | Object ID |
176
- | `"sz"` | integer | `headers+sizes`+ | Payload size in bytes |
177
- | `"pl"` | byte string | `headers+data`+ | Payload bytes (or zeroed if masked) |
148
+ | Key | Type | Detail Level | Description |
149
+ | ------- | ----------- | ---------------- | ---------------------------------------------------------- |
150
+ | `"sid"` | integer | `headers+sizes`+ | Stream ID |
151
+ | `"g"` | integer | `headers+sizes`+ | Group ID |
152
+ | `"o"` | integer | `headers+sizes`+ | Object ID |
153
+ | `"sz"` | integer | `headers+sizes`+ | Payload size in bytes |
154
+ | `"pl"` | byte string | `headers+data`+ | Payload bytes (or zeroed if masked). Payload-bearing. |
178
155
 
179
- #### Event 5: State Change
156
+ ### Event 5: State Change
180
157
 
181
158
  | Key | Type | Detail Level | Description |
182
159
  | -------- | ----------- | ------------ | ---------------------- |
183
160
  | `"from"` | text string | `control`+ | Previous session phase |
184
161
  | `"to"` | text string | `control`+ | New session phase |
185
162
 
186
- #### Event 6: Error
163
+ ### Event 6: Error
187
164
 
188
165
  | Key | Type | Detail Level | Description |
189
166
  | ---------- | ----------- | ------------ | --------------------- |
190
167
  | `"ec"` | integer | `control`+ | Error code |
191
168
  | `"reason"` | text string | `control`+ | Human-readable reason |
192
169
 
193
- #### Event 7: Annotation
170
+ ### Event 7: Annotation
194
171
 
195
172
  | Key | Type | Detail Level | Description |
196
173
  | --------- | ----------- | ------------ | --------------------------------- |
197
174
  | `"label"` | text string | any | User-defined label |
198
175
  | `"data"` | any | any | User-defined data (any CBOR type) |
199
176
 
177
+ ### Event 8: Peer Connected
178
+
179
+ Emitted by `"relay-tap"` recorders when a new peer session is established through the relay.
180
+
181
+ | Key | Type | Detail Level | Description |
182
+ | -------------- | ----------- | ------------ | ------------------------------------------------------------------------------------ |
183
+ | `"p"` | text string | `control`+ | Peer identifier (source-local); subsequent events from this peer carry the same `"p"`. |
184
+ | `"endpoint"` | text string | `control`+ | Peer-reported endpoint URI or remote address (best-effort). Optional. |
185
+ | `"transport"` | text string | `control`+ | Transport type (`"webtransport"`, `"raw-quic"`, etc.). Optional. |
186
+ | `"role"` | text string | `control`+ | `"publisher"`, `"subscriber"`, or `"both"` if known at connection time; else absent. |
187
+ | `"side"` | text string | `control`+ | `"downstream"` if the peer connected *to* this relay; `"upstream"` if this relay connected *to* the peer (origin or upstream relay). Absent for non-relay perspectives. Identifies the connection direction, not subscription causality — see Event 10 for subscription chains. |
188
+
189
+ ### Event 9: Peer Disconnected
190
+
191
+ | Key | Type | Detail Level | Description |
192
+ | ---------- | ----------- | ------------ | -------------------------------------------------------- |
193
+ | `"p"` | text string | `control`+ | Peer identifier (matches the corresponding event 8). |
194
+ | `"ec"` | integer | `control`+ | Error/close code (0 = clean close). |
195
+ | `"reason"` | text string | `control`+ | Human-readable reason. Optional. |
196
+
197
+ ### Event 10: Subscription Derivation
198
+
199
+ Emitted by `"relay-tap"` recorders when an upstream subscription is causally created or extended to satisfy one or more downstream subscriptions. This is the structural primitive for multi-hop correlation: the collector reconstructs end-to-end trace trees from these derivation links plus `"traceId"` propagation (see [Trace ID Propagation](#trace-id-propagation)).
200
+
201
+ The same event is emitted again when an additional downstream subscription is satisfied by an existing upstream subscription (subscription fan-in over time) — the `"d"` array reflects all currently linked downstream subs at emission time.
202
+
203
+ | Key | Type | Detail Level | Description |
204
+ | ----------- | --------------------- | ------------ | --------------------------------------------------------------------------------------- |
205
+ | `"u"` | array `[text, uint]` | `control`+ | `[upstream-peer-id, upstream-request-id]` — the upstream subscription. Peer IDs are source-local. |
206
+ | `"d"` | array of arrays | `control`+ | List of `[downstream-peer-id, downstream-request-id]` pairs causing/sharing this upstream sub. |
207
+ | `"kind"` | text string | `control`+ | `"created"` (new upstream sub) or `"shared"` (existing upstream sub now serving an additional downstream sub). |
208
+ | `"traceId"` | byte string | `control`+ | Optional. 16-byte trace ID propagated via the `MOQTAP_TRACE_ID` extension parameter (see [Trace ID Propagation](#trace-id-propagation)). |
209
+ | `"ns"` | array of byte strings | `control`+ | Optional. Track namespace the subscription targets, one byte string per namespace field. |
210
+ | `"tn"` | byte string | `control`+ | Optional. Track name the subscription targets. |
211
+ | `"tdr"` | integer | `control`+ | Optional. Timestamp at which the downstream SUBSCRIBE was received. |
212
+ | `"tus"` | integer | `control`+ | Optional. Timestamp at which the upstream SUBSCRIBE was transmitted. Absent for terminal subscriptions, where the relay is the content origin. |
213
+ | `"tuo"` | integer | `control`+ | Optional. Timestamp at which SUBSCRIBE_OK was received from upstream. Absent for terminal or still-in-flight subscriptions. |
214
+ | `"tdo"` | integer | `control`+ | Optional. Timestamp at which SUBSCRIBE_OK was transmitted downstream. Absent for still-in-flight subscriptions. |
215
+
216
+ **Timestamps.** `"tdr"`, `"tus"`, `"tuo"` and `"tdo"` use the same timebase as the common `"t"` field — microseconds since the containing segment's `startTime`, on the emitting source's clock. Differences between them are therefore meaningful without any cross-hop clock agreement, which is the point: they measure how long *this* relay took, not when two relays each think it was. A consumer MUST NOT subtract a timestamp produced by one source from one produced by another.
217
+
218
+ **Emission and update.** A source SHOULD emit this event as soon as the downstream SUBSCRIBE is received, carrying whichever timestamps are known at that moment, and MAY emit further events for the same `(u, d)` pair as later timestamps become available. A consumer MUST treat a later event for the same pair as an update, overwriting earlier field values, rather than recording a second derivation.
219
+
220
+ **Track identity is optional but recommended.** `"ns"` and `"tn"` are not required — the pair can in principle be recovered by correlating request IDs against the SUBSCRIBE control-message events. Including them makes the event self-contained, which matters when the trace is filtered, sampled at the control-message level, or joined across sources that do not share a control-message stream.
221
+
222
+ ## Sampling
223
+
224
+ When the source dropped, sampled, or filtered events before writing them, the header SHOULD include a `"sampling"` map so consumers know they are seeing a partial view.
225
+
226
+ | Key | CBOR Type | Required | Description |
227
+ | ------------------ | ---------------- | -------- | -------------------------------------------------------------------------------------------------------- |
228
+ | `"effectiveRate"` | float | No | Effective fraction of source events retained, in `(0.0, 1.0]`. `1.0` means no rate-based dropping. |
229
+ | `"maxEventsPerSec"`| unsigned integer | No | Per-segment cap that triggered drops, if rate-limited. |
230
+ | `"dropPolicy"` | text string | No | Drop strategy when the cap was exceeded: `"head"`, `"tail"`, or `"sampled"`. |
231
+ | `"droppedTotal"` | unsigned integer | No | Cumulative events dropped since `startTime`. In segmented traces, this is the running total at segment start. |
232
+ | `"droppedSegment"` | unsigned integer | No | Events dropped within this segment only. Only meaningful when `"segment"` is present. |
233
+ | `"rule"` | text string | No | Source-side filter rule that selected events (e.g., `"namespace prefix=foo/bar"`). |
234
+ | `"ruleLang"` | text string | No | Filter language identifier: `"prefix"`, `"suffix"`, `"glob"`, `"cel"`, etc. |
235
+ | `"appliesTo"` | array of int | When sampling is active | Event type IDs to which the drop policy was applied. |
236
+
237
+ A reader that sees a `"sampling"` map SHOULD surface the fact that the trace is partial. A trace without `"sampling"` is presumed complete relative to the declared detail level.
238
+
239
+ ### Non-droppable Event Types
240
+
241
+ Some event types carry causal or structural information that is destroyed by random sampling — a SUBSCRIBE without its SUBSCRIBE_OK, a state change with no following events, a peer-disconnect with no matching peer-connect — yielding a misleading trace. Sources MUST NOT drop or sample the following event types under any drop policy:
242
+
243
+ - Event `0` — Control message
244
+ - Event `5` — State change
245
+ - Event `6` — Error
246
+ - Event `8` — Peer connected
247
+ - Event `9` — Peer disconnected
248
+ - Event `10` — Subscription derivation
249
+
250
+ If a source cannot keep up with the rate of these event types, it MUST refuse the recording or subscription rather than emit a sampled trace. Source-side **filtering** (e.g., dropping all events for namespace `foo/bar`) is not sampling and remains permitted — filtered traces are still causally complete with respect to what passed the filter.
251
+
252
+ Sampling MAY apply to the high-volume event types: `1` (Stream opened), `2` (Stream closed), `3` (Object header), `4` (Object payload), and `7` (Annotation). When sampling is applied, `"appliesTo"` MUST list the affected event types; readers MAY then assume non-listed event types are complete.
253
+
254
+ ## Trace ID Propagation
255
+
256
+ For end-to-end correlation across multiple relays and clients, moqtap defines a `MOQTAP_TRACE_ID` extension parameter (see the moqtap draft for the assigned parameter ID) carried on SUBSCRIBE messages. The trace ID is a **16-byte opaque identifier** that:
257
+
258
+ - Is generated by the originating endpoint (publisher client, subscriber client, or first moqtap-aware relay if no upstream party set one).
259
+ - MUST be propagated unchanged onto upstream SUBSCRIBEs that a relay creates to satisfy a downstream SUBSCRIBE carrying it.
260
+ - SHOULD appear as the `"traceId"` field on Event 10 (Subscription Derivation) and MAY appear on any other event whose triggering subscription carries one.
261
+ - Allows a moqtap collector to stitch traces from multiple sources (clients pushing moqtraces, relays exposing tap tracks) into a unified end-to-end view, with no topology configuration required.
262
+
263
+ **The trace ID is carried as raw bytes at every layer** — a 16-byte parameter value on the wire, a 16-byte CBOR byte string in `"traceId"`. It is never hex-encoded, base64-encoded, or otherwise transformed. This is not a stylistic preference: stitching only works if two independently written implementations produce byte-identical values for the same subscription chain, and every encoding step is a place for them to disagree. A source holding a trace ID whose length is not exactly 16 bytes MUST treat it as malformed rather than pad, truncate, or re-encode it.
264
+
265
+ Privacy: trace IDs reveal subscription patterns across operator boundaries. Relay operators MAY hash, replace, or strip the trace ID at egress per their policy. The moqtap draft documents the recommended defaults.
266
+
267
+ ## Privacy Considerations
268
+
269
+ Trace recording ranges from protocol-metadata-only (low sensitivity) to full media capture (very high sensitivity). Implementers and operators MUST treat the following as elevated-risk capabilities:
270
+
271
+ 1. **Payload-bearing events and fields.** These carry or reveal user content:
272
+ - Event 4 `"pl"` (object payload bytes, `"headers+data"`+)
273
+ - Event 0 `"raw"` (raw control-message bytes, `"full"`)
274
+ - Detail levels `"headers+data"` and `"full"` in aggregate
275
+
276
+ Capturing them is **optional** for all conformant recorders. Observer- and tap-based capture of these fields is equivalent to intercepting user media; recorders MUST NOT enable it by default and SHOULD require explicit operator opt-in per session or per track. The default detail level for general-purpose tools SHOULD be `"control"` or `"headers"`.
277
+
278
+ 2. **Payload size leakage.** Even without payload bytes, `"headers+sizes"` reveals object sizes, which can fingerprint media content. Operators handling sensitive traffic SHOULD evaluate size leakage as part of their threat model.
279
+
280
+ 3. **Trace ID correlation.** `MOQTAP_TRACE_ID` enables cross-operator correlation of subscription behavior. See [Trace ID Propagation](#trace-id-propagation) for egress mitigations.
281
+
282
+ 4. **Peer identifiers.** The `"p"` field is source-local but may encode address- or user-identifying information if the source chooses to. Publishers of traces SHOULD redact or opaquely hash `"p"` before sharing traces outside the recording operator's trust boundary.
283
+
284
+ 5. **Passive (observer / relay-tap) capture.** Perspectives `"observer"` and `"relay-tap"` record traffic that the recording party is not a direct participant in. Operators MUST ensure they have authorization to record such traffic under applicable law and session-setup consent; this specification does not grant such authorization.
285
+
286
+ Retention, access control, and disclosure of `.moqtrace` files are out of scope here but are expected to be governed by the deploying operator's policy.
287
+
288
+ ---
289
+
290
+ # Part 2: `.moqtrace` Binary File Format
291
+
292
+ A `.moqtrace` file serializes the event model above to a self-describing CBOR container.
293
+
294
+ ## File Layout
295
+
296
+ ```
297
+ Offset Length Content
298
+ ------ ------ -------
299
+ 0 8 Magic bytes: "MOQTRACE" (0x4d4f515452414345)
300
+ 8 4 Format version (uint32 LE) — currently 2
301
+ 12 4 Header length (uint32 LE) — byte length of the CBOR header blob
302
+ 16 N Header (CBOR map) — session metadata and recording configuration
303
+ 16+N ... Event stream (CBOR sequence) — concatenated CBOR items, one per event
304
+ ```
305
+
306
+ A **segmented** trace is a concatenation of one or more such layouts back-to-back; each segment begins with magic bytes and is independently parseable. See [Segmented Traces](#segmented-traces).
307
+
308
+ ### Magic Bytes
309
+
310
+ The 8-byte ASCII string `MOQTRACE` identifies the file format. Any tool can check these bytes before attempting to parse. In segmented traces, the same bytes also serve as a segment delimiter — readers can resync to the next segment by scanning for the magic.
311
+
312
+ ### Format Version
313
+
314
+ uint32 little-endian. Currently `2`. See [Versioning and Compatibility](#versioning-and-compatibility) for which versions a reader must accept.
315
+
316
+ ### Header Length
317
+
318
+ uint32 little-endian. The byte length of the CBOR-encoded header that immediately follows. This allows readers to extract metadata without scanning the event stream — useful for file browsers, search indexing, and quick filtering ("show me all draft-17 client traces").
319
+
320
+ ## Header
321
+
322
+ A single CBOR map with the following keys:
323
+
324
+ | Key | CBOR Type | Required | Description |
325
+ | --------------- | ----------- | -------- | -------------------------------------------------------------------------------------------------------- |
326
+ | `"protocol"` | text string | Yes | MoQT version identifier (e.g., `"moq-transport-17"`, `"moq-transport-rfc9999"`) |
327
+ | `"perspective"` | text string | Yes | Recording viewpoint: `"client"`, `"server"`, `"observer"`, or `"relay-tap"` |
328
+ | `"detail"` | text string | Yes | Detail level (see [Detail Levels](#detail-levels)) |
329
+ | `"startTime"` | integer | Yes | Recording start time (Unix epoch milliseconds) |
330
+ | `"transport"` | text string | No | Transport type (e.g., `"webtransport"`, `"raw-quic"`) |
331
+ | `"endTime"` | integer | No | Recording end time (Unix epoch milliseconds). Written when trace is finalized. |
332
+ | `"source"` | text string | No | Software that produced the trace (e.g., `"moqtap-devtools/0.1.0"`, `"my-relay/2.3.1"`). Also serves as the namespace for source-local `"p"` values. |
333
+ | `"endpoint"` | text string | No | Remote peer URI (e.g., `"https://relay.example.com/moq"`) |
334
+ | `"sessionId"` | text string | No | Capture-correlation identifier — see [Identifier Scoping](#identifier-scoping). |
335
+ | `"segment"` | map | No | Segment metadata; present iff this trace is one segment of a segmented stream. See [Segmented Traces](#segmented-traces). |
336
+ | `"sampling"` | map | No | Sampling/filter metadata; present iff events were sampled or filtered at the source. See [Sampling](#sampling). |
337
+ | `"custom"` | map | No | User-defined metadata (arbitrary key-value pairs). `"payloadMasked": true` here indicates payload masking is active. |
338
+
339
+ ### Protocol Identifier
340
+
341
+ The `"protocol"` field identifies the MoQT version without assuming lifecycle stage. Use the IETF document name minus the `draft-ietf-` prefix for drafts, and the RFC number for published standards:
342
+
343
+ - Draft phase: `"moq-transport-07"`, `"moq-transport-17"`
344
+ - RFC phase: `"moq-transport-rfc9999"`
345
+
346
+ ### Perspective
347
+
348
+ The `"perspective"` field describes the vantage point of the recording:
349
+
350
+ - `"client"` — captured at the MoQT client (initiator of the QUIC connection)
351
+ - `"server"` — captured at the MoQT server or relay (single-session view from inside the endpoint)
352
+ - `"observer"` — passive capture (e.g., DevTools extension, network tap) that intercepts traffic without participating in the session
353
+ - `"relay-tap"` — active capture from inside a relay, reporting on **multiple** concurrent peer sessions through that relay. Distinct from `"server"` because events span peers; see the `"p"` field on events.
354
+
355
+ A reader that encounters a perspective value it does not recognize MUST NOT reject the trace. New values may be added without a version bump, and every event in the file remains parseable regardless. Preserve the value verbatim and surface it to the caller.
356
+
357
+ ## Event Stream
358
+
359
+ After the header, the remainder of the file (or segment) is a sequence of concatenated CBOR items (one per event). This is a standard CBOR sequence (RFC 8742) — no array wrapper, no separators. Event contents are defined in [Part 1](#part-1-event-model-normative).
360
+
361
+ ## Segmented Traces
362
+
363
+ A **segmented trace** is a sequence of self-contained `.moqtrace` blobs concatenated back-to-back. Each segment is independently parseable and includes its own magic bytes, format version, header, and event sequence. This enables:
364
+
365
+ - **On-disk capture rotation.** A long-running recorder can rotate to a new segment every N seconds or M megabytes without finalizing the file; consumers process completed segments while the recorder writes the next.
366
+ - **Live carriage.** A transport that wants to carry a `.moqtrace` stream live uses segment boundaries as its cut points — see [Appendix A](#appendix-a-moqt-carriage-convention-informative).
367
+ - **Best-effort recovery.** If a segment is truncated or corrupt, readers can skip forward to the next magic-byte sequence and resume.
368
+
369
+ A trace is segmented iff its header contains a `"segment"` map:
370
+
371
+ | Key | CBOR Type | Required | Description |
372
+ | ------------------ | ---------------- | -------- | ---------------------------------------------------------------------------------------------------------- |
373
+ | `"sequence"` | unsigned integer | Yes | 0-based sequence number of this segment within the stream. Monotonically increasing, no gaps in a single source. |
374
+ | `"durationMs"` | unsigned integer | No | Nominal segment duration in milliseconds (hint to readers; actual duration may differ). |
375
+ | `"streamId"` | text string | No | Opaque identifier shared across all segments of the same logical stream. |
376
+ | `"continues"` | boolean | No | `true` if this segment continues a previous one with the same `streamId`. `false` (or absent) for the first segment. |
377
+
378
+ Within a single segmented stream:
379
+
380
+ - The `"protocol"`, `"perspective"`, `"detail"`, and `"sessionId"` fields MUST be identical across all segments.
381
+ - `"startTime"` in each segment header refers to that segment's start, NOT the stream's start.
382
+ - Event `"n"` (sequence number) and `"t"` (timestamp) are **segment-local** in segmented traces: they restart at 0 in each segment. Readers stitching segments together compute global ordering from `(segment.sequence, n)`.
383
+ - `"sampling.droppedTotal"` is cumulative across the stream; `"sampling.droppedSegment"` is local to the segment.
384
+
385
+ A non-segmented trace MUST NOT include a `"segment"` field. Readers MUST treat the absence of `"segment"` as "this is a complete, single-segment file."
386
+
200
387
  ## Reading a `.moqtrace` File
201
388
 
202
389
  ```
@@ -204,16 +391,24 @@ Object header events and object payload events (event 4) for the same object sha
204
391
  2. Read 4 bytes (uint32 LE) → format version. Reject if unsupported.
205
392
  3. Read 4 bytes (uint32 LE) → header length N.
206
393
  4. Read N bytes → CBOR-decode to get header map.
207
- 5. Read remaining bytes as a CBOR sequence:
208
- - Repeatedly CBOR-decode one item at a time until EOF.
209
- - Each item is one event (a CBOR map).
394
+ 5. Read CBOR items one at a time:
395
+ - If the next 8 bytes are the magic "MOQTRACE", a new segment begins —
396
+ loop back to step 2 (segmented trace).
397
+ - Otherwise, CBOR-decode one item → one event map.
398
+ - Stop at EOF.
210
399
  ```
211
400
 
401
+ A reader that does not implement segmented mode MAY treat the first occurrence of `MOQTRACE` after byte 0 as EOF for the current trace.
402
+
403
+ If the stream ends part-way through a CBOR item, the file was truncated — a crashed recorder, an interrupted transfer. Readers SHOULD report this distinctly from a clean end-of-file, and SHOULD return the events decoded up to that point rather than discarding them. A truncated trace is still evidence, and the events before the cut are exactly as valid as they were.
404
+
212
405
  ## Writing a `.moqtrace` File
213
406
 
407
+ ### Single-segment (file) writer
408
+
214
409
  ```
215
410
  1. Write magic bytes "MOQTRACE"
216
- 2. Write format version (uint32 LE, currently 1)
411
+ 2. Write format version (uint32 LE, currently 2)
217
412
  3. CBOR-encode the header map → get header bytes
218
413
  4. Write header length (uint32 LE)
219
414
  5. Write header bytes
@@ -225,18 +420,34 @@ Object header events and object payload events (event 4) for the same object sha
225
420
  - Or accept that endTime may be absent in crash-truncated files
226
421
  ```
227
422
 
423
+ ### Segmented writer (file rotation)
424
+
425
+ ```
426
+ 1. For each segment:
427
+ a. Construct the segment header (include "segment.sequence", increment per segment).
428
+ b. Emit magic + version + headerLen + header bytes.
429
+ c. Emit one or more event CBOR items.
430
+ d. Close the segment (file rotation, etc.).
431
+ 2. The next segment begins fresh at step 1a; do NOT carry over event "n" or "t".
432
+ ```
433
+
228
434
  ## Versioning and Compatibility
229
435
 
230
- - **Version 1** is defined by this document.
231
- - Readers MUST check the format version before parsing.
436
+ - **Version 2** is defined by this document. **Version 1** is defined by the previous revision.
437
+ - **Writers SHOULD write version 2.**
438
+ - **Readers MUST accept both version 1 and version 2, and MUST reject any other version.** A version-1 file is exactly a non-segmented version-2 trace without the keys this revision adds, and every added key is optional. Accepting version 1 therefore costs a reader nothing and keeps every previously recorded capture readable.
232
439
  - Unknown keys in header maps or event maps MUST be ignored (forward-compatible).
233
- - Optional keys MAY be added to an existing event type without a version bump —
234
- `"sid"` on event 0 was added this way. A reader must therefore treat any
235
- optional key as absent-by-default rather than assuming files of a given
236
- version all carry the same keys.
237
- - New event types (`"e"` values) MAY be added in future versions; readers SHOULD skip unknown event types.
440
+ - Optional keys MAY be added to an existing event type without a version bump — `"sid"` on event 0 was added this way. A reader must therefore treat any optional key as absent-by-default rather than assuming files of a given version all carry the same keys.
441
+ - New event types (`"e"` values) MAY be added without a version bump; readers MUST skip unknown event types rather than failing. A reader that rejects an unknown `"e"` value turns every future addition into a breaking change.
442
+ - New `"perspective"` and `"detail"` values MAY likewise be added; see [Perspective](#perspective).
238
443
  - The detail level in the header is informational — readers MUST handle missing fields gracefully regardless of declared level.
239
444
 
445
+ ### Why version 2 exists
446
+
447
+ Every field this revision adds is additive and would have been legal in a version-1 file under the rules above. **Segmentation is the exception, and it is the sole reason for the bump.** A segmented stream splices a preamble — magic bytes, version, header length, header — into the middle of what a version-1 reader believes is an uninterrupted CBOR sequence. That reader decodes the `M` of `MOQTRACE` (`0x4D`) as the start of a 13-byte CBOR byte string, consumes 13 bytes of the preamble, and desynchronizes with no way to recover. Nothing in version 1 lets it detect the boundary or resynchronize past it.
448
+
449
+ Declaring version 2 converts that silent corruption into an explicit rejection at byte 8. A source that never writes segmented traces could in principle keep writing version 1, but SHOULD NOT: one declared version per revision keeps the compatibility matrix a line rather than a grid.
450
+
240
451
  ## Interoperability
241
452
 
242
453
  The `.moqtrace` format is shared between the JavaScript (`@moqtap/trace`) and Rust (`moqtap-trace`) implementations. A file written by either is readable by both. CBOR libraries used:
@@ -245,3 +456,59 @@ The `.moqtrace` format is shared between the JavaScript (`@moqtap/trace`) and Ru
245
456
  - Rust: `ciborium`
246
457
 
247
458
  Both produce deterministic CBOR output (canonical CBOR is NOT required, but map keys SHOULD be sorted for consistency).
459
+
460
+ Two encoding choices are **normative**, because a CBOR library's defaults are not necessarily readable by another CBOR library:
461
+
462
+ - **An integral value MUST be written as a CBOR integer (major type 0 or 1), not as a float.** CBOR permits a float carrying an integral value, and some encoders emit one for any number beyond 32 bits — which every epoch-millisecond timestamp is. A reader MUST nevertheless accept the float form for any integer field, rejecting only a value that is fractional or outside the range a float represents exactly, since files written that way exist.
463
+ - **A byte string MUST be written as major type 2, not wrapped in the typed-array tag 64 of RFC 8746.** A reader MUST nevertheless accept tag 64 wrapping a byte string, for the same reason.
464
+
465
+ Both rules exist because both were broken in opposite directions by the two implementations, and neither test suite could see it: each read only bytes it had written itself, so each agreed with a convention the other did not implement.
466
+
467
+ This claim is maintained by a shared corpus of `.moqtrace` files that both implementations read and write as part of their test suites, covering at minimum: a version-1 file, a non-segmented version-2 file, a segmented version-2 stream, a file carrying an unknown event type, a file carrying an unknown perspective, and a truncated file. An implementation that cannot round-trip the corpus is not conformant, whatever this document says.
468
+
469
+ ---
470
+
471
+ # Appendix A: MoQT Carriage Convention (informative)
472
+
473
+ > **Scoping note.** This appendix describes how a live `.moqtrace` stream maps onto a MoQT track. It is transport behavior, not file-format behavior, and is expected to migrate into a separate **moqtap** protocol draft. It is included here only so early implementers have a single reference while the drafts stabilize. The normative Part 1 and Part 2 of this document do not depend on anything in this appendix, and no released implementation currently provides it.
474
+
475
+ When a `.moqtrace` stream is carried over MoQT as a track:
476
+
477
+ - One MoQT **group** carries one segment (as defined in [Segmented Traces](#segmented-traces)).
478
+ - **Object 0** of each group carries the segment header bytes: `MOQTRACE` + version + headerLen + headerCBOR.
479
+ - **Objects 1..N** of each group each carry exactly one CBOR-encoded event.
480
+ - Concatenating all object payloads of a group, in order, reproduces a valid single-segment `.moqtrace` file.
481
+ - A subscriber joining at the latest group receives object 0 first and can begin parsing immediately without prior context.
482
+
483
+ Implementations MAY batch multiple events into one object for efficiency at high event rates; if so, each object payload remains a valid CBOR sequence.
484
+
485
+ **Late-subscriber join latency.** A subscriber joining mid-segment using SUBSCRIBE Filter Type `Largest Object` (0x2) starts at the next object after the current largest, missing the segment header object — its trace is unparseable until the next group begins. Subscribers SHOULD therefore use Filter Type `Next Group Start` (0x1), which delivers object 0 (the header) of the next group first. Worst-case time-to-first-event is one segment duration. Producers SHOULD pick segment duration to balance join latency against per-segment overhead; 1 s suits interactive debugging, longer suits archival capture.
486
+
487
+ **On-demand production.** A source SHOULD NOT produce trace events when no subscriber is attached. The producer launches when the first subscriber arrives, segment 0 begins at that moment, and the producer stops (or pauses) when the last subscriber leaves. Sources requiring retroactive history (FETCH-style backfill) need a ring buffer; this is out of scope for this specification.
488
+
489
+ **Refusing recording on overload.** When a source would otherwise drop a non-droppable event type (see [Non-droppable Event Types](#non-droppable-event-types)), the carriage convention is to return `REQUEST_ERROR` on the subscription rather than emit a sampled trace.
490
+
491
+ ---
492
+
493
+ # Changes from version 1
494
+
495
+ - **Format version bumped to 2**, solely to make segmentation detectable. See [Why version 2 exists](#why-version-2-exists).
496
+ - **Readers must now accept both versions.** Version 1 said readers "MUST reject files with a version they don't support", which reads as license to reject version 1 once version 2 exists. It is now explicit that a conformant reader accepts 1 and 2.
497
+ - **Document restructured** into a normative event model (Part 1), a normative binary container (Part 2), and an informative MoQT carriage appendix. Event semantics are no longer fused with container semantics, so the model can be reused in other containers.
498
+ - **New Identifier Scoping section** distinguishing source-local `"p"`, capture-correlation `"sessionId"`, and federated `"traceId"`, with their orthogonality and presence rules.
499
+ - **New Privacy Considerations section** flagging payload-bearing events (Event 4 `"pl"`, Event 0 `"raw"`, detail levels `"headers+data"` and `"full"`) as optional and high-risk, requiring operator opt-in and recommending default-off for general-purpose tools.
500
+ - **New header field `"segment"`** enabling [Segmented Traces](#segmented-traces).
501
+ - **New header field `"sampling"`** declaring effective rate, drop policy, drop counters, and the source-side filter rule.
502
+ - **New `"perspective"` value `"relay-tap"`**, distinguishing a relay reporting on multiple concurrent peer sessions from `"server"` (single-session view) and `"observer"` (passive sniffer).
503
+ - **New common event field `"p"`** — peer identifier, required for `"relay-tap"` so multi-peer events can be demultiplexed.
504
+ - **New event types 8 (Peer connected), 9 (Peer disconnected) and 10 (Subscription derivation).**
505
+ - **Event 10 carries track identity and per-hop timestamps** — `"ns"`, `"tn"`, `"tdr"`, `"tus"`, `"tuo"`, `"tdo"` — making per-hop latency attribution possible from the event alone, plus explicit update semantics for events emitted before all four timestamps are known.
506
+ - **Event 10's trace ID is `"traceId"`, a 16-byte byte string.** An earlier revision of this proposal spelled it `"trace"` and left the type open; a text-string spelling invites per-implementation encodings, which defeats the cross-operator stitching the field exists for.
507
+ - **Unknown event types, perspectives and detail levels must be tolerated, not rejected.** Version 1 said readers "SHOULD skip unknown event types"; neither reference implementation did, which is why the purely additive parts of this revision would have broken them anyway.
508
+ - Event `"n"` and `"t"` clarified as **segment-local** in segmented traces.
509
+ - **Non-droppable event types rule** added to the Sampling section.
510
+ - **Trace ID Propagation section** defining the role of the `MOQTAP_TRACE_ID` extension parameter.
511
+ - **Truncated files** now have defined reader behavior: report distinctly, return what decoded.
512
+ - **Two CBOR encoding choices are now normative** — integral values as integers, byte strings as major type 2 — with readers required to accept the other form as well. See [Interoperability](#interoperability).
513
+ - `"payloadMasked"` advertisement strengthened from SHOULD to MUST when masking is active.
514
+ - The header table's `"protocol"` examples (`"draft-14"`, `"rfc9999"`) contradicted the Protocol Identifier section below it; corrected to `"moq-transport-17"` / `"moq-transport-rfc9999"`.