@moqtap/trace 0.3.0 → 0.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -151
- package/SPEC.md +707 -13
- package/dist/index.cjs +506 -95
- package/dist/index.d.cts +545 -6
- package/dist/index.d.ts +545 -6
- package/dist/index.js +504 -93
- package/package.json +6 -4
- package/src/binary.ts +776 -105
- package/src/index.ts +9 -4
- package/src/json.ts +7 -0
- package/src/reader.ts +286 -0
- package/src/recorder.ts +189 -12
- package/src/types.ts +263 -2
package/SPEC.md
CHANGED
|
@@ -32,11 +32,19 @@ The `"detail"` field (see Part 2 header) declares what was recorded. Each level
|
|
|
32
32
|
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- |
|
|
33
33
|
| `"control"` | Control messages only (setup, subscribe, publish, goaway, etc.). No data stream events, no object payloads. | Lightweight protocol flow analysis. DevTools default. |
|
|
34
34
|
| `"headers"` | Control messages + data stream headers (subgroup/fetch/datagram headers, object metadata: group, object ID, priority, status). No payload bytes. | Delivery pattern analysis, timing. |
|
|
35
|
-
| `"headers+sizes"` | Everything in `"headers"` + payload byte lengths for each object. | Bandwidth analysis without storing media. |
|
|
35
|
+
| `"headers+sizes"` | Everything in `"headers"` + payload byte lengths for each object, and the size of the bytes behind an error ([Event 6](#event-6-error) `"rawlen"`). | Bandwidth analysis without storing media. |
|
|
36
36
|
| `"headers+data"` | Everything in `"headers"` + full payload bytes for each object. | Full session replay, debugging media corruption. |
|
|
37
|
-
| `"full"` | Everything above + raw wire bytes for every message (pre-decode).
|
|
37
|
+
| `"full"` | Everything above + raw wire bytes for every *control* message (pre-decode), and the bytes behind an error event ([Event 6](#event-6-error) `"raw"`). See the note below on data-stream framing. | Wire-level debugging, compliance testing. |
|
|
38
38
|
|
|
39
|
-
Levels `"headers+data"` and `"full"`, and the `"raw"` field on
|
|
39
|
+
Levels `"headers+data"` and `"full"`, and the `"raw"` field on Event 0 and [Event 6](#event-6-error), are **payload-bearing** — see [Privacy Considerations](#privacy-considerations). So is [Event 6](#event-6-error)'s `"raw"`, which is why it is gated at `"full"` and not at Event 6's own `control`+: an error naming a *data* stream has subgroup framing and object payload behind it, which is media, and inheriting the event's level would have put it in traces whose declared level excludes payloads entirely.
|
|
40
|
+
|
|
41
|
+
**No level carries data-stream framing bytes.** `"raw"` exists only on Event 0,
|
|
42
|
+
and Event 4's `"pl"` begins after the object header, so the bytes of a
|
|
43
|
+
`SUBGROUP_HEADER`, a fetch header or a datagram header are recorded at no level
|
|
44
|
+
including `"full"`. What those headers carry is recorded as decoded fields
|
|
45
|
+
instead — see [Event 1](#event-1-stream-opened). Closing that gap needs a new
|
|
46
|
+
key and is not in this revision; the level's description no longer implies
|
|
47
|
+
otherwise.
|
|
40
48
|
|
|
41
49
|
At any detail level, the recorder MAY be configured to **mask payloads** — replacing payload bytes with zeroes before writing. This preserves payload sizes for bandwidth analysis while stripping media content. Sources MUST advertise masking via `"payloadMasked": true` in the header's `"custom"` map when active.
|
|
42
50
|
|
|
@@ -96,8 +104,8 @@ Because `"p"` is source-local, a collector correlating traces from multiple sour
|
|
|
96
104
|
| ------- | ----------- | ------------ | ------------------------------------------------- |
|
|
97
105
|
| `"d"` | integer | `control`+ | Direction: `0` = sent (tx), `1` = received (rx) |
|
|
98
106
|
| `"mt"` | integer | `control`+ | Wire message type ID (e.g., `0x03` for SUBSCRIBE) |
|
|
99
|
-
| `"msg"` | map | `control`+ | Decoded message fields (
|
|
100
|
-
| `"sid"` | integer
|
|
107
|
+
| `"msg"` | map | `control`+ | Decoded message fields, keyed in snake_case (see below) |
|
|
108
|
+
| `"sid"` | unsigned integer | `control`+ | QUIC stream ID the message travelled on. Optional (see below). |
|
|
101
109
|
| `"raw"` | byte string | `full` only | Raw wire bytes (including type and length prefix). Payload-bearing — see [Privacy Considerations](#privacy-considerations). |
|
|
102
110
|
|
|
103
111
|
`"sid"` is optional: a recorder that sits at the session level rather than the
|
|
@@ -114,6 +122,52 @@ track names, aliases, request outcomes — is lost to the reader. Through
|
|
|
114
122
|
draft-16 all control messages share one stream and the field is merely
|
|
115
123
|
informative.
|
|
116
124
|
|
|
125
|
+
#### `"msg"` field naming and shape
|
|
126
|
+
|
|
127
|
+
`"msg"` MUST be a CBOR map. Its keys are the message's field names **in
|
|
128
|
+
snake_case** — `request_id`, `track_alias`, `group_order` — which is what the
|
|
129
|
+
drafts themselves use and what the shared codec vectors carry, so a reader can
|
|
130
|
+
address a field without knowing which implementation wrote the file.
|
|
131
|
+
|
|
132
|
+
A writer that has no decoded fields — the message type is one it cannot parse,
|
|
133
|
+
or the recorder is not decoding bodies at all — MUST write an empty map `{}`
|
|
134
|
+
rather than omit the key. Omission costs a reader more than the one byte it
|
|
135
|
+
saves: Event 0 is one of the types [sampling](#sampling) MUST NOT drop, so a
|
|
136
|
+
reader that treats a missing `"msg"` as a malformed event discards exactly the
|
|
137
|
+
events the format promises to keep.
|
|
138
|
+
|
|
139
|
+
Readers MUST be more tolerant than writers, because files predating this rule
|
|
140
|
+
exist:
|
|
141
|
+
|
|
142
|
+
- An **absent** `"msg"` MUST be read as an empty map, not as an error.
|
|
143
|
+
- A `"msg"` that is **not a map** MUST be preserved verbatim and offered to the
|
|
144
|
+
caller unchanged. It MUST NOT cause the event to be rejected. Recordings
|
|
145
|
+
written before this revision carry a text rendering of the decoded message
|
|
146
|
+
here — every `capture-*` case in the conformance corpus is such a file — and
|
|
147
|
+
they remain readable, with the field simply not addressable by key.
|
|
148
|
+
|
|
149
|
+
CBOR `null` is a value, not an absence: a writer put it there, so it falls
|
|
150
|
+
under "not a map" and MUST be preserved. CBOR `undefined` (major type 7,
|
|
151
|
+
simple value 23) is **not** preservable here, and writers MUST NOT emit it.
|
|
152
|
+
Neither reference implementation can represent it distinctly *as a `"msg"`* —
|
|
153
|
+
the TypeScript reader cannot tell it from a field the caller never set, and the
|
|
154
|
+
Rust reader's CBOR library decodes it to `null` before the reader sees it — so
|
|
155
|
+
a reader MAY normalise it to an empty map or to `null`, and the two will
|
|
156
|
+
disagree about which. There is no file this matters for unless one is written
|
|
157
|
+
deliberately; the rule exists so that nobody writes one expecting it to
|
|
158
|
+
survive. It is one row of a general table — see [Shapes a CBOR library may
|
|
159
|
+
normalise before you see them](#shapes-a-cbor-library-may-normalise-before-you-see-them),
|
|
160
|
+
which also explains why the answer differs by *where* in a trace the value sits:
|
|
161
|
+
the same `undefined` inside an unrecognised-key store is preserved by one
|
|
162
|
+
implementation and not the other.
|
|
163
|
+
|
|
164
|
+
**A tool that reads a trace and writes it back MUST preserve a non-map `"msg"`
|
|
165
|
+
as it found it**, and MUST NOT replace it with an empty map to satisfy the
|
|
166
|
+
writer rule above. Replacing it would discard the only record of a message that
|
|
167
|
+
will never be seen again. This is one instance of a general rule stated under
|
|
168
|
+
[Versioning and Compatibility](#versioning-and-compatibility): on a rewrite the
|
|
169
|
+
reader rule outranks the writer rule.
|
|
170
|
+
|
|
117
171
|
### Event 1: Stream Opened
|
|
118
172
|
|
|
119
173
|
| Key | Type | Detail Level | Description |
|
|
@@ -121,6 +175,73 @@ informative.
|
|
|
121
175
|
| `"sid"` | integer | `headers`+ | QUIC stream ID |
|
|
122
176
|
| `"d"` | integer | `headers`+ | Direction: `0` = outgoing, `1` = incoming |
|
|
123
177
|
| `"st"` | integer | `headers`+ | Stream type: `0` = subgroup, `1` = datagram, `2` = fetch |
|
|
178
|
+
| `"ta"` | integer | `headers`+ | Track alias the stream carries. Optional. |
|
|
179
|
+
| `"sg"` | integer | `headers`+ | Subgroup ID. Optional; only meaningful when `st == 0`. |
|
|
180
|
+
| `"fri"` | integer | `headers`+ | Fetch request ID. Optional; only meaningful when `st == 2`. |
|
|
181
|
+
| `"g"` | integer | `headers`+ | Group ID. Optional; only meaningful when `st == 1`. |
|
|
182
|
+
|
|
183
|
+
#### Why these four keys exist
|
|
184
|
+
|
|
185
|
+
A `SUBGROUP_HEADER` carries a track alias, a group, a subgroup and a publisher
|
|
186
|
+
priority. A fetch header carries a request ID. A datagram header carries a track
|
|
187
|
+
alias, a group, an object and a priority. Before this revision the model could
|
|
188
|
+
express group, object, priority and status and nothing else — **track alias,
|
|
189
|
+
subgroup ID and fetch request ID had nowhere to live**, and at `"headers"` there
|
|
190
|
+
are no payload bytes to re-parse them from. A `"headers"` recording therefore
|
|
191
|
+
could not answer which track a stream belonged to, which is most of what the
|
|
192
|
+
level exists for.
|
|
193
|
+
|
|
194
|
+
**A writer MUST write `"fri"` on a fetch stream** (`st == 2`), where it is the
|
|
195
|
+
only correlation between the stream and the FETCH that asked for it. A reader
|
|
196
|
+
MUST NOT reject a fetch stream that lacks it: recordings predating this revision
|
|
197
|
+
have none, and Event 1 is not a type a reader may discard on a missing optional
|
|
198
|
+
key. The same asymmetry as [`"msg"`](#msg-field-naming-and-shape) — writers
|
|
199
|
+
conform, readers tolerate.
|
|
200
|
+
|
|
201
|
+
The other three are written when known. `"sg"`, `"fri"` and `"g"` are each
|
|
202
|
+
scoped to one stream type because on the others they have no source; a writer
|
|
203
|
+
MUST NOT write one outside its scope, and a reader that meets one there MUST
|
|
204
|
+
**keep** it — read it into the field it names, and write it back — rather than
|
|
205
|
+
reject the event or route it elsewhere.
|
|
206
|
+
|
|
207
|
+
"Keep", not "ignore": a reader is free to disregard the *meaning* of an
|
|
208
|
+
out-of-scope key, but it does not get to drop the value. Dropping would violate
|
|
209
|
+
[the preservation rule](#versioning-and-compatibility), and routing it to the
|
|
210
|
+
unrecognised-key store would be wrong twice over — the key *is* recognised, its
|
|
211
|
+
value *is* usable, and a key held in both places is written back twice.
|
|
212
|
+
|
|
213
|
+
Wrong *place* is not wrong *shape*, and they part company here. A `"sg"` on a
|
|
214
|
+
fetch stream is a usable value in a position that means nothing: it reaches the
|
|
215
|
+
field. A `"sg"` carrying a text string is not a usable value at all, wherever it
|
|
216
|
+
sits, and goes to the unrecognised-key store under [the rule for unusable
|
|
217
|
+
types](#versioning-and-compatibility). A reader has to answer both questions,
|
|
218
|
+
in that order.
|
|
219
|
+
|
|
220
|
+
**Event 3 wins.** Where a value appears both here and on an Event 3 for the same
|
|
221
|
+
stream — a group ID, most obviously — **Event 3 is authoritative** and this copy
|
|
222
|
+
is a convenience for readers that have not yet seen an object. A reader MUST NOT
|
|
223
|
+
treat a disagreement as corruption; it MUST prefer Event 3.
|
|
224
|
+
|
|
225
|
+
`"g"` is scoped to datagrams for that reason: on a subgroup stream every object
|
|
226
|
+
carries the same group by construction, so a copy here would be a second field
|
|
227
|
+
with no independent source. `"pp"` is deliberately **not** added for the same
|
|
228
|
+
reason — Event 3's `"pp"` is the stream header's publisher priority copied per
|
|
229
|
+
object, and a second copy would have no tiebreak.
|
|
230
|
+
|
|
231
|
+
#### Integer keys decode to the same type as their Event 3 counterparts
|
|
232
|
+
|
|
233
|
+
`"ta"`, `"sg"`, `"fri"` and `"g"` MUST decode to whatever language type an
|
|
234
|
+
implementation gives Event 3's `"g"`, `"o"` and `"sid"` — not to whatever it
|
|
235
|
+
gives Event 1's existing `"st"` and `"d"`.
|
|
236
|
+
|
|
237
|
+
Those two groups differ today, for a good reason that does not extend here:
|
|
238
|
+
`"st"` and `"d"` are small enumerations, and an implementation may reasonably
|
|
239
|
+
hold them in a narrow type. The keys added above are wire identifiers with the
|
|
240
|
+
same range as Event 3's. An implementer following the enclosing event's local
|
|
241
|
+
convention would type Event 1's `"g"` as a JavaScript `number` while Event 3's
|
|
242
|
+
is a `bigint`, and `event1.g === event3.groupId` would then evaluate `42 === 42n`
|
|
243
|
+
as `false` — silently, and for every reader that made the same reasonable
|
|
244
|
+
choice.
|
|
124
245
|
|
|
125
246
|
### Event 2: Stream Closed
|
|
126
247
|
|
|
@@ -162,10 +283,160 @@ Object header events and object payload events (event 4) for the same object sha
|
|
|
162
283
|
|
|
163
284
|
### Event 6: Error
|
|
164
285
|
|
|
165
|
-
| Key | Type
|
|
166
|
-
| ---------- |
|
|
167
|
-
| `"ec"` | integer
|
|
168
|
-
| `"reason"` | text string
|
|
286
|
+
| Key | Type | Detail Level | Description |
|
|
287
|
+
| ---------- | ---------------- | ------------ | --------------------- |
|
|
288
|
+
| `"ec"` | integer | `control`+ | Error code |
|
|
289
|
+
| `"reason"` | text string | `control`+ | Human-readable reason |
|
|
290
|
+
| `"sid"` | unsigned integer | `control`+ | QUIC stream ID the error was observed on. **Optional**, on the same terms as [Event 0's `"sid"`](#event-0-control-message): absent means there was no stream or none is known, and readers MUST distinguish that from stream `0`. |
|
|
291
|
+
| `"ek"` | text string | `control`+ | Error kind. **Optional.** See below. |
|
|
292
|
+
| `"rawlen"` | unsigned integer | `headers+sizes`+ | Byte length of the input the recorder held for this error, before any truncation. **Optional.** See below. |
|
|
293
|
+
| `"raw"` | byte string | `full` only | The offending bytes, truncated to the cap below. **Optional.** Payload-bearing — see [Privacy Considerations](#privacy-considerations). |
|
|
294
|
+
|
|
295
|
+
#### Why an error event carries bytes
|
|
296
|
+
|
|
297
|
+
A peer that sends something malformed is one of the few things a shared trace
|
|
298
|
+
is uniquely good for: the recording party can see it and the sending party
|
|
299
|
+
cannot. A report saying "your SUBSCRIBE_OK did not parse" is an assertion; the
|
|
300
|
+
same report carrying the bytes that did not parse is evidence, and the other
|
|
301
|
+
party can run it against their own encoder without reproducing the session.
|
|
302
|
+
|
|
303
|
+
Before these keys the only field able to hold bytes was Event 0's `"raw"`, and
|
|
304
|
+
a recorder wanting to keep the evidence had to claim the violation was a
|
|
305
|
+
control message in order to have somewhere to put it. That is a worse record
|
|
306
|
+
than none: it asserts a decodable message where there was a protocol
|
|
307
|
+
violation, and no reader can tell the two apart. One recorder in this project
|
|
308
|
+
declined the trade and left the note behind in its source — *"the offending
|
|
309
|
+
bytes stay behind: Event 6 has no field to carry them"* — which is the gap
|
|
310
|
+
these keys close.
|
|
311
|
+
|
|
312
|
+
#### `"ek"` — error kind
|
|
313
|
+
|
|
314
|
+
An open vocabulary. This revision names `"protocol"` (the peer violated the
|
|
315
|
+
protocol), `"transport"` (the QUIC or WebTransport layer failed), and
|
|
316
|
+
`"decode"` (bytes that would not parse as any message this recorder knows).
|
|
317
|
+
Other values MAY be used and MAY be added without a version bump; readers MUST
|
|
318
|
+
preserve a value they do not recognise and MUST NOT reject the event, as with
|
|
319
|
+
[`"perspective"`](#perspective).
|
|
320
|
+
|
|
321
|
+
`"ek"` is spelled in full rather than as `"kind"` because `"kind"` is already a
|
|
322
|
+
top-level key on [Event 10](#event-10-subscription-derivation) with the
|
|
323
|
+
vocabulary `"created"` / `"shared"`. Two vocabularies under one key at the same
|
|
324
|
+
map level survive in a typed reader and break in every flat projection of a
|
|
325
|
+
trace — a JSONL export, a warehouse load, a column-oriented viewer — where
|
|
326
|
+
`kind` becomes one column holding both.
|
|
327
|
+
|
|
328
|
+
#### `"raw"` is capped, and `"rawlen"` says whether the cap bit
|
|
329
|
+
|
|
330
|
+
**A recorder MUST NOT write more than 4096 bytes in `"raw"`.** Where it holds
|
|
331
|
+
more, it writes the first 4096 and records the full length in `"rawlen"`.
|
|
332
|
+
|
|
333
|
+
The cap is a MUST and a fixed number rather than a SHOULD and a suggestion,
|
|
334
|
+
because Event 6 is [non-droppable](#non-droppable-event-types). Every other
|
|
335
|
+
high-volume field in this format is bounded by sampling; this one cannot be,
|
|
336
|
+
so if it is not bounded here it is not bounded at all. A peer opening ten
|
|
337
|
+
thousand streams and sending garbage on each is not a hypothetical — it is
|
|
338
|
+
what a fuzzer does to a relay, and a trace of it should not be larger than the
|
|
339
|
+
attack. A number a reader can test also means the corpus can hold a case
|
|
340
|
+
proving the cap was applied rather than merely described; "SHOULD keep it
|
|
341
|
+
small" cannot be tested by anyone.
|
|
342
|
+
|
|
343
|
+
4096 was chosen against two known quantities. It is far above what control
|
|
344
|
+
messages actually measure: the largest encoded control message across every
|
|
345
|
+
draft in the shared codec vectors is 95 bytes, and each of the four real
|
|
346
|
+
third-party captures in the conformance corpus is under 3 kB *in total*,
|
|
347
|
+
control messages, object headers and all. And it sits well below
|
|
348
|
+
`MAX_MESSAGE_LENGTH`, the 65535-byte ceiling the protocol itself puts on a
|
|
349
|
+
control message from draft-11 onward — deliberately below, because `"raw"`
|
|
350
|
+
exists for diagnosis and not for replay. A message can legitimately be larger
|
|
351
|
+
than the cap; the cap is not a claim that it cannot. A fault that leaves no trace in the
|
|
352
|
+
first 4096 bytes is a real possibility, and it is what `"rawlen"` is for: the
|
|
353
|
+
reader learns that the record is partial and how much is missing, which is
|
|
354
|
+
enough to go and ask for the rest.
|
|
355
|
+
|
|
356
|
+
**`"rawlen"` sits at `headers+sizes`, not at `full` and not at `control`.** It
|
|
357
|
+
is available in every trace where `"raw"` MUST NOT appear, which is the point
|
|
358
|
+
of having it: how large a malformed message was is often enough on its own to
|
|
359
|
+
separate a truncated message from a mistyped one, and it carries no content.
|
|
360
|
+
|
|
361
|
+
It is nonetheless a **size**, and this format gates sizes deliberately —
|
|
362
|
+
`headers+sizes` exists precisely to mean "how much, but not what". Putting
|
|
363
|
+
`"rawlen"` at `control` would let a `control`-level trace carry a fact about
|
|
364
|
+
data-stream volume that its own declared level excludes, because Event 6's
|
|
365
|
+
`"sid"` may name a data stream: the length would then be the size of a run of
|
|
366
|
+
media framing. One number is a small leak, but the latch below bounds `"raw"`
|
|
367
|
+
and not `"rawlen"`, so on a repeatedly failing stream it is a *sequence* of
|
|
368
|
+
sizes, which is the shape traffic analysis wants. Gating it with the other
|
|
369
|
+
sizes costs a `control`-level trace one diagnostic and keeps each level's
|
|
370
|
+
promise exactly what it says.
|
|
371
|
+
|
|
372
|
+
A recorder at `headers+sizes` or above SHOULD write it whether or not it is
|
|
373
|
+
also writing the bytes.
|
|
374
|
+
|
|
375
|
+
When `"raw"` is present, `"rawlen"` SHOULD be present too, and a reader MAY
|
|
376
|
+
take `rawlen > len(raw)` as the definition of a truncated capture. If
|
|
377
|
+
`"rawlen"` is absent, a `"raw"` of exactly 4096 bytes MUST be treated as
|
|
378
|
+
possibly truncated, since that is the one length the cap makes ambiguous.
|
|
379
|
+
|
|
380
|
+
**A recorder that does not know the true length MUST omit `"rawlen"` rather
|
|
381
|
+
than write the length it happens to hold.** This is the case the key is most
|
|
382
|
+
easily got wrong in, and it defeats the key entirely. A recorder that stops
|
|
383
|
+
*reading* at the cap — which is the sensible thing to do with a stream that
|
|
384
|
+
will not parse, and what the recorder cited below does — never holds more than
|
|
385
|
+
4096 bytes. Writing `rawlen` as 4096 there says `rawlen == len(raw)`, which is
|
|
386
|
+
this section's own signal for **not truncated**: the field would assert
|
|
387
|
+
completeness precisely when the recorder has the least idea whether the input
|
|
388
|
+
was complete. Omitting it falls through to the rule above, where a `"raw"` of
|
|
389
|
+
exactly the cap is treated as possibly truncated, which is the truth.
|
|
390
|
+
|
|
391
|
+
Where the length *is* knowable without keeping the bytes, write it. A
|
|
392
|
+
length-prefixed control message supplies it from its own length field, and a
|
|
393
|
+
recorder that read that prefix and then gave up on the body knows exactly how
|
|
394
|
+
much it did not keep. That is the case `"rawlen"` exists for, and it is common:
|
|
395
|
+
the fault is often the body, and the prefix is what got the recorder as far as
|
|
396
|
+
noticing.
|
|
397
|
+
|
|
398
|
+
**The cap binds the recorder, and nothing else.** It is addressed to the party
|
|
399
|
+
deciding what to say about traffic it has just observed. It is *not* addressed
|
|
400
|
+
to a tool that reads an event and serializes it again: a reader meeting a
|
|
401
|
+
`"raw"` longer than 4096 bytes MUST NOT reject the event, and a rewrite MUST
|
|
402
|
+
NOT shorten it. That is [the reader rule outranking the writer
|
|
403
|
+
rule](#versioning-and-compatibility), and this is the case it exists for.
|
|
404
|
+
Re-truncating someone else's file destroys evidence in order to make it conform
|
|
405
|
+
to a rule that was never addressed to the tool doing the truncating. Report the
|
|
406
|
+
non-conformance if it is worth reporting; do not repair it.
|
|
407
|
+
|
|
408
|
+
The distinction has a consequence worth stating for implementers, because
|
|
409
|
+
getting it wrong is easy and the result is silent. **A serializer is the wrong
|
|
410
|
+
place to enforce this cap.** A serializer cannot tell a freshly recorded event
|
|
411
|
+
from one that arrived by being read, so a cap applied there either truncates
|
|
412
|
+
evidence on rewrite or rejects a file the reader was required to accept —
|
|
413
|
+
whichever it does, it does to the wrong events. The cap belongs where the event
|
|
414
|
+
is *constructed from observed bytes*.
|
|
415
|
+
|
|
416
|
+
#### One `"raw"` per flow
|
|
417
|
+
|
|
418
|
+
**A recorder MUST NOT emit `"raw"` more than once per flow** — per stream where
|
|
419
|
+
the error names one, per peer where it does not, and once for the whole
|
|
420
|
+
recording where it writes no peer identifier either. That last case is every
|
|
421
|
+
recorder but a relay tap, since `"p"` is only required of one: the three
|
|
422
|
+
collapse to "the narrowest scope the recorder can name", and a recorder that
|
|
423
|
+
can name none has one flow. Later errors on a flow that
|
|
424
|
+
has already carried its bytes are still recorded; they simply carry no `"raw"`.
|
|
425
|
+
|
|
426
|
+
The latch is on the field, not on the event, and the distinction is the whole
|
|
427
|
+
point. Event 6 is non-droppable: suppressing the *event* would discard the
|
|
428
|
+
causal record this document elsewhere forbids discarding, and would do it
|
|
429
|
+
precisely when a peer is misbehaving repeatedly, which is when the record
|
|
430
|
+
matters most. What actually grows without bound is the bytes, so that is what
|
|
431
|
+
is bounded. The first failure on a flow is the diagnostic one in any case —
|
|
432
|
+
once a stream will not parse, a recorder is no longer interpreting what
|
|
433
|
+
follows, so the "bytes" of the second error are the same undifferentiated run
|
|
434
|
+
as the first.
|
|
435
|
+
|
|
436
|
+
This matches what a recorder in this project already does, one level up: its
|
|
437
|
+
control-stream reader latches on the first undecodable run and stops reading
|
|
438
|
+
that stream, capping the kept bytes at 4096. This section takes that behaviour,
|
|
439
|
+
which was chosen for a panel display, and makes it the format's rule.
|
|
169
440
|
|
|
170
441
|
### Event 7: Annotation
|
|
171
442
|
|
|
@@ -271,11 +542,16 @@ Trace recording ranges from protocol-metadata-only (low sensitivity) to full med
|
|
|
271
542
|
1. **Payload-bearing events and fields.** These carry or reveal user content:
|
|
272
543
|
- Event 4 `"pl"` (object payload bytes, `"headers+data"`+)
|
|
273
544
|
- Event 0 `"raw"` (raw control-message bytes, `"full"`)
|
|
545
|
+
- [Event 6](#event-6-error) `"raw"` (the bytes behind an error, `"full"`).
|
|
546
|
+
Easy to overlook, because an error event sounds like metadata: its
|
|
547
|
+
`"sid"` may name a *data* stream, in which case the bytes behind the
|
|
548
|
+
error are subgroup framing and object payload — media, arriving through
|
|
549
|
+
the one event type sampling may never drop.
|
|
274
550
|
- Detail levels `"headers+data"` and `"full"` in aggregate
|
|
275
551
|
|
|
276
552
|
Capturing them is **optional** for all conformant recorders. Observer- and tap-based capture of these fields is equivalent to intercepting user media; recorders MUST NOT enable it by default and SHOULD require explicit operator opt-in per session or per track. The default detail level for general-purpose tools SHOULD be `"control"` or `"headers"`.
|
|
277
553
|
|
|
278
|
-
2. **Payload size leakage.** Even without payload bytes, `"headers+sizes"` reveals object sizes, which can fingerprint media content. Operators handling sensitive traffic SHOULD evaluate size leakage as part of their threat model.
|
|
554
|
+
2. **Payload size leakage.** Even without payload bytes, `"headers+sizes"` reveals object sizes, which can fingerprint media content. [Event 6](#event-6-error)'s `"rawlen"` belongs to this class rather than to the one above — it is gated with the sizes for that reason — and it leaks a little differently: the one-`"raw"`-per-flow latch does not bind it, so a stream that fails repeatedly yields a *sequence* of sizes rather than a single one. Operators handling sensitive traffic SHOULD evaluate size leakage as part of their threat model.
|
|
279
555
|
|
|
280
556
|
3. **Trace ID correlation.** `MOQTAP_TRACE_ID` enables cross-operator correlation of subscription behavior. See [Trace ID Propagation](#trace-id-propagation) for egress mitigations.
|
|
281
557
|
|
|
@@ -326,9 +602,9 @@ A single CBOR map with the following keys:
|
|
|
326
602
|
| `"protocol"` | text string | Yes | MoQT version identifier (e.g., `"moq-transport-17"`, `"moq-transport-rfc9999"`) |
|
|
327
603
|
| `"perspective"` | text string | Yes | Recording viewpoint: `"client"`, `"server"`, `"observer"`, or `"relay-tap"` |
|
|
328
604
|
| `"detail"` | text string | Yes | Detail level (see [Detail Levels](#detail-levels)) |
|
|
329
|
-
| `"startTime"` | integer
|
|
605
|
+
| `"startTime"` | unsigned integer | Yes | Recording start time (Unix epoch milliseconds). Unsigned: a trace from before 1970 is a clock fault, not a recording, and a reader that accepts one has no way to say so. |
|
|
330
606
|
| `"transport"` | text string | No | Transport type (e.g., `"webtransport"`, `"raw-quic"`) |
|
|
331
|
-
| `"endTime"` | integer
|
|
607
|
+
| `"endTime"` | unsigned integer | No | Recording end time (Unix epoch milliseconds). Written when trace is finalized. |
|
|
332
608
|
| `"source"` | text string | No | Software that produced the trace (e.g., `"moqtap-devtools/0.1.0"`, `"my-relay/2.3.1"`). Also serves as the namespace for source-local `"p"` values. |
|
|
333
609
|
| `"endpoint"` | text string | No | Remote peer URI (e.g., `"https://relay.example.com/moq"`) |
|
|
334
610
|
| `"sessionId"` | text string | No | Capture-correlation identifier — see [Identifier Scoping](#identifier-scoping). |
|
|
@@ -336,6 +612,186 @@ A single CBOR map with the following keys:
|
|
|
336
612
|
| `"sampling"` | map | No | Sampling/filter metadata; present iff events were sampled or filtered at the source. See [Sampling](#sampling). |
|
|
337
613
|
| `"custom"` | map | No | User-defined metadata (arbitrary key-value pairs). `"payloadMasked": true` here indicates payload masking is active. |
|
|
338
614
|
|
|
615
|
+
### Unrecognised keys in the header
|
|
616
|
+
|
|
617
|
+
[Versioning and Compatibility](#versioning-and-compatibility) requires a reader
|
|
618
|
+
that writes a trace back out to preserve the unrecognised keys it read, in
|
|
619
|
+
"header maps or event maps". This section says what that means for the header,
|
|
620
|
+
because the header is where the rule was least obviously binding and where both
|
|
621
|
+
reference implementations broke it: each read a header, dropped every key it
|
|
622
|
+
did not know, and wrote out a file that looked as though it had never carried
|
|
623
|
+
them.
|
|
624
|
+
|
|
625
|
+
**Three maps in the header have keys this document names**: the header map
|
|
626
|
+
itself, `"segment"`, and `"sampling"`. Each MUST carry its own
|
|
627
|
+
unrecognised-key store, and each store MUST be written back into the map it
|
|
628
|
+
came from. One store for the whole header would not do — a private key on
|
|
629
|
+
`"segment"` and a key of the same name at the top level are different keys, and
|
|
630
|
+
re-emitting either in the other's map changes what the file says.
|
|
631
|
+
|
|
632
|
+
Everything the [key-preservation rules](#versioning-and-compatibility) say
|
|
633
|
+
about events applies here unchanged, reading "header" for "event" and "the
|
|
634
|
+
segment" for "the event":
|
|
635
|
+
|
|
636
|
+
- A key this document does not define goes to the store of the map it appeared
|
|
637
|
+
in, and is written back from there.
|
|
638
|
+
- A key this document *does* define, carrying a value the reader cannot use,
|
|
639
|
+
goes to that same store. Knowing more about a key must not mean preserving it
|
|
640
|
+
less. `"transport": 42` is not a transport; it is also not nothing.
|
|
641
|
+
- Which values are usable is settled by [the same
|
|
642
|
+
list](#versioning-and-compatibility) — integral floats yes, fractional no,
|
|
643
|
+
negative no for a key defined as unsigned, and no rounding of a value the
|
|
644
|
+
reader cannot represent.
|
|
645
|
+
- **The range clause binds only where this document states a range.** In the
|
|
646
|
+
header that is one key: `"effectiveRate"`, defined as lying in `(0.0, 1.0]`.
|
|
647
|
+
A rate of `1.5`, of `0.0`, or of NaN is unusable and goes to the sampling
|
|
648
|
+
store. This is a semantic check rather than a type check and it is the only
|
|
649
|
+
one in the header, deliberately — every other key here is bounded by its type
|
|
650
|
+
alone, and a reader inventing bounds this document does not state would start
|
|
651
|
+
rejecting values a future revision means to allow. The reason `"effectiveRate"`
|
|
652
|
+
earns the check is that consumers divide by it: a rate of `0.0` handed to a
|
|
653
|
+
caller reconstructing true event counts is a division by zero, and `1.5` is a
|
|
654
|
+
count larger than what was recorded.
|
|
655
|
+
- **A value under a CBOR tag is unusable**, tag 64 excepted. This format uses
|
|
656
|
+
no tags; a writer MUST NOT emit one. Tag 64 is the exception readers already
|
|
657
|
+
MUST accept for byte strings (see [Interoperability](#interoperability)), and
|
|
658
|
+
a reader that unwraps it and re-emits the bytes as major type 2 has done what
|
|
659
|
+
that rule asks — so "written back unchanged" binds the value, not its
|
|
660
|
+
encoding. Any other tag is a shape this document gives no meaning, and it
|
|
661
|
+
goes to the store with its tag intact where the reader can hold one.
|
|
662
|
+
|
|
663
|
+
#### A composite value is usable only as a whole
|
|
664
|
+
|
|
665
|
+
`"sampling.appliesTo"` is an array of event type IDs. If one element is not
|
|
666
|
+
one, the array is unusable and goes to the store **entire**. A reader MUST NOT
|
|
667
|
+
keep the elements it liked and discard the rest.
|
|
668
|
+
|
|
669
|
+
This is the shape where discarding is not merely lossy but wrong. An
|
|
670
|
+
`"appliesTo"` of `[3, "x", 5]` read as `[3, 5]` is not a partial answer: the
|
|
671
|
+
key names the event types the drop policy touched, and every type absent from
|
|
672
|
+
it may be treated as complete. Shortening the array therefore reports a sampled
|
|
673
|
+
event type as fully recorded — the opposite of what the file said, stated with
|
|
674
|
+
the same confidence. The same reasoning governs `"ns"` on Event 10, and any
|
|
675
|
+
array or map a future key defines.
|
|
676
|
+
|
|
677
|
+
#### `"custom"`, and values a reader cannot hold
|
|
678
|
+
|
|
679
|
+
`"custom"` is not one of the three maps above: every key in it belongs to
|
|
680
|
+
whoever wrote the trace, so there is no such thing as an unrecognised key
|
|
681
|
+
there and it needs no store. It is a passthrough. A reader MUST hand back what
|
|
682
|
+
the file carried, key for key and value for value, and MUST write back what it
|
|
683
|
+
was handed.
|
|
684
|
+
|
|
685
|
+
A reader whose representation cannot hold that map exactly MUST treat
|
|
686
|
+
`"custom"` as unusable and route the whole value to the *header's* store,
|
|
687
|
+
rather than hand back a version of it that lost something. Both reference
|
|
688
|
+
implementations declare `"custom"` as a string-keyed map, so for them a
|
|
689
|
+
`"custom"` that is not a map at all is unusable — and so is one carrying a
|
|
690
|
+
non-text key, *where the reader can see that it does*; the next subsection
|
|
691
|
+
explains why that qualifier is not a loophole. Losing typed access is the
|
|
692
|
+
smaller harm: nothing in this document gives `"custom"` keys meaning, so there
|
|
693
|
+
is nothing to lose but convenience, and the bytes survive.
|
|
694
|
+
|
|
695
|
+
A reader whose decoder happens to hand back a shape its own declared type
|
|
696
|
+
cannot describe — a map object where a string-keyed record was promised — is
|
|
697
|
+
not thereby conformant. It is preserving the bytes by accident, through a type
|
|
698
|
+
that lies to every caller reading the field, and the first caller to iterate
|
|
699
|
+
the keys finds a shape it was told could not occur. Route the value to the
|
|
700
|
+
store and keep the field's type true.
|
|
701
|
+
|
|
702
|
+
That generalises to a test worth applying to every key in this section:
|
|
703
|
+
**can this reader write back exactly what it read?** Where it can, the value
|
|
704
|
+
belongs in the field. Where it cannot, the value belongs in the store. A
|
|
705
|
+
reader that answers "no" and keeps the field anyway is the defect this section
|
|
706
|
+
exists to name.
|
|
707
|
+
|
|
708
|
+
#### Keys in these maps are text
|
|
709
|
+
|
|
710
|
+
Every key this document defines — in the header map, in `"segment"`, in
|
|
711
|
+
`"sampling"`, and by the nature of the field in `"custom"` — is a text string,
|
|
712
|
+
and **a writer MUST NOT emit a key of any other kind.**
|
|
713
|
+
|
|
714
|
+
What a reader does with one is deliberately left undefined, and nothing may
|
|
715
|
+
depend on two readers agreeing about it. The reason is worth stating, because
|
|
716
|
+
the obvious rule is one that cannot be implemented: **some CBOR decoders
|
|
717
|
+
cannot tell.** A decoder that returns maps as language-native objects coerces
|
|
718
|
+
the integer key `1` into the text key `"1"` before any code written against
|
|
719
|
+
this document runs, at which point it is indistinguishable from a text `"1"`
|
|
720
|
+
the same map might also carry — and the two collide. One of the two reference
|
|
721
|
+
implementations decodes exactly this way. Requiring it to report the
|
|
722
|
+
difference would be requiring something it cannot observe, and a specification
|
|
723
|
+
that mandates the impossible teaches implementers which of its rules to skip.
|
|
724
|
+
|
|
725
|
+
This is one instance of a general problem, not a special case: see [Shapes a
|
|
726
|
+
CBOR library may normalise before you see them](#shapes-a-cbor-library-may-normalise-before-you-see-them)
|
|
727
|
+
for the full list and the rule that governs all of it. A non-text key is the
|
|
728
|
+
most obvious member; the text key `"__proto__"`, a duplicate key, and CBOR
|
|
729
|
+
`undefined` are others, and each is invisible to at least one of the two
|
|
730
|
+
reference implementations.
|
|
731
|
+
|
|
732
|
+
So the requirement lands on the writer, where it can be met. What a reader
|
|
733
|
+
MUST NOT do is **re-key** — turn an integer key into a text key and write that
|
|
734
|
+
back — because the result is a file asserting a key the original never carried,
|
|
735
|
+
possibly colliding with a real one. A reader that can hold the key as it found
|
|
736
|
+
it SHOULD do exactly that and preserve it in the store; a reader whose decoder
|
|
737
|
+
has already re-keyed it before it was asked is not thereby non-conformant,
|
|
738
|
+
having never had the choice.
|
|
739
|
+
|
|
740
|
+
Rejecting the header is *not* the recommended answer here, which is worth
|
|
741
|
+
saying because an earlier draft of this section recommended it. Failing the
|
|
742
|
+
header loses every event behind it, and this subsection would then be
|
|
743
|
+
prescribing, for the most capable reader, the most destructive outcome — the
|
|
744
|
+
same harm the `"segment"`/`"sampling"` subsection below forbids for a value it
|
|
745
|
+
merely cannot use. Preserve it and move on. This is the one place in this
|
|
746
|
+
section where the two implementations are permitted to differ, and it is
|
|
747
|
+
permitted only because no conformant file reaches it.
|
|
748
|
+
|
|
749
|
+
#### `"segment"` and `"sampling"` are optional keys
|
|
750
|
+
|
|
751
|
+
A `"segment"` or `"sampling"` that is not a map MUST NOT fail the file. It goes
|
|
752
|
+
to the store like any other unusable optional value. A reader that rejects the
|
|
753
|
+
file instead has turned one unreadable metadata value into the loss of every
|
|
754
|
+
event behind it.
|
|
755
|
+
|
|
756
|
+
**But an unusable `"segment"` is not an absent one.** [Segmented
|
|
757
|
+
Traces](#segmented-traces) requires a reader to treat an absent `"segment"` as
|
|
758
|
+
"this is a complete, single-segment file", and that assertion MUST NOT be made
|
|
759
|
+
here: the file said it was a segment and the reader merely could not place it.
|
|
760
|
+
Making it would be the same invention as defaulting `"sequence"` to `0`, and
|
|
761
|
+
worse in a segmented stream, where a segment read as non-segmented has its
|
|
762
|
+
`"n"` and `"t"` taken as file-global when they are segment-local, silently
|
|
763
|
+
misordering every event in it.
|
|
764
|
+
|
|
765
|
+
No new field is needed to tell the two apart: the unusable value is sitting in
|
|
766
|
+
the header's store under the key `"segment"`, so a reader that finds it there
|
|
767
|
+
knows exactly which case it is in. What it MUST NOT do is answer "complete
|
|
768
|
+
single-segment file" to a caller that asks.
|
|
769
|
+
|
|
770
|
+
`"segment.sequence"` is the exception, and the only key in the header whose
|
|
771
|
+
absence or unusability makes the header itself malformed. It is the sole
|
|
772
|
+
ordering key of a segmented stream: a reader that cannot read it cannot place
|
|
773
|
+
the segment, and a default invents an order the file never had. The four
|
|
774
|
+
required top-level keys — `"protocol"`, `"perspective"`, `"detail"`,
|
|
775
|
+
`"startTime"` — are malformed on the same terms, there being no header to
|
|
776
|
+
construct without them.
|
|
777
|
+
|
|
778
|
+
Malformed here means malformed *for that segment*. A reader MUST report it and
|
|
779
|
+
MUST NOT present the segment as read; it MAY continue to the next segment, on
|
|
780
|
+
the same terms as [a malformed event](#versioning-and-compatibility). What it
|
|
781
|
+
MUST NOT do is return a header it invented — a required key filled in with a
|
|
782
|
+
default, an empty string, or a language's null standing in for a value the file
|
|
783
|
+
never carried. That reader reports a fabricated trace as a real one, which is
|
|
784
|
+
worse than reporting nothing.
|
|
785
|
+
|
|
786
|
+
#### The store is written last, and never twice
|
|
787
|
+
|
|
788
|
+
A store entry whose key the surrounding map already writes MUST NOT be written
|
|
789
|
+
a second time. A reader never produces such an entry — a key it recognised and
|
|
790
|
+
used is by definition not unrecognised — but a caller assembling a header by
|
|
791
|
+
hand can, and a CBOR map carrying one key twice is a map no two readers need
|
|
792
|
+
agree on. The field wins and the store entry is dropped. Event serialization
|
|
793
|
+
already resolves the collision this way.
|
|
794
|
+
|
|
339
795
|
### Protocol Identifier
|
|
340
796
|
|
|
341
797
|
The `"protocol"` field identifies the MoQT version without assuming lifecycle stage. Use the IETF document name minus the `draft-ietf-` prefix for drafts, and the RFC number for published standards:
|
|
@@ -436,10 +892,87 @@ If the stream ends part-way through a CBOR item, the file was truncated — a cr
|
|
|
436
892
|
- **Version 2** is defined by this document. **Version 1** is defined by the previous revision.
|
|
437
893
|
- **Writers SHOULD write version 2.**
|
|
438
894
|
- **Readers MUST accept both version 1 and version 2, and MUST reject any other version.** A version-1 file is exactly a non-segmented version-2 trace without the keys this revision adds, and every added key is optional. Accepting version 1 therefore costs a reader nothing and keeps every previously recorded capture readable.
|
|
439
|
-
- Unknown keys in header maps or event maps MUST be ignored (forward-compatible).
|
|
895
|
+
- Unknown keys in header maps or event maps MUST be ignored (forward-compatible). For the header specifically, see [Unrecognised keys in the header](#unrecognised-keys-in-the-header) — the header has three maps with named keys, not one, and each keeps its own.
|
|
896
|
+
- **A reader that writes a trace back out MUST preserve the unrecognised keys it read**, on a recognised event type as much as on an unrecognised one, and in the header as much as in an event. "Ignore" above means read past, not discard. Skipping a key costs a reader nothing; dropping it costs every reader downstream, because the output is a valid file that looks like it never carried the key — and the tools that read and rewrite traces are exactly the ones a trace passes through on its way to someone else: a redaction pass, a filter, a re-segmentation, a download with annotations applied.
|
|
440
897
|
- Optional keys MAY be added to an existing event type without a version bump — `"sid"` on event 0 was added this way. A reader must therefore treat any optional key as absent-by-default rather than assuming files of a given version all carry the same keys.
|
|
441
898
|
- New event types (`"e"` values) MAY be added without a version bump; readers MUST skip unknown event types rather than failing. A reader that rejects an unknown `"e"` value turns every future addition into a breaking change.
|
|
442
899
|
- New `"perspective"` and `"detail"` values MAY likewise be added; see [Perspective](#perspective).
|
|
900
|
+
- **A defined key whose value has an unusable type is treated as unrecognised.**
|
|
901
|
+
If this document says `"ta"` is an integer and a file carries `"ta": "hello"`,
|
|
902
|
+
a reader cannot use the value — but it MUST NOT delete it. The key goes to the
|
|
903
|
+
unrecognised-key store, is ignored for meaning, and is written back unchanged,
|
|
904
|
+
exactly as a key the reader had never heard of would be. A reader that knows
|
|
905
|
+
more about a key must not therefore preserve it less. "Unchanged" binds the
|
|
906
|
+
value, not its position: such a key is written wherever the store is emitted,
|
|
907
|
+
which is generally after the keys the event type owns rather than where the
|
|
908
|
+
key originally sat. CBOR maps are unordered, so this loses nothing — but a
|
|
909
|
+
byte-for-byte diff against the input will show it.
|
|
910
|
+
|
|
911
|
+
This matters most where it is least expected: adding a key to a reader's
|
|
912
|
+
vocabulary must never *reduce* what that reader preserves. Before a key is
|
|
913
|
+
defined, a wrong-typed value survives as an unrecognised key; after, a reader
|
|
914
|
+
that simply type-checks and moves on drops it. Both reference implementations
|
|
915
|
+
did precisely that when Event 1's keys were added, and lost values they had
|
|
916
|
+
previously kept.
|
|
917
|
+
|
|
918
|
+
Two bounds on the rule. A **required** key with an unusable type is a
|
|
919
|
+
malformed event, indistinguishable from that key being absent, because there
|
|
920
|
+
is no event to construct without it. And an unusable **optional** key MUST NOT fail
|
|
921
|
+
the event at all — it is the case this whole rule exists for, and the value
|
|
922
|
+
goes to the store.
|
|
923
|
+
|
|
924
|
+
That bound is about the event, not the API. Whether a malformed event ends the
|
|
925
|
+
read is a property of how a reader is driven: a streaming reader whose
|
|
926
|
+
idiomatic use collects into a fallible result stops at the first one, so for
|
|
927
|
+
that caller failing an event *is* failing the file. Readers MAY stop and MAY
|
|
928
|
+
continue, but MUST make which one is happening visible to the caller, and MUST
|
|
929
|
+
NOT report a partial event list as a complete one. Two conformant readers can
|
|
930
|
+
return different event counts for the same malformed file; nothing may depend
|
|
931
|
+
on them agreeing.
|
|
932
|
+
|
|
933
|
+
**What "unusable" means**, because two readers that draw this line differently
|
|
934
|
+
disagree about which values reach a field and which reach the store — and that
|
|
935
|
+
disagreement is invisible until someone diffs their output:
|
|
936
|
+
|
|
937
|
+
- A CBOR integer and a float with an integral value are both usable for an
|
|
938
|
+
integer key. Files carrying the float form exist and readers already MUST
|
|
939
|
+
accept them (see [Interoperability](#interoperability)); this rule does not
|
|
940
|
+
take that back.
|
|
941
|
+
- A number with a fractional part is **unusable** for an integer key.
|
|
942
|
+
- A negative value is **unusable** for a key this document defines as
|
|
943
|
+
unsigned. Every identifier here is a wire varint and has no negative form.
|
|
944
|
+
- A value outside the range the key's meaning allows is **unusable** where the
|
|
945
|
+
key is optional — `"sg"` as `-1`. Where the key is **required**, a reader
|
|
946
|
+
that can represent the value MUST still build the event: `"pp": 999` is a
|
|
947
|
+
number no conformant writer produces, but discarding the event loses far
|
|
948
|
+
more than the odd priority does. Report it if you like; do not delete it.
|
|
949
|
+
A required value the reader genuinely cannot represent remains malformed,
|
|
950
|
+
which is the same outcome as the key being absent.
|
|
951
|
+
- An implementation that holds a key in a type narrower than the key's range
|
|
952
|
+
MUST treat an unrepresentable value as unusable rather than rounding it.
|
|
953
|
+
Rounding invents a value the file never carried. This is why the identifier
|
|
954
|
+
keys must be held in a 64-bit-capable type: in a language where the ordinary
|
|
955
|
+
number type stops being exact at 2^53, using it for a `u64` identifier makes
|
|
956
|
+
that reader disagree with every other one above 2^53 — silently, and only
|
|
957
|
+
for large values, which is to say only in production.
|
|
958
|
+
- **On a rewrite, the reader rule outranks the writer rule.** A tool that reads
|
|
959
|
+
a trace and writes it back is bound by what it read, not by what a recorder
|
|
960
|
+
should have written. Where this document tells a writer MUST NOT emit
|
|
961
|
+
something — a `"msg"` that is not a map, a key outside its stream-type scope —
|
|
962
|
+
and tells a reader to keep it, a rewrite **keeps it**. The writer rules bind a
|
|
963
|
+
recorder deciding what to say about traffic it just saw; they do not license a
|
|
964
|
+
redaction pass, a filter or a format converter to quietly correct someone
|
|
965
|
+
else's file into one that no longer says what happened. A tool that wants to
|
|
966
|
+
report non-conformance should report it, not silently repair it.
|
|
967
|
+
- **Keys beginning `"x-"` are reserved for private use.** No revision of this
|
|
968
|
+
specification will define one, so a tool may add its own without ever
|
|
969
|
+
colliding with a future key — and a conformance test can rely on such a key
|
|
970
|
+
staying unrecognised. Every rule above applies to them unchanged: ignore them,
|
|
971
|
+
and preserve them on a rewrite. Absent this reservation there is no key an
|
|
972
|
+
implementation can be *sure* stays unknown, which matters more than it sounds:
|
|
973
|
+
a test fixture built from keys borrowed from a live proposal silently stops
|
|
974
|
+
testing anything the moment that proposal ships, and this specification did
|
|
975
|
+
exactly that to its own corpus once.
|
|
443
976
|
- The detail level in the header is informational — readers MUST handle missing fields gracefully regardless of declared level.
|
|
444
977
|
|
|
445
978
|
### Why version 2 exists
|
|
@@ -462,10 +995,168 @@ Two encoding choices are **normative**, because a CBOR library's defaults are no
|
|
|
462
995
|
- **An integral value MUST be written as a CBOR integer (major type 0 or 1), not as a float.** CBOR permits a float carrying an integral value, and some encoders emit one for any number beyond 32 bits — which every epoch-millisecond timestamp is. A reader MUST nevertheless accept the float form for any integer field, rejecting only a value that is fractional or outside the range a float represents exactly, since files written that way exist.
|
|
463
996
|
- **A byte string MUST be written as major type 2, not wrapped in the typed-array tag 64 of RFC 8746.** A reader MUST nevertheless accept tag 64 wrapping a byte string, for the same reason.
|
|
464
997
|
|
|
998
|
+
The first rule is about the **value**, not about the type this document gives
|
|
999
|
+
the key. `"effectiveRate"` is defined as a float and its most common value is
|
|
1000
|
+
`1.0` — "no rate-based dropping" — which is integral, so it is written as the
|
|
1001
|
+
CBOR integer `1`. Conversely a reader MUST accept a CBOR integer for a key this
|
|
1002
|
+
document types as a float, which is the mirror of the sentence above and was
|
|
1003
|
+
left implicit for one revision too long: the two reference implementations
|
|
1004
|
+
disagreed on exactly this value, one writing a float64 and one an integer, for
|
|
1005
|
+
a key whose commonest setting triggers it.
|
|
1006
|
+
|
|
1007
|
+
Both directions are byte-level differences that preserve the number exactly.
|
|
1008
|
+
That is the general shape of these rules: they exist so that two encoders
|
|
1009
|
+
produce the same bytes for the same trace, not because either encoding loses
|
|
1010
|
+
anything.
|
|
1011
|
+
|
|
465
1012
|
Both rules exist because both were broken in opposite directions by the two implementations, and neither test suite could see it: each read only bytes it had written itself, so each agreed with a convention the other did not implement.
|
|
466
1013
|
|
|
467
1014
|
This claim is maintained by a shared corpus of `.moqtrace` files that both implementations read and write as part of their test suites, covering at minimum: a version-1 file, a non-segmented version-2 file, a segmented version-2 stream, a file carrying an unknown event type, a file carrying an unknown perspective, and a truncated file. An implementation that cannot round-trip the corpus is not conformant, whatever this document says.
|
|
468
1015
|
|
|
1016
|
+
The corpus is [`moqtrace/` in the `test-traces` repository](https://github.com/moqtap/test-traces/tree/master/moqtrace), published as `@moqtap/test-traces`. Its `manifest.json` indexes every case with the format version, segment count, header fields, event count and event-type histogram a conformant reader must agree on, so a third implementation can use it without reading either reference implementation. Two files there carry the non-canonical encodings the rules above require readers to accept — integers as floats, and byte strings under tag 64 — because a rule with no file exercising it is a rule nobody is held to.
|
|
1017
|
+
|
|
1018
|
+
### Shapes a CBOR library may normalise before you see them
|
|
1019
|
+
|
|
1020
|
+
Every rule in this document about preserving a value assumes the reader is
|
|
1021
|
+
handed what the file carried. Sometimes it is not. A CBOR library may normalise
|
|
1022
|
+
a decoded value before any code written against this specification runs, and in
|
|
1023
|
+
each case below the normalisation is invisible: the reader cannot report a
|
|
1024
|
+
difference it was never shown.
|
|
1025
|
+
|
|
1026
|
+
The two reference implementations are known to do the following, and neither
|
|
1027
|
+
can be configured out of it without changing decoders:
|
|
1028
|
+
|
|
1029
|
+
| Shape in the file | What one library hands back |
|
|
1030
|
+
| ----------------- | --------------------------- |
|
|
1031
|
+
| An integer map key, `1` | The text key `"1"` — which then collides with a real `"1"` |
|
|
1032
|
+
| The text key `"__proto__"` | The text key `"__proto_"` — which then collides with a real `"__proto_"` |
|
|
1033
|
+
| The same key twice † | One entry — `cbor-x` keeps the last. **`ciborium` keeps both**, so a Rust reader *can* see this one |
|
|
1034
|
+
| CBOR `undefined` (major type 7, value 23) | CBOR `null` |
|
|
1035
|
+
| A byte string under RFC 8746 tag 64 | A bare byte string |
|
|
1036
|
+
| A float carrying an integral value | An integer |
|
|
1037
|
+
|
|
1038
|
+
† **The duplicate-key row is asymmetric, and that asymmetry is the point.**
|
|
1039
|
+
`ciborium` models a map as a list of pairs and hands back both entries, so part
|
|
1040
|
+
2 binds a Rust reader there: it can see the duplicate and must not delete
|
|
1041
|
+
either entry. `cbor-x` has already collapsed the pair before its caller runs,
|
|
1042
|
+
so part 3 applies to a JavaScript reader on the same file. A row in this table
|
|
1043
|
+
is not automatically excused for every implementation — read parts 2 and 3
|
|
1044
|
+
against the library actually in use, and expect a shape to be observable to one
|
|
1045
|
+
reader and invisible to the other.
|
|
1046
|
+
|
|
1047
|
+
**The rule, in three parts.**
|
|
1048
|
+
|
|
1049
|
+
1. **A writer MUST NOT *choose* any shape in that table.** Nothing this format
|
|
1050
|
+
defines needs one, so a recorder composing a trace out of what it observed
|
|
1051
|
+
has no reason to produce one, and this is where the requirement can actually
|
|
1052
|
+
be met.
|
|
1053
|
+
|
|
1054
|
+
This binds what a writer **originates**, not what it relays. A store may
|
|
1055
|
+
legitimately hold one of these shapes — the Rust reader hands back
|
|
1056
|
+
`"__proto__"` and an integer map key intact — and writing that store back
|
|
1057
|
+
offers exactly three options: drop the entry, rename it, or emit it as it
|
|
1058
|
+
stands. The first loses a value and the second invents one, and part 2
|
|
1059
|
+
forbids both. So the third is what a conformant rewrite does, and this rule
|
|
1060
|
+
does not stand in its way: **a stored entry is written as it stands.**
|
|
1061
|
+
Anything else would make a reader non-conformant for faithfully preserving
|
|
1062
|
+
what it was given, which is the opposite of what this section is for.
|
|
1063
|
+
|
|
1064
|
+
**Two rows are exceptions, for opposite reasons.** The last two — an
|
|
1065
|
+
integral float, and a byte string under tag 64 — are exceptions because a
|
|
1066
|
+
value-preserving re-encoding *exists*: rewriting them costs nothing but
|
|
1067
|
+
bytes, and the paragraph after this one requires it. The duplicate-key row
|
|
1068
|
+
is an exception because no such re-encoding exists at all: a map cannot
|
|
1069
|
+
carry one key twice and still be a map any two readers agree on, so a
|
|
1070
|
+
rewrite must drop an entry, and dropping it loses a value. That is the one
|
|
1071
|
+
place this document knowingly accepts a loss, and it is set out at the end
|
|
1072
|
+
of this section rather than waved at here.
|
|
1073
|
+
2. **Where a reader can observe the shape, it MUST NOT substitute a different
|
|
1074
|
+
one.** Route it to the unrecognised-key store and preserve it, or report the
|
|
1075
|
+
value as unusable — but do not hand back a value the file did not carry. A
|
|
1076
|
+
reader that turns `undefined` into `null`, or `"__proto__"` into
|
|
1077
|
+
`"__proto_"`, has invented data, and it has done so in the one code path
|
|
1078
|
+
whose entire purpose is preservation.
|
|
1079
|
+
3. **Where a reader cannot observe it, the reader is not non-conformant, and
|
|
1080
|
+
nothing may depend on the outcome.** A rule that requires reporting a
|
|
1081
|
+
difference the reader was never shown is a rule no one can implement, and
|
|
1082
|
+
this document does not issue one.
|
|
1083
|
+
|
|
1084
|
+
Part 3 is not an escape hatch for the merely inconvenient. It applies only
|
|
1085
|
+
where the normalisation happens *below* the implementation — where the value
|
|
1086
|
+
reaching this specification's code is already the normalised one. Anything the
|
|
1087
|
+
reader can still see, part 2 governs.
|
|
1088
|
+
|
|
1089
|
+
**The last two rows are different from the rest**, and are the exception that
|
|
1090
|
+
proves the rule: this document already requires readers to accept them and
|
|
1091
|
+
writers not to produce them (see the two normative encoding choices above).
|
|
1092
|
+
They are listed here because a reader that *preserves* one — a byte string
|
|
1093
|
+
still wrapped in tag 64, sitting in a store — is preserving a shape its own
|
|
1094
|
+
writer rule forbids, and would then emit it.
|
|
1095
|
+
|
|
1096
|
+
**So the two encoding rules apply to stored values too.** When a store is
|
|
1097
|
+
written back, an integral float in it is written as an integer and a tag-64
|
|
1098
|
+
byte string as major type 2, at any depth. That is a change of encoding and not
|
|
1099
|
+
of value, which is exactly the line this document already draws elsewhere:
|
|
1100
|
+
"written back unchanged" binds the value, not its encoding. The alternative —
|
|
1101
|
+
verbatim bytes — cannot be implemented by a reader whose decoder already
|
|
1102
|
+
normalised them on the way in, so requiring it would make one implementation
|
|
1103
|
+
permanently non-conformant while the other did nothing useful with its
|
|
1104
|
+
compliance.
|
|
1105
|
+
|
|
1106
|
+
One value does change under this rule: negative zero, whose sign is lost when
|
|
1107
|
+
`-0.0` is written as `0`. It is called out here rather than carved out, because
|
|
1108
|
+
a trace has no field where negative zero means anything, and an exception would
|
|
1109
|
+
cost more than it buys.
|
|
1110
|
+
|
|
1111
|
+
**Duplicate keys.** A writer MUST NOT emit a map carrying the same key twice.
|
|
1112
|
+
Readers are not required to agree about which entry of a duplicate pair wins,
|
|
1113
|
+
and no file may depend on it. RFC 8949 already calls such a map invalid; this
|
|
1114
|
+
document adds that a conformant tool must not *emit* one, having read one.
|
|
1115
|
+
|
|
1116
|
+
Two places produce one without anybody meaning to. A store built by hand is an
|
|
1117
|
+
ordered list of pairs rather than a map, so nothing stops a caller putting the
|
|
1118
|
+
same key in twice. And a map **nested inside** a stored value can carry a
|
|
1119
|
+
duplicate that came from the file itself — which is the more likely of the two,
|
|
1120
|
+
since it needs no mistake at all, only a non-conformant peer. The rule reaches
|
|
1121
|
+
both: a tool that read one must not emit one, at any depth.
|
|
1122
|
+
|
|
1123
|
+
**This is the one rule in this document that loses data, and it does so
|
|
1124
|
+
knowingly.** Collapsing a duplicate pair to a single entry discards the other
|
|
1125
|
+
entry's value. The alternative is emitting a map RFC 8949 calls invalid, which
|
|
1126
|
+
every downstream reader would then resolve differently — so the choice is not
|
|
1127
|
+
between losing a value and keeping it, but between losing it once, here, and
|
|
1128
|
+
losing it unpredictably at each reader thereafter. The input was already
|
|
1129
|
+
invalid; no rewrite of it can be lossless. Where the loss matters, report the
|
|
1130
|
+
duplicate rather than passing it on silently.
|
|
1131
|
+
|
|
1132
|
+
**And the two implementations will not lose the same one.** The distinction
|
|
1133
|
+
worth keeping straight, because the table above makes it and this paragraph
|
|
1134
|
+
once contradicted it: `ciborium` keeps **both** entries and makes no choice at
|
|
1135
|
+
all, so the Rust *reader* is the thing that keeps the first, by the first-match
|
|
1136
|
+
lookup it uses for every key. `cbor-x` has kept only the last before its caller
|
|
1137
|
+
runs. So the same input file rewritten through each yields two files differing
|
|
1138
|
+
in **value**, not merely in encoding.
|
|
1139
|
+
|
|
1140
|
+
That is a worse divergence than anything the two encoding rules address, and it
|
|
1141
|
+
is stated here rather than left to be discovered because it cannot be fixed:
|
|
1142
|
+
making one of them normative would require the other to report a difference its
|
|
1143
|
+
decoder discarded before any of this document's code ran. Keeping the first
|
|
1144
|
+
entry is the better behaviour where the choice exists, and it is what a
|
|
1145
|
+
first-match reader does anyway — but it is a SHOULD only the implementation
|
|
1146
|
+
that can still see both entries can be held to.
|
|
1147
|
+
|
|
1148
|
+
That asymmetry is exactly why this is worth a rule. A reader handed both
|
|
1149
|
+
entries can *delete* one silently, which a reader handed one cannot; the
|
|
1150
|
+
capable implementation is the one at risk, and both reference implementations
|
|
1151
|
+
did lose a value here — the Rust one on its read path, for a key a field had
|
|
1152
|
+
already taken — until it was found by inspection rather than by any test.
|
|
1153
|
+
|
|
1154
|
+
What keeps this from mattering is the rule at the top of this paragraph: **no
|
|
1155
|
+
conformant writer produces a duplicate key**, so a file carrying one came from
|
|
1156
|
+
a non-conformant peer, and a trace of a non-conformant peer is exactly the
|
|
1157
|
+
thing [Event 6](#event-6-error) exists to record. Record it there; do not rely
|
|
1158
|
+
on the rewrite.
|
|
1159
|
+
|
|
469
1160
|
---
|
|
470
1161
|
|
|
471
1162
|
# Appendix A: MoQT Carriage Convention (informative)
|
|
@@ -505,6 +1196,9 @@ Implementations MAY batch multiple events into one object for efficiency at high
|
|
|
505
1196
|
- **Event 10 carries track identity and per-hop timestamps** — `"ns"`, `"tn"`, `"tdr"`, `"tus"`, `"tuo"`, `"tdo"` — making per-hop latency attribution possible from the event alone, plus explicit update semantics for events emitted before all four timestamps are known.
|
|
506
1197
|
- **Event 10's trace ID is `"traceId"`, a 16-byte byte string.** An earlier revision of this proposal spelled it `"trace"` and left the type open; a text-string spelling invites per-implementation encodings, which defeats the cross-operator stitching the field exists for.
|
|
507
1198
|
- **Unknown event types, perspectives and detail levels must be tolerated, not rejected.** Version 1 said readers "SHOULD skip unknown event types"; neither reference implementation did, which is why the purely additive parts of this revision would have broken them anyway.
|
|
1199
|
+
- **Unrecognised keys must survive a read-modify-write.** Version 1 said only that unknown keys must be ignored, which both implementations read as permission to drop them — so every optional key either revision adds was safe to read past and silently destroyed by any tool that rewrote the file.
|
|
1200
|
+
- **The key-preservation rule now says what it means for the header**, in a section of its own. "Header maps" was plural and neither implementation read it that way: both dropped every unknown header key outright, which is the same data loss as on an event and strictly larger, since it applies to every unknown key rather than only a wrong-typed one. The header has three maps with named keys — the header itself, `"segment"` and `"sampling"` — and each keeps its own store. See [Unrecognised keys in the header](#unrecognised-keys-in-the-header).
|
|
1201
|
+
- **Event 6 can carry the bytes behind the error** — `"sid"`, `"ek"`, a `"rawlen"` gated at `"headers+sizes"` and a `"raw"` gated at `"full"`, with a normative 4096-byte cap and a one-per-flow latch. Version 1's error event held a code and a sentence, so a recorder that wanted to keep the offending bytes had to record the violation as a control message to have a field to put them in. See [Event 6](#event-6-error).
|
|
508
1202
|
- Event `"n"` and `"t"` clarified as **segment-local** in segmented traces.
|
|
509
1203
|
- **Non-droppable event types rule** added to the Sampling section.
|
|
510
1204
|
- **Trace ID Propagation section** defining the role of the `MOQTAP_TRACE_ID` extension parameter.
|