@downtrace/protocol 0.7.1 → 0.9.0

package/README.md

@@ -1,5 +1,7 @@
 # @downtrace/protocol
 
+> ⚠️ **Not production ready.** Downtrace is a closed free pilot: no plans, no billing and no SLA. The ingestion protocol is `v0` and can still change between minor versions. Install it where a dependency that is still moving is acceptable.
+
 The ingestion contract between Downtrace agents and the Downtrace cloud, as a JSON Schema (draft 2020-12) plus the TypeScript types generated from it.
 
 - `schema/v0/aggregates.schema.json` — the contract for what an agent sends. Everything else derives from it.
@@ -16,6 +18,8 @@ Since **0.2.0** an endpoint may also carry `postgres`: how many queries each req
 
 `POST {DOWNTRACE_URL}/v0/aggregates` · `Authorization: Bearer <token>` · JSON · 202 accepted, 400 invalid, 401 bad token, 429 rate limit or daily budget. The 202 body is an `IngestResponse`.
 
+`intervals` may be empty whenever the batch carries anything else: a process that serves no requests closes no interval, and nothing else a batch carries has to wait for one — what such a process throws or reports included. A batch that carries nothing at all is a `400` that says so.
+
 ## How the protocol changes
 
 **The package version is the protocol version.** `@downtrace/protocol@0.5.0` speaks protocol `0.5.0`, and CI refuses to publish a release where the two disagree, so you can read the contract off the version in your lockfile. A release that changes the package without touching the contract moves the patch segment only.
@@ -26,6 +30,12 @@ Since **0.7.0** `agent` may also carry `observers`: which of `pg`, `http`, `redi
 
 Also since **0.7.0**, `RuntimeHealth` requires none of its fields and at least one of them. It used to demand all six, which made the one language-dependent corner of the protocol: a Go runtime has garbage collection, a heap and a resident set, and no event loop, so it could either invent one or send none of the three it does have. Relaxing a constraint is additive — a sender reporting all six still validates.
 
+Since **0.8.0** a captured request may carry `poolWaitMs`: how long it waited for a connection from a pool before it could talk to the dependency at all. The instrumentation measures this per request already, and a consumer that only sees the total time cannot tell a slow database from one it never got to talk to. **Absent means the request asked no pool**, which is not a wait of zero — zero is a request that asked and was served at once — and the two have to stay apart, because a route that never queues must not read as one that queues for nothing.
+
+Since **0.9.0** an error says **how the instrumentation came to see it**, and may carry the structural context the application attached to it. `Operation.kind` and `ProcessException.kind` each gain `framework` — the exception reached the framework's error path and was turned into a 5xx — and `explicit` — the application handed it over itself, from a `catch` that no hook can see. Both may carry a `context`: a flat object of at most **8** keys whose values are strings of at most **64** characters, already sanitised by the sender under the same rule as an error message, so anything that looked like a value arrives as `?` and nothing in it is per user. It is the context of the **first** occurrence of that signature in the window, because what travels here is counted per signature and never per occurrence. Absent is the normal case, and an instrumentation older than this sends neither the new kinds nor the field.
+
+And since **0.9.0** a batch may carry `ending`: `signal`, `exit` or `idle`, meaning **this is the last batch of this process** and it handed over what it was holding before leaving. Absent means the sender did not say, exactly as with `observers`, and never that the process ended badly — an instrumentation older than this never says it, and a process an exception killed cannot say anything at all. It carries no instant on purpose: the one worth showing is when the batch arrived, and the cloud measures that itself. It exists because from the outside every ending looks alike — batches stop — and telling a process that handed over from one that died is the difference between «nothing was lost» and «if it died, what it had not sent is lost».
+
 Also since **0.7.0**, a capture's evidence has a contract of its own: `schema/v0/capture-evidence.schema.json`, exported as `CAPTURE_EVIDENCE_SCHEMA_V0` with the types `CaptureEvidence`, `CapturedRequest`, `CapturedOperation` and `CaptureCoverage`, and its path as `CAPTURE_EVIDENCE_PATH` / `captureEvidencePath(id)`. It is the black box's fine detail, frozen: requests with their operations, in order, with **starts and ends** rather than durations — order and overlap are the whole reason for capturing detail. Hashes only; the text of a query never travels here. And it declares **two coverages**, never a total: what it observed from its effective start, and what it attached from detail still retained.
 
 And since **0.7.0** that evidence may carry `reference`: a few requests per endpoint the instrumentation kept as something to compare the captured ones against, with `selection` —an enum, because a sender writing «representative» in a free string means nothing— the `population` they were drawn from, and `renewalPaused` when requests were observed that it deliberately did not consider. A sample has the same shape as a captured request. Being earlier does not make one healthy, and nothing here says it does: `product.md:100`.
@@ -36,7 +46,21 @@ And since **0.7.0** `agent` may carry `resources`: what the instrumentation says
 
 Also since **0.7.0**, **what the cloud answers is part of the contract**: `schema/v0/ingest-response.schema.json`, exported as `INGEST_RESPONSE_SCHEMA_V0` with the types `IngestResponse` and `PendingCapture`. It carries `accepted` and `inserted` as it always did, and may carry `captures` — the captures the cloud is waiting for in the environment the batch came from, each with what to watch, for how long, and until when. An agent that ignores the body behaves exactly as it did, which is what lets the cloud accept before any agent sends (ADR 0008).
 
-Fields are only ever **added**, and always optional. Each addition bumps the minor version, and the batch's `protocol` field is an enum of every published minor, so a cloud that speaks 0.2 still accepts an agent that speaks 0.1. Removing or renaming a field, or moving a histogram's bucket bounds, would be a major version on a new path. Bucket bounds are declared in the schema itself (`x-latency-boundaries-ms`, `x-calls-per-request-boundaries`), and so are the versions themselves and the ingestion path (`x-ingest-path`), so they are part of the generated contract rather than a constant someone has to keep in sync.
+And since **0.9.0** a process exception may say what it already said. `ProcessException` may carry `total`: how many times its signature has happened in its instance since that instance started, a running total every delivery of the signature carries again. `count` keeps its meaning —what happened since the last delivery the sender heard land— so `total - count` is what the sender knows was applied. With it the cloud applies a delivery once: it keeps the largest total it has applied per instance and signature and counts only what a delivery adds beyond it, and never more than its `count`. So a batch sent again because its answer was lost, sent twice, or overtaken by a later one adds nothing, and one that never arrived is counted by the next. It relies on `instance.id` being generated once per process, which is what the contract already says. Without it, which is what an older sender sends, the cloud counts `count` as delivered, as it always did, and says so where that count is shown.
+
+And still protocol **0.9.0**, `resources` may carry `droppedExceptions`: the exception events the instrumentation discarded because they did not fit in its caps of distinct signatures — the register's window, and the sender's accumulation while batches do not land — since the previous batch. An occurrence, never a signature: what did not fit cannot be named without being remembered, and remembering is the growth the cap exists to stop. It is the number that keeps a process that threw 40 distinct errors from reading as one that threw 32 — the loss for budget, told apart from the absence of errors (COB-01) — and the cloud adds it up per instance, as it adds the batch counters.
+
+Also since **0.9.0**, a capture's evidence may carry `coarse`: the coarse half of the black box, frozen — the last few minutes, second by second and per endpoint (counts, errors, latency sums and a maximum, operations per second), plus the process's event loop delay in its own series, and the count of the routes the register had no row for. It is what `product.md:122` says a capture freezes alongside the fine detail and the reference samples, and what `product.md:94` gives its reason: seeing how something detected late began. Sums and a maximum, never percentiles; a route's series starts at the first second the process saw it, so a watched quiet second is a zero and an unwatched second is absent; and a second with no event loop reading is absent, because a zero would say the loop was idle. The bounds are the register's own — 129 routes at most, 300 seconds — and the summary is optional, so an older instrumentation that sends none is as valid as it was.
+
+And still protocol **0.9.0**, a capture's `coverage` may carry `shed`: `{ "ms": integer ≥ 1, "reason": "latency" | "memory" }` — the time inside the capture's window when the instrumentation was not writing the fine detail, neither the ring nor the reserve, and why. `ms` is how long that lasted, rounded up, and `reason` is the last reason in force in the window, the same enum `agent.resources.shedReason` uses in a batch. It is the declaration that keeps a capture taken while shedding from reading as «the route was quiet»: the requests of that time are not in the evidence and do not count as observed, while the coarse summary does count them, and the capture's result says exactly that. It is optional, because three different things all read as «not said»: no shedding at all, a shedding the instrumentation did not decide (configuration, the benchmark), and a sender older than the field.
+
+And still protocol **0.9.0**, `agent` may carry `errorSources`: `{ "inRequest": [...], "outsideRequest": [...] }` — which of the sources of error the process has **connected**, as of the moment the batch is sealed. `inRequest` holds the values of `Operation.kind` that are errors — `error`, `framework`, `explicit` — and `outsideRequest` the values of `ProcessException.kind` — `uncaught`, `unhandled-rejection`, `framework`, `explicit`. A value is on its list when the process has, at that moment, the instrumentation that produces it installed; the application calling `captureException` does not make it one without the hook connected. Both lists are required when the field is present, and no value appears twice. **Absent is «did not declare»**, exactly as with `observers`: an older sender says nothing, and that is not the same as the two lists left empty, which are a statement — the sender says it has connected no source of error. It exists because the protocol only bounds what a batch can carry from above: an instrumentation that speaks a protocol and does not send everything it carries has no other way to say so, and a reader that attributed to it the kinds its protocol allows would be saying what it does not send.
+
+And still protocol **0.9.0**, a captured request may carry `dependenciesTruncated`: its list of dependencies did not fit and was truncated to what the register keeps per request, and the capture's `coverage` counts such rows in its own `dependenciesTruncated`, beside `truncated` and `detailLost`. It is the mark that keeps a row that entered a capture of a dependency because its list did not fit from reading as a complete one: a dependency that is not in the list is not proven not to have been used, and `truncated` means operations, so the row says it in a field of its own. Absent means false, as with the other two: a sender older than the field sends nothing, and an evidence without the mark is as valid as it was.
+
+And still protocol **0.9.0**, an operation may be an **outgoing HTTP call** or a **Redis command**: `Operation.kind` gains `call`, identified by its method and its host, and `command`, identified by its name and its server, beside `query` as the things a route ran. Their text is a word and the place it went — `POST api.stripe.com`, `HGETALL cache:6379` — and never a path, a query string, a key or an argument: the schema refuses a label with anything after the place, or a slash, a question mark or a hash in it, and the text is absent in minimal mode or when the dependency's name is withheld, with the hash still the identity. Which kinds are errors is said beside the enum as `x-error`, one boolean per value: `error`, `framework` and `explicit` are, the three things a route ran are not, and `errorSources.inRequest` takes exactly the ones that are. A captured operation may say its `kind` too, from the same enum. **Absent there means the sender did not say**, never a query: an instrumentation older than this wrote every operation it recorded into the black box without one, the error signature beside a failed query among them, so a reader that took them for queries would be naming what it was not told.
+
+Fields are only ever **added**, and always optional. Each addition bumps the minor version, and the batch's `protocol` field is an enum of every published minor, so a cloud that speaks 0.2 still accepts an agent that speaks 0.1. Removing or renaming a field, or moving a histogram's bucket bounds, would be a major version on a new path. Bucket bounds are declared in the schema itself (`x-latency-boundaries-ms`, `x-calls-per-request-boundaries`), and so are the versions themselves and the ingestion path (`x-ingest-path`), so they are part of the generated contract rather than a constant someone has to keep in sync. So is the first minor that carries each kind of operation and of process exception (`x-since`, beside each `kind` enum), which is how a consumer tells a kind of error a sender cannot send from one it did not send, and whether each kind of operation is an error (`x-error`). A validator in strict mode has to be told these annotations are keywords.
 
 ## Source