@observertc/observer-js 1.0.0-beta.13 → 1.0.0-beta.15

package/README.md

@@ -40,11 +40,9 @@ and emits a single, unified stream of typed events the application can react to.
 > works whether your project uses `import` (ESM) or `require()` (CommonJS). Everything — including
 > the built-in file sink — is exported from the single `@observertc/observer-js` entry.
 
-> **For AI agents:** [`llms.txt`](./llms.txt) is a curated map of these docs (with a one-file
-> [`llms-full.txt`](./llms-full.txt) export for one-shot ingestion); [`AGENTS.md`](./AGENTS.md)
-> covers build/test commands and conventions for working **in** this repository. `llms-full.txt`
-> ships in the npm package (so it's available wherever the library is installed); `llms.txt` and
-> `AGENTS.md` live in the repo (and `llms.txt` belongs at the root of the docs site).
+> **For AI agents:** [`llms.txt`](./llms.txt) is a curated map of these docs (it belongs at the root
+> of the docs site); [`AGENTS.md`](./AGENTS.md) covers build/test commands and the conventions for
+> working **in** this repository.
 
 ---
 
@@ -55,7 +53,7 @@ and emits a single, unified stream of typed events the application can react to.
 3. [Data flow](#data-flow)
 4. [Entity hierarchy](#entity-hierarchy)
 5. [Ingestion: `accept()`, context & lifecycle](#ingestion-accept-context--lifecycle)
-6. [Update policies](#update-policies)
+6. [When things update](#when-things-update)
 7. [The event bus](#the-event-bus) ← the core of the API
 8. [API reference](#api-reference)
 9. [Schema types (`ClientSample`)](#schema-types-clientsample)
@@ -65,8 +63,9 @@ and emits a single, unified stream of typed events the application can react to.
 13. [Sinks (per-client sample persistence)](#sinks-per-client-sample-persistence)
 14. [Injecting data into a client](#injecting-data-into-a-client)
 15. [Logging](#logging)
-16. [Error-handling philosophy](#error-handling-philosophy)
-17. [Development & extension guide](#development--extension-guide)
+16. [Design notes](#design-notes)
+17. [Error-handling philosophy](#error-handling-philosophy)
+18. [Development & extension guide](#development--extension-guide)
 
 ---
 
@@ -105,10 +104,9 @@ import { Observer, ClientSample } from '@observertc/observer-js';
 
 // 1. Create an observer.
 const observer = new Observer({
-  // when the observer aggregates call/client metrics:
-  updatePolicy: 'update-when-all-call-updated',
-  // default policy applied to calls created automatically by accept():
-  defaultCallUpdatePolicy: 'update-on-any-client-updated',
+  // a call updates when any of its clients does, and the observer when any of its calls does —
+  // both default to true, so this line is only here to show the knob exists:
+  autoUpdateOnCallUpdate: true,
   // optional auto-teardown:
   closeCallIfEmptyForMs: 20_000,
   closeClientIfIdleForMs: 60_000,
@@ -292,30 +290,38 @@ These return `undefined` (and warn) when the parent is closed; `createObservedCa
 
 ---
 
-## Update policies
+## When things update
 
-"Update" means *recompute aggregated metrics and emit the `*-updated` event* at that level.
-Both the observer and each call have a configurable trigger. Updates are **event-driven** — there
-is no built-in timer. An app that wants a fixed cadence can call `observer.update()` /
-`call.update()` from its own `setInterval`. With `'none'`, **nothing auto-updates** — the level
-updates only when the application calls the public `update()` itself.
+"Update" means *recompute aggregated metrics, run the detectors, and emit the `*-updated` event* at
+that level. Updates are **event-driven** — there is no built-in timer.
 
-**Observer-level** (`ObserverConfig.updatePolicy`, default `update-when-all-call-updated`):
+The rule is structural rather than configurable:
 
-| Policy | Triggers `observer.update()` when… |
-|--------|-------------------------------------|
-| `update-on-any-call-updated` | any call updates |
-| `update-when-all-call-updated` | every call has updated since the last observer update |
-| `none` | never automatically — only when the app calls `observer.update()` |
+> **A call is updated when any of its clients is updated. The observer is updated when any of its
+> calls is updated.** Composed, that means the observer is updated exactly when any client anywhere
+> is updated.
 
-**Call-level** (`ObservedCallSettings.updatePolicy`, defaulted from
-`ObserverConfig.defaultCallUpdatePolicy`):
+Two booleans, both defaulting to `true`, let you opt out of a link in that chain:
 
-| Policy | Triggers `call.update()` when… |
-|--------|--------------------------------|
-| `update-on-any-client-updated` | any client in the call updates |
-| `update-when-all-client-updated` | every client has updated since the last call update |
-| `none` | never automatically — only when the app calls `call.update()` |
+| Setting | Where | Effect when `false` |
+|---------|-------|---------------------|
+| `autoUpdateOnClientUpdate` | `ObservedCallSettings` | the call updates only when you call `call.update()` |
+| `autoUpdateOnCallUpdate` | `ObserverConfig` | the observer updates only when you call `observer.update()` |
+
+An app that wants a fixed cadence sets both to `false` and drives `observer.update()` from its own
+`setInterval`. Note that **observer-scoped detectors and validators run nowhere else** — if the
+observer never updates, they never run.
+
+```ts
+const observer = new Observer({ autoUpdateOnCallUpdate: false });
+
+setInterval(() => observer.update(), 5_000);
+```
+
+> Earlier versions had an `updatePolicy` / `defaultCallUpdatePolicy` enum (`'update-on-any-…'`,
+> `'update-when-all-…'`, `'none'`) and a pluggable `Updater` object. Both are gone. "When all clients
+> have updated" sounds appealing and deadlocks on the first client that stops sending — one silent
+> participant froze the whole call's aggregation until it timed out.
 
 ---
 
@@ -334,7 +340,7 @@ contains the ancestry from the observer down to the entity that raised it, plus
 specific subject:
 
 ```ts
-type ObserverEventBase            = { observer: Observer };
+type ObserverEventBase            = { observer: Observer, context?: AcceptContext };
 type ObservedCallScope            = ObserverEventBase            & { observedCall: ObservedCall };
 type ObservedClientScope          = ObservedCallScope            & { observedClient: ObservedClient };
 type ObservedPeerConnectionScope  = ObservedClientScope          & { observedPeerConnection: ObservedPeerConnection };
@@ -360,9 +366,11 @@ additional field(s) on top of that scope.
 
 | Event | Extra payload | Fires when |
 |-------|---------------|-----------|
-| `observer-updated` | — | `observer.update()` ran (per the observer update policy) |
+| `observer-updated` | — | `observer.update()` ran (see [When things update](#when-things-update)) |
 | `observer-closed` | — | `observer.close()` |
 | `sample-rejected` | `{ reason: 'observer-closed' \| 'missing-callId' \| 'missing-clientId', sample: ClientSample }` | a sample was dropped by `accept()` |
+| `observer-issue` | `{ issue: ObserverIssue }` | `observer.addIssue(...)` — a cross-call / SFU-wide finding (see [observer-level detectors](#observer-level-detectors-cross-call--sfu-wide)) |
+| `validation-ready` | `{ validator: string, report: ValidationReport }` | a [validator](#validators--one-shot-structural-checks) settled — fires once per check, not per tick |
 
 #### Mediasoup level — scope `{ observer, observedMediasoupRouter }`
 
@@ -383,7 +391,7 @@ See [Mediasoup router observation](#mediasoup-router-observation) for the full d
 | `call-closed` | — | the call closed |
 | `call-empty` | — | last client left the call |
 | `call-not-empty` | — | first client joined a previously-empty call |
-| `call-issue` | `{ issue: ClientIssue }` | `call.addIssue(...)` (server-side detector finding) |
+| `call-issue` | `{ issue: ObserverIssue }` | `call.addIssue(...)` (server-side detector finding) |
 
 #### Client level — scope `{ observer, observedCall, observedClient }`
 
@@ -396,7 +404,8 @@ See [Mediasoup router observation](#mediasoup-router-observation) for the full d
 | `client-joined` | — | first `CLIENT_JOINED` event seen |
 | `client-left` | — | `CLIENT_LEFT` seen (or inferred on close) |
 | `client-rejoined` | `{ timestamp: number }` | a later `CLIENT_JOINED` after an earlier join |
-| `client-issue` | `{ issue: ClientIssue }` | a client-reported issue arrived, or `client.addIssue(...)` |
+| `client-issue` | `{ issue: ClientIssue }` | a client-reported issue arrived, or `client.addIssue(...)`. A keyed issue also opens an entry in `observedClient.activeIssues` |
+| `client-issue-resolved` | `{ resolvedIssue: ResolvedActiveClientIssue }` | a stateful issue ended — the client sent its `<type>-resolved` companion, or the observer force-closed it. Carries the finished interval (`durationInMs`, `resolvedBy`) — see [client issues](#client-issues-the-lifecycle-and-the-division-of-labour) |
 | `client-metadata` | `{ metaData: ClientMetaData }` | a client meta item arrived |
 | `client-extension-stats` | `{ extensionStats: ExtensionStat }` | an app-defined extension stat arrived |
 | `client-event` | `{ event: ClientEvent }` | any client event was processed |
@@ -449,8 +458,8 @@ listen to them, but prefer the bus equivalents above for application logic.
 new Observer<AppData>(config?: ObserverConfig<AppData>)
 
 type ObserverConfig<AppData = Record<string, unknown>> = {
-    updatePolicy?: 'update-on-any-call-updated' | 'update-when-all-call-updated' | 'none';
-    defaultCallUpdatePolicy?: ObservedCallSettings['updatePolicy'];
+    // a call updates when any client does; the observer when any call does. Default true.
+    autoUpdateOnCallUpdate?: boolean;
     appData?: AppData;
     closeClientIfIdleForMs?: number;
     closeCallIfEmptyForMs?: number;
@@ -487,7 +496,15 @@ Key members:
 - `createObservedCall<T>(settings): ObservedCall<T> | undefined`
 - `getOrCreateObservedCall<T>(settings): ObservedCall<T> | undefined`
 - `update(): void` — force an aggregation/`observer-updated` tick
+- `addObserverDetector(name, config?): this` — build a cross-call detector onto `observer.detectors`
+- `addCallDetector(name, config?): this` / `removeCallDetector(name): this` — register a call-scoped
+  detector for every call created **from now on**
+- `addValidator(name, config?): this` — start a one-shot structural check
 - `close(): void`
+- `readonly detectors: Detectors` — observer-scoped registry. **Starts empty**; nothing is implicit
+- `readonly callDetectorConfigs: Map<name, config>` — what `addCallDetector` recorded
+- `readonly validators: Set<RunningValidator>` — normally empty; each removes itself on finishing
+- `readonly activeIssuesRegistry: ActiveIssuesRegistry` — the fleet's open client issues
 - `readonly observedCalls: Map<string, ObservedCall>`
 - `readonly observedTURN: ObservedTURN`
 - `get appData()`, `get numberOfCalls()`
@@ -500,7 +517,8 @@ Key members:
 
 ```ts
 type ObservedCallSettings<AppData = Record<string, unknown>> = {
-    updatePolicy?: 'update-on-any-client-updated' | 'update-when-all-client-updated' | 'none';
+    // update this call whenever one of its clients accepts a sample. Default true.
+    autoUpdateOnClientUpdate?: boolean;
     callId: string;
     appData?: AppData;
     closeCallIfEmptyForMs?: number;
@@ -512,8 +530,11 @@ Key members:
 - `readonly callId: string`, `appData: AppData`
 - `readonly observedClients: Map<string, ObservedClient>`, `get numberOfClients()`
 - `getObservedClient<T>(clientId)`, `createObservedClient<T>(settings)`, `getOrCreateObservedClient<T>(settings)` (all `… | undefined`)
-- `addIssue(issue: ClientIssue): void` — raise a **call-level** issue → emits `call-issue`
+- `addIssue(issue: ObserverIssue): void` — raise a **call-level** issue → emits `call-issue`
+- `addDetector(name, config?): this` — build a call-scoped detector onto this call only
 - `readonly detectors: Detectors` — server-side detector registry (empty by default; see [Detectors](#detectors-server-side-extension-point))
+- `readonly activeIssuesRegistry: ActiveIssuesRegistry` — this call's open client issues, propagating into the observer's
+- `readonly unconsumedOutboundTracks: Set<ObservedOutboundTrack>` — maintained by the resolver
 - `scoreCalculator: ScoreCalculator`, `get score()`, `readonly calculatedScore`
 - `remoteTrackResolver?: RemoteTrackResolver` — set from `ObserverConfig.createRemoteTrackResolver` at call creation (see [Remote track resolution](#remote-track-resolution-mediasoup--sfu))
 - aggregates: `numberOfIssues`, `numberOfPeerConnections`, `numberOfInboundRtpStreams`,
@@ -562,11 +583,28 @@ Key members:
   `iceCandidates`, `iceCandidatePairs`, `certificates`, `selectedIceCandidatePairs`,
   `selectedIceCandiadtePairForTurn`
 - State: `connectionState?`, `iceConnectionState?`, `iceGatheringState?`, `usingTURN`, `usingTCP`
-- Metrics: `currentRttInMs?`, `currentJitter?`, `availableIncomingBitrate`,
-  `availableOutgoingBitrate`, sending/receiving bitrates, packet rates, and `total*` / `delta*`
-  byte/packet counters
+- Metrics: `currentRttInMs?`, `iceRttInMs?`, `rtcpRttInMs?`, `sfuHopRttInMs?`, `currentJitter?`,
+  `availableIncomingBitrate`, `availableOutgoingBitrate`, sending/receiving bitrates, packet rates,
+  and `total*` / `delta*` byte/packet counters
 - `accept(pcSample, context?)`, `close()`, `get score()`
 
+**Two different round trips — don't mix them.** `iceRttInMs` comes from ICE/STUN consent checks and
+measures the trip to *whatever terminates ICE*: **in an SFU topology that is the SFU**, so it is the
+client↔SFU leg. `rtcpRttInMs` comes from RTCP receiver reports and is an **end-to-end** media-path
+round trip. They are not interchangeable, and averaging them together produces a number that moves
+as streams come and go for reasons unrelated to the network. `currentRttInMs` therefore *prefers*
+RTCP and falls back to ICE — always one kind within a tick, never a blend. `sfuHopRttInMs`
+(`rtcp − ice`) estimates everything past the SFU, which separates "this client's last mile is slow"
+from "the path beyond the SFU is slow".
+
+**Counter-reset boundaries.** Chrome resets an SSRC's cumulative counters when the codec switches
+([crbug/webrtc/5361](https://bugs.chromium.org/p/webrtc/issues/detail?id=5361), open since 2015),
+which otherwise shows up as a sawtooth spike or a negative bitrate. `ObservedInboundRtp` /
+`ObservedOutboundRtp` therefore set **`counterResetBoundary`** on any tick where `codecId`,
+`encoder`/`decoderImplementation` or `scalabilityMode` changed, and suppress every delta for that
+tick. Without this, a room-wide codec rollout fires a synchronized fake-degradation alert across
+every participant at once.
+
 **Remote-RTP correlation (derived).** During `accept()`, receiver/sender reports are linked
 to the local streams by `remoteId` (fallback SSRC) and surfaced as fields:
 
@@ -701,12 +739,28 @@ snapshot happens only once.
 
 ## Detectors (server-side extension point)
 
-`observer-js` deliberately ships **no built-in detectors**. Per-client signals — packet loss,
-jitter, RTT, freezes, etc. — are already detectable on the client and arrive on samples as
-`clientIssues` (surfaced via `client-issue`). Server-side detection should focus on what only
-the server can see by **correlating data across the clients of a call**.
+`observer-js` ships [ten detectors](#built-in-detectors), each an opt-in extension you register
+explicitly with `addObserverDetector` / `addCallDetector` / `addDetector` (see
+[Registering detectors](#registering-detectors)) — none are created automatically. All of them
+correlate **across** the clients of a call or the calls of a fleet, because that is the only thing a
+server can do better than a browser: per-client signals — packet loss, jitter, RTT, freezes — are
+already detected on the client and arrive on samples as `clientIssues` (surfaced via `client-issue`).
 
-The hook lives on **`ObservedCall`**:
+Findings are raised as **`ObserverIssue`** — `{ type, timestamp, payload? }` — and the payload is the
+**object**, not a JSON string. A server-raised finding is delivered to an in-process handler, so
+there is nothing to serialise for:
+
+```ts
+observer.on('call-issue', ({ observedCall, issue }) => {
+  issue.payload;                  // the object; no JSON.parse
+  issuePayloadOf(issue);          // if you want to accept a string payload too
+  issuePayloadAsString(issue);    // only at an edge that needs text (log, HTTP, queue)
+});
+```
+
+(`ClientIssue`, the type on samples, keeps its string payload — that one really is a wire format.)
+
+The registry is also an open extension point, on **`ObservedCall`**:
 
 ```ts
 import { Observer, Detector } from '@observertc/observer-js';
@@ -730,21 +784,462 @@ observer.on('call-added', ({ observedCall }) => {
 observer.on('call-issue', ({ observedCall, issue }) => { /* react */ });
 ```
 
-`Detector` interface and the registry:
+### Client issues: the lifecycle, and the division of labour
+
+The most important thing to understand about detection in this library is **what it deliberately
+does not do**. A client running
+[`client-monitor-js`](https://github.com/ObserveRTC/client-monitor-js) already ships ~20 detectors
+that decide *what is wrong with that endpoint* — `congestion`, `cpulimitation`, `audio-concealment`,
+`freezed-video-track`, `keyframe-storm`, `video-decoder-overloaded`, `stuck-decoder`,
+`ice-disconnected`, and so on. Those verdicts are better than anything re-derived from raw counters
+server-side, because they carry hysteresis and multi-signal confirmation: `audio-concealment`
+subtracts silent concealment (raw `concealedSamples` rises during ordinary silence, so a naive
+detector flags every quiet moment); `audio-jitter-buffer-stress` requires the buffer to be grown
+**and** NetEQ to be time-stretching (a grown buffer alone means NetEQ is *succeeding*);
+`ice-disconnected` only fires once `disconnected` has persisted, so the blips ICE heals on its own
+never surface.
+
+**observer-js does not repeat that work.** Its job is the question no browser can answer: *who else
+is in this state right now, what do they have in common, and where in publisher → SFU → subscriber
+does the fault begin?*
+
+#### The wire format
+
+From client-monitor-js **4.6.0** the whole issue lifecycle reaches the server. A stateful issue
+arrives as two `clientIssues[]` entries sharing a `key`:
+
+```
+raise:      { type: 'stuck-decoder',          key, payload,                                 timestamp: raisedAt }
+resolution: { type: 'stuck-decoder-resolved', key, payload: { raisedAt, comment, …final },  timestamp: resolvedAt }
+```
+
+The observer opens an entry in `observedClient.activeIssues` on the raise and closes it on the
+matching key, emitting **`client-issue-resolved`** with the finished interval. Handled for you:
+
+- the `-resolved` **suffix** is stripped, so both entries share one logical `type`;
+- a **re-raise** of a live key refreshes the payload without restarting `raisedAt`;
+- **keyless** entries are one-shot — reported via `client-issue`, never tracked;
+- issues still open when a client closes are **force-resolved** (`resolvedBy: 'client-closed'`), and
+  the registry additionally expires stale entries, so a crashed participant can't leave an issue
+  "active" forever.
+
+#### Why intervals beat windows
+
+This turns point-in-time symptom reports into **intervals**, and that is the whole game. *"Several
+clients reported congestion in the last 10 seconds"* is a heuristic that has to guess whether the
+symptoms are still happening. *"Several clients are congested **right now, simultaneously**"* is
+ground truth, because the client says when the episode ends. Overlapping intervals are far stronger
+evidence of a shared cause than near-in-time reports.
+
+```ts
+observer.on('client-issue', ({ observedClient, issue }) => { /* opened (or one-shot) */ });
+observer.on('client-issue-resolved', ({ resolvedIssue }) => {
+  resolvedIssue.type;          // 'stuck-decoder' — suffix stripped
+  resolvedIssue.durationInMs;  // how long the episode lasted
+  resolvedIssue.resolvedBy;    // 'client' | 'timeout' | 'client-closed'
+});
+
+// the live per-client mirror
+observedClient.activeIssues;   // ObservedClientIssueRegistry, keyed by issue.key
+```
+
+> **client-monitor-js >= 4.6.0 is required** for every issue-driven detector. There is no fallback
+> path that infers these conditions from raw counters — the client decides better, and maintaining a
+> worse second implementation to be polite to old clients is how both end up wrong. Issues without a
+> `key` have no lifecycle (nothing could ever close them), so they stay one-shot: reported on
+> `client-issue`, never registered.
+
+#### `ActiveIssuesRegistry` — issues are **pushed**, not polled
+
+A detector does not go looking for the issues it cares about. It implements `ActiveIssueTracker` and
+registers for the types it consumes; the registry hands them over as they open and close.
+
+```ts
+observedCall.activeIssuesRegistry   // this meeting
+observer.activeIssuesRegistry       // the fleet; every call's registry propagates into it
+
+observer.activeIssuesRegistry.addIssueTracker('congestion', myDetector);
+observer.activeIssuesRegistry.removeIssueTracker(myDetector);
+
+registry.values();   // the open issues in this scope, oldest first
+registry.size;       // how many
+```
+
+The cost of a detector is then proportional to the issues it actually receives, not to the number of
+participants: a healthy 500-client fleet does no per-tick work at all, because nothing was pushed.
+
+**There is no wildcard.** A tracker names its types and sees nothing else. "Feed me everything and
+I'll work out what matters" moves the decision from the application — which knows its client build
+and its issue vocabulary — onto a detector that has to guess, and it makes the cost of a
+subscription unbounded and invisible. If a detector should watch five types, list five types.
+
+Onset spread is measured on the **observer clock**, never the client's. `raisedAt` comes from each
+participant's own machine, and comparing those across clients makes clock skew look like a
+synchronized infrastructure event.
+
+### Observer-level detectors (cross-call / SFU-wide)
+
+Some findings only exist **above** call scope — "many calls on the same SFU degraded at once" is far
+more actionable than fifty individual client alerts. The same detector registry exists on the `Observer`,
+runs on every `observer.update()`, and raises findings through `observer.addIssue(...)`, surfaced on
+the bus as **`observer-issue`**:
+
+```ts
+observer.detectors.add({
+  name: 'sfu-wide-degradation',
+  update: () => {
+    const degradedCalls = [ ...observer.observedCalls.values() ].filter(isDegraded);
+
+    if (observer.numberOfCalls > 3 && degradedCalls.length / observer.numberOfCalls > 0.6) {
+      observer.addIssue({ type: 'SFU_WIDE_QUALITY_DEGRADATION', timestamp: Date.now() });
+    }
+  },
+});
+
+observer.on('observer-issue', ({ issue }) => alert(issue));
+```
+
+### Publisher → subscribers: the resolver links
+
+The question a single browser can never answer is *"did **everyone** receiving Alice see the same
+degradation?"*. The join is the publisher↔subscriber links maintained by a
+[`RemoteTrackResolver`](#remote-track-resolution-mediasoup--sfu), and detectors walk them directly:
+
+```ts
+outboundTrack.remoteInboundTracks;      // Set<ObservedInboundTrack> — every subscriber of this source
+inboundTrack.remoteOutboundTrack;       // the publisher, or undefined if unlinked
+inboundTrack.getInboundRtp();           // that receiver's RTP stats
+observedCall.unconsumedOutboundTracks;  // published tracks with no subscriber at all
+```
+
+> A `TrackDistributionAggregator` class used to sit in front of these links and summarise every
+> published track against all of its receivers, on every tick. It is gone. It scanned the majority
+> (all published tracks) to find the interesting minority, which is the wrong axis — the detectors
+> now start from the handful of *affected* tracks the issue registry pushed at them and resolve only
+> those. The statistics helpers it used (`percentile`, `median`, `summarize`, `counterDelta`,
+> `robustZScore`, `SlidingWindow`, `TrendTester`) are all still exported for building your own.
+
+### Call health: `CallHealthAggregator`
+
+The **client** axis. Where the resolver links answer "how was *this source* delivered?", this asks
+"how is *each participant* doing, sending vs receiving?":
+
+```ts
+import { CallHealthAggregator } from '@observertc/observer-js';
+
+const health = new CallHealthAggregator(observedCall).aggregate();
+
+health.degradedRatio;             // 0.82 — the number that distinguishes shared faults from individual ones
+health.inboundDegradedRatio;      // receiving side → egress/downstream suspicion
+health.outboundDegradedRatio;     // sending side  → ingress suspicion
+health.rttInMs?.median;           // percentile rollups, never means
+health.qualityLimitation;         // { cpu, bandwidth, other } client counts
+health.clients;                   // per-client entries with `reasons`, direction flags, TURN/TCP
+```
+
+### Registering detectors
+
+**Nothing is created implicitly.** A `new Observer()` has zero detectors. There is no detector
+configuration in `ObserverConfig` and no default set — an application says what it wants to watch, or
+it watches nothing.
+
+```ts
+const observer = new Observer({
+  createRemoteTrackResolver: createDefaultMediasoupRemoteTrackResolverFactory(),
+});
+
+// observer-scoped (cross-call) — built immediately onto `observer.detectors`
+observer.addObserverDetector('observer-concurrent-issue-detector', {
+  issueTypes: [ 'congestion', 'ice-disconnected', 'ice-connection-failed' ],
+  minAffectedCalls: 3,
+});
+observer.addObserverDetector('turn-server-outage-detector', { minClientsAtPeak: 10 });
+
+// call-scoped — recorded in `observer.callDetectorConfigs`, applied to every call created AFTER this
+observer.addCallDetector('call-concurrent-issue-detector', {
+  issueTypes: [ 'congestion', 'ice-disconnected' ],
+});
+observer.removeCallDetector('unconsumed-track-detector');   // undo, for future calls
+
+// one specific call
+observedCall.addDetector('issue-fan-out-detector', { issueTypes: [ 'freezed-video-track' ] });
+```
+
+Detectors are named by their kebab-case `NAME`, and the name types the config — an unknown name or a
+key that belongs to a different detector will not compile. Each detector owns its defaults in its own
+constructor, beside the doc explaining what each threshold means; there is no central table to keep
+in sync.
+
+> **Why no defaults?** A detector nobody asked for is a detector nobody will act on. It costs time on
+> every tick and raises findings into a handler that was not written to expect them. Earlier versions
+> auto-created everything from a three-state config slot; the result was applications receiving
+> finding types they had never heard of.
+
+Every issue-driven detector takes an explicit, non-empty `issueTypes` (or
+`publisherIssueTypes`/`receiverIssueTypes`). There is no "watch everything" option — see
+[the registry](#activeissuesregistry--issues-are-pushed-not-polled).
+
+> **🔗 marks detectors that require a
+> [`RemoteTrackResolver`](#remote-track-resolution-mediasoup--sfu).** They reason about a published
+> track and its subscribers, so without the publisher↔subscriber links they see nothing and stay
+> **silent forever** — which looks exactly like "no problems found". Configure
+> `ObserverConfig.createRemoteTrackResolver`, and start the
+> [`remote-track-resolver` validator](#validators--one-shot-structural-checks) to prove it is wired.
+
+### Built-in detectors
+
+They consume the verdicts `client-monitor-js` >= 4.6.0 already ships (raise + `<type>-resolved`) and
+add only the cross-participant conclusion. **None re-derives a per-endpoint verdict from raw
+counters** — that is the rule the whole design hangs on:
+
+> **If a condition is detectable on the client, the client's issue is the source of truth.**
+
+| Detector | 🔗 | Scope | Raises |
+|----------|:--:|-------|--------|
+| `CallConcurrentIssueDetector` | | call | `CONCURRENT_CLIENT_ISSUES`, `ISSUE_ONSET_BURST` |
+| `ObserverConcurrentIssueDetector` | | **observer** | `CROSS_CALL_CONCURRENT_ISSUES`, `CROSS_CALL_ISSUE_ONSET_BURST` |
+| `IssueFanOutDetector` | 🔗 | call | `PUBLISHED_TRACK_ISSUE_FAN_OUT`, `SINGLE_RECEIVER_ISSUE` |
+| `PublisherFaultCorroborationDetector` | 🔗 | call | `CORROBORATED_PUBLISHER_FAULT` |
+| `TrackDeliveryMismatchDetector` | 🔗 | call | `PUBLISHED_TRACK_NOT_DELIVERED`, `RECEIVER_TRACK_NOT_DELIVERED`, `PUBLISHER_TRACK_DRY` |
+| `UnconsumedTrackDetector` | 🔗 | call | `UNCONSUMED_PUBLISHED_TRACK` |
+| `ClientPopulationIssueDetector` | | **observer** | `CLIENT_POPULATION_ISSUE` |
+| `SfuCongestionDetector` | | **observer** | `sfu-congestion` |
+| `TurnServerHealthDetector` | | **observer** | `TURN_SERVER_DEGRADED` |
+| `TurnServerOutageDetector` | | **observer** | `TURN_SERVER_OUTAGE` |
+
+What each adds that no endpoint can know:
+
+- **`CallConcurrentIssueDetector`** — *who else in this meeting is in this state right now?* The
+  difference between "one person's Wi-Fi" and "this room is broken".
+- **`ObserverConcurrentIssueDetector`** — *is our infrastructure in trouble?* A **separate class**,
+  not the call one with a bigger denominator, because it is a different question with different
+  gates. It requires the group to span at least `minAffectedCalls` **independent calls** (default
+  `2`) and raises its own `CROSS_CALL_*` types. Without that gate, one thirty-person meeting where
+  everyone is congested clears every client threshold and pages you for a single bad room the
+  call-scoped detector already reported. Clients in different calls share no room, no publisher and
+  no host — only the servers, which is what makes the finding conclusive. Note there is deliberately
+  **no participant ratio** at this scope: six broken calls out of forty is a small share of all
+  clients, and a ratio gate would hide exactly the event you want.
+- **`IssueFanOutDetector`** — *does this issue follow one published source, or one receiver?*
+- **`PublisherFaultCorroborationDetector`** — *do **both ends** of one track agree the source is at
+  fault?* Fan-out sees one end and infers; this sees the publisher reporting `encoder-bottleneck`
+  about its own send path *while* its subscribers report `freezed-video-track` about receiving it.
+  Two independent parties, one conclusion, nothing left to deduce — hence the highest confidence in
+  the library. Run both: fan-out is broader and catches the case where the publisher is fine and the
+  SFU's forwarding is not.
+- **`ClientPopulationIssueDetector`** — *is this concentrated on one **kind of client**?* The one
+  correlation here that is neither per-call nor per-server. Every other observer-scoped detector
+  reasons "clients in unrelated calls share only the infrastructure, so it must be us" — right for
+  network symptoms, **wrong for endpoint ones**. `cpulimitation` across six unrelated calls is not an
+  SFU event; CPU is owned by the endpoint, so what those endpoints share is a browser version or a
+  client release. Groups by `browser` / `engine` / `platform` / `operationSystem`, one axis per
+  instance. The gate is **relative risk**, not share: "30% of Chrome 141 is unhappy" means nothing if
+  30% of everyone is, and a share-based rule simply indicts whichever browser is most popular.
+- **`SfuCongestionDetector`** — *is congestion spiking across the fleet right now?* Counts distinct
+  clients reporting congestion in fixed wall-clock buckets and compares each bucket against a
+  median+MAD baseline of the ones before it. Buckets rather than update ticks on purpose: the tick is
+  unevenly spaced and shorter than a client's sampling period, so counting on it compares windows of
+  different lengths and calls the difference a signal. Only add it when the observer's calls all come
+  from the **same SFU** — the finding's meaning is "these clients share only that server".
+- **`TrackDeliveryMismatchDetector`** — *are the two ends of a track disagreeing?*
+- **`UnconsumedTrackDetector`** — *is anyone actually subscribed?* (reads the resolver's silence)
+- **`TurnServerHealthDetector`** — *does trouble cluster on one relay?*
+- **`TurnServerOutageDetector`** — covers the case the health detector structurally cannot. The
+  health detector groups clients by the server relaying them and asks how many report issues — it
+  needs clients *on* the server to ask. When a TURN server dies, allocation fails: existing sessions
+  drop and new clients never obtain a relay candidate through it, so they are never attributed to it
+  at all. Its population goes to zero and the health detector falls silent for the worst possible
+  reason. **Degradation makes clients unhappy; an outage makes them disappear.** Absence is a
+  dangerous signal, so the **control group** is the heart of the design: a call ending, everyone
+  leaving at 6pm, and a fleet-wide network event all look identical to an outage. It refuses to blame
+  a server unless clients *not* relayed through it are demonstrably still connected
+  (`requireControlGroup`, on by default).
+
+#### There is no ICE detector
+
+ICE trouble is reported by `client-monitor-js` >= 4.6.0 as the keyed issues `ice-disconnected`,
+`ice-connection-failed`, `ice-transport-stalled` and `unstable-ice-path`, each with hysteresis and
+multi-signal confirmation behind it. An `IceDisruptionDetector` used to re-derive that server-side
+from raw state transitions; it has been removed, because the server sees less and guesses more. The
+client knows whether `disconnected` persisted or healed in 200 ms; the observer does not.
+
+Correlating ICE trouble is now configuration, not a class:
+
+```ts
+observer.addObserverDetector('observer-concurrent-issue-detector', {
+  issueTypes: [ 'ice-disconnected', 'ice-connection-failed', 'ice-transport-stalled' ],
+});
+```
+
+### Validators — one-shot structural checks
+
+Every detector above answers *"is something wrong right now?"* and runs on every tick, because the
+answer legitimately changes. A **validator** answers *"is this deployment built correctly?"* — which
+only changes when you deploy. So it is not configured on and left running: you **start** one, it runs
+until it can decide, reports once, and the observer drops it.
 
 ```ts
-interface Detector { readonly name: string; update(): void; }
-
-class Detectors {
-  add(d: Detector): void;
-  remove(d: Detector): void;
-  clear(): void;
-  update(): void;          // called by ObservedCall.update(); guards each detector in try/catch
-  get listOfNames(): string[];
+observer.addValidator('simulcast-receivers', { minChecks: 5 });
+
+observer.on('validation-ready', ({ validator, report }) => {
+  if (!report.ready) return;
+  if (report.verdict === 'layer-decided-lowest-common-denominator') page(validator, report);
+});
+
+onDeploy(() => observer.addValidator('simulcast-receivers'));   // check again
+```
+
+`observer.validators` is the set currently running — normally empty, since each removes itself on
+finishing. There is no revalidation timer: a deploy, not elapsed time, is what makes a structural
+verdict stale, so re-checking means starting another.
+
+| Validator | `addValidator` name | Question | Also raises |
+|-----------|---------------------|----------|-------------|
+| `SimulcastReceiverValidator` 🔗 | `simulcast-receivers` | Does the SFU pick layers per receiver, or drag the publisher down to the worst one? | `WORST_RECEIVER_CONTAGION` |
+| `RemoteTrackResolverValidator` | `remote-track-resolver` | Is the resolver actually linking anything? | `REMOTE_TRACK_LINKS_UNRESOLVED` |
+| `CodecConsistencyValidator` | `codec-consistency` | Is everyone on the same codec — and is it the one you think you negotiated? | `CODEC_INCONSISTENCY` |
+
+**`SimulcastReceiverValidator`** — simulcast (or SVC) exists so one slow
+participant doesn't set everyone's quality: with several encodings the server hands the struggling
+receiver a lower layer and leaves the rest alone. Without it — or with a server that relays RTCP end
+to end, so the publisher's bandwidth estimate collapses to the slowest receiver — the only way to
+serve them is to make the *source* send less. Both causes look identical from outside; what the check
+establishes is whether per-receiver adaptation happens at all.
+
+| `verdict` | meaning |
+|-----------|---------|
+| `layer-decided-per-receiver` | verified — a receiver fell far behind and the publisher carried on |
+| `layer-decided-lowest-common-denominator` | the publisher followed its worst receiver; everyone gets the slowest participant's quality |
+| `inconclusive` | cancelled, or the observer closed, before it could decide |
+
+**Not finishing is not a pass.** The check only runs when a publisher has 3+ receivers and one is at
+most half the median; plenty of healthy deployments never present that. A validator that never sees it
+simply keeps running and never reports — it does not quietly succeed. `report.checks` counts the times
+the check genuinely ran, so an `inconclusive` with `checks: 0` says plainly that nothing was verified.
+
+**`RemoteTrackResolverValidator`** exists because of a specific, nasty failure mode. Four things here
+are built on publisher↔subscriber links — `IssueFanOutDetector`,
+`PublisherFaultCorroborationDetector`, `TrackDeliveryMismatchDetector`, `UnconsumedTrackDetector`
+(and `SimulcastReceiverValidator`) — and every one of them correctly does *nothing* when the links
+are missing rather than guessing. So a resolver wired to the wrong id field leaves all of them
+permanently silent, and **silence is what a healthy deployment looks like too**: you would conclude
+your calls were clean when in fact nothing was ever examined. Verdicts: `links-resolved` /
+`no-links-resolved` / `inconclusive`. Run it at start-up and after changing the resolver or the SFU's
+id scheme.
+
+**`CodecConsistencyValidator`** answers two things at once. A **split** — participants of one call on
+different codecs — is a real fault with a confusing symptom: an SFU that forwards without transcoding
+cannot serve them all, so some pairs see each other and some do not, with no error anywhere. Only
+something holding every participant at once can see it. The quieter half is the silent fallback: a
+deployment configured for VP9 or AV1 drops to VP8 whenever one endpoint cannot negotiate the
+preference, the call keeps working at a higher bitrate than budgeted, and the team believes it
+shipped AV1 months ago. Give it `expected` and it says so. Verdicts: `codec-consistent` /
+`codec-split` / `unexpected-codec` / `inconclusive`.
+
+```ts
+observer.addValidator('remote-track-resolver');
+observer.addValidator('codec-consistency', { expected: { video: 'video/VP9', audio: 'audio/opus' } });
+```
+
+#### Conclusions
+
+Every issue-driven finding carries a `conclusion` in its payload — the interpretation step, so the
+person reading the alert doesn't have to perform it:
+
+```jsonc
+{
+  "type": "CROSS_CALL_ISSUE_ONSET_BURST",
+  "issueType": "congestion",
+  "calls": 40, "affectedCalls": 6,
+  "perCall": [ { "callId": "…", "affectedClients": 4, "totalClients": 9 } ],
+  "conclusion": {
+    "faultDomain": "infrastructure",
+    "summary": "network congestion is open across independent calls at the same time — 6 of 40 calls (11/300 clients)",
+    "recommendation": "check SFU egress bandwidth and host network saturation before looking at any single participant",
+    "confidence": 0.85
+  }
 }
 ```
 
----
+`faultDomain` is one of `infrastructure`, `call`, `published-track`, `endpoint`, `client-population`
+or `unknown`, and it comes from the **spread**, not the issue type — congestion in one call is a
+meeting problem, congestion in six calls is a server problem, and the client reported the identical
+symptom in both.
+
+One case is worth knowing about because it inverts the usual reading: **`cpu-limitation` spread
+across many independent calls concludes `client-population`, not `infrastructure`.** Endpoint CPU is
+owned by the endpoint, so breadth there points at what those endpoints share — a recent client
+release, a browser version, shared VDI hardware — and paging the SFU on-call would be wrong. The
+conclusion table encodes that so nobody has to rediscover it during an incident.
+
+Unknown issue types (your own custom client detectors) still produce a structurally valid conclusion
+from the spread alone; they just get generic wording.
+
+Two functions are exported, one per scope: `concludeCallIssue()` and `concludeObserverIssue()`. They
+are separate because a detector already knows its scope, and a single generic function forced every
+caller to pass the other scope's fields as placeholders — call-scoped detectors passing
+`affectedCalls: 1, totalCalls: 1` forever, observer-scoped ones passing a participant ratio that was
+deliberately never read. Placeholders like that invite being read as if they meant something.
+
+#### Cost
+
+Detectors run inside `call.update()`, on your event loop, so their cost matters. Two things keep it
+off the participant axis:
+
+- **Issues are pushed, not polled.** A detector holds only what the registry handed it, so an
+  `update()` that finds `size === 0` — the overwhelmingly common case — costs one comparison,
+  whatever the participant count. Nothing iterates clients looking for trouble.
+- **So are unconsumed tracks.** `observedCall.unconsumedOutboundTracks` is maintained by the resolver
+  as tracks gain and lose subscribers, so `UnconsumedTrackDetector` reads a set that is normally
+  empty instead of walking every published track (529 µs → 65 µs per tick at 1 200 tracks).
+- **Track lookups start from the affected minority.** A detector resolving an issue to its published
+  track searches the *reporting client's* peer connections (typically one or two), not the call.
+
+At 20 calls × 12 participants (2 640 subscriptions) the whole detector pass costs ~1.3 ms per tick.
+`yarn bench` prints a per-detector breakdown for your own shape.
+
+#### Worked examples
+
+[`examples/detectors.ts`](./examples/detectors.ts) (`yarn example:detectors`) runs one scenario per
+detector — the question it answers, its full config, the synthetic traffic that makes it fire, and
+the finding with its conclusion. It asserts every expected finding is produced, so it doubles as a
+smoke test. [`examples/sfu-observer.ts`](./examples/sfu-observer.ts) (`yarn example`) is the end-to-end
+tour instead: ingest → correlate → react, with the mediasoup wiring alongside.
+
+#### `TrackDeliveryMismatchDetector` — resolving an ambiguous symptom
+
+A dry track ("no bytes are arriving") is the clearest symptom there is and, on its own, completely
+ambiguous. A receiver seeing silence cannot distinguish *the camera was switched off* from *the SFU
+stopped forwarding* from *my own consumer wedged* — all three look identical from the browser.
+
+Joining the two ends of the published track resolves it:
+
+| publisher | subscribers | verdict |
+|---|---|---|
+| sending | **all** dry | `PUBLISHED_TRACK_NOT_DELIVERED` — the forwarding path |
+| sending | **some** dry | `RECEIVER_TRACK_NOT_DELIVERED` — those consumers (in mediasoup: recreate them) |
+| dry | any dry | `PUBLISHER_TRACK_DRY` — the source stopped; **not** an SFU fault |
+
+The publisher side is judged from both available signals: its own `dry-outbound-track` issue when the
+client reports one, and the observed outbound RTP (`deltaPacketsSent`) as fallback and corroboration.
+That combination is what makes the first row trustworthy — the server can state that packets
+demonstrably left the publisher during the same interval in which every receiver got nothing.
+
+This is the "SFU forwarding mismatch" check, and it needs **no mediasoup instrumentation at all** —
+the clients' own dry-track verdicts plus the resolver links are sufficient.
+
+#### `UnconsumedTrackDetector` — reading the resolver's silence
+
+The one detector where the *absence* of links is the signal: a track still pushing packets whose
+`remoteInboundTracks` set is empty, i.e. uplink and SFU ingress spent on media nobody receives
+(everyone has the publisher hidden, a simulcast layer no viewer selects, or an app that forgot to
+stop a track). It waits `minUnconsumedDurationInMs` first, since a gap between publishing and the
+first subscription is normal at join time.
+
+Note the trap this one has to guard against, and why it checks `call.remoteTrackResolver` at runtime
+rather than trusting the flag alone: **"no subscribers" and "no resolver configured" produce the
+identical observation.** Without a resolver it would report every published track in the call as
+unconsumed.
 
 ## Remote track resolution (mediasoup / SFU)
 
@@ -836,6 +1331,80 @@ on your own cadence read `observedRouter.sample` (snapshot/serialize/persist wha
 you don't, and close routers you no longer track. (mediasoup also typically shards routers across
 workers/cores, which keeps any one router small.)
 
+### Extending the sample, and building your own report
+
+The sample is yours to annotate. Every entity — the router, each transport, producer, consumer, data
+producer and data consumer — has an `attachments?: Record<string, unknown>` slot, and there are three
+ways to fill it, from most declarative to most ad-hoc.
+
+**1. `enrich` — mirror mediasoup's own `appData`.** The common case: your application already keeps
+`participantId`, `purpose` and similar on the mediasoup objects, and you want them on the sample.
+Runs once per entity at creation, before the corresponding event:
+
+```ts
+observer.createObservedMediasoupRouter({
+  router,
+  enrich: {
+    producer: (producer) => ({ participantId: producer.appData.participantId, purpose: producer.appData.purpose }),
+    consumer: (consumer) => ({ subscriberId: consumer.appData.subscriberId }),
+    transport: (transport) => ({ role: transport.appData.role }),
+  },
+});
+```
+
+A throwing enricher is caught and logged — it can't take the router's bookkeeping down with it.
+
+**2. Lifecycle events — enrich on the fly.** Each entity announces itself as
+`<entity>-sample-added` and `<entity>-sample-closed`, carrying **the live sample object** (not a
+copy) plus the mediasoup object it came from. Mutating it in the handler is the intended pattern:
+
+```ts
+observedRouter.on('producer-sample-added', ({ sample, producer, transport }) => {
+  sample.attachments = { ...sample.attachments, participantId: lookup(producer.id) };
+});
+
+observedRouter.on('producer-sample-closed', ({ sample }) => {
+  archive(sample);   // its `closedAt` is set
+});
+```
+
+Events: `transport-sample-added` / `-closed`, `producer-sample-added` / `-closed`,
+`consumer-sample-added` / `-closed`, `data-producer-sample-added` / `-closed`,
+`data-consumer-sample-added` / `-closed`.
+
+**3. `attachTo(id, attachments)` — annotate later, from anywhere.** When the knowledge arrives after
+the entity did (a signalling message, a database lookup that resolved):
+
+```ts
+observedRouter.attachTo(producerId, { participantId, joinedFrom: 'mobile' });   // merges
+```
+
+Ids are unique across mediasoup entity kinds, so one method covers all of them. It returns `false`
+for an unknown id rather than failing quietly — which matters when application events race the
+mediasoup ones. For direct access there are typed accessors: `getTransportSample(id)`,
+`getProducerSample(id)`, `getConsumerSample(id)`, `getDataProducerSample(id)`,
+`getDataConsumerSample(id)`. They index the *same* objects the arrays hold, so a lookup is O(1)
+instead of a `sample.producers.find(...)` scan.
+
+#### Building your own report
+
+`observedRouter.sample` is live — arrays grow and `history` entries are appended as the router runs,
+so a report built directly on it keeps changing after you think you're done. Use **`snapshot()`** for
+a detached deep copy:
+
+```ts
+const report = {
+  ...observedRouter.snapshot(),        // never moves again
+  generatedAt: Date.now(),
+  region: process.env.REGION,
+};
+```
+
+> **Note on typing.** The sample types no longer carry a `Record<string, unknown>` index signature.
+> That signature allowed arbitrary top-level keys but also silently accepted typos on real fields and
+> weakened autocomplete. Custom data belongs in `attachments`, which is typed as such. If you were
+> assigning ad-hoc keys directly onto a sample object, move them into `attachments`.
+
 ### Matching peer connections — by **event**, not by storage
 
 The observer correlates the SFU side with the client side **at the peer-connection level**: a
@@ -1151,6 +1720,16 @@ filtering, per-module routing, and full silencing.
 
 ---
 
+## Design notes
+
+**[`docs/design-notes.md`](./docs/design-notes.md)** covers the reasoning behind the library rather
+than its API: why client-detectable conditions are never re-derived server-side, why each shipped
+detector exists, what was deliberately *not* built and why, the WebRTC domain facts that shaped the
+implementation (ICE-Lite disconnect waves, counter resets, why ICE and RTCP RTT must never be
+blended), and an operational threshold reference.
+
+---
+
 ## Error-handling philosophy
 
 The library **warns and degrades; it does not throw** on operational problems:
@@ -1184,10 +1763,13 @@ lint + typecheck + **build** + test on every push/PR.
 
 **Project layout** (`src/`): `Observer.ts`, `ObservedCall.ts`, `ObservedClient.ts`,
 `ObservedPeerConnection.ts`, the `Observed*` sub-stat classes, `ObserverEvents.ts` (the typed
-event map + scope types), `detectors/` (`Detector`, `Detectors`), `scores/`, `updaters/`
-(update-policy strategies), `utils/` (remote-track resolvers), `common/` (`logger`, `utils`,
-`Middleware`), `schema/` (sample/event/meta types), and `sinks/` (the `ClientSampleSink` base +
-`JsonlFileSink` / `InMemorySink`, re-exported from the package root).
+event map + scope types), `detectors/` (`Detector`, `Detectors`, and one file per detector),
+`validators/` (`Validator`, `Validators`, one file per validator), `issues/` (`ActiveClientIssue`,
+`ActiveIssueTracker`, `ActiveIssuesRegistry`, `ObservedClientIssueRegistry`), `scores/`,
+`resolvers/` (remote-track resolvers), `utils/` (`stats`, `SlidingWindow`, `TrendTester`,
+`CallHealthAggregator`), `common/` (`logger`, `utils`, `Middleware`), `schema/`
+(sample/event/meta types), and `sinks/` (the `ClientSampleSink` base + `JsonlFileSink` /
+`InMemorySink`, re-exported from the package root).
 
 **Conventions to follow when developing further:**