@smthrs/observability 0.35.0 → 1.0.0-rc.3

package/docs/concepts/instrumentation-and-export.md

@@ -0,0 +1,68 @@
+---
+title: "Instrumentation and export"
+description: "Why the Smithers packages instrument themselves but export nothing, what that split buys, and the three independent ways to read the same run."
+sidebar:
+  order: 1
+---
+
+Instrumentation and export are two different jobs, and Smithers puts them in
+different packages on purpose.
+
+## Every other package is already instrumented
+
+A flow execution opens spans without being asked. The flow lifecycle, the
+engine dispatch, a run claim, a heartbeat, a journal write, down to individual
+`sql.execute` statements: each is a span on Effect's own tracer. Alongside
+them, each store package declares and updates its own metric handles on its hot
+paths: `JournalMetrics` in [`@smthrs/journal`](/api/journal), `RunStoreMetrics`
+in [`@smthrs/run-store`](/api/run-store), `CacheStoreMetrics` in
+[`@smthrs/step-cache`](/api/step-cache), `ArtifactStoreMetrics` in
+[`@smthrs/artifacts`](/api/artifacts), and `DatabaseMetrics` in
+[`@smthrs/database`](/api/database).
+
+None of those packages allocates an exporter. Effect's tracer and metric
+registry are collection points; with no exporter installed they collect and
+discard, at a cost close to zero.
+
+## This package is the exporter
+
+`Otlp.layerFetch` installs Effect's OTLP logger, metrics exporter, and tracer
+against a collector endpoint, with a service identity filled in. That is the
+whole wiring, and it is why telemetry is additive rather than invasive:
+providing one layer turns on delivery for instrumentation that was already
+there, and removing the line turns it off without touching a flow body.
+
+The split also keeps the dependency in one place. Because `Otlp` composes only
+what `effect` already ships, a package that wants a counter takes on no
+OpenTelemetry dependency to get one, and the process pays for an SDK only if a
+host chooses `NodeOtel` or `BrowserOtel`.
+
+## The producer owns the meaning, the exporter owns the delivery
+
+A metric handle belongs to the package that knows when to advance it. This
+package holds only the four cross-package runtime signals nobody else owns,
+and it holds them so the producers can share one identifier:
+`flows_run_throughput` is updated in [`@smthrs/run-store`](/api/run-store),
+`flows_seat_active` and `flows_quota_park` in [`@smthrs/agent`](/api/agent),
+and `flows_observability_log_dropped` here. Step-cache counters stay in
+[`@smthrs/step-cache`](/api/step-cache); this package does not duplicate them.
+
+The consequence for a reader: a metric's name tells you which package to open
+to learn what advances it, and this package tells you only how it leaves the
+process.
+
+## Three ways to read one run
+
+Export is one read path, not the only one, and the other two need no collector:
+
+1. **The OTLP trace.** Every span and series posts to your collector. Filter on
+   the `service.name` you passed to the layer.
+2. **The durable journal.** `Journal.entries` reads a run's lifecycle events in
+   process, and `JournalLogger` puts the run's log records on that same
+   history. See [Forward logs to the run journal](../guides/forward-logs-to-the-journal.md).
+3. **The metric handles.** Effect's `Metric.value` reads a counter in process,
+   with no exporter installed at all. See
+   [Read the runtime metrics](../guides/read-runtime-metrics.md).
+
+The three tell the same story about the same run, which is what makes a
+disagreement between them worth investigating.

package/docs/concepts/layer-map.md

@@ -0,0 +1,72 @@
+---
+title: "The layer map"
+description: "The five telemetry layer builders this package ships, what each one allocates, and the rule that keeps the root entry point free of Node modules."
+sidebar:
+  order: 2
+---
+
+Five builders produce a telemetry layer, and they differ in one thing: how much
+they allocate for you.
+
+## The five builders
+
+| Builder                 | Allocates                                                                  | Reach for it when                                                                 |
+| ----------------------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
+| `Otlp.layerFetch`       | Effect's three OTLP exporters, plus the host's global `fetch`              | You have a collector and no other OpenTelemetry requirement. This is the default. |
+| `Otlp.layer`            | Effect's three OTLP exporters. You provide the `HttpClient`                | You need a specific HTTP client, such as Undici on Node.                          |
+| `NodeOtel.layerOtel`    | The Node OpenTelemetry SDK: three OTLP/HTTP exporters and their processors | Node code that must run through the OpenTelemetry SDK.                            |
+| `BrowserOtel.layerOtel` | The web OpenTelemetry SDK around processors and readers you construct      | Browser code that must run through the OpenTelemetry SDK.                         |
+| `Otel.layerOtel`        | Effect bridges; optional caller factories build providers                  | The application controls SDK provider construction.                               |
+
+Each has an explicit do-nothing counterpart: `Otlp.layerNoop` and
+`Otel.layerNoop` both provide nothing and export nothing, so wiring code
+switches layers instead of branching around a missing one.
+
+## Why `Otlp` is the default
+
+`Otlp` composes only `effect/unstable/observability/Otlp`, so it adds no
+OpenTelemetry SDK to the process, and it delivers over Effect's `HttpClient`
+rather than a platform transport. That makes it the cheapest wiring and the
+only one that runs unchanged in Node, in Bun, and in a browser.
+
+Reach past it when something other than a collector endpoint is in play: an
+existing OpenTelemetry instrumentation you must feed, a vendor exporter that is
+not OTLP/HTTP JSON, or a provider a framework hands you.
+
+## Why the root entry point stops at `Otel`
+
+The root entry point exports `Otlp`, `Endpoint`, `JournalLogger`, `Logger`,
+`Metric`, `Otel`, and `Resource`. It stops there because those seven resolve
+nothing host-specific.
+
+`NodeOtel` does. It reaches `@effect/opentelemetry/NodeSdk` and
+`@opentelemetry/sdk-trace-node`, and the latter pulls
+`@opentelemetry/context-async-hooks`, which imports the bare specifier
+`async_hooks`. A browser bundler cannot resolve that, so re-exporting `NodeOtel`
+from the root would break the root's browser bundle for every consumer,
+including those that never wanted an SDK. `BrowserOtel` is kept beside it for
+symmetry: a module that binds one host belongs on a subpath.
+
+Nothing the root entry point reaches resolves a `node:` built-in, as an ESM
+import or as a `require("node:...")` call, so a browser bundle of the root
+carries no host module.
+
+## `Resource` sits under all of them
+
+Every builder decodes the same `Resource.Configuration`. `Otlp`, `NodeOtel`,
+and `BrowserOtel` attach its attributes to all enabled signals. `Otel.layerOtel`
+passes that validated resource to provider factories and the metric producer;
+factories must use it when constructing tracer and logger providers.
+
+Already-created providers retain their own resources. For those providers,
+`Otel.layerOtel`'s resource option sets only the metric resource and tracer
+instrumentation scope. Construct them with the intended resource before
+injection to give all three signals the same service identity.
+
+`Resource.layer` provides the resource on its own, for a composition that
+assembles the rest itself.
+
+`Otlp` is the one builder with defaults: `Otlp.defaultServiceName` is `flows`
+and `Otlp.defaultServiceVersion` is this package's release version. Name your
+own service in production, so a collector holding several Smithers processes
+can tell them apart.

package/docs/concepts/validated-acquisition.md

@@ -0,0 +1,105 @@
+---
+title: "Validation at layer acquisition"
+description: "Export failure is absorbed by design, so a misconfigured exporter looks exactly like a working one. Endpoints and resources are therefore decoded when the layer is built, and refused with typed errors."
+sidebar:
+  order: 3
+---
+
+An exporter that cannot deliver must not take the application down with it. A
+collector outage is not an application outage, so Effect's exporter absorbs
+export failure: it retries a transient failure three times, then temporarily
+disables delivery, and the effect that produced the span never learns any of
+it.
+
+That is the right behavior, and it creates one hazard. A layer built against
+`""`, `localhost:4318`, or an endpoint with a trailing newline behaves
+identically to a working one: it builds, it runs, it fails nothing, and it
+delivers nothing forever. Nobody notices until an operator goes looking for a
+trace that was never sent.
+
+## Acquisition is the only place to report it
+
+So both configuration inputs are decoded while the layer is being built, before
+any exporter exists:
+
+- The collector endpoint, by `Endpoint.decode`, failing with
+  `Endpoint.InvalidExporterEndpoint`.
+- The service identity and resource attributes, by `Resource.decode`, failing
+  with `Resource.InvalidResourceConfiguration`.
+
+Both are typed failures in the layer's error channel, which means a composition
+sees them at startup, in the place where the value was written, rather than as
+silence in a dashboard weeks later. `Otlp.layer`, `Otlp.layerFetch`, and
+`NodeOtel.layerOtel` all decode their endpoint; every builder decodes its
+resource.
+
+## What an endpoint must be
+
+An absolute `http://` or `https://` URL, at most 2,048 characters, carrying no
+username, password, query, fragment, backslashes, spaces, or control characters.
+Base paths such as `https://collector.example/tenant/9` are supported.
+
+Queries and fragments are refused because exporters append `/v1/<signal>` to
+the base. Appending it after `?tenant=9` or `#notes` would leave the HTTP request
+path unchanged, silently sending telemetry to the wrong route. Use exporter
+headers for authentication, not credentials embedded in the URL.
+
+The last clause is the one that surprises people, and it is the whole reason
+`Endpoint` exists rather than a bare `new URL` check. The WHATWG URL parser is a
+repairer, not a validator: it strips leading and trailing spaces and C0
+controls, and removes tab, newline, and carriage return from anywhere in its
+input. So `new URL("http://collector:4318\n")` succeeds, and reports a
+well-formed URL, while the untrimmed original is what a builder would hand its
+exporter. An endpoint read from a config file with a trailing newline, or
+pasted with a leading space, is the most common shape of this mistake, and it
+produces exactly the silent no-delivery the refusal exists to prevent. No legal
+collector endpoint carries one of those characters, so refusing them costs
+nothing.
+
+Repeated trailing separators are normalized away instead of refused:
+`http://host//` and `http://host` both post to `http://host/v1/traces`.
+
+## What a resource must be
+
+`Resource.Configuration` bounds the identity so an accepted value cannot become
+an unencodable payload later:
+
+- `serviceName` and `serviceVersion` are non-empty, well formed, and at most
+  1,024 UTF-16 code units.
+- At most 256 attributes, with non-empty, well formed keys of at most 1,024
+  code units.
+- Attribute values are finite numbers, booleans, strings of at most 65,536 code
+  units, or homogeneous arrays of one of those scalar types with at most 256
+  elements (`Resource.maximumAttributeArrayLength`).
+- The complete OTLP JSON resource is at most 128 KiB
+  (`Resource.maximumResourceBytes`), including identity, SDK-added
+  `telemetry.sdk.name` and `telemetry.sdk.language`, keys, values, array wrappers,
+  escaping and UTF-8. SDK fields are reserved even for the default Effect
+  exporter. Counting stops as soon as the budget is exceeded;
+  the value-free refusal names `attributes`, the measured lower bound and limit.
+  Per-field limits alone cannot guarantee this cumulative bound.
+- Resource metadata is explicit. The default OTLP and Node SDK layers isolate
+  ambient resource configuration so it cannot enlarge metadata after admission.
+- The 1 MiB transport reserves 8 KiB for the JSON envelope and escaped scope
+  name, leaving 888 KiB (`Otlp.reservedBatchBytes`) for a signal batch. This is
+  about 909 bytes per record at the upstream 1,000-record log or span limit,
+  enough for ordinary batches; application records can still be too large.
+- NUL and unpaired UTF-16 surrogates are refused anywhere. Valid astral Unicode
+  is preserved.
+
+An empty `serviceName` is refused rather than defaulted, because anonymous
+telemetry in a shared collector is worse than none.
+
+## Refusals never carry the rejected value
+
+Both errors report a stable `code`, the `path` of the option the bad value
+arrived on, and a message. Neither retains the value itself: `baseUrl` is the
+field most likely to carry a token in a query string or a credential in its
+userinfo, and an error message is the least controlled surface in a system. The
+`path` is what you need to find the value, and you already have the value.
+
+The `path` names the caller's own option, so a refusal from `Otlp` says
+`baseUrl` and one from `NodeOtel` says `endpoint`.
+
+For what to do when you see one, see
+[Troubleshooting](../troubleshooting.md).

package/docs/guides/export-to-a-collector.md

@@ -0,0 +1,167 @@
+---
+title: "Export to an OTLP collector"
+description: "Wire Otlp.layerFetch into a running program: the collector endpoint, the service identity, authentication headers, export cadence, the shutdown flush, and what happens when the collector is down."
+sidebar:
+  order: 1
+---
+
+`Otlp` posts logs, metrics, and traces to an OTLP/HTTP collector as JSON. This
+guide covers the options you set once at startup and the behavior you should
+expect from them afterwards.
+
+## Provide the layer
+
+```ts
+import * as Otlp from "@smthrs/observability/Otlp"
+import * as Effect from "effect/Effect"
+
+const telemetry = Otlp.layerFetch({
+  baseUrl: "http://localhost:4318",
+  serviceName: "deploy-status"
+})
+
+// `program` is your application effect, unchanged from its non-telemetry form.
+const outcome = program.pipe(Effect.provide(telemetry), Effect.scoped)
+```
+
+The layer is scoped: give it a lifetime with `Effect.scoped`, or build it as
+part of a runtime that owns a scope. Signals post below `baseUrl` at
+`/v1/logs`, `/v1/metrics`, and `/v1/traces`, so a base URL with a path prefix
+is honored: `http://collector:4318/tenant/9` posts to
+`http://collector:4318/tenant/9/v1/traces`.
+
+`Otlp.layer` is the same wiring without the `fetch` binding. Use it when the
+host must supply its own HTTP client:
+
+```ts
+import * as NodeHost from "@smthrs/platform-node/NodeHost"
+import * as Layer from "effect/Layer"
+
+const telemetry = Otlp.layer({ baseUrl: "http://localhost:4318" }).pipe(
+  Layer.provide(NodeHost.NodeHttpClient.layerUndici)
+)
+```
+
+## Name the service
+
+`serviceName` and `serviceVersion` become the `service.name` and
+`service.version` resource attributes on every exported signal, and they are
+how a collector holding several processes tells them apart. Omitted, they fall
+back to `Otlp.defaultServiceName` (`flows`) and `Otlp.defaultServiceVersion`
+(this package's release version), which is a reasonable default for a Smithers
+CLI process and a poor one for your application.
+
+Add deployment facts as `attributes`:
+
+```ts
+const telemetry = Otlp.layerFetch({
+  baseUrl: "https://otlp.example.com",
+  serviceName: "deploy-status",
+  serviceVersion: "2.4.1",
+  attributes: {
+    "deployment.environment.name": "production",
+    "service.instance.id": process.env.HOSTNAME ?? "local"
+  }
+})
+```
+
+Attribute values are strings, finite numbers, booleans, or homogeneous arrays
+of those. The full bounds are in
+[Validation at layer acquisition](../concepts/validated-acquisition.md).
+
+## Authenticate
+
+`headers` is sent with every export request, which is where a vendor token
+goes:
+
+```ts
+const telemetry = Otlp.layerFetch({
+  baseUrl: "https://otlp.example.com",
+  serviceName: "deploy-status",
+  headers: { authorization: "Bearer YOUR_TOKEN" }
+})
+```
+
+Read the token from the environment or your secret store. A credential in the
+URL is refused: an endpoint carrying userinfo fails layer acquisition. A
+credential header (`authorization`, `x-api-key`, `x-honeycomb-team`, and
+similar names) over plaintext `http://` is refused too, unless the collector is
+on `localhost`, `127.0.0.0/8`, or `[::1]`. Non-credential headers such as a
+tenant id work on any endpoint. The check reads header names only, so use
+`https://` for every remote collector.
+
+Fetch redirects are refused, including same-origin redirects. Configure the
+collector's final URL. Other HTTP transports must refuse redirects themselves.
+For fetch options such as `credentials: "omit"`, provide
+`FetchHttpClient.RequestInit` around the whole exporting effect:
+
+```ts
+import * as FetchHttpClient from "effect/unstable/http/FetchHttpClient"
+
+const outcome = program.pipe(
+  Effect.provide(telemetry),
+  Effect.provideService(FetchHttpClient.RequestInit, { credentials: "omit" }),
+  Effect.scoped
+)
+```
+
+Options supplied only to the inner HTTP-client layer are replaced by the
+redirect policy. Ambient options are retained except `redirect`, which is
+always `"manual"`. Redirect responses fail export through the existing failure
+handling below; they never forward credentials or signal bodies.
+
+## Control the cadence and the flush
+
+`exportInterval` sets how often all three signals batch and post. Omitted, each
+signal keeps Effect's own default. `shutdownTimeout` bounds the final flush
+when the layer's scope closes.
+
+```ts
+const telemetry = Otlp.layerFetch({
+  baseUrl: "http://localhost:4318",
+  serviceName: "deploy-status",
+  exportInterval: "10 seconds",
+  shutdownTimeout: "5 seconds"
+})
+```
+
+Closing the scope is the deterministic flush. A short program may deliver
+nothing on the interval and everything at shutdown, so read what a collector
+received only after the scope has closed.
+
+## Turn export off explicitly
+
+`Otlp.layerNoop` provides nothing and exports nothing. It exists so a host that
+has no collector, or a browser build that has not opted in, switches layers
+rather than branching around a missing provide:
+
+```ts
+const telemetry = collectorUrl === undefined
+  ? Otlp.layerNoop
+  : Otlp.layerFetch({ baseUrl: collectorUrl, serviceName: "deploy-status" })
+```
+
+## What happens when the collector is down
+
+Export failure never fails your program, and it never surfaces as a rejected
+promise. Effect's exporter retries a transient failure three times, honoring a
+`retry-after` response header when one is present, then disables that exporter
+for 60 seconds and logs a `Debug` record saying so. Spans and records produced
+inside that window are dropped, and delivery resumes afterwards.
+
+Two consequences worth planning for:
+
+- A dead collector is invisible at `Info`. Run at `Debug` while you are
+  bringing an exporter up, or watch the collector side.
+- A configuration mistake and a collector outage look the same after
+  acquisition. That is why the endpoint is decoded before any exporter exists;
+  see [Troubleshooting](../troubleshooting.md) when nothing arrives.
+
+## Next steps
+
+- [Test telemetry without a collector](./testing.md): assert on what would have
+  been posted.
+- [Wire an OpenTelemetry SDK](./wire-an-opentelemetry-sdk.md): when a collector
+  endpoint is not enough.
+- [Read the runtime metrics](./read-runtime-metrics.md): what the exported
+  series mean.

package/docs/guides/forward-logs-to-the-journal.md

@@ -0,0 +1,226 @@
+---
+title: "Forward logs to the run journal"
+description: "Install JournalLogger.layerJournalForwarding so a run's Effect log records become durable telemetry.log entries, and read them back: capacity, bounds, redaction, observed losses, and admission limits."
+sidebar:
+  order: 3
+---
+
+`JournalLogger` turns a run's Effect log records into durable, structured
+`telemetry.log` entries on that run's journal. It is the second delivery path
+beside [OTLP export](./export-to-a-collector.md), and it is the one an operator
+reads without a collector: the records land next to the run's lifecycle events
+and are read back with the same `Journal.entries` call.
+
+## Install the layer
+
+The layer names one run, and requires a `Journal.Journal` service from
+[`@smthrs/journal`](/api/journal):
+
+```ts
+import * as JournalEvent from "@smthrs/journal/JournalEvent"
+import * as JournalLogger from "@smthrs/observability/JournalLogger"
+import * as Schema from "effect/Schema"
+
+const runId = Schema.decodeUnknownSync(JournalEvent.RunId)("deploy-status-1")
+
+const journalLogs = JournalLogger.layerJournalForwarding({ runId })
+```
+
+`runId` is branded, so it comes from decoding a string rather than from a bare
+literal. Decoding it here means a malformed id is reported where it was
+written.
+
+Provide the layer over your journal. In a test, the deterministic bundle is
+enough:
+
+```ts
+import * as TestJournal from "@smthrs/journal/test/TestJournal"
+import * as Layer from "effect/Layer"
+
+const layer = Layer.provideMerge(journalLogs, TestJournal.layer())
+```
+
+In production, the journal is the one your durable engine already composed;
+provide the forwarder over it the same way.
+
+## Read the records back
+
+Each record is one journal entry with `eventType` `telemetry.log`, written on
+the journal's lossy channel by source `flows/observability/logger`. The payload
+decodes with the exported schema:
+
+```ts
+import * as Journal from "@smthrs/journal/Journal"
+import * as Effect from "effect/Effect"
+
+const read = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  const page = yield* journal.entries({ runId, limit: 200 })
+  return page.entries
+    .filter((entry) => entry.eventType === "telemetry.log")
+    .map((entry) => Schema.decodeUnknownSync(JournalLogger.TelemetryLog)(entry.payload))
+})
+```
+
+A decoded record carries the level, the logged message, the log annotations,
+the fiber id, an ISO-8601 timestamp, the active span's `traceId` and `spanId`
+when there was one, and a structural `cause` that preserves the order and the
+kind of every failure reason:
+
+```text
+{
+  version: 1,
+  level: "Error",
+  message: [ "deploy failed" ],
+  annotations: { lane: "api", attempt: 2 },
+  cause: {
+    version: 1,
+    reasons: [
+      { _tag: "Fail", error: { code: "timeout" } },
+      { _tag: "Interrupt", fiberId: 7 }
+    ]
+  },
+  fiberId: 12,
+  traceId: "0af7651916cd43dd8448eb211c80319c",
+  spanId: "b7ad6b7169203331",
+  timestamp: "2026-09-03T12:00:00.000Z"
+}
+```
+
+The durable journal allocates the per-source sequence, so restarting the layer
+or running two of them for one run cannot make two records collide on one
+identity.
+
+## Choose the queue depth
+
+The logger callback never blocks. It snapshots the record synchronously when the
+bounded queue has room, then offers it to that queue, which a forked worker
+drains into the journal.
+
+```ts
+const journalLogs = JournalLogger.layerJournalForwarding({
+  runId,
+  capacity: 4096,
+  minimumLogLevel: "Debug",
+  mergeWithExisting: true
+})
+```
+
+- `capacity` defaults to 256 and accepts 1 through
+  `JournalLogger.maximumCapacity` (65,536). Raise it for a chatty run on a slow
+  journal; a full queue drops the incoming record before snapshotting it, so
+  saturation costs no traversal of the logged value.
+- `minimumLogLevel` pins `References.MinimumLogLevel` for the whole
+  application. Omitted, the layer leaves that reference alone and Effect's own
+  `Info` default applies.
+- `mergeWithExisting` defaults to `false`, which replaces the ambient logger
+  set. Pass `true` to keep a console logger alongside the forwarder. See
+  [Install a logger](./install-a-logger.md).
+
+An invalid run id or capacity fails layer acquisition with
+`InvalidJournalLoggerOptions`, before any worker starts.
+
+## What the snapshot does to a value
+
+The record is detached, bounded, and redacted before it joins the queue, so
+mutating a logged object after the log call cannot change what is persisted,
+and a hostile value cannot stall the logger.
+
+| Ceiling                  | Value | What happens past it                             |
+| ------------------------ | ----- | ------------------------------------------------ |
+| `maximumSnapshotBytes`   | 1 MiB | Text is cut and marked `[Truncated]`             |
+| `maximumSnapshotMembers` | 4,096 | Remaining members become one `[Truncated]` entry |
+| `maximumSnapshotDepth`   | 64    | Deeper values become `[Deep]`                    |
+
+Values that cannot be read at all, such as a throwing accessor or a revoked
+proxy, become `[Unrenderable]`. Functions, symbols, and binary data are named
+rather than serialized (`[Function]`, `[Symbol]`, `[Binary]`), and a cycle
+becomes `[Circular]`. Text is bounded in one pass that never cuts a surrogate
+pair, so a truncated string is still well formed.
+
+An `Error` is projected to `name`, `message`, `stack`, and `cause` first, then
+to every other own data property it carries, under the same member and byte
+budgets. A `Schema.TaggedError` therefore keeps its `_tag` and its declared
+fields, and a Node system error keeps `code`, `syscall`, and `path`. Accessor
+properties become `[Unrenderable]`, and members past the budget become one
+`[Truncated]` entry.
+
+Container-shaped fields keep their shape. Annotations whose snapshot spent the
+whole budget arrive as `{ "[Truncated]": "[Truncated]" }` rather than as a
+scalar the schema would refuse. A projection that still fails `TelemetryLog`
+degrades to a total record with an empty cause, so every durable
+`telemetry.log` row decodes.
+
+The journal's own redaction rules run before queue admission, the same scrub
+durable events get, so a credential in a logged object never reaches a
+permanent row.
+
+## Watch for lost records
+
+Forwarding is lossy on purpose: a telemetry backlog must not become application
+backpressure. Each of these observed losses advances `Metric.droppedLogRecords`
+(`flows_observability_log_dropped`) once:
+
+1. **Forwarder queue overflow.** The queue was full when the record arrived.
+   Counted only.
+2. **Journal `Dropped` receipt.** The journal refused admission under its
+   `drop-newest` policy. Counted only; `Accepted` and `Duplicate` receipts do
+   not advance the counter.
+3. **Journal delivery failure.** Admission failed. Counted, and reported as a
+   warning with the run id and code `journal_forwarding_failed`.
+4. **A defect from the journal implementation.** Counted and warned with the
+   run id and code `journal_forwarding_defect`. The worker keeps draining.
+
+Warnings carry fixed diagnostic codes instead of raw errors or causes, so
+failure credentials do not reach ambient warning sinks. The worker is forked
+before the forwarding logger is installed, so a warning cannot enqueue itself.
+
+`emitLossy` reports admission, not durable delivery. The counter cannot observe
+later `drop-oldest` evictions of admitted telemetry, asynchronous persistence
+failures, or journal shutdown losses. An `Accepted.evicted` summary does not
+identify the evicted record, so it cannot attribute that loss to telemetry.
+
+Interruption stays fatal: closing the layer's scope ends the worker and can
+drop records queued behind an in-flight write without advancing this counter.
+See [Assert on a short-lived run](#assert-on-a-short-lived-run) for the shape a
+test needs because of that.
+
+A nonzero `droppedLogRecords` with a healthy journal usually means the capacity
+is too small for the run's log volume. See
+[Read the runtime metrics](./read-runtime-metrics.md) for how to read the
+counter.
+
+## Assert on a short-lived run
+
+The forwarder exposes no drain barrier. `journal.flush` settles the entries the
+journal has already admitted, so it returns while a record is still waiting in
+the forwarder's queue or inside an admission the worker has not finished.
+Closing the layer's scope then interrupts that worker and discards what is
+left, so flushing before an assertion proves nothing about a record the journal
+never saw.
+
+Poll the journal for the records you expect while the layer's scope is still
+open, and let the scope close after the assertion:
+
+```ts
+const forwarded = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  yield* Effect.logInfo("first")
+  yield* Effect.logInfo("second")
+  for (let attempt = 0; attempt < 200; attempt++) {
+    const page = yield* journal.entries({ runId, limit: 100 })
+    const records = page.entries.filter((entry) => entry.eventType === "telemetry.log")
+    if (records.length >= 2) return records
+    yield* Effect.sleep("5 millis")
+  }
+  return []
+}).pipe(
+  Effect.provide(Layer.provideMerge(JournalLogger.layerJournalForwarding({ runId }), TestJournal.layer())),
+  Effect.scoped
+)
+```
+
+A bounded poll fails the assertion when a record never arrives, where a fixed
+sleep only makes that failure intermittent. The same pattern is in
+[Test telemetry without a collector](./testing.md), and the loss it guards
+against is cataloged in [Troubleshooting](../troubleshooting.md).