@smthrs/observability 0.35.0 → 1.0.0-rc.12

package/docs/guides/forward-logs-to-the-journal.md

@@ -0,0 +1,226 @@
+---
+title: "Forward logs to the run journal"
+description: "Install JournalLogger.layerJournalForwarding so a run's Effect log records become durable telemetry.log entries, and read them back: capacity, bounds, redaction, observed losses, and admission limits."
+sidebar:
+  order: 3
+---
+
+`JournalLogger` turns a run's Effect log records into durable, structured
+`telemetry.log` entries on that run's journal. It is the second delivery path
+beside [OTLP export](./export-to-a-collector.md), and it is the one an operator
+reads without a collector: the records land next to the run's lifecycle events
+and are read back with the same `Journal.entries` call.
+
+## Install the layer
+
+The layer names one run, and requires a `Journal.Journal` service from
+[`@smthrs/journal`](/api/journal):
+
+```ts
+import * as JournalEvent from "@smthrs/journal/JournalEvent"
+import * as JournalLogger from "@smthrs/observability/JournalLogger"
+import * as Schema from "effect/Schema"
+
+const runId = Schema.decodeUnknownSync(JournalEvent.RunId)("deploy-status-1")
+
+const journalLogs = JournalLogger.layerJournalForwarding({ runId })
+```
+
+`runId` is branded, so it comes from decoding a string rather than from a bare
+literal. Decoding it here means a malformed id is reported where it was
+written.
+
+Provide the layer over your journal. In a test, the deterministic bundle is
+enough:
+
+```ts
+import * as TestJournal from "@smthrs/journal/test/TestJournal"
+import * as Layer from "effect/Layer"
+
+const layer = Layer.provideMerge(journalLogs, TestJournal.layer())
+```
+
+In production, the journal is the one your durable engine already composed;
+provide the forwarder over it the same way.
+
+## Read the records back
+
+Each record is one journal entry with `eventType` `telemetry.log`, written on
+the journal's lossy channel by source `flows/observability/logger`. The payload
+decodes with the exported schema:
+
+```ts
+import * as Journal from "@smthrs/journal/Journal"
+import * as Effect from "effect/Effect"
+
+const read = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  const page = yield* journal.entries({ runId, limit: 200 })
+  return page.entries
+    .filter((entry) => entry.eventType === "telemetry.log")
+    .map((entry) => Schema.decodeUnknownSync(JournalLogger.TelemetryLog)(entry.payload))
+})
+```
+
+A decoded record carries the level, the logged message, the log annotations,
+the fiber id, an ISO-8601 timestamp, the active span's `traceId` and `spanId`
+when there was one, and a structural `cause` that preserves the order and the
+kind of every failure reason:
+
+```text
+{
+  version: 1,
+  level: "Error",
+  message: [ "deploy failed" ],
+  annotations: { lane: "api", attempt: 2 },
+  cause: {
+    version: 1,
+    reasons: [
+      { _tag: "Fail", error: { code: "timeout" } },
+      { _tag: "Interrupt", fiberId: 7 }
+    ]
+  },
+  fiberId: 12,
+  traceId: "0af7651916cd43dd8448eb211c80319c",
+  spanId: "b7ad6b7169203331",
+  timestamp: "2026-09-03T12:00:00.000Z"
+}
+```
+
+The durable journal allocates the per-source sequence, so restarting the layer
+or running two of them for one run cannot make two records collide on one
+identity.
+
+## Choose the queue depth
+
+The logger callback never blocks. It snapshots the record synchronously when the
+bounded queue has room, then offers it to that queue, which a forked worker
+drains into the journal.
+
+```ts
+const journalLogs = JournalLogger.layerJournalForwarding({
+  runId,
+  capacity: 4096,
+  minimumLogLevel: "Debug",
+  mergeWithExisting: true
+})
+```
+
+- `capacity` defaults to 256 and accepts 1 through
+  `JournalLogger.maximumCapacity` (65,536). Raise it for a chatty run on a slow
+  journal; a full queue drops the incoming record before snapshotting it, so
+  saturation costs no traversal of the logged value.
+- `minimumLogLevel` pins `References.MinimumLogLevel` for the whole
+  application. Omitted, the layer leaves that reference alone and Effect's own
+  `Info` default applies.
+- `mergeWithExisting` defaults to `false`, which replaces the ambient logger
+  set. Pass `true` to keep a console logger alongside the forwarder. See
+  [Install a logger](./install-a-logger.md).
+
+An invalid run id or capacity fails layer acquisition with
+`InvalidJournalLoggerOptions`, before any worker starts.
+
+## What the snapshot does to a value
+
+The record is detached, bounded, and redacted before it joins the queue, so
+mutating a logged object after the log call cannot change what is persisted,
+and a hostile value cannot stall the logger.
+
+| Ceiling                  | Value | What happens past it                             |
+| ------------------------ | ----- | ------------------------------------------------ |
+| `maximumSnapshotBytes`   | 1 MiB | Text is cut and marked `[Truncated]`             |
+| `maximumSnapshotMembers` | 4,096 | Remaining members become one `[Truncated]` entry |
+| `maximumSnapshotDepth`   | 64    | Deeper values become `[Deep]`                    |
+
+Values that cannot be read at all, such as a throwing accessor or a revoked
+proxy, become `[Unrenderable]`. Functions, symbols, and binary data are named
+rather than serialized (`[Function]`, `[Symbol]`, `[Binary]`), and a cycle
+becomes `[Circular]`. Text is bounded in one pass that never cuts a surrogate
+pair, so a truncated string is still well formed.
+
+An `Error` is projected to `name`, `message`, `stack`, and `cause` first, then
+to every other own data property it carries, under the same member and byte
+budgets. A `Schema.TaggedError` therefore keeps its `_tag` and its declared
+fields, and a Node system error keeps `code`, `syscall`, and `path`. Accessor
+properties become `[Unrenderable]`, and members past the budget become one
+`[Truncated]` entry.
+
+Container-shaped fields keep their shape. Annotations whose snapshot spent the
+whole budget arrive as `{ "[Truncated]": "[Truncated]" }` rather than as a
+scalar the schema would refuse. A projection that still fails `TelemetryLog`
+degrades to a total record with an empty cause, so every durable
+`telemetry.log` row decodes.
+
+The journal's own redaction rules run before queue admission, the same scrub
+durable events get, so a credential in a logged object never reaches a
+permanent row.
+
+## Watch for lost records
+
+Forwarding is lossy on purpose: a telemetry backlog must not become application
+backpressure. Each of these observed losses advances `Metric.droppedLogRecords`
+(`flows_observability_log_dropped`) once:
+
+1. **Forwarder queue overflow.** The queue was full when the record arrived.
+   Counted only.
+2. **Journal `Dropped` receipt.** The journal refused admission under its
+   `drop-newest` policy. Counted only; `Accepted` and `Duplicate` receipts do
+   not advance the counter.
+3. **Journal delivery failure.** Admission failed. Counted, and reported as a
+   warning with the run id and code `journal_forwarding_failed`.
+4. **A defect from the journal implementation.** Counted and warned with the
+   run id and code `journal_forwarding_defect`. The worker keeps draining.
+
+Warnings carry fixed diagnostic codes instead of raw errors or causes, so
+failure credentials do not reach ambient warning sinks. The worker is forked
+before the forwarding logger is installed, so a warning cannot enqueue itself.
+
+`emitLossy` reports admission, not durable delivery. The counter cannot observe
+later `drop-oldest` evictions of admitted telemetry, asynchronous persistence
+failures, or journal shutdown losses. An `Accepted.evicted` summary does not
+identify the evicted record, so it cannot attribute that loss to telemetry.
+
+Interruption stays fatal: closing the layer's scope ends the worker and can
+drop records queued behind an in-flight write without advancing this counter.
+See [Assert on a short-lived run](#assert-on-a-short-lived-run) for the shape a
+test needs because of that.
+
+A nonzero `droppedLogRecords` with a healthy journal usually means the capacity
+is too small for the run's log volume. See
+[Read the runtime metrics](./read-runtime-metrics.md) for how to read the
+counter.
+
+## Assert on a short-lived run
+
+The forwarder exposes no drain barrier. `journal.flush` settles the entries the
+journal has already admitted, so it returns while a record is still waiting in
+the forwarder's queue or inside an admission the worker has not finished.
+Closing the layer's scope then interrupts that worker and discards what is
+left, so flushing before an assertion proves nothing about a record the journal
+never saw.
+
+Poll the journal for the records you expect while the layer's scope is still
+open, and let the scope close after the assertion:
+
+```ts
+const forwarded = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  yield* Effect.logInfo("first")
+  yield* Effect.logInfo("second")
+  for (let attempt = 0; attempt < 200; attempt++) {
+    const page = yield* journal.entries({ runId, limit: 100 })
+    const records = page.entries.filter((entry) => entry.eventType === "telemetry.log")
+    if (records.length >= 2) return records
+    yield* Effect.sleep("5 millis")
+  }
+  return []
+}).pipe(
+  Effect.provide(Layer.provideMerge(JournalLogger.layerJournalForwarding({ runId }), TestJournal.layer())),
+  Effect.scoped
+)
+```
+
+A bounded poll fails the assertion when a record never arrives, where a fixed
+sleep only makes that failure intermittent. The same pattern is in
+[Test telemetry without a collector](./testing.md), and the loss it guards
+against is cataloged in [Troubleshooting](../troubleshooting.md).

package/docs/guides/install-a-logger.md

@@ -0,0 +1,111 @@
+---
+title: "Install a logger"
+description: "Pick a Logger layer for a development shell, a production process, or a silent test, and understand the two rules every one of them follows: the ambient set is replaced, and the minimum level is left alone."
+sidebar:
+  order: 4
+---
+
+`Logger` ships four layers over Effect's own logger machinery. They differ in
+the sink; they agree on the two rules that decide what installing one does to
+the rest of the application.
+
+## Pick a layer
+
+```ts
+import * as Logger from "@smthrs/observability/Logger"
+
+// A development shell: colored, human-readable lines.
+const development = Logger.layerPrettyDev()
+
+// A production process: one structured JSON object per line.
+const production = Logger.layerStructuredJson()
+
+// A test that should print nothing.
+const silent = Logger.layerNoop()
+```
+
+`Logger.layerPrettyDev` chooses colors and mode from the terminal it finds, so
+it degrades to plain text when the output is not a TTY.
+`Logger.layerStructuredJson` is the one to ship: a log shipper can parse it, and
+it is what a collector's log pipeline expects.
+
+To install a logger you wrote, or one from another package, use `Logger.layer`:
+
+```ts
+import * as EffectLogger from "effect/Logger"
+
+const custom = Logger.layer(
+  EffectLogger.make((options) => {
+    /* your sink */
+  })
+)
+```
+
+## Rule one: the ambient set is replaced
+
+`mergeWithExisting` defaults to `false`, matching Effect's own `Logger.layer`,
+so installing one of these layers replaces the ambient logger set rather than
+adding to it. That is what you want for a process that has exactly one sink,
+and it is the usual explanation for logs disappearing after a layer was added.
+
+Pass `true` to keep the loggers already installed:
+
+```ts
+const both = Logger.layerStructuredJson({ mergeWithExisting: true })
+```
+
+The same option is on
+[`JournalLogger.layerJournalForwarding`](./forward-logs-to-the-journal.md), and
+merging is how a control-plane run gets console output and durable
+`telemetry.log` rows from one log call.
+
+## Rule two: the minimum level is left alone
+
+A layer that adds a sink has no business overriding the level an application
+chose elsewhere, so `References.MinimumLogLevel` is provided only when you name
+`minimumLogLevel`. Omitted, whatever the application set stands, and Effect's
+own `Info` default applies when nothing set it.
+
+```ts
+// Leaves the application's chosen level in place.
+const inherits = Logger.layerStructuredJson()
+
+// Pins Warn for the whole application.
+const pinned = Logger.layerStructuredJson({ minimumLogLevel: "Warn" })
+```
+
+`Logger.layerNoop` is the deliberate exception. Silence is its purpose, so it
+pins `None` unless you name another level. Passing one is how a test silences
+output while keeping a level assertion meaningful:
+
+```ts
+const quietButFiltered = Logger.layerNoop({ minimumLogLevel: "Debug" })
+```
+
+## Merge when you install more than one sink
+
+Installing a logger decides what is rendered locally. It exports nothing on its
+own: log records reach a collector because [`Otlp`](./export-to-a-collector.md)
+adds its own log exporter to the same ambient set, and they reach a run's
+history because
+[`JournalLogger`](./forward-logs-to-the-journal.md) adds a forwarder to it.
+
+That shared set is why rule one matters here. A `Logger` layer with the default
+`mergeWithExisting: false` discards every logger the set already held, and
+whether that includes the OTLP exporter depends on which layer the composition
+builds last. Pass `mergeWithExisting: true` on the `Logger` layer whenever
+another sink is installed, and the ordering question does not arise:
+
+```ts
+import * as Effect from "effect/Effect"
+
+const outcome = program.pipe(
+  Effect.provide(Logger.layerStructuredJson({ mergeWithExisting: true })),
+  Effect.provide(telemetry),
+  Effect.scoped
+)
+```
+
+One `Effect.logInfo` call then renders locally, posts to the collector's
+`/v1/logs` path, and, with the journal forwarder installed, lands on the run's
+durable history.

package/docs/guides/read-runtime-metrics.md

@@ -0,0 +1,83 @@
+---
+title: "Read the runtime metrics"
+description: "The four runtime metric handles this package declares, the exact series names a dashboard binds to, which package advances each one, and how to read a counter in process without an exporter."
+sidebar:
+  order: 5
+---
+
+`Metric` declares the runtime signals that cross package boundaries: the
+handles a producer updates and a dashboard reads. Every other metric in
+Smithers belongs to the package that owns the behavior it measures.
+
+## The six handles
+
+| Handle                       | Series name                                | Kind    | Advances when                                                                                                                                                      |
+| ---------------------------- | ------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `runThroughput`              | `flows_run_throughput`                     | counter | A run's terminal transition commits in [`@smthrs/run-store`](/api/run-store).                                                                                      |
+| `activeSeats`                | `flows_seat_active`                        | gauge   | Held for the lifetime of a production `Agent.run` stream in [`@smthrs/agent`](/api/agent), and released on success, failure, or interruption.                      |
+| `quotaParks`                 | `flows_quota_park`                         | counter | A sealed quota decision is first executed, and not when that decision is replayed after a wake or a process restart.                                               |
+| `structuredOutputRejections` | `flows_agent_structured_output_rejections` | counter | An agent answer is refused by its declared output schema, in the same session or a fresh one.                                                                      |
+| `structuredOutputExhausted`  | `flows_agent_structured_output_exhausted`  | counter | An agent step fails after spending every output correction and its repair.                                                                                         |
+| `droppedLogRecords`          | `flows_observability_log_dropped`          | counter | A log record is lost before durable delivery: once per queue overflow, once per journal delivery failure, and once per defect the forwarding worker recovers from. |
+
+`Metric.registry` is the same six as one object, so a host can enumerate them.
+The series names are the dashboard contract; treat them as public API.
+
+Step-cache lookup and write counters belong to
+[`@smthrs/step-cache`](/api/step-cache), and the store packages keep their own
+handles. This package does not duplicate any of them.
+
+## Read a counter in process
+
+A metric handle is readable without an exporter, which makes it a fine
+assertion target in a test and a fine health check in a host:
+
+```ts
+import * as ObservabilityMetric from "@smthrs/observability/Metric"
+import * as Effect from "effect/Effect"
+import * as Metric from "effect/Metric"
+
+const dropped = Effect.gen(function*() {
+  const value = yield* Metric.value(ObservabilityMetric.droppedLogRecords)
+  return value.count
+})
+```
+
+A counter's value carries `count`; a gauge's carries `value`.
+
+## Read a dimensioned counter through its attribute view
+
+Some packages update a counter through a tagged view rather than the bare
+handle. The dispatch counter in [`@smthrs/engine-store`](/api/engine-store) is
+the common example: it updates only the series tagged with the outcome, so
+reading the bare handle reads the attribute-less series and always sees zero.
+
+```ts
+import { EngineStoreMetrics } from "@smthrs/engine-store"
+
+const dispatches = Effect.gen(function*() {
+  const value = yield* Metric.value(EngineStoreMetrics.dispatch.Success)
+  return value.count
+})
+```
+
+Read the exported view, not the parent handle, whenever the owning package
+exports one.
+
+## Isolate the registry in a test
+
+Effect keeps metric state in a `MetricRegistry` reference with a
+process-default value, so counters accumulate across tests in one process.
+Provide a fresh map to isolate one run:
+
+```ts
+Effect.provideService(Metric.MetricRegistry, new Map())
+```
+
+## Where the series go
+
+Every registered series exports with the rest of the telemetry: with
+[`Otlp` provided](./export-to-a-collector.md), they post to the collector's
+`/v1/metrics` path under the `service.name` you configured. With no exporter
+installed, they are still collected and still readable in process, and nothing
+leaves the machine.

package/docs/guides/testing.md

@@ -0,0 +1,134 @@
+---
+title: "Test telemetry without a collector"
+description: "Assert on what would have been exported: a recording fetch for OTLP payloads, TestJournal for forwarded log records, a fresh metric registry, and the noop layers that stand in for absent telemetry."
+sidebar:
+  order: 6
+---
+
+Telemetry is testable because every outbound edge is a service. Replace the
+edge, keep the wiring, and the assertions are about the payloads a collector
+would have received.
+
+## Record what the exporter posts
+
+`Otlp.layerFetch` delivers over the `FetchHttpClient.Fetch` service, so
+providing your own function captures every export request and touches no
+network:
+
+```ts
+import * as Otlp from "@smthrs/observability/Otlp"
+import * as Effect from "effect/Effect"
+import * as FetchHttpClient from "effect/http/FetchHttpClient"
+import * as Metric from "effect/Metric"
+
+const recordingFetch = () => {
+  const requests: Array<{ url: string; body: string }> = []
+  const fetch: typeof globalThis.fetch = (input, init) => {
+    const body = typeof init?.body === "string"
+      ? init.body
+      : new TextDecoder().decode(init?.body as Uint8Array)
+    requests.push({ url: String(input), body })
+    return Promise.resolve(new Response("{}", { status: 200 }))
+  }
+  return { requests, fetch }
+}
+
+const collector = recordingFetch()
+
+await Effect.runPromise(
+  program.pipe(
+    Effect.provide(Otlp.layerFetch({ baseUrl: "http://collector.invalid:4318" })),
+    Effect.provideService(FetchHttpClient.Fetch, collector.fetch),
+    Effect.provideService(Metric.MetricRegistry, new Map()),
+    Effect.orDie,
+    Effect.scoped
+  )
+)
+```
+
+Assert after the scope closes, because the shutdown flush is what delivers a
+short program's signals. Filter `collector.requests` by URL suffix to separate
+the three signals, and parse a body to read span or series names. An
+unresolvable host such as `collector.invalid` is a deliberate choice: if the
+service is ever missed, the test fails rather than reaching the network.
+
+The fresh `Metric.MetricRegistry` map keeps one test's counters out of the
+next test's export.
+
+## Test the journal forwarder
+
+`TestJournal.layer()` from [`@smthrs/journal`](/api/journal) provides the
+production SQLite journal over an in-memory database, so a forwarding test
+exercises the real write path:
+
+```ts
+import * as Journal from "@smthrs/journal/Journal"
+import * as TestJournal from "@smthrs/journal/test/TestJournal"
+import * as JournalLogger from "@smthrs/observability/JournalLogger"
+import * as Effect from "effect/Effect"
+import * as Layer from "effect/Layer"
+
+// `runId` is a decoded JournalEvent.RunId, as in the forwarding guide.
+const entries = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  yield* Effect.logInfo("hello")
+  const page = yield* journal.entries({ runId, limit: 100 })
+  return page.entries
+}).pipe(
+  Effect.provide(Layer.provideMerge(JournalLogger.layerJournalForwarding({ runId }), TestJournal.layer())),
+  Effect.scoped
+)
+```
+
+Forwarding is asynchronous, so poll until the expected number of entries has
+landed rather than sleeping a fixed interval. To assert on a loss instead,
+read `Metric.value(ObservabilityMetric.droppedLogRecords)`.
+
+## Control time for the retry window
+
+Export retries and the 60-second disable window run on Effect's clock, so
+`TestClock` makes them deterministic:
+
+```ts
+import { TestClock } from "effect/testing"
+
+const exported = Effect.gen(function*() {
+  yield* Effect.void.pipe(Effect.withSpan("exported"))
+  // One initial request plus three transient retries completes within four seconds.
+  yield* TestClock.adjust("5 seconds")
+})
+```
+
+Provide `TestClock.layer()` alongside the exporter, then advance the clock and
+count the recorded requests.
+
+## Stand telemetry down explicitly
+
+A test that is not about telemetry should say so rather than leave the layer
+out and let a type error decide:
+
+| Layer                | What it does                                                    |
+| -------------------- | --------------------------------------------------------------- |
+| `Otlp.layerNoop`     | Provides nothing and exports nothing.                           |
+| `Otel.layerNoop`     | The empty provider-neutral OTEL slot.                           |
+| `Logger.layerNoop()` | Removes the logger set and pins `None` unless you name a level. |
+
+`Journal.layerNoop` from [`@smthrs/journal`](/api/journal) is the matching stub
+for the forwarder when the assertion is about queue behavior rather than about
+persistence.
+
+## Assert on refusals at acquisition
+
+Both configuration refusals happen while the layer is built, so a test builds
+the layer and inspects the exit rather than running a program:
+
+```ts
+import * as Layer from "effect/Layer"
+
+const exit = await Effect.runPromiseExit(
+  Effect.scoped(Layer.build(Otlp.layerFetch({ baseUrl: "collector:4318" })))
+)
+```
+
+The failure is an `Endpoint.InvalidExporterEndpoint` whose `path` is `baseUrl`.
+See [Troubleshooting](../troubleshooting.md) for the catalog.