@smthrs/observability 0.35.0 → 1.0.0-rc.12

package/docs/concepts/instrumentation-and-export.md

@@ -0,0 +1,98 @@
+---
+title: "Instrumentation and export"
+description: "Why the Smithers packages instrument themselves but export nothing, what that split buys, and the three independent ways to read the same run."
+sidebar:
+  order: 1
+---
+
+Instrumentation and export are two different jobs, and Smithers puts them in
+different packages on purpose.
+
+## Every other package is already instrumented
+
+A flow execution opens spans without being asked. The flow lifecycle, the
+engine dispatch, a run claim, a heartbeat, a journal write, down to individual
+SQL statements: each is a span on Effect's own tracer. Alongside
+them, each store package declares and updates its own metric handles on its hot
+paths: `JournalMetrics` in [`@smthrs/journal`](/api/journal), `RunStoreMetrics`
+in [`@smthrs/run-store`](/api/run-store), `CacheStoreMetrics` in
+[`@smthrs/step-cache`](/api/step-cache), `ArtifactStoreMetrics` in
+[`@smthrs/artifacts`](/api/artifacts), and `DatabaseMetrics` in
+[`@smthrs/database`](/api/database).
+
+None of those packages allocates an exporter. Effect's tracer and metric
+registry are collection points; with no exporter installed they collect and
+discard, at a cost close to zero.
+
+## Spans Effect names
+
+A Smithers package names its own spans for the operation, such as
+`RunStore.claim` or `Journal.emitDurable`. Effect names the spans of the SQL,
+HTTP and RPC clients those packages use:
+
+| Span            | Name                                               | Kind               | What it carries                                                                                                              |
+| --------------- | -------------------------------------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
+| SQL statement   | The database system: `sqlite` or `postgresql`      | `client`           | `db.system.name`, `db.query.text` and `effect.sql.method`                                                                    |
+| SQL transaction | `sql.transaction`                                  | `internal`         | The events `effect.sql.transaction.commit`, `effect.sql.transaction.savepoint` and `effect.sql.transaction.rollback`         |
+| HTTP request    | The request method, such as `POST`                 | `client`           | `http.request.method`, `url.full`, `url.path`, `url.scheme`, `server.address`, `server.port` and `http.response.status_code` |
+| RPC call        | The procedure's tag, such as `Projection.Snapshot` | `client`, `server` | `rpc.system.name` (`effect_rpc`) and `rpc.method`                                                                            |
+
+On an HTTP request span, `server.address` is the host name without the scheme,
+and `server.port` is always present, as the scheme's default when the URL names
+none.
+
+An HTTP request span records no request or response header. A host that wants
+them provides `HttpClient.TracerHeaderFilter`. The model, artifact and
+step-cache packages add their credential header names to
+`Headers.CurrentRedactedNames`, so a recorded credential header of theirs reads
+`<redacted>`. Every other recorded header keeps its value, apart from the names
+Effect redacts itself, such as `authorization`, `cookie`, `set-cookie` and
+`x-api-key`.
+
+A `4xx` or `5xx` answer ends an HTTP request span as failed, with the status in
+`error.type`. That holds even where the calling package treats the status as an
+ordinary answer: a shared-cache miss is a failed `GET` span with `error.type`
+`404` inside a lookup span that succeeded.
+
+## This package is the exporter
+
+`Otlp.layerFetch` installs Effect's OTLP logger, metrics exporter, and tracer
+against a collector endpoint, with a service identity filled in. That is the
+whole wiring, and it is why telemetry is additive rather than invasive:
+providing one layer turns on delivery for instrumentation that was already
+there, and removing the line turns it off without touching a flow body.
+
+The split also keeps the dependency in one place. Because `Otlp` composes only
+what `effect` already ships, a package that wants a counter takes on no
+OpenTelemetry dependency to get one, and the process pays for an SDK only if a
+host chooses `NodeOtel` or `BrowserOtel`.
+
+## The producer owns the meaning, the exporter owns the delivery
+
+A metric handle belongs to the package that knows when to advance it. This
+package holds only the four cross-package runtime signals nobody else owns,
+and it holds them so the producers can share one identifier:
+`flows_run_throughput` is updated in [`@smthrs/run-store`](/api/run-store),
+`flows_seat_active` and `flows_quota_park` in [`@smthrs/agent`](/api/agent),
+and `flows_observability_log_dropped` here. Step-cache counters stay in
+[`@smthrs/step-cache`](/api/step-cache); this package does not duplicate them.
+
+The consequence for a reader: a metric's name tells you which package to open
+to learn what advances it, and this package tells you only how it leaves the
+process.
+
+## Three ways to read one run
+
+Export is one read path, not the only one, and the other two need no collector:
+
+1. **The OTLP trace.** Every span and series posts to your collector. Filter on
+   the `service.name` you passed to the layer.
+2. **The durable journal.** `Journal.entries` reads a run's lifecycle events in
+   process, and `JournalLogger` puts the run's log records on that same
+   history. See [Forward logs to the run journal](../guides/forward-logs-to-the-journal.md).
+3. **The metric handles.** Effect's `Metric.value` reads a counter in process,
+   with no exporter installed at all. See
+   [Read the runtime metrics](../guides/read-runtime-metrics.md).
+
+The three tell the same story about the same run, which is what makes a
+disagreement between them worth investigating.

package/docs/concepts/layer-map.md

@@ -0,0 +1,73 @@
+---
+title: "The layer map"
+description: "The five telemetry layer builders this package ships, what each one allocates, and the rule that keeps the root entry point free of Node modules."
+sidebar:
+  order: 2
+---
+
+Five builders produce a telemetry layer, and they differ in one thing: how much
+they allocate for you.
+
+## The five builders
+
+| Builder                 | Allocates                                                                  | Reach for it when                                                                 |
+| ----------------------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
+| `Otlp.layerFetch`       | Effect's three OTLP exporters, plus the host's global `fetch`              | You have a collector and no other OpenTelemetry requirement. This is the default. |
+| `Otlp.layer`            | Effect's three OTLP exporters. You provide the `HttpClient`                | You need a specific HTTP client, such as Undici on Node.                          |
+| `NodeOtel.layerOtel`    | The Node OpenTelemetry SDK: three OTLP/HTTP exporters and their processors | Node code that must run through the OpenTelemetry SDK.                            |
+| `BrowserOtel.layerOtel` | The web OpenTelemetry SDK around processors and readers you construct      | Browser code that must run through the OpenTelemetry SDK.                         |
+| `Otel.layerOtel`        | Effect bridges; optional caller factories build providers                  | The application controls SDK provider construction.                               |
+
+Each has an explicit do-nothing counterpart: `Otlp.layerNoop` and
+`Otel.layerNoop` both provide nothing and export nothing, so wiring code
+switches layers instead of branching around a missing one.
+
+## Why `Otlp` is the default
+
+`Otlp` composes only `effect/observability/Otlp`, so it adds no
+OpenTelemetry SDK to the process, and it delivers over Effect's `HttpClient`
+rather than a platform transport. That makes it the cheapest wiring and the
+only one that runs unchanged in Node, in Bun, and in a browser.
+
+Reach past it when something other than a collector endpoint is in play: an
+existing OpenTelemetry instrumentation you must feed, a vendor exporter that is
+not OTLP/HTTP JSON, or a provider a framework hands you.
+
+## Why the root entry point stops at `Otel`
+
+The root entry point exports `Otlp`, `Endpoint`, `JournalLogger`, `Logger`,
+`Metric`, `Otel`, and `Resource`. It stops there because those seven resolve
+nothing host-specific.
+
+`NodeOtel` does. It reaches `@effect/opentelemetry/NodeSdk` and
+`@opentelemetry/sdk-trace-node`, and the latter pulls
+`@opentelemetry/context-async-hooks`, which imports the bare specifier
+`async_hooks`. A browser bundler cannot resolve that, so re-exporting `NodeOtel`
+from the root would break the root's browser bundle for every consumer,
+including those that never wanted an SDK. `BrowserOtel` is kept beside it for
+symmetry: a module that binds one host belongs on a subpath.
+
+Nothing the root entry point reaches resolves a `node:` built-in, as an ESM
+import or as a `require("node:...")` call, so a browser bundle of the root
+carries no host module.
+
+## `Resource` sits under all of them
+
+Every builder decodes the same `Resource.Configuration`. `Otlp`, `NodeOtel`,
+and `BrowserOtel` attach its attributes to all enabled signals. `Otel.layerOtel`
+passes that validated resource to provider factories and the metric producer;
+factories must use it when constructing tracer and logger providers.
+
+Already-created providers retain their own resources. For those providers,
+`Otel.layerOtel`'s resource option sets only the metric resource. Construct
+them with the intended resource before injection to give all three signals the
+same service identity. Spans and log records carry the instrumentation scope
+`effect` with the Effect version, whatever the resource.
+
+`Resource.layer` provides the resource on its own, for a composition that
+assembles the rest itself.
+
+`Otlp` is the one builder with defaults: `Otlp.defaultServiceName` is `flows`
+and `Otlp.defaultServiceVersion` is this package's release version. Name your
+own service in production, so a collector holding several Smithers processes
+can tell them apart.

package/docs/concepts/validated-acquisition.md

@@ -0,0 +1,106 @@
+---
+title: "Validation at layer acquisition"
+description: "Export failure is absorbed by design, so a misconfigured exporter looks exactly like a working one. Endpoints and resources are therefore decoded when the layer is built, and refused with typed errors."
+sidebar:
+  order: 3
+---
+
+An exporter that cannot deliver must not take the application down with it. A
+collector outage is not an application outage, so Effect's exporter absorbs
+export failure: it retries a transient failure three times, then temporarily
+disables delivery, and the effect that produced the span never learns any of
+it.
+
+That is the right behavior, and it creates one hazard. A layer built against
+`""`, `localhost:4318`, or an endpoint with a trailing newline behaves
+identically to a working one: it builds, it runs, it fails nothing, and it
+delivers nothing forever. Nobody notices until an operator goes looking for a
+trace that was never sent.
+
+## Acquisition is the only place to report it
+
+So both configuration inputs are decoded while the layer is being built, before
+any exporter exists:
+
+- The collector endpoint, by `Endpoint.decode`, failing with
+  `Endpoint.InvalidExporterEndpoint`.
+- The service identity and resource attributes, by `Resource.decode`, failing
+  with `Resource.InvalidResourceConfiguration`.
+
+Both are typed failures in the layer's error channel, which means a composition
+sees them at startup, in the place where the value was written, rather than as
+silence in a dashboard weeks later. `Otlp.layer`, `Otlp.layerFetch`, and
+`NodeOtel.layerOtel` all decode their endpoint; every builder decodes its
+resource.
+
+## What an endpoint must be
+
+An absolute `http://` or `https://` URL, at most 2,048 characters, carrying no
+username, password, query, fragment, backslashes, spaces, or control characters.
+Base paths such as `https://collector.example/tenant/9` are supported.
+
+Queries and fragments are refused because exporters append `/v1/<signal>` to
+the base. Appending it after `?tenant=9` or `#notes` would leave the HTTP request
+path unchanged, silently sending telemetry to the wrong route. Use exporter
+headers for authentication, not credentials embedded in the URL.
+
+The last clause is the one that surprises people, and it is the whole reason
+`Endpoint` exists rather than a bare `new URL` check. The WHATWG URL parser is a
+repairer, not a validator: it strips leading and trailing spaces and C0
+controls, and removes tab, newline, and carriage return from anywhere in its
+input. So `new URL("http://collector:4318\n")` succeeds, and reports a
+well-formed URL, while the untrimmed original is what a builder would hand its
+exporter. An endpoint read from a config file with a trailing newline, or
+pasted with a leading space, is the most common shape of this mistake, and it
+produces exactly the silent no-delivery the refusal exists to prevent. No legal
+collector endpoint carries one of those characters, so refusing them costs
+nothing.
+
+Repeated trailing separators are normalized away instead of refused:
+`http://host//` and `http://host` both post to `http://host/v1/traces`.
+
+## What a resource must be
+
+`Resource.Configuration` bounds the identity so an accepted value cannot become
+an unencodable payload later:
+
+- `serviceName` and `serviceVersion` are non-empty, well formed, and at most
+  1,024 UTF-16 code units.
+- At most 256 attributes, with non-empty, well formed keys of at most 1,024
+  code units.
+- Attribute values are finite numbers, booleans, strings of at most 65,536 code
+  units, or homogeneous arrays of one of those scalar types with at most 256
+  elements (`Resource.maximumAttributeArrayLength`).
+- The complete OTLP JSON resource is at most 128 KiB
+  (`Resource.maximumResourceBytes`), including identity, SDK-added
+  `telemetry.sdk.name`, `telemetry.sdk.language` and `telemetry.sdk.version`,
+  keys, values, array wrappers, escaping and UTF-8. SDK fields are reserved even for the default Effect
+  exporter. Counting stops as soon as the budget is exceeded;
+  the value-free refusal names `attributes`, the measured lower bound and limit.
+  Per-field limits alone cannot guarantee this cumulative bound.
+- Resource metadata is explicit. The default OTLP and Node SDK layers isolate
+  ambient resource configuration so it cannot enlarge metadata after admission.
+- The 1 MiB transport reserves 8 KiB for the JSON envelope and escaped scope
+  name, which in a metrics request repeats the service name, leaving 888 KiB
+  (`Otlp.reservedBatchBytes`) for a signal batch. This is
+  about 909 bytes per record at the upstream 1,000-record log or span limit,
+  enough for ordinary batches; application records can still be too large.
+- NUL and unpaired UTF-16 surrogates are refused anywhere. Valid astral Unicode
+  is preserved.
+
+An empty `serviceName` is refused rather than defaulted, because anonymous
+telemetry in a shared collector is worse than none.
+
+## Refusals never carry the rejected value
+
+Both errors report a stable `code`, the `path` of the option the bad value
+arrived on, and a message. Neither retains the value itself: `baseUrl` is the
+field most likely to carry a token in a query string or a credential in its
+userinfo, and an error message is the least controlled surface in a system. The
+`path` is what you need to find the value, and you already have the value.
+
+The `path` names the caller's own option, so a refusal from `Otlp` says
+`baseUrl` and one from `NodeOtel` says `endpoint`.
+
+For what to do when you see one, see
+[Troubleshooting](../troubleshooting.md).

package/docs/guides/export-to-a-collector.md

@@ -0,0 +1,190 @@
+---
+title: "Export to an OTLP collector"
+description: "Wire Otlp.layerFetch into a running program: the collector endpoint, the service identity, authentication headers, export cadence, the shutdown flush, and what happens when the collector is down."
+sidebar:
+  order: 1
+---
+
+`Otlp` posts logs, metrics, and traces to an OTLP/HTTP collector as JSON. This
+guide covers the options you set once at startup and the behavior you should
+expect from them afterwards.
+
+## Provide the layer
+
+```ts
+import * as Otlp from "@smthrs/observability/Otlp"
+import * as Effect from "effect/Effect"
+
+const telemetry = Otlp.layerFetch({
+  baseUrl: "http://localhost:4318",
+  serviceName: "deploy-status"
+})
+
+// `program` is your application effect, unchanged from its non-telemetry form.
+const outcome = program.pipe(Effect.provide(telemetry), Effect.scoped)
+```
+
+The layer is scoped: give it a lifetime with `Effect.scoped`, or build it as
+part of a runtime that owns a scope. Signals post below `baseUrl` at
+`/v1/logs`, `/v1/metrics`, and `/v1/traces`, so a base URL with a path prefix
+is honored: `http://collector:4318/tenant/9` posts to
+`http://collector:4318/tenant/9/v1/traces`.
+
+`Otlp.layer` is the same wiring without the `fetch` binding. Use it when the
+host must supply its own HTTP client:
+
+```ts
+import * as NodeHost from "@smthrs/platform-node/NodeHost"
+import * as Layer from "effect/Layer"
+
+const telemetry = Otlp.layer({ baseUrl: "http://localhost:4318" }).pipe(
+  Layer.provide(NodeHost.NodeHttpClient.layerUndici)
+)
+```
+
+## Name the service
+
+`serviceName` and `serviceVersion` become the `service.name` and
+`service.version` resource attributes on every exported signal, and they are
+how a collector holding several processes tells them apart. Omitted, they fall
+back to `Otlp.defaultServiceName` (`flows`) and `Otlp.defaultServiceVersion`
+(this package's release version), which is a reasonable default for a Smithers
+CLI process and a poor one for your application.
+
+Add deployment facts as `attributes`:
+
+```ts
+const telemetry = Otlp.layerFetch({
+  baseUrl: "https://otlp.example.com",
+  serviceName: "deploy-status",
+  serviceVersion: "2.4.1",
+  attributes: {
+    "deployment.environment.name": "production",
+    "service.instance.id": process.env.HOSTNAME ?? "local"
+  }
+})
+```
+
+Attribute values are strings, finite numbers, booleans, or homogeneous arrays
+of those. The full bounds are in
+[Validation at layer acquisition](../concepts/validated-acquisition.md).
+
+## Authenticate
+
+`headers` is sent with every export request, which is where a vendor token
+goes:
+
+```ts
+const telemetry = Otlp.layerFetch({
+  baseUrl: "https://otlp.example.com",
+  serviceName: "deploy-status",
+  headers: { authorization: "Bearer YOUR_TOKEN" }
+})
+```
+
+Read the token from the environment or your secret store. A credential in the
+URL is refused: an endpoint carrying userinfo fails layer acquisition. A
+credential header (`authorization`, `x-api-key`, `x-honeycomb-team`, and
+similar names) over plaintext `http://` is refused too, unless the collector is
+on `localhost`, `127.0.0.0/8`, or `[::1]`. Non-credential headers such as a
+tenant id work on any endpoint. The check reads header names only, so use
+`https://` for every remote collector.
+
+Fetch redirects are refused, including same-origin redirects. Configure the
+collector's final URL. Other HTTP transports must refuse redirects themselves.
+For fetch options such as `credentials: "omit"`, provide
+`FetchHttpClient.RequestInit` around the whole exporting effect:
+
+```ts
+import * as FetchHttpClient from "effect/http/FetchHttpClient"
+
+const outcome = program.pipe(
+  Effect.provide(telemetry),
+  Effect.provideService(FetchHttpClient.RequestInit, { credentials: "omit" }),
+  Effect.scoped
+)
+```
+
+Options supplied only to the inner HTTP-client layer are replaced by the
+redirect policy. Ambient options are retained except `redirect`, which is
+always `"manual"`. Redirect responses fail export through the existing failure
+handling below; they never forward credentials or signal bodies.
+
+## Control the cadence and the flush
+
+`exportInterval` sets how often all three signals batch and post. Omitted, each
+signal keeps Effect's own default. `shutdownTimeout` bounds the final flush
+when the layer's scope closes.
+
+```ts
+const telemetry = Otlp.layerFetch({
+  baseUrl: "http://localhost:4318",
+  serviceName: "deploy-status",
+  exportInterval: "10 seconds",
+  shutdownTimeout: "5 seconds"
+})
+```
+
+Closing the scope is the deterministic flush. A short program may deliver
+nothing on the interval and everything at shutdown, so read what a collector
+received only after the scope has closed.
+
+## Turn export off explicitly
+
+`Otlp.layerNoop` provides nothing and exports nothing. It exists so a host that
+has no collector, or a browser build that has not opted in, switches layers
+rather than branching around a missing provide:
+
+```ts
+const telemetry = collectorUrl === undefined
+  ? Otlp.layerNoop
+  : Otlp.layerFetch({ baseUrl: collectorUrl, serviceName: "deploy-status" })
+```
+
+## What the collector receives
+
+Every request is OTLP/HTTP JSON. Its `user-agent` is
+`OTel-OTLP-Exporter-JavaScript-Effect-OtlpTracer/` and the Effect version, with
+`OtlpLogger` or `OtlpMetrics` for the other two signals, after any `user-agent`
+you set in `headers`.
+
+| Part        | What it carries                                                                                                                                                                                                                                               |
+| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Resource    | `telemetry.sdk.name` (`effect`), `telemetry.sdk.language` (`nodejs` or `webjs`) and `telemetry.sdk.version` (the Effect version), then your `attributes`, `service.name` and `service.version`.                                                               |
+| Scope       | Spans and log records name their scope `effect`, with the Effect version. Metrics name their scope for your service.                                                                                                                                          |
+| Spans       | Status `1` on success. Status `2` on failure, with the first error's message and one `exception` event per error. Status `0` and the attribute `effect.fiber.interrupted` on interruption. Failure text and string attributes are redacted before they leave. |
+| Log records | Your annotations as attributes, plus `effect.fiberId` and, for each log span, `effect.log_span.<label>` holding the elapsed milliseconds as a number. A record logged with a cause adds `exception.type`, `exception.message` and `exception.stacktrace`.     |
+| Metrics     | Counters, gauges and histograms under the names the producing package declares. A timer's unit is `ms`. A histogram lists every boundary it declares in `explicitBounds` and counts values above the last one in a final bucket.                              |
+
+The names of the spans themselves are in
+[Instrumentation and export](../concepts/instrumentation-and-export.md#spans-effect-names).
+
+## What happens when the collector is down
+
+Export failure never fails your program, and it never surfaces as a rejected
+promise. Effect's exporter retries a transient failure three times, honoring a
+`retry-after` response header on any retried status when one is present, then
+disables that exporter for 60 seconds and logs a `Debug` record saying so.
+Spans and records produced inside that window are dropped, and delivery resumes
+afterwards.
+
+The exporter reads a response's status and `retry-after` header and nothing
+else. Each exchange ends when the status arrives, so a collector that accepts a
+batch and never finishes its answer does not hold up the next one.
+
+Two consequences worth planning for:
+
+- A dead collector is invisible at `Info`. Run at `Debug` while you are
+  bringing an exporter up, or watch the collector side.
+- A configuration mistake and a collector outage look the same after
+  acquisition. That is why the endpoint is decoded before any exporter exists;
+  see [Troubleshooting](../troubleshooting.md) when nothing arrives.
+
+## Next steps
+
+- [Test telemetry without a collector](./testing.md): assert on what would have
+  been posted.
+- [Wire an OpenTelemetry SDK](./wire-an-opentelemetry-sdk.md): when a collector
+  endpoint is not enough.
+- [Read the runtime metrics](./read-runtime-metrics.md): what the exported
+  series mean.