@smthrs/observability 0.35.0 → 1.0.0-rc.4

package/docs/quickstart.md

@@ -0,0 +1,145 @@
+---
+title: "Quickstart"
+description: "Export a trace, a log record, and a metric series to a collector you can watch, using a thirty-line collector stand-in and one Effect layer."
+sidebar:
+  order: 2
+---
+
+This quickstart exports all three signals from one small program and shows you
+what arrived. It needs no Docker, no vendor account, and no OpenTelemetry SDK:
+a twelve-line Node server stands in for the collector so you can read the
+delivery with your own eyes, and swapping in a real collector is a one-line
+change at the end.
+
+## Prerequisites
+
+- Node.js 26.4.0 or later, which runs a `.ts` file directly by stripping its
+  types.
+- A package whose `package.json` sets `"type": "module"`, with the
+  dependencies installed. Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
+
+## Start a collector stand-in
+
+An OTLP/HTTP collector is an HTTP server that accepts JSON on three paths.
+Create `collector.mjs`:
+
+```js
+import { createServer } from "node:http"
+
+createServer((request, response) => {
+  const chunks = []
+  request.on("data", (chunk) => chunks.push(chunk))
+  request.on("end", () => {
+    const body = JSON.parse(Buffer.concat(chunks).toString("utf8"))
+    console.log(request.url, Object.keys(body).join(" "))
+    response.writeHead(200, { "content-type": "application/json" })
+    response.end("{}")
+  })
+}).listen(4318, "127.0.0.1")
+```
+
+Run it in its own terminal, and leave it running:
+
+```bash
+node collector.mjs
+```
+
+## Write the program
+
+Create `quickstart.ts`. The program opens a span, writes a log record, and
+advances a counter. None of those three lines mentions telemetry: they are
+ordinary Effect, and they are exactly what the Smithers packages already do on
+their own hot paths.
+
+```ts
+import * as Otlp from "@smthrs/observability/Otlp"
+import * as Effect from "effect/Effect"
+import * as Metric from "effect/Metric"
+
+const deployments = Metric.counter("quickstart/deployments", {
+  description: "Deployments this program recorded"
+})
+
+const deploy = Effect.gen(function*() {
+  yield* Effect.logInfo("deploying the API")
+  yield* Metric.update(deployments, 1)
+}).pipe(Effect.withSpan("deploy"))
+```
+
+Now add the exporter and run it. `layerFetch` posts over the host's global
+`fetch`, so there is nothing else to provide:
+
+```ts
+const telemetry = Otlp.layerFetch({
+  baseUrl: "http://localhost:4318",
+  serviceName: "quickstart",
+  serviceVersion: "1.0.0"
+})
+
+await Effect.runPromise(
+  deploy.pipe(Effect.provide(telemetry), Effect.orDie, Effect.scoped)
+)
+```
+
+`Effect.orDie` says that an unusable collector endpoint is a startup defect
+rather than something this program recovers from. The layer is scoped, so
+`Effect.scoped` gives it a lifetime, and closing that scope is what flushes.
+
+## Run it
+
+```bash
+node quickstart.ts
+```
+
+The collector terminal prints one line per signal. The three arrive in
+whatever order the exporters flush:
+
+```text
+/v1/traces resourceSpans
+/v1/logs resourceLogs
+/v1/metrics resourceMetrics
+```
+
+If you print a whole body instead of its top-level key, every payload carries
+the resource you named: `service.name` is `quickstart` and `service.version` is
+`1.0.0`.
+
+## What just happened
+
+One layer installed three of Effect's own OTLP exporters against your base URL.
+Each signal posts below it on its own path, so `http://localhost:4318` becomes
+`/v1/traces`, `/v1/logs`, and `/v1/metrics`. The exporters batch, so nothing
+was delivered while the program ran; closing the layer's scope force-flushed
+all three, which is why `Effect.scoped` is where the output comes from.
+
+The program never mentioned a tracer, a meter, or a logger provider. Effect's
+own tracer and metric registry were already collecting; the layer is only the
+delivery.
+
+## Point it at a real collector
+
+Change one field:
+
+```ts
+const telemetry = Otlp.layerFetch({
+  baseUrl: "https://otlp.example.com",
+  serviceName: "quickstart",
+  headers: { authorization: "Bearer YOUR_TOKEN" }
+})
+```
+
+Replace `YOUR_TOKEN` with the token your vendor issued. The endpoint must be an
+absolute `http:` or `https:` URL carrying no credentials; anything else refuses
+the layer at acquisition rather than failing silently later. See
+[Export to an OTLP collector](./guides/export-to-a-collector.md) for the rest
+of the options.
+
+## Next steps
+
+- [Export to an OTLP collector](./guides/export-to-a-collector.md): service
+  identity, authentication headers, export cadence, and what happens when a
+  collector is down.
+- [Instrumentation and export](./concepts/instrumentation-and-export.md): why
+  the exporter is one layer and what is already instrumented without it.
+- [Forward logs to the run journal](./guides/forward-logs-to-the-journal.md):
+  the second delivery path, durable and readable in process.

package/docs/troubleshooting.md

@@ -0,0 +1,168 @@
+---
+title: "Troubleshooting"
+description: "The typed refusals @smthrs/observability reports and the silent failures it cannot: nothing arriving at a collector, a bundler that cannot resolve async_hooks, logs that stopped, and a counter that reads zero."
+---
+
+Two kinds of problem show up here. A typed refusal names its own cause and
+happens while the layer is built. A silent failure is the harder one, because
+export is absorbed by design and a broken exporter looks exactly like a working
+one.
+
+## InvalidExporterEndpoint
+
+**What happened.** A builder was handed a collector endpoint it will not
+export to. The failure carries `code` `invalid_exporter_endpoint`, the `path`
+of the option it arrived on (`baseUrl` for `Otlp`, `endpoint` for `NodeOtel`),
+and a message. The rejected value is not in the error.
+
+**What to change.** Pass an absolute `http:` or `https:` URL of at most 2,048
+characters that carries no username or password and no spaces or control
+characters. The usual culprits, in order:
+
+- A trailing newline off a config file, or a leading space off a paste. Trim
+  the value. This one is refused rather than repaired precisely because the
+  URL parser would repair it into a URL nobody typed.
+- A missing scheme: `collector:4318` instead of `http://collector:4318`.
+- A signal path already appended. Pass the base URL; the builder adds
+  `/v1/traces`, `/v1/metrics`, and `/v1/logs`.
+- Credentials in the URL. Move them to the `headers` option.
+
+## InvalidResourceConfiguration
+
+**What happened.** The service identity or the resource attributes failed
+decoding. The failure carries `code` `invalid_resource_configuration` and a
+`path` naming the offending field, such as `serviceName` or an attribute key.
+
+**What to change.** Check the value against the bounds: a non-empty service
+name and version of at most 1,024 UTF-16 code units, at most 256 attributes,
+attribute values that are strings of at most 65,536 code units, finite numbers,
+booleans, or homogeneous arrays of those with at most 256 elements. The complete
+resource must fit 128 KiB of OTLP JSON, including keys, identity, the SDK-added
+`telemetry.sdk.name` and `telemetry.sdk.language` fields, wrappers,
+escaping and UTF-8. Many individually valid values can exceed this budget;
+remove or shorten attributes. The refusal names `attributes`, the measured
+lower bound in bytes and the limit, without retaining values. NUL and unpaired surrogates are
+refused anywhere. An empty `serviceName` is the common one, usually an
+environment variable that was not set.
+
+## InvalidJournalLoggerOptions
+
+**What happened.** `JournalLogger.layerJournalForwarding` was given a run id or
+a capacity it will not start a worker for. The failure carries `code`
+`invalid_journal_logger_options` and the offending `path`.
+
+**What to change.** Decode the run id through `JournalEvent.RunId` rather than
+casting a string, and keep `capacity` between 1 and 65,536. The refusal happens
+before any record is queued, so nothing was silently dropped.
+
+## Nothing arrives at the collector
+
+**What happened.** The layer built, the program ran, and the collector holds
+nothing. Export failure is absorbed by design, so this is not reported anywhere
+by default.
+
+**What to check, in order.**
+
+1. **Did the scope close?** Exporters batch. A short program delivers on the
+   shutdown flush, so read the collector only after `Effect.scoped` completes.
+   A program killed with `SIGKILL` never flushes.
+2. **Is the base URL what you think?** Signals post below it, so
+   `http://collector:4318` posts to `http://collector:4318/v1/traces`. If your
+   collector expects a prefix, include it in `baseUrl`.
+3. **Is the collector answering?** Run at `Debug`. The exporter logs
+   `Disabling exporter for 60 seconds` when it gives up after three retries.
+4. **Is the layer actually provided?** A composition that swapped in
+   `Otlp.layerNoop` behaves exactly like a healthy one that delivers nothing.
+
+## An OTLP export batch was discarded
+
+A `Warn` record with code `otlp_export_discarded` means the transport discarded
+a batch because it exceeded 1 MiB (`oversized`), stalled for ten seconds
+(`stalled`), or found all four request slots occupied (`saturated`). The
+`flows_observability_otlp_dropped` counter records every discarded batch. The
+warning carries the request byte size, limit and running drop count, and is
+limited to one per minute per transport.
+
+Warnings go only to the loggers captured before exporter acquisition, without
+logging into this exporter. Keep an ambient sink outside OTLP at level `Warn`
+or lower: it remains useful even when the drop counter cannot reach the
+collector. An empty ambient logger set or a higher minimum level suppresses
+warnings. Reduce unusually large records or metric cardinality for oversized
+batches; check collector availability and latency for stalls and saturation.
+The resource budget reserves 888 KiB for ordinary batches, but cannot bound
+application record sizes.
+
+## Log records stopped reaching the collector
+
+**What happened.** Spans and metric series still arrive, and logs do not, after
+a `Logger` layer was added.
+
+**What to change.** `Otlp` adds its log exporter to the ambient logger set, and
+a `Logger` layer with the default `mergeWithExisting: false` replaces that set.
+Whether the exporter survives depends on which layer the composition builds
+last, which is not worth reasoning about: pass `mergeWithExisting: true` on the
+`Logger` layer whenever another sink is installed. See
+[Install a logger](./guides/install-a-logger.md).
+
+## A bundler cannot resolve `async_hooks`
+
+**What happened.** A browser or edge build failed on `async_hooks`,
+`node:async_hooks`, or a module under `@opentelemetry/context-async-hooks`.
+
+**What to change.** Something in the graph imports
+`@smthrs/observability/NodeOtel`, directly or through a module that does.
+`NodeOtel` binds the Node OpenTelemetry SDK and cannot bundle for a browser.
+Use `Otlp.layerFetch` for browser delivery, or `BrowserOtel` when you need the
+web SDK, and keep the `NodeOtel` import behind a Node-only entry point. The
+root entry point itself resolves no `node:` built-in and bundles for a browser
+as it is.
+
+## `Metric.value` reads zero
+
+**What happened.** The system is clearly doing work, and a counter reads zero.
+
+**What to check.**
+
+- **A dimensioned counter read through its bare handle.** Some packages update
+  only an attribute-tagged series, so the attribute-less parent stays at zero.
+  Read the exported view, for example `EngineStoreMetrics.dispatch.Success`
+  from [`@smthrs/engine-store`](/api/engine-store).
+- **A registry provided per test.** Providing `Metric.MetricRegistry` with a
+  fresh map isolates state; a read outside that scope sees a different
+  registry.
+- **A gauge read as a counter.** A gauge's value is `value`, a counter's is
+  `count`.
+
+## `droppedLogRecords` keeps climbing
+
+**What happened.** `flows_observability_log_dropped` advances during a run,
+which means log records are not reaching the journal.
+
+**What to change.** The counter advances for three distinct losses, and the
+last two also emit a warning annotated with the run id:
+
+- **Queue overflow**, with no warning. Raise `capacity` above the default 256,
+  up to 65,536.
+- **A journal delivery failure.** Read the warning; the journal itself is the
+  problem, not the logger.
+- **A defect from the journal implementation.** Read the warning. The worker
+  keeps draining, so the run continues with gaps.
+
+Records queued behind an in-flight write can also be dropped when the layer's
+scope closes, and the forwarder exposes no drain barrier. `journal.flush`
+settles only the entries the journal has already admitted, so it returns while
+a record is still waiting in the forwarder's queue. Poll the journal for the
+records you expect while the layer's scope is open, as
+[Forward logs to the run journal](./guides/forward-logs-to-the-journal.md#assert-on-a-short-lived-run)
+shows.
+
+## Reading a refusal's path
+
+`path` is a dotted route into the value that failed, built from the decoder's
+own issue path. `serviceName` names the identity field;
+`attributes.region` names the `region` entry inside the `attributes` record. When the decoder cannot attribute the failure to a field,
+the path is the bare `resource`, which means the whole configuration was
+malformed: `null`, or not an object at all.
+
+Neither refusal carries the rejected value, so pair the path with the value you
+passed on that field.

package/package.json

@@ -1,48 +1,237 @@
 {
   "name": "@smthrs/observability",
-  "version": "0.35.0",
-  "description": "Concrete Smithers metrics, logging, tracing, and observability integrations",
   "type": "module",
-  "sideEffects": false,
+  "version": "1.0.0-rc.4",
+  "license": "MIT",
+  "smthrs": {
+    "group": "engine"
+  },
+  "description": "Observability for flows: default OTLP export wiring plus Effect-native logger, metric, and OpenTelemetry SDK layers",
+  "homepage": "https://github.com/smithersai/smithers",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/smithersai/smithers.git",
+    "directory": "packages/smithers/flows/observability"
+  },
+  "bugs": {
+    "url": "https://github.com/smithersai/smithers/issues"
+  },
+  "tags": [
+    "typescript",
+    "effect",
+    "observability",
+    "opentelemetry"
+  ],
+  "keywords": [
+    "typescript",
+    "effect",
+    "observability",
+    "opentelemetry"
+  ],
+  "engines": {
+    "node": ">=26.4.0"
+  },
+  "sideEffects": [],
   "exports": {
+    "./package.json": "./package.json",
     ".": {
-      "types": "./src/index.d.ts",
-      "import": "./src/index.js",
-      "default": "./src/index.js"
-    },
-    "./metrics": {
-      "types": "./src/metrics/index.d.ts",
-      "import": "./src/metrics/index.js",
-      "default": "./src/metrics/index.js"
-    },
-    "./*": {
-      "types": "./src/index.d.ts",
-      "import": "./src/*.js",
-      "default": "./src/*.js"
+      "import": {
+        "types": "./dist/esm/index.d.ts",
+        "default": "./dist/esm/index.js"
+      },
+      "require": {
+        "types": "./dist/cjs/index.d.ts",
+        "default": "./dist/cjs/index.js"
+      }
+    },
+    "./BrowserOtel": {
+      "import": {
+        "types": "./dist/esm/BrowserOtel.d.ts",
+        "default": "./dist/esm/BrowserOtel.js"
+      },
+      "require": {
+        "types": "./dist/cjs/BrowserOtel.d.ts",
+        "default": "./dist/cjs/BrowserOtel.js"
+      }
+    },
+    "./Endpoint": {
+      "import": {
+        "types": "./dist/esm/Endpoint.d.ts",
+        "default": "./dist/esm/Endpoint.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Endpoint.d.ts",
+        "default": "./dist/cjs/Endpoint.js"
+      }
+    },
+    "./JournalLogger": {
+      "import": {
+        "types": "./dist/esm/JournalLogger.d.ts",
+        "default": "./dist/esm/JournalLogger.js"
+      },
+      "require": {
+        "types": "./dist/cjs/JournalLogger.d.ts",
+        "default": "./dist/cjs/JournalLogger.js"
+      }
+    },
+    "./Logger": {
+      "import": {
+        "types": "./dist/esm/Logger.d.ts",
+        "default": "./dist/esm/Logger.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Logger.d.ts",
+        "default": "./dist/cjs/Logger.js"
+      }
+    },
+    "./Metric": {
+      "import": {
+        "types": "./dist/esm/Metric.d.ts",
+        "default": "./dist/esm/Metric.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Metric.d.ts",
+        "default": "./dist/cjs/Metric.js"
+      }
+    },
+    "./NodeOtel": {
+      "import": {
+        "types": "./dist/esm/NodeOtel.d.ts",
+        "default": "./dist/esm/NodeOtel.js"
+      },
+      "require": {
+        "types": "./dist/cjs/NodeOtel.d.ts",
+        "default": "./dist/cjs/NodeOtel.js"
+      }
+    },
+    "./Otel": {
+      "import": {
+        "types": "./dist/esm/Otel.d.ts",
+        "default": "./dist/esm/Otel.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Otel.d.ts",
+        "default": "./dist/cjs/Otel.js"
+      }
+    },
+    "./Otlp": {
+      "import": {
+        "types": "./dist/esm/Otlp.d.ts",
+        "default": "./dist/esm/Otlp.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Otlp.d.ts",
+        "default": "./dist/cjs/Otlp.js"
+      }
+    },
+    "./Resource": {
+      "import": {
+        "types": "./dist/esm/Resource.d.ts",
+        "default": "./dist/esm/Resource.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Resource.d.ts",
+        "default": "./dist/cjs/Resource.js"
+      }
+    },
+    "./index": {
+      "import": {
+        "types": "./dist/esm/index.d.ts",
+        "default": "./dist/esm/index.js"
+      },
+      "require": {
+        "types": "./dist/cjs/index.d.ts",
+        "default": "./dist/cjs/index.js"
+      }
     }
   },
   "files": [
-    "src/",
-    "docker-compose.otel.yml",
-    "otel-collector-config.yml",
-    "prometheus/",
-    "tempo/",
-    "loki/",
-    "grafana/"
+    "src/**/*.ts",
+    "dist/**/*.js",
+    "dist/**/*.js.map",
+    "dist/**/*.d.ts",
+    "dist/**/*.d.ts.map",
+    "dist/**/package.json",
+    "LICENSE",
+    "README.md",
+    "CHANGELOG.md",
+    "docs/"
   ],
+  "publishConfig": {
+    "access": "public",
+    "provenance": true,
+    "tag": "next"
+  },
   "dependencies": {
-    "@effect/opentelemetry": "4.0.0-beta.105",
-    "@effect/platform-bun": "4.0.0-beta.105",
-    "@effect/platform-node-shared": "4.0.0-beta.105",
-    "effect": "4.0.0-beta.105"
+    "@opentelemetry/sdk-logs": "0.222.0",
+    "@opentelemetry/sdk-metrics": "2.11.0",
+    "@smthrs/journal": "1.0.0-rc.4"
   },
   "devDependencies": {
-    "@types/bun": "latest",
-    "typescript": "~6.0.3"
+    "@effect/language-service": "0.87.2",
+    "@effect/opentelemetry": "4.0.0-rc.115",
+    "@effect/vitest": "4.0.0-rc.115",
+    "@eslint/js": "9.39.5",
+    "@opentelemetry/api": "1.9.1",
+    "@opentelemetry/exporter-logs-otlp-http": "0.222.0",
+    "@opentelemetry/exporter-metrics-otlp-http": "0.222.0",
+    "@opentelemetry/exporter-trace-otlp-http": "0.222.0",
+    "@opentelemetry/sdk-trace-base": "2.11.0",
+    "@opentelemetry/sdk-trace-node": "2.11.0",
+    "@opentelemetry/sdk-trace-web": "2.11.0",
+    "@types/node": "26.4.1",
+    "@vitest/coverage-v8": "5.0.0",
+    "dprint": "0.57.1",
+    "effect": "4.0.0-rc.115",
+    "esbuild": "0.28.2",
+    "eslint": "9.39.5",
+    "eslint-import-resolver-typescript": "4.4.5",
+    "eslint-plugin-import": "2.32.0",
+    "eslint-plugin-jsdoc": "64.3.4",
+    "eslint-plugin-unicorn": "65.0.0",
+    "madge": "8.0.0",
+    "typescript": "5.9.3",
+    "typescript-eslint": "8.69.0",
+    "vitest": "5.0.0"
+  },
+  "peerDependencies": {
+    "@effect/opentelemetry": "4.0.0-rc.115",
+    "@opentelemetry/api": ">=1.9.1 <2.0.0",
+    "@opentelemetry/exporter-logs-otlp-http": ">=0.222.0 <0.300.0",
+    "@opentelemetry/exporter-metrics-otlp-http": ">=0.222.0 <0.300.0",
+    "@opentelemetry/exporter-trace-otlp-http": ">=0.222.0 <0.300.0",
+    "@opentelemetry/sdk-trace-base": ">=2.11.0 <3.0.0",
+    "@opentelemetry/sdk-trace-node": ">=2.11.0 <3.0.0",
+    "@opentelemetry/sdk-trace-web": ">=2.11.0 <3.0.0",
+    "effect": "4.0.0-rc.115"
+  },
+  "peerDependenciesMeta": {
+    "@opentelemetry/exporter-logs-otlp-http": {
+      "optional": true
+    },
+    "@opentelemetry/exporter-metrics-otlp-http": {
+      "optional": true
+    },
+    "@opentelemetry/exporter-trace-otlp-http": {
+      "optional": true
+    },
+    "@opentelemetry/sdk-trace-base": {
+      "optional": true
+    },
+    "@opentelemetry/sdk-trace-node": {
+      "optional": true
+    },
+    "@opentelemetry/sdk-trace-web": {
+      "optional": true
+    }
   },
   "scripts": {
-    "build": "rm -f src/index.d.ts src/metrics/index.d.ts src/*-*.d.ts && tsup --dts-only",
-    "test": "bun test tests",
-    "typecheck": "tsc -p tsconfig.json --noEmit"
+    "lint": "eslint src --max-warnings=0 && dprint check",
+    "format": "dprint fmt",
+    "build": "node scripts/build.mjs",
+    "check": "tsc -b tsconfig.json && tsc -p tsconfig.test.json --noEmit",
+    "circular": "node scripts/circular.mjs",
+    "test": "vitest",
+    "coverage": "vitest --coverage"
   }
 }

package/src/BrowserOtel.ts

@@ -0,0 +1,51 @@
+/**
+ * Browser OpenTelemetry setup with explicitly injected providers/readers.
+ *
+ * @since 0.1.0
+ */
+
+import * as WebSdk from "@effect/opentelemetry/WebSdk"
+import type { LogRecordProcessor } from "@opentelemetry/sdk-logs"
+import type { MetricReader } from "@opentelemetry/sdk-metrics"
+import type { SpanProcessor } from "@opentelemetry/sdk-trace-base"
+import * as Effect from "effect/Effect"
+import * as Layer from "effect/Layer"
+import * as Resource from "./Resource.ts"
+import type { Configuration as ResourceConfiguration } from "./Resource.ts"
+
+/**
+ * Browser OTEL layer options. Processors and readers are created by the
+ * caller's factories, which keeps this module free of Node imports and browser
+ * exporter policy. Every factory is owned: it runs once during layer
+ * acquisition, after resource validation, and the layer scope shuts down
+ * everything it returned. An empty array installs nothing for that signal.
+ *
+ * @category models
+ * @since 0.1.0
+ */
+export interface Options {
+  readonly resource: ResourceConfiguration
+  readonly spanProcessor?: (() => SpanProcessor | ReadonlyArray<SpanProcessor>) | undefined
+  readonly logRecordProcessor?: (() => LogRecordProcessor | ReadonlyArray<LogRecordProcessor>) | undefined
+  readonly metricReader?: (() => MetricReader | ReadonlyArray<MetricReader>) | undefined
+  readonly loggerMergeWithExisting?: boolean | undefined
+}
+
+/**
+ * Builds a scoped browser OpenTelemetry layer. The web SDK owns every
+ * processor and reader the factories return and shuts them down on release.
+ *
+ * @category layers
+ * @since 0.1.0
+ */
+export const layerOtel = (options: Options): Layer.Layer<never, Resource.InvalidResourceConfiguration> =>
+  Layer.unwrap(
+    Effect.map(Resource.decode(options.resource), (decoded) =>
+      WebSdk.layer(() => ({
+        resource: Resource.toOpenTelemetryConfiguration(decoded),
+        spanProcessor: options.spanProcessor?.(),
+        logRecordProcessor: options.logRecordProcessor?.(),
+        metricReader: options.metricReader?.(),
+        loggerMergeWithExisting: options.loggerMergeWithExisting
+      })))
+  )