@objectstack/observability 17.0.0 → 17.2.0

package/CHANGELOG.md

@@ -1,5 +1,321 @@
 # @objectstack/observability
 
+## 17.2.0
+
+### Minor Changes
+
+- 914c413: fix(observability): **BREAKING** — `http_request_errors_total` is retired (ADR-0049 enforce-or-remove, #9834)
+  
+  **⛔ If you have a Grafana panel, an alert rule or a recording rule keyed on
+  `http_request_errors_total`, it will read a FLAT ZERO after this upgrade.** That
+  zero is the removal, not a healthy server, and it is the one way this change can
+  hurt you — nothing throws, nothing warns, the series simply stops receiving
+  samples. Rewrite the query before you deploy.
+  
+  Maintainer ruling 2026-08-20: **RETIRE**. The name was declared in `SEMCONV` as
+  part of a stable namespace *"so hosts can wire alerts/dashboards against it"*,
+  but the only emitter was `@objectstack/runtime`'s `instrumentRouteHandler`,
+  applied only by the dispatcher's own route Proxy — so the series never saw
+  auth's `getRawApp()` mount, the REST data API via `RouteManager`, or any other
+  inbound surface. Its two siblings in the same HTTP family moved to the
+  `IHttpServer.afterResponse` transport seam (`http_requests_total`, #9650/#9835;
+  `http_request_duration_ms`, #9834/#10004) and this one could not follow:
+  `HttpResponseObservation` carries `{method, routePattern, status, elapsedMs}`
+  and **no throw signal of any kind**, so every transport-side shape would have
+  counted a *different* population rather than the same one more widely.
+  
+  Migration (FROM → TO):
+  
+  | Wrote | Write instead |
+  |---|---|
+  | `rate(http_request_errors_total[5m])` in a panel or alert | `rate(http_requests_total{status=~"5.."}[5m])` — emitted by the transport, so it covers every inbound surface instead of the dispatcher's routes only |
+  | `sum by (route) (http_request_errors_total)` | `sum by (route) (http_requests_total{status=~"5.."})` |
+  | `SEMCONV.httpRequestErrorsTotal` / `RUNTIME_METRICS.httpRequestErrorsTotal` in host code | Delete the read. Both members are gone; `tsc` reports the missing property at the read site. |
+  
+  One-line fix: replace the metric name with `http_requests_total{status=~"5.."}`.
+  
+  <!-- adr-0087: registered http-request-errors-total-retired -->
+  
+  **The replacement is wider, not merely different.** The retired counter was
+  divergent from a 5xx rate in *both* directions, measured: the dispatcher answers
+  its own errors through `errorResponseBase`, which sets a status and does **not**
+  re-throw — so the counter **missed** those — while its `catch` incremented
+  unconditionally, so a **thrown 4xx WAS counted** as an error. And
+  `http_requests_total` already carries a `status` label, so a status-class error
+  counter was fully derivable from data the transport already publishes. Prove the
+  new query wider rather than merely non-empty: make an auth route or a REST
+  data-API route answer 5xx and confirm it moves, where the retired counter would
+  not have moved at all.
+  
+  **If what you were actually alerting on was "a handler threw rather than
+  returning an error envelope"** — the one signal this counter uniquely carried —
+  that is the `errorReporter`, not a metric. Wire an `ErrorReporter` adapter
+  (Sentry / Datadog / your own); it still fires on every 5xx throw and is
+  untouched by this change.
+  
+  What is NOT removed: `http_requests_total`, `http_request_duration_ms`,
+  request-id propagation, the 5xx error reporter, and the
+  `res.__obsRecordedError` side channel that carries a swallowed error to it. The
+  dispatcher still instruments every route it mounts; it just no longer publishes
+  a fourth series whose name promised more coverage than it had.
+
+### Patch Changes
+
+- Updated dependencies [6936d07]
+- Updated dependencies [59eb04d]
+- Updated dependencies [9f05b7d]
+- Updated dependencies [7d2d112]
+- Updated dependencies [5fa0d72]
+- Updated dependencies [02b3b07]
+- Updated dependencies [914c413]
+- Updated dependencies [55809a0]
+- Updated dependencies [52db1d1]
+- Updated dependencies [5649efb]
+- Updated dependencies [2306a76]
+- Updated dependencies [e5ea701]
+- Updated dependencies [a40dcc1]
+- Updated dependencies [def0d3e]
+- Updated dependencies [8d0bb79]
+- Updated dependencies [5acb58d]
+- Updated dependencies [2e3cf95]
+- Updated dependencies [4c93387]
+- Updated dependencies [a037f7c]
+- Updated dependencies [3ee8ddf]
+- Updated dependencies [16cef97]
+- Updated dependencies [a79bd35]
+- Updated dependencies [6ceaa4b]
+- Updated dependencies [15ea214]
+- Updated dependencies [de19489]
+- Updated dependencies [c684d00]
+- Updated dependencies [923c424]
+- Updated dependencies [1ec36b7]
+- Updated dependencies [5f2e54c]
+- Updated dependencies [189373b]
+- Updated dependencies [35ad101]
+- Updated dependencies [ceb33a9]
+- Updated dependencies [73d9795]
+- Updated dependencies [8012960]
+- Updated dependencies [f34f56b]
+- Updated dependencies [f399618]
+- Updated dependencies [75e9301]
+- Updated dependencies [2810695]
+  - @objectstack/spec@17.2.0
+
+## 17.1.0
+
+### Patch Changes
+
+- 899052a: docs(observability): record what a zero on `cache_*` MEANS — "no configured consumer", not "no cache activity" (#9954)
+  
+  `SEMCONV` declares the `cache_*` families as a stable namespace explicitly so
+  hosts can wire alerts and dashboards against it. An operator who does that gets
+  a flat zero on `cache_lookups_total` in a default install — and nothing in the
+  declaration or the operator docs said why, so the only available readings were
+  "0% hit rate" or "the adapter is broken". Both are wrong.
+  
+  **Nothing about emission changes.** The path is proven working: #9832 wired the
+  cache adapter to the host's registry and #9951 pins a real lookup observing
+  `cache_lookups_total{adapter=memory,result=miss}`. The zero is *true*; what it
+  failed to communicate is its cause.
+  
+  The cause, re-measured on `origin/main` rather than taken from the card: no
+  consumer of the `cache` service is unconditional. Every production consumer is
+  a rate-limit or budget counter store, and each is gated on a declaration
+  somebody has to write:
+  
+  - `packages/plugins/plugin-auth/src/auth-plugin.ts` — better-auth's per-IP
+    counters, reached only when `rate_limit_max` or `rate_limit_window_seconds`
+    is explicitly supplied in auth settings.
+  - `packages/runtime/src/dispatcher-plugin.ts` — the inbound rate limiter and
+    the declarative per-endpoint buckets. Both register *nothing at all* when no
+    budget is declared (`createInboundRateLimitMiddleware` returns `null`;
+    `limiterFor` returns `null` on an endpoint with no armed `rateLimit`), so an
+    unmetered deployment never reaches the cache.
+  - the per-number OTP send budget, reached only on an SMS send path.
+  
+  Declare none of them — the default slate — and the family sits at 0 while the
+  server handles traffic normally.
+  
+  So the annotation states the invariant rather than a roster: *every* consumer
+  is an explicitly-declared rate-limit/budget counter store. That sentence stays
+  true when another conditional consumer is added, and goes false exactly when an
+  unconditional one appears — which is when it should be revisited.
+  
+  This is the mirror image of the HTTP note that landed alongside it: there a
+  zero means "not instrumented"; here a zero is a true count of a service nothing
+  asked anything of. The two are deliberately worded so they cannot be read as
+  the same statement.
+  
+  The note is attached as per-family JSDoc rather than a `//` group header, and
+  that placement is load-bearing: `tsup`'s dts generation drops line comments, so
+  only the JSDoc form reaches `dist/index.d.ts` — i.e. the operator's IDE hover,
+  which is where somebody wiring a `cache_lookups_total` panel actually meets the
+  declaration. Verified in the built artifact, not assumed.
+- 1e050a5: fix(observability): emit `http_request_duration_ms` from the transport seam, so
+  p95 latency sees every inbound surface instead of dispatcher routes only
+  (#9834)
+  
+  #9835 moved `http_requests_total` to the `IHttpServer.afterResponse` seam and
+  stopped there, which left the two derived signals the operator guidance names
+  inconsistent with each other: 5xx rate covered auth's `getRawApp()` mount and
+  the REST data API, while p95 latency still saw only the routes the dispatcher's
+  own `Proxy` wrapped. A missing latency panel is at least loud; the worse
+  reading is the p95 that IS drawn, computed from dispatcher routes only and
+  presented as the server's.
+  
+  - `@objectstack/observability`: new `armHttpRequestDurationHistogram(server,
+    metrics)`, the duration family's counterpart to `armHttpRequestCounter` —
+    same `afterResponse` seam, its own `Symbol.for` first-wins latch, so a host
+    that armed one family can still arm the other. `ArmHttpMetricResult` is the
+    family-neutral spelling of the result union; `ArmHttpRequestCounterResult`
+    stays as an alias.
+  - `@objectstack/plugin-hono-server`: `installHttpMetricsSeam` arms both
+    families, so the transport owns every transport-observable HTTP metric in the
+    shipped composition rather than splitting ownership across layers.
+  - `@objectstack/runtime`: the dispatcher offers its registry to the histogram
+    seam as well, and `instrumentRouteHandler` gains
+    `emitHttpRequestDurationMs` (default `true`) — passed `false` exactly when
+    the transport implements the seam, which is what keeps the dispatcher's own
+    routes at ONE observation instead of reintroducing the #9833 double count one
+    family over.
+  
+  **⚠️ The observation window changes with the emitter.** The per-route wrapper
+  timed `await handler(req, res)` — handler latency. The transport times from
+  first seeing the request to the response existing, so the middleware chain and
+  body parse are now included and samples can only move UP. This is the number a
+  latency panel should show (the request's latency, not one layer's share of it),
+  but it is a visible shift in an existing series: compare p95 across the upgrade
+  boundary deliberately. Documented in `docs/OBSERVABILITY.md` and the
+  production-readiness guide.
+  
+  `http_request_errors_total` is deliberately NOT moved. The observation carries
+  no throw signal — only `{method, routePattern, status, elapsedMs}` — so a
+  transport-side emitter would have to key off a status class, which counts a
+  different population than "the handler threw". That is a semantics decision,
+  recorded on #9834 rather than guessed at here; the counter stays ungated on
+  every transport and keeps its documented meaning.
+- 7ff3975: feat(spec): `IHttpServer` gains an optional `afterResponse` response-observing
+  hook so HTTP metrics are transport-agnostic instead of Hono-only (#9835)
+  
+  The contract addition (additive — a new optional member plus the
+  `HttpResponseObservation` / `HttpResponseObserver` types and the reserved
+  `UNMATCHED_ROUTE_PATTERN` label): a transport invokes each registered observer
+  exactly once per answered request with `{ method, routePattern, status,
+  elapsedMs }`, after the response exists — the observation point the `use()`
+  middleware contract cannot express (it runs before dispatch and never sees a
+  status). `routePattern` is REQUIRED to be the registered route pattern
+  (`/api/v1/data/:id`), never the concrete path, so no adapter re-decides metric
+  cardinality. Optionality is feature-detected runtime-real
+  (`typeof server.afterResponse === 'function'`); a transport that does not
+  implement the seam reports **no** HTTP metrics — zero there means "not
+  instrumented", never "no traffic".
+  
+  Implementations and consumers in the same change:
+  
+  - `@objectstack/plugin-hono-server`: `HonoHttpServer` implements the seam (the
+    ruled #9650 raw-app middleware becomes its delivery path — same reach,
+    including `getRawApp()` mounts and middleware-refused 429s); unrouted
+    requests are now labelled with the reserved `unmatched` pattern (previously
+    they could surface as `/*`).
+  - `@objectstack/observability`: new `armHttpRequestCounter(server, metrics)`
+    arms the `http_requests_total` counter through the seam at most once per
+    server (first caller wins), which is what makes "exactly one counter per
+    server" structural.
+  - `@objectstack/runtime`: the dispatcher offers its `observability.metrics`
+    registry to the seam (a host that wires only the dispatcher now counts every
+    inbound surface) and suppresses its own per-route copy of
+    `http_requests_total` when the transport implements the seam — retiring the
+    #9833 double count. Request-id echo, the duration histogram, the error
+    counter and the error reporter are unchanged.
+  - `@objectstack/http-conformance`: `NodeHttpServer` implements the seam, and a
+    new cross-adapter conformance suite locks the semantics for both adapters.
+  - `@objectstack/core`: re-exports the new contract types/constant.
+- Updated dependencies [56656aa]
+- Updated dependencies [07e630e]
+- Updated dependencies [2f65b1b]
+- Updated dependencies [720ee95]
+- Updated dependencies [f287435]
+- Updated dependencies [9aa8890]
+- Updated dependencies [7c9c1dd]
+- Updated dependencies [75b7c24]
+- Updated dependencies [d5552ca]
+- Updated dependencies [d9813a9]
+- Updated dependencies [8640fb2]
+- Updated dependencies [2420641]
+- Updated dependencies [2ad91c3]
+- Updated dependencies [f57fb38]
+- Updated dependencies [00777a0]
+- Updated dependencies [d491625]
+- Updated dependencies [420804d]
+- Updated dependencies [716ac9b]
+- Updated dependencies [62b1427]
+- Updated dependencies [7ea1372]
+- Updated dependencies [23abe27]
+- Updated dependencies [985a9cd]
+- Updated dependencies [a8189ae]
+- Updated dependencies [26e70fb]
+- Updated dependencies [42b05af]
+- Updated dependencies [2b292ce]
+- Updated dependencies [abcf853]
+- Updated dependencies [8b9eba5]
+- Updated dependencies [d575779]
+- Updated dependencies [94f7ef8]
+- Updated dependencies [c5ac5e4]
+- Updated dependencies [a777944]
+- Updated dependencies [dd88e1c]
+- Updated dependencies [856527c]
+- Updated dependencies [870f710]
+- Updated dependencies [79c46da]
+- Updated dependencies [7ff3975]
+- Updated dependencies [29d055b]
+- Updated dependencies [65589d6]
+- Updated dependencies [2c86fe3]
+- Updated dependencies [e196c6a]
+- Updated dependencies [4ab7523]
+- Updated dependencies [19539b4]
+- Updated dependencies [11b779e]
+- Updated dependencies [739fe5b]
+- Updated dependencies [4bfe1a5]
+- Updated dependencies [2065e31]
+- Updated dependencies [b69d0f5]
+- Updated dependencies [4d47afe]
+- Updated dependencies [e4e5c6e]
+- Updated dependencies [9a56784]
+- Updated dependencies [d00d2f6]
+- Updated dependencies [df0c12d]
+- Updated dependencies [d31785f]
+- Updated dependencies [c308a4f]
+- Updated dependencies [e2899f6]
+- Updated dependencies [3851f87]
+- Updated dependencies [2a29caa]
+- Updated dependencies [09a6eee]
+- Updated dependencies [1a7f907]
+- Updated dependencies [cd455c8]
+- Updated dependencies [30d3752]
+- Updated dependencies [c80e7ae]
+- Updated dependencies [09a9a8a]
+- Updated dependencies [07026cf]
+- Updated dependencies [5d4f3d5]
+- Updated dependencies [4d80e8b]
+- Updated dependencies [30b1c63]
+- Updated dependencies [079b457]
+- Updated dependencies [e43b211]
+- Updated dependencies [890b38f]
+- Updated dependencies [8bee54b]
+- Updated dependencies [7a537ce]
+- Updated dependencies [593c4bf]
+- Updated dependencies [ff08691]
+- Updated dependencies [60e0f90]
+- Updated dependencies [90c5285]
+- Updated dependencies [7901b2d]
+- Updated dependencies [56bca91]
+- Updated dependencies [79394d7]
+- Updated dependencies [730fd9a]
+- Updated dependencies [44bc51d]
+- Updated dependencies [73cfddf]
+- Updated dependencies [d634e66]
+  - @objectstack/spec@17.1.0
+
 ## 17.0.0
 
 ### Patch Changes

package/dist/index.cjs

@@ -37,6 +37,8 @@ __export(index_exports, {
   RUNTIME_METRICS: () => RUNTIME_METRICS,
   SEMCONV: () => SEMCONV,
   allowPerfDisclosure: () => allowPerfDisclosure,
+  armHttpRequestCounter: () => armHttpRequestCounter,
+  armHttpRequestDurationHistogram: () => armHttpRequestDurationHistogram,
   countServerTiming: () => countServerTiming,
   currentPerfTiming: () => currentPerfTiming,
   formatServerTiming: () => formatServerTiming,
@@ -59,16 +61,35 @@ var OBSERVABILITY_ERRORS_SERVICE = "observability:errors";
 
 // src/semconv.ts
 var SEMCONV = {
-  // ── HTTP — emitted by `@objectstack/runtime`'s instrumentRouteHandler ──
-  /** Counter, labels: `method`, `route`, `status`. */
+  // ── HTTP — emitter differs per family, see each ────────────────────
+  /**
+   * Counter, labels: `method`, `route`, `status`. Emitted by the TRANSPORT
+   * through the `IHttpServer.afterResponse` seam (#9835), so it covers
+   * every inbound request on the server rather than only the routes the
+   * runtime dispatcher registers.
+   */
   httpRequestsTotal: "http_requests_total",
-  /** Histogram (ms), labels: `method`, `route`. */
-  httpRequestDurationMs: "http_request_duration_ms",
   /**
-   * Counter, labels: `method`, `route`. Incremented when an
-   * in-flight handler throws after the response is sent.
+   * Histogram (ms), labels: `method`, `route`. Emitted by the TRANSPORT
+   * through the same seam (#9834). It measures the REQUEST as the transport
+   * sees it — first sight to the response existing, middleware chain and
+   * body parse included — not the handler's share of it.
    */
-  httpRequestErrorsTotal: "http_request_errors_total",
+  httpRequestDurationMs: "http_request_duration_ms",
+  // ⛔ RETIRED — `http_request_errors_total` was removed in
+  // `@objectstack/observability` 17.2.0 (#9834, ADR-0049 enforce-or-remove).
+  // ⛔ Do not re-add the name. It was DECLARED here as a stable server-wide
+  // signal and EMITTED only from `@objectstack/runtime`'s per-route wrapper,
+  // on a THROWN handler — so it never saw auth's `getRawApp()` mount, the
+  // REST data API, or any error a handler answered politely through
+  // `errorResponseBase` (which sets a status and does not re-throw). No
+  // transport-side emitter could preserve that population either: the
+  // `IHttpServer.afterResponse` observation carries `{method, routePattern,
+  // status, elapsedMs}` and no throw signal at all.
+  // ⇒ Read the 5xx rate from `http_requests_total{status=~"5.."}` instead.
+  // The transport emits that family through the seam, so it covers every
+  // inbound surface (#9650 / #9835 / #10004) and carries the status label
+  // this counter only stood in for. Maintainer ruling 2026-08-20.
   // ── Storage — emitted by `@objectstack/service-storage` adapters ──
   /** Counter, labels: `adapter` (`local`|`s3`|…), `op` (`get`|`put`|`delete`|`head`), `result` (`ok`|`error`). */
   storageOperationsTotal: "storage_operations_total",
@@ -77,11 +98,41 @@ var SEMCONV = {
   /** Counter, labels: `adapter`, `op`, `errorClass`. */
   storageErrorsTotal: "storage_errors_total",
   // ── Cache — emitted by `@objectstack/service-cache` adapters ──
-  /** Counter, labels: `adapter` (`memory`|`redis`), `result` (`hit`|`miss`). */
+  // Uniform emitter; what varies is who CONSULTS the service — see below.
+  /**
+   * Counter, labels: `adapter` (`memory`|`redis`), `result` (`hit`|`miss`).
+   *
+   * ⚠️ A flat zero means "NO CONFIGURED CONSUMER", not "no cache activity",
+   * and — unlike the HTTP families above — it is NOT an instrumentation gap.
+   * The adapters hold the host's registry and count every call they receive
+   * (#9832 wired that; #9951 pins it), so a zero here is TRUE. What it fails
+   * to communicate is WHY.
+   *
+   * The why: nothing consults the `cache` service unconditionally. Every
+   * production consumer is a rate-limit / budget counter store, and each is
+   * gated on a declaration somebody has to write — better-auth's per-IP
+   * counters (`rate_limit_max` / `rate_limit_window_seconds` in auth
+   * settings), the dispatcher's inbound limiter and its declarative
+   * per-endpoint buckets (an armed `rateLimit` budget; with none declared
+   * the dispatcher registers no limiter at all), and the per-number OTP send
+   * budget (an SMS send path). A default install declares none of them, so
+   * this counter stays at 0 while the server handles traffic normally.
+   *
+   * ⇒ Read a flat `cache_*` as a question about CONFIGURATION, never as a 0%
+   * hit rate or a broken adapter. Before trusting a cache hit-rate panel,
+   * confirm at least one consumer above is actually armed.
+   */
   cacheLookupsTotal: "cache_lookups_total",
-  /** Counter, labels: `adapter`, `op` (`set`|`delete`|`clear`). */
+  /**
+   * Counter, labels: `adapter`, `op` (`set`|`delete`|`clear`). Same
+   * "zero = no configured consumer" reading as `cacheLookupsTotal` above.
+   */
   cacheWritesTotal: "cache_writes_total",
-  /** Counter, labels: `adapter`, `op`, `errorClass`. */
+  /**
+   * Counter, labels: `adapter`, `op`, `errorClass`. Same "zero = no
+   * configured consumer" reading as `cacheLookupsTotal` above — a zero is
+   * "nothing was asked of the cache", not "every call succeeded".
+   */
   cacheErrorsTotal: "cache_errors_total",
   // ── Background jobs — emitted by `@objectstack/runtime`'s AppPlugin ──
   /**
@@ -101,10 +152,49 @@ var SEMCONV = {
 };
 var RUNTIME_METRICS = {
   httpRequestsTotal: SEMCONV.httpRequestsTotal,
-  httpRequestDurationMs: SEMCONV.httpRequestDurationMs,
-  httpRequestErrorsTotal: SEMCONV.httpRequestErrorsTotal
+  httpRequestDurationMs: SEMCONV.httpRequestDurationMs
+  // `httpRequestErrorsTotal` retired with its SEMCONV declaration above
+  // (#9834). The alias is not a compatibility window of its own: there is no
+  // emitter left to read, so keeping the name here would hand callers a
+  // string nothing ever writes.
 };
 
+// src/http-transport-metrics.ts
+var HTTP_REQUEST_COUNTER_ARMED = /* @__PURE__ */ Symbol.for(
+  "objectstack.observability.httpRequestCounterArmed"
+);
+function armHttpRequestCounter(server, metrics) {
+  if (typeof server.afterResponse !== "function") return "unsupported";
+  const latched = server;
+  if (latched[HTTP_REQUEST_COUNTER_ARMED]) return "already-armed";
+  latched[HTTP_REQUEST_COUNTER_ARMED] = true;
+  server.afterResponse((observation) => {
+    metrics.counter(RUNTIME_METRICS.httpRequestsTotal, {
+      method: observation.method,
+      route: observation.routePattern,
+      status: String(observation.status)
+    });
+  });
+  return "armed";
+}
+var HTTP_REQUEST_DURATION_ARMED = /* @__PURE__ */ Symbol.for(
+  "objectstack.observability.httpRequestDurationArmed"
+);
+function armHttpRequestDurationHistogram(server, metrics) {
+  if (typeof server.afterResponse !== "function") return "unsupported";
+  const latched = server;
+  if (latched[HTTP_REQUEST_DURATION_ARMED]) return "already-armed";
+  latched[HTTP_REQUEST_DURATION_ARMED] = true;
+  server.afterResponse((observation) => {
+    metrics.histogram(
+      RUNTIME_METRICS.httpRequestDurationMs,
+      observation.elapsedMs,
+      { method: observation.method, route: observation.routePattern }
+    );
+  });
+  return "armed";
+}
+
 // src/metrics-exporters.ts
 var NoopMetricsRegistry = class {
   counter() {
@@ -704,6 +794,8 @@ function isPerfDisclosurePrincipal(ec) {
   RUNTIME_METRICS,
   SEMCONV,
   allowPerfDisclosure,
+  armHttpRequestCounter,
+  armHttpRequestDurationHistogram,
   countServerTiming,
   currentPerfTiming,
   formatServerTiming,