@decocms/start 6.0.1 → 6.1.0

package/MIGRATION_TOOLING_PLAN.md

@@ -121,6 +121,15 @@ this plan.
 | 2026-05-07 | **D6.1 — Cloudflare credentials never leave `deco-start`** | Same-day refinement of D6 after the first central deploy on `baggagio-tanstack` failed with `Secret CLOUDFLARE_API_TOKEN is required, but not provided while calling`. The original D6 design used `secrets: inherit` from the storefront stub and required `CLOUDFLARE_*` to live in the `deco-sites` org, which broke the principle that *the only secrets a storefront repo holds are the secrets that go into wrangler secrets, not the ones used to deploy*. First-pass refinement: the central `deploy.yml` / `preview.yml` / `sync-secrets.yml` jobs declared `environment: production` to try to make `${{ secrets.CLOUDFLARE_* }}` resolve from `decocms/deco-start`'s `production` Environment. **Found broken empirically on 2026-05-07** — the deployment registers in the *caller* repo, not the called workflow's repo, so the environment lookup uses the caller's `production` env (auto-created with no secrets). Superseded by D6.2 the same evening. |
 | 2026-05-07 | **D6.2 — App-mediated dispatch + no per-site registry (supersedes D6 + D6.1)** | After D6.1's `environment:` mechanism was empirically shown not to work cross-repo, the architecture pivoted: a `decocms-deployer` GitHub App is installed on `decocms/deco-start` (`actions:write`) and on each storefront repo (`contents:read`, optionally `pull-requests:write`). The storefront caller stub mints a short-lived App-installation token and calls `gh workflow run deploy.yml --repo decocms/deco-start --ref v3 -f site_owner=… -f site_name=…`. The central workflow runs in `decocms/deco-start`'s context, so `CLOUDFLARE_API_TOKEN` / `CLOUDFLARE_ACCOUNT_ID` are ordinary repo secrets. For runtime `SECRET_*` values, each storefront has a `<site_name>-secrets` GitHub Environment in `decocms/deco-start` (S1 design); `sync-secrets.yml` binds to that environment and pushes to `wrangler secret put`. The per-site registry under `deploy/sites/<repo>.jsonc` was dropped entirely (Pure C): worker name = repo basename by convention; the App being installed on the storefront repo is the deploy authorization gate; rare per-worker derived fields (like AE dataset name) use `$WORKER_*` substitution tokens in the template. Force-rollback is impossible for production deploys because the central workflow ignores caller-supplied `site_sha` and resolves the storefront's current default-branch HEAD itself. See [`deploy/README.md`](./deploy/README.md) for the full trust model. **Operational migrations required by Pure C:** `miess-01-tanstack` repo's worker shifts from `miess-tanstack` to `miess-01-tanstack` (CF-side cutover); `lebiscuit-tanstack` AE dataset shifts from `deco_metrics_lebiscuit` to `deco_metrics_lebiscuit_tanstack` (orphans old data). |
 | 2026-05-07 | **D6.3 — Revert D6/D6.1/D6.2; deploys move to Cloudflare Workers Builds** | The whole D6 family (centralized GitHub Actions reusable workflows + `decocms-deployer` GitHub App + per-storefront GitHub Environments + central `deploy/wrangler-template.jsonc` + `deco-wrangler` CLI + per-site caller stubs) is being **reverted**. Trigger: GitHub Free orgs do not propagate org-level secrets to private repos, which forced the App private key to live as a per-storefront repo secret in every storefront — that key gives the holder the ability to mint installation tokens that can trigger workflows on `decocms/deco-start`, which in turn have the only Cloudflare credentials in the system. Per-repo distribution + rotation of that key across N customer storefronts didn't scale and concentrated blast radius on one credential. **Replacement (chosen, to be detailed in a follow-up D-record once shipped):** [Cloudflare Workers Builds](https://developers.cloudflare.com/workers/ci-cd/builds/) owns the deploy/preview pipelines per-worker. Verified empirically on `baggagio-tanstack` 2026-05-07: a malicious `wrangler.jsonc` `name` field pointing at a different worker (`americanas-tanstack`) is **ignored** by CF Builds — the deploy lands on the connected worker (`baggagio-tanstack`), CF surfaces a warning banner in the dashboard, and CF auto-opens a PR to fix the config (deco-sites/baggagio-tanstack#34). The dashboard repo<->worker connection is the source of truth; the in-repo config is treated as a secondary input. Per-storefront wiring (one CF dashboard click per worker) is acceptable at our scale; revisit when CF's [git-integration enable API](https://github.com/cloudflare/workers-sdk/issues/12058) lands. The `deco-build` CLI (regenerates `wrangler.jsonc` bindings from a central template) and runtime-secrets management remain to be designed in a separate PR. |
+| 2026-05-22 | **D-9 — Stable `request.id` propagation across all observability channels** | The fragmentation surfaced during the May 2026 error triage: tail-worker logs, direct-POST metrics, CF-Destinations spans, and structured `console.log` JSON all carry overlapping information but no shared join key. A single user-visible 5xx required hand-correlating timestamps across four ClickHouse tables. **Decision:** the framework generates a stable `request.id` once at request entry (precedence: inbound `x-request-id` → `cf-ray` → `crypto.randomUUID()`) inside `RequestContext.run`, then stamps it on (a) the root span as the `request.id` attribute, (b) every log line via the logger attribute floor, (c) the response as `X-Request-Id` (read by `deco-otel-tail`), and (d) the metric labels via `extra`. Symmetric for `trace.id` — read from the root span's spanContext, echoed as `X-Trace-Id`. This re-establishes a single join key across all channels: pick any row from any table, filter by `request.id`, and reconstruct the full request lifecycle. **Files:** `RequestContext.requestId` + logger floor + workerEntry response-header echo + tail-worker enrichment. **Captured in Phase 1 of [the observability refinement plan](../../.cursor/plans/observability_refinement_plan_4fa41548.plan.md)**. |
+| 2026-05-22 | **D-10 — Server-side log normalization at the ingest worker, cost-neutral** | CF Destinations wraps every `console.log(JSON.stringify(...))` line into an OTLP LogRecord with the JSON body in `body.stringValue`. Querying by structured fields requires `JSONExtract` everywhere — slow, query-fragile, and tied to whatever the producer happens to embed. **Two design choices considered:** (a) migrate all framework `logger.{info,warn,debug}` calls to direct-POST native OTLP, ditching the CF Destinations sampled path. Substantially more code volume in production direct-POST traffic; bypasses the head sampling that keeps fleet cost bounded. (b) lift the JSON-in-body into native OTLP `LogAttributes` server-side at the `deco-otel-ingest` worker. Same wire volume, same cost. **Decision: option (b).** The ingest worker's `logsToRows` now detects JSON-shaped `body.stringValue`, lifts `level`/`msg`/`trace_id`/`span_id` plus arbitrary keys into native OTLP attributes, reduces `Body` to the human-readable `msg`, and falls back unchanged for non-JSON strings (third-party `console.log`). Dashboards drop `JSONExtractString(Body, 'level') = 'error'` in favor of `SeverityText = 'ERROR'`. **Files:** `stats-lake/ingestion/otel-ingest/src/index.ts`. Phase 4 of the observability refinement plan. |
+| 2026-05-22 | **D-11 — Outcome metrics layer becomes the truth source for "did we serve users today?"** | Earlier metric labels (`method`, `path`, `status`) couldn't answer "5xx rate per route per site" without joining metrics to tail-worker logs. The path label was raw-URL (unbounded cardinality risk); status was opaque (no class bucketing); no cache decision / cache layer; no commerce histogram in the framework (only apps-start sites that bumped to a recent version had it). **Decision:** expand the canonical label set for `http_requests_total` / `http_request_duration_ms` / `http_request_errors_total` to `{ method, route_pattern, status, status_class, outcome?, cache_decision?, cache_layer?, region?, …extra }`. `route_pattern` is the TanStack closed-set pattern (`/_products/$slug/p`); fallback is the normalized path. `status_class` is `2xx`/.../`5xx`/`unknown`. Cache labels lift the existing `X-Cache` / `X-Cache-Profile` headers up to the metric so dashboards answer cache-hit rate per route from the counter alone. Move `commerce_request_duration_ms` declaration into `@decocms/start` so every site emits it as soon as the framework is bumped, regardless of apps-start version (apps register operation strings only). Labels: `{ provider, operation, status_class?, cached? }`. **Files:** `src/middleware/observability.ts` (`statusClassFor`, `RequestMetricLabels`, `CacheLayer`, `recordCommerceMetric`, expanded `recordCacheMetric` signature). Phase 2 of the observability refinement plan. |
+| 2026-05-22 | **D-12 — Direct-POST OTLP trace exporter for framework `deco.*` spans** | Empirical verification (May 2026) confirmed the framework's 10+ `withTracing` calls produced zero rows in `otel_traces`. Root cause: the bridge tracer in `instrumentWorker` delegates to `trace.getTracer(...)` on the `@opentelemetry/api` global. With no `TracerProvider` registered (the common case — CF Workers only auto-installs a provider when `observability.traces.destinations` is set), every framework span is silently discarded. **Decision:** introduce `otelHttpTracer.ts` — a direct-POST OTLP/HTTP trace exporter that mirrors the existing meter + error-log adapters. Same transport: per-isolate buffer, ctx.waitUntil flush, FNV-1a hash sampling at `headSamplingRate` (default 0.01 matches CF Destinations recommendation). Consistent per-trace decision so child spans are kept iff their root is kept. Honors inbound W3C `traceparent` — if the remote parent arrived sampled, every span in that trace is exported regardless of the rate. Wired alongside the existing `@opentelemetry/api` bridge via `configureTracerStack` — CF auto-spans still flow to the CF dashboard, framework spans direct-POST to ClickHouse. Default-on `injectTraceContext` inside `createInstrumentedFetch` was already in place. **Files:** `src/sdk/otelHttpTracer.ts`, `src/sdk/otel.ts` (`configureTracerStack`), `src/sdk/workerEntry.ts` (traceparent parsing). Phase 3 of the observability refinement plan. |
+| 2026-05-22 | **D-13 — Per-site Grafana dashboards + alert rules are auto-provisioned from `dim_sites`** | Hand-built dashboards drift the moment a new site lands: a fleet of 100 sites can't be maintained by a human curator. **Decision:** the canonical observability provisioning lives in [`stats-lake/observability/`](../../../stats-lake/observability/) — a single dashboard template + a single alert-rule template, parameterized by `{{site}}`/`{{team}}`/`{{datasource_uid}}` and rendered once per site by `scripts/provision-dashboards.ts` (reads `dim_sites` joined to `dim_teams`, writes to `dashboards/dist/<team>/<site>.json` + `alerts/dist/<team>/<site>.yaml`). Alerts use **anomaly bands, not thresholds** — current 5-min mean vs 24h rolling mean ± 3σ, fires after 10 minutes outside the band. Same rule set runs on every site, but the baseline is per-site so a noisy storefront doesn't false-positive against a quiet one. Every alert carries a `runbook_url` annotation pointing at [`deco-start/docs/runbooks/`](../docs/runbooks/) — the runbook is part of the alert, not a separate artifact. **Decision points open** (Phase 5 of the refinement plan): alerting venue (Grafana → email, Linear MCP tickets, both, or none for v1). **Files:** `stats-lake/observability/{dashboards,alerts,scripts,README.md}` + `deco-start/docs/runbooks/`. |
+| 2026-05-22 | **D-17 — Alerting venue: none for v1; ship dashboards-only** | The action layer (Phase 5) generates anomaly-band alert rule templates per site, but every alert needs a pager and we don't have an on-call rotation. **Decision:** ship dashboards-only for v1. The alert templates in `stats-lake/observability/alerts/templates/site-rules.yaml` stay versioned so they evolve with the dashboards, but `provision-dashboards.ts` only templates them into `alerts/dist/` when `--with-alerts` is passed, and no Grafana → email / Linear MCP / PagerDuty receiver is wired up. Rejected alternatives: (a) Grafana → email — emails get muted within a week without a triage owner; (b) Linear MCP ticket-per-fire — creates noise during active incidents and you can't triage a ticket while firefighting; (c) both — overkill before we know which storefronts will be noisiest. **Revisit when:** an on-call rotation exists, OR a specific incident class earns dedicated paging (e.g., billing-critical sites that need 24/7 coverage). **Files:** `stats-lake/observability/scripts/provision-dashboards.ts` (`--with-alerts` flag), `stats-lake/observability/README.md`. Phase 5 of the observability refinement plan. |
+| 2026-05-22 | **D-16 — `deco-audit-observability` is warn-by-default; promote to block once the fleet is clean** | The audit (D-14) detects drift in `tail_consumers`, `version_metadata`, `DECO_METRICS`, and `DECO_OTEL_*_ENDPOINT` vars across every storefront wrangler.jsonc. Pre-merge blocking on day one would fail PRs that have nothing to do with observability — storefronts are upgraded over weeks, not all at once. Advisory comments alone get ignored. **Decision:** `--mode warn` (default) annotates findings via `::warning::` GitHub Actions lines but always exits 0, so observability drift surfaces in CI without blocking ship. `--mode block` exits 1 on any `error`-severity finding for use once the fleet has been pulled current. Storefronts opt into `block` per-repo by wiring `--mode block` in their workflow when they've cleared their findings. Includes `--github` flag to emit native annotations. **Files:** `scripts/audit-observability-config.ts` (parseArgs `--mode` / `--github`, main exit policy), test coverage in the matching `.test.ts` (6 new CLI smoke tests via tsx subprocess). Phase 6 of the observability refinement plan. |
+| 2026-05-22 | **D-15 — OTel Collector swap is a documented target state, not a committed milestone** | The current `deco-otel-ingest` Worker is a hand-rolled OTLP/HTTP parser + ClickHouse inserter. It works at ~14M POSTs/month, but each new OTLP protocol revision, each new receiver (gRPC OTLP, Prometheus remote-write), and each new sink would cost us code to write and maintain — code the upstream OTel Collector + `clickhouseexporter` ship as a maintained product. **Decision:** mark the Collector swap as the eventual target state, **with no committed timeline**, and capture the explicit revisit-triggers so we know when to act rather than relying on "we should think about this sometime." The decision is cheap because the ClickHouse schema is **already** the canonical `clickhouseexporter` shape — that was a deliberate design choice in `clickhouse/schema/otel/` so the ingest path stays swappable. Migration when triggered is config-only at the data layer: stand up a Collector in the same CF account, configure `otlphttp`/`otlpgrpc` receivers + `clickhouse/v1` exporter + `transform` processors for the existing PII redaction and JSON-body lift (1:1 from current Worker logic), DNS-cutover the `DECO_OTEL_*_ENDPOINT` vars. **Revisit triggers:** OTLP 2.0 ships, we need gRPC OTLP, we need a non-ClickHouse sink, ingest volume exceeds 100M POSTs/mo, or a hand-rolled parser develops a defect we can't fix quickly. **Files:** [`stats-lake/ingestion/otel-ingest/COLLECTOR_TARGET.md`](../../../stats-lake/ingestion/otel-ingest/COLLECTOR_TARGET.md) holds the full migration runbook + rollback story. Phase 7 of the observability refinement plan; explicitly **optional**. |
+| 2026-05-22 | **D-14 — `deco-audit-observability` covers fleet bindings, not just the `observability` block** | The existing audit only checked the `observability` block (sampling rates, persist, destinations). Phase 1+2+3 made several other wrangler keys load-bearing: `tail_consumers` must list `deco-otel-tail` (Phase 1 enrichment is a no-op without it), `version_metadata` must bind `CF_VERSION_METADATA` (no `service.version` without it = no deploy correlation), `analytics_engine_datasets` must bind `DECO_METRICS` (no AE meter), `vars.DECO_OTEL_{METRICS,TRACES,LOGS}_ENDPOINT` must resolve (direct-POST channels silently no-op otherwise). **Decision:** expand the audit with six new rules under a sibling function `auditFleetBindings` and a composing `auditWranglerConfig`. Severity tuned to the impact: tail consumer + version_metadata are `error` (operational coverage gap); the rest are `warn` (degraded mode, not total failure). Drift is detected today; the matching `--fix` codemod and CI gate hardness (block / warn / advisory) are Phase 6 decision points still open. **Files:** `scripts/audit-observability-config.ts` (`auditFleetBindings`, `auditWranglerConfig`). |
 | 2026-05-19 | **D-8 — Cloudflare Tail Worker (Strategy B) is the canonical 100% error capture mechanism** | At fleet scale (100 sites, 2.5B req/month) head sampling forces a tradeoff: 1% sampling makes the `head_sampling_rate * 5B-event-cap` math work, but 99% of error traces and 99% of error-correlated logs get dropped at the CF Destinations head. The framework already covers framework-emitted errors via the in-Worker direct-POST channel (`DECO_OTEL_LOGS_ENDPOINT`) — that's 100% of `logger.error(...)` regardless of `head_sampling_rate`. But three structural gaps remain that *no* in-Worker code can close from inside its own request handler: (a) uncaught throws (the worker isolate is already unwinding when the throw bubbles out of `instrumentWorker`), (b) `exceededCpu` / `exceededMemory` outcomes (the runtime kills the producer before any in-Worker code can run), (c) raw `console.error(...)` from third-party SDKs that bypass the framework logger. **Decision:** introduce [`deco-otel-tail`](https://github.com/decocms/stats-lake/tree/main/ingestion/otel-tail) — a Cloudflare Tail Worker in `stats-lake/ingestion/otel-tail/`. CF invokes it on every execution of any producer worker that lists it under `tail_consumers` (`wrangler.jsonc`). The handler filters TraceItems down to the interesting subset (`outcome !== "ok" \|\| exceptions.length > 0 \|\| logs.some(l => l.level === "error")`), translates each to OTLP LogRecords (one per exception, one per `error`-level log line, plus a synthetic LogRecord for non-ok outcomes that didn't surface either), and forwards them to `deco-otel-ingest` via an in-account service binding (no public hop). Rows land in `otel_logs` with `Attributes['_source'] = 'tail-worker'` so dashboards can split tail-captured errors from direct-POST + CF-Destinations errors. **Rejected alternatives:** (1) **Codemod + lint to enforce `logger.error` calls** — structural coverage gap; can't catch uncaught throws or 1101s by definition, and a lint can't enforce calls inside third-party code. (2) **Logpush + ingest pipeline** — bypassed because Logpush isn't OTLP-shaped and the pricing curve loses to tail-worker at our scale. (3) **CF dashboard log retention only** — no fan-out to ClickHouse, no fleet-wide query surface. (4) **DO-buffered tail-on-error** — ~$8K/mo at fleet scale per the cost model in `docs/observability.md`. **Coverage matrix lives in [`docs/observability.md`](./docs/observability.md) → "Error capture — three-channel model".** Producer-side wiring is one line per `wrangler.jsonc`: `tail_consumers: [{ service: "deco-otel-tail" }]`. **Operational dependency:** the tail worker MUST be deployed to the same Cloudflare account as `deco-otel-ingest` (currently `c95fc4cec7fc52453228d9db170c372c`) so the `[[services]]` binding resolves. If `deco-otel-ingest` ever moves accounts, the service binding collapses to a public HTTPS POST and the model needs revisiting. **Agent behaviour:** when designing error capture for new Worker-deployed code, default to Strategy B for the long tail; don't reach for codemod/lint enforcement unless there's a specific code-quality concern beyond capture. |
 
 The full text of the constitutional rule (loaded into every agent

package/docs/observability.md

@@ -71,17 +71,24 @@ This makes it possible to filter traces by cache decision directly in ClickStack
 
 ## What's measured
 
-| Metric                         | Type      | Source                              | Labels                          |
-| ------------------------------ | --------- | ----------------------------------- | ------------------------------- |
-| `http_requests_total`          | counter   | `workerEntry`                       | `method`, `path`, `status`      |
-| `http_request_duration_ms`     | histogram | `workerEntry`                       | `method`, `path`, `status`      |
-| `http_request_errors_total`    | counter   | `workerEntry` (status >= 500)       | `method`, `path`, `status`      |
-| `cache_hit_total`              | counter   | edge cache decision                 | `profile`, `decision`           |
-| `cache_miss_total`             | counter   | edge cache decision                 | `profile`, `decision`           |
-| `resolve_duration_ms`          | histogram | `resolveDecoPage`                   | —                               |
+| Metric                          | Type      | Source                              | Labels (canonical, Phase 2 / D-11)                                                              |
+| ------------------------------- | --------- | ----------------------------------- | ----------------------------------------------------------------------------------------------- |
+| `http_requests_total`           | counter   | `workerEntry`                       | `method`, `route_pattern`, `status`, `status_class`, `outcome?`, `cache_decision?`, `cache_layer?`, `region?` |
+| `http_request_duration_ms`      | histogram | `workerEntry`                       | same as `http_requests_total`                                                                  |
+| `http_request_errors_total`     | counter   | `workerEntry` (status >= 500)       | same as `http_requests_total`                                                                  |
+| `cache_hit_total`               | counter   | edge cache decision                 | `profile`, `decision`, `layer` (`edge` \| `cachedLoader` \| `vtex-swr`)                         |
+| `cache_miss_total`              | counter   | edge cache decision                 | `profile`, `decision`, `layer`                                                                  |
+| `commerce_request_duration_ms`  | histogram | commerce clients (vtex/shopify/…)   | `provider`, `operation`, `status_class?`, `cached?`                                             |
+| `resolve_duration_ms`           | histogram | `resolveDecoPage`                   | —                                                                                              |
 
 `decision` values mirror the `X-Cache` response header: `HIT`, `STALE-HIT`, `STALE-ERROR`, `MISS`, `BYPASS`.
 
+`route_pattern` is the TanStack route pattern (e.g. `/_products/$slug/p`) rather than the raw URL path — bounded cardinality, joinable to the route table. Callers that don't supply one get a normalized path with dynamic segments collapsed (`/products/:slug/p`).
+
+`status_class` is the canonical `2xx`/.../`5xx`/`unknown` bucket. Dashboards aggregate by `status_class` for SLO panels and by `status` for incident drill-down.
+
+`commerce_request_duration_ms` owned by the framework (Phase 2 / D-11) so every site emits it as soon as `@decocms/start` is bumped, regardless of `@decocms/apps` version. Apps register operation strings via `recordCommerceMetric`; the framework owns the cardinality contract.
+
 ### Metrics: AE vs OTLP (the two-meter split)
 
 `instrumentWorker` plugs **up to two meters in parallel**, composed via `createCompositeMeter`:
@@ -181,8 +188,9 @@ The direct-POST channels are wired automatically when the relevant env vars reso
 | ---------------------------------- | -------------------- | ---------------------- | ---------------------------------------------------- |
 | `DECO_OTEL_METRICS_ENDPOINT`       | OTLP metrics POST    | `""` (unset)           | OTLP meter is not created; AE-only metrics           |
 | `DECO_OTEL_LOGS_ENDPOINT`          | OTLP error-log POST  | `""` (unset)           | Error logs ride CF Destinations only (head-sampled)  |
+| `DECO_OTEL_TRACES_ENDPOINT`        | OTLP traces POST     | `""` (unset)           | Framework `deco.*` spans drop unless CF Traces is on |
 
-Both are opt-out via `OtelOptions.otlpMetricsEnabled: false` / `otlpErrorLogsEnabled: false` if you need to disable them at boot for a specific environment without changing the env vars.
+All three are opt-out via `OtelOptions.otlpMetricsEnabled: false` / `otlpErrorLogsEnabled: false` / `otlpTracesEnabled: false` if you need to disable them at boot for a specific environment without changing the env vars. Traces honor `OtelOptions.otlpTracesSamplingRate` (default `0.01` to match CF Destinations) — sampling decisions are consistent per trace (`FNV-1a` hash of `trace_id`), so child spans are kept iff their root is kept. Remote parents that arrive sampled (`traceparent` flags `01`) override the rate and are always exported.
 
 ## Log shape (and how to query it)
 
@@ -213,7 +221,9 @@ In the ClickStack UI you can also filter logs panel by `trace_id` directly — p
 
 ## Outbound trace propagation
 
-For any outbound `fetch` issued during a request (VTEX, Shopify, internal APIs), inject a W3C `traceparent` header so upstream services that participate in OTel can join your trace:
+For commerce clients (VTEX, Shopify), `createInstrumentedFetch` injects the W3C `traceparent` header by default. To opt out for a specific endpoint that rejects unknown headers, pass `injectTraceparent: false`.
+
+For any other outbound `fetch` issued during a request, inject a `traceparent` header manually so upstream services that participate in OTel can join your trace:
 
 ```ts
 import { injectTraceContext } from "@decocms/start/sdk/observability";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@decocms/start",
-  "version": "6.0.1",
+  "version": "6.1.0",
   "type": "module",
   "description": "Deco framework for TanStack Start - CMS bridge, admin protocol, hooks, schema generation",
   "main": "./src/index.ts",

package/src/middleware/observability.test.ts

@@ -0,0 +1,237 @@
+/**
+ * Phase 2 (D-11) coverage for the metric surface — canonical label set,
+ * cache_layer, commerce_request_duration_ms. The Phase 1 logger/trace
+ * tests live under `src/sdk/logger.test.ts` and `src/sdk/otel.test.ts`;
+ * this file focuses on the middleware-level helpers.
+ */
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import {
+  configureMeter,
+  type MeterAdapter,
+  MetricNames,
+  recordCacheMetric,
+  recordCommerceMetric,
+  recordRequestMetric,
+  statusClassFor,
+} from "./observability";
+
+interface Counter {
+  name: string;
+  value: number;
+  labels?: Record<string, unknown>;
+}
+interface Histogram {
+  name: string;
+  value: number;
+  labels?: Record<string, unknown>;
+}
+
+function captureMeter(): {
+  adapter: MeterAdapter;
+  counters: Counter[];
+  histograms: Histogram[];
+} {
+  const counters: Counter[] = [];
+  const histograms: Histogram[] = [];
+  const adapter: MeterAdapter = {
+    counterInc(name, value, labels) {
+      counters.push({ name, value: value ?? 1, labels });
+    },
+    histogramRecord(name, value, labels) {
+      histograms.push({ name, value, labels });
+    },
+  };
+  return { adapter, counters, histograms };
+}
+
+describe("statusClassFor", () => {
+  it("maps 2xx / 3xx / 4xx / 5xx to canonical class labels", () => {
+    expect(statusClassFor(200)).toBe("2xx");
+    expect(statusClassFor(204)).toBe("2xx");
+    expect(statusClassFor(301)).toBe("3xx");
+    expect(statusClassFor(404)).toBe("4xx");
+    expect(statusClassFor(500)).toBe("5xx");
+    expect(statusClassFor(503)).toBe("5xx");
+  });
+
+  it("returns 'unknown' for out-of-range / NaN / non-numeric inputs", () => {
+    expect(statusClassFor(-1)).toBe("unknown");
+    expect(statusClassFor(99)).toBe("unknown");
+    expect(statusClassFor(600)).toBe("unknown");
+    expect(statusClassFor(Number.NaN)).toBe("unknown");
+    expect(statusClassFor(Infinity)).toBe("unknown");
+  });
+});
+
+describe("recordRequestMetric — canonical labels (D-11)", () => {
+  afterEach(() => {
+    // Reset meter so other tests start clean.
+    configureMeter({ counterInc: () => {} });
+  });
+
+  it("stamps method + route_pattern + status + status_class by default", () => {
+    const { adapter, counters, histograms } = captureMeter();
+    configureMeter(adapter);
+
+    recordRequestMetric("GET", "/products/abc123/p", 200, 42);
+
+    expect(counters).toHaveLength(1);
+    expect(counters[0]?.name).toBe(MetricNames.HTTP_REQUESTS_TOTAL);
+    expect(counters[0]?.labels).toMatchObject({
+      method: "GET",
+      // Default normalization: dynamic segments collapsed.
+      route_pattern: "/products/:slug/p",
+      status: 200,
+      status_class: "2xx",
+    });
+    expect(histograms).toHaveLength(1);
+    expect(histograms[0]?.name).toBe(MetricNames.HTTP_REQUEST_DURATION_MS);
+    expect(histograms[0]?.value).toBe(42);
+  });
+
+  it("prefers caller-supplied route_pattern over normalized path", () => {
+    const { adapter, counters } = captureMeter();
+    configureMeter(adapter);
+
+    recordRequestMetric("GET", "/anything/random/123", 200, 5, {
+      route_pattern: "/_products/$slug/p",
+    });
+
+    expect(counters[0]?.labels?.route_pattern).toBe("/_products/$slug/p");
+  });
+
+  it("emits http_request_errors_total on 5xx", () => {
+    const { adapter, counters } = captureMeter();
+    configureMeter(adapter);
+
+    recordRequestMetric("POST", "/checkout", 503, 120);
+
+    const errCounter = counters.find((c) => c.name === MetricNames.HTTP_REQUEST_ERRORS);
+    expect(errCounter).toBeDefined();
+    expect(errCounter?.labels?.status_class).toBe("5xx");
+  });
+
+  it("propagates optional labels (outcome, cache_decision, cache_layer, region, extra)", () => {
+    const { adapter, counters } = captureMeter();
+    configureMeter(adapter);
+
+    recordRequestMetric("GET", "/", 200, 10, {
+      outcome: "ok",
+      cache_decision: "STALE-HIT",
+      cache_layer: "edge",
+      region: "GRU",
+      extra: { ab_variant: "B" },
+    });
+
+    expect(counters[0]?.labels).toMatchObject({
+      outcome: "ok",
+      cache_decision: "STALE-HIT",
+      cache_layer: "edge",
+      region: "GRU",
+      ab_variant: "B",
+    });
+  });
+
+  it("is a no-op when no meter is configured", () => {
+    // We can't easily prove a no-op other than verifying no throw —
+    // safer than calling configureMeter(null), which would mask real
+    // bugs. The previous test's `afterEach` reset already gives us a
+    // bare meter; this test confirms the call is benign.
+    expect(() => recordRequestMetric("GET", "/", 200, 1)).not.toThrow();
+  });
+});
+
+describe("recordCacheMetric — cache_layer label", () => {
+  beforeEach(() => {
+    configureMeter({ counterInc: () => {} });
+  });
+
+  it("stamps profile + decision + layer when all are provided", () => {
+    const { adapter, counters } = captureMeter();
+    configureMeter(adapter);
+
+    recordCacheMetric(true, "product", "HIT", "edge");
+
+    expect(counters).toHaveLength(1);
+    expect(counters[0]?.name).toBe(MetricNames.CACHE_HIT);
+    expect(counters[0]?.labels).toMatchObject({
+      profile: "product",
+      decision: "HIT",
+      layer: "edge",
+    });
+  });
+
+  it("emits cache_miss_total when hit=false", () => {
+    const { adapter, counters } = captureMeter();
+    configureMeter(adapter);
+
+    recordCacheMetric(false, "search", "MISS", "edge");
+
+    expect(counters[0]?.name).toBe(MetricNames.CACHE_MISS);
+  });
+
+  it("supports the legacy 3-arg signature for backward compat", () => {
+    const { adapter, counters } = captureMeter();
+    configureMeter(adapter);
+
+    recordCacheMetric(true, "static");
+
+    expect(counters[0]?.labels).toEqual({ profile: "static" });
+  });
+
+  it("distinguishes cachedLoader vs edge vs vtex-swr layers", () => {
+    const { adapter, counters } = captureMeter();
+    configureMeter(adapter);
+
+    recordCacheMetric(true, "loader-x", "HIT", "cachedLoader");
+    recordCacheMetric(true, "vtex-product", "HIT", "vtex-swr");
+
+    expect(counters[0]?.labels?.layer).toBe("cachedLoader");
+    expect(counters[1]?.labels?.layer).toBe("vtex-swr");
+  });
+});
+
+describe("recordCommerceMetric (D-11)", () => {
+  beforeEach(() => {
+    configureMeter({ counterInc: () => {} });
+  });
+
+  it("emits commerce_request_duration_ms with provider + operation labels", () => {
+    const { adapter, histograms } = captureMeter();
+    configureMeter(adapter);
+
+    recordCommerceMetric(123, {
+      provider: "vtex",
+      operation: "intelligent-search.product_search",
+      status_class: "2xx",
+    });
+
+    expect(histograms).toHaveLength(1);
+    expect(histograms[0]?.name).toBe(MetricNames.COMMERCE_REQUEST_DURATION_MS);
+    expect(histograms[0]?.value).toBe(123);
+    expect(histograms[0]?.labels).toMatchObject({
+      provider: "vtex",
+      operation: "intelligent-search.product_search",
+      status_class: "2xx",
+    });
+  });
+
+  it("includes the cached boolean when provided", () => {
+    const { adapter, histograms } = captureMeter();
+    configureMeter(adapter);
+
+    recordCommerceMetric(5, {
+      provider: "shopify",
+      operation: "graphql.cart_query",
+      cached: true,
+    });
+
+    expect(histograms[0]?.labels?.cached).toBe(true);
+  });
+
+  it("is a no-op when no meter is configured", () => {
+    expect(() =>
+      recordCommerceMetric(1, { provider: "vtex", operation: "test" }),
+    ).not.toThrow();
+  });
+});

package/src/middleware/observability.ts

@@ -239,25 +239,121 @@ export const MetricNames = {
   CACHE_MISS: "cache_miss_total",
   RESOLVE_DURATION_MS: "resolve_duration_ms",
   FETCH_DURATION_MS: "fetch_duration_ms",
+  /**
+   * Per-provider outbound commerce fetch duration. Owned by
+   * `@decocms/start` (not `@decocms/apps`) so every site emits this
+   * histogram unconditionally as soon as it bumps the framework,
+   * regardless of apps-start version. Apps register operation strings
+   * (`vtex.intelligent-search.product_search`,
+   * `shopify.graphql.cart_create`, ...) via `recordCommerceMetric`
+   * below; the framework owns the cardinality contract.
+   *
+   * Canonical labels: `provider`, `operation`, `status_class`, `cached`.
+   * See `recordCommerceMetric` for the full label set and Phase 2 in
+   * `MIGRATION_TOOLING_PLAN.md` for the rationale.
+   */
+  COMMERCE_REQUEST_DURATION_MS: "commerce_request_duration_ms",
 } as const;
 
+/**
+ * Map an HTTP status code to its canonical class label (`2xx` / ... /
+ * `5xx`). Out-of-range numbers (e.g. -1 from a thrown fetch) fall back
+ * to `"unknown"` so dashboards don't break on edge cases.
+ *
+ * Exported because callers occasionally need the same mapping for
+ * non-metric purposes (logging, tail enrichment).
+ */
+export function statusClassFor(status: number): string {
+  if (typeof status !== "number" || !Number.isFinite(status)) return "unknown";
+  if (status < 100 || status >= 600) return "unknown";
+  return `${Math.floor(status / 100)}xx`;
+}
+
+/**
+ * Optional dimensions stamped on `http_requests_total` /
+ * `http_request_duration_ms` / `http_request_errors_total`. All fields
+ * are optional — callers pass what they have, the framework fills in
+ * the rest from defaults.
+ *
+ * Cardinality discipline: every field here is bounded. `route_pattern`
+ * comes from the TanStack router (a closed set), `outcome` is the CF
+ * Workers Observability enum, `cache_decision` / `cache_layer` are
+ * union types declared in this module, `region` is a small set of CF
+ * colo codes. Status is unbounded by spec but bounded in practice; the
+ * `status_class` label bounds the cardinality further for dashboards
+ * that don't need the raw value.
+ */
+export interface RequestMetricLabels {
+  /** TanStack route pattern (`/_products/$slug/p`) — closed set. */
+  route_pattern?: string;
+  /** Cloudflare Workers Observability `outcome` (`ok`, `exception`, ...). */
+  outcome?: string;
+  /** Cache layer + decision when known. */
+  cache_decision?: CacheDecision;
+  cache_layer?: CacheLayer;
+  /** Cloudflare colo (`GRU`, `IAD`, ...). */
+  region?: string;
+  /**
+   * Arbitrary extra labels — callers should avoid this and add fields
+   * to the typed surface above instead. Kept as an escape hatch so
+   * non-canonical experiments don't require a framework release.
+   */
+  extra?: Record<string, string | number | boolean>;
+}
+
 /**
  * Record an HTTP request metric.
- * Call in middleware after the response is produced.
+ *
+ * Call in middleware after the response is produced. Two-call surface
+ * for backward compat:
+ *
+ *   recordRequestMetric(method, path, status, durationMs)
+ *   recordRequestMetric(method, path, status, durationMs, labels)
+ *
+ * The labels argument is optional — sites that haven't bumped to the
+ * Phase 2 metric shape still emit the original three labels
+ * (`method`, `route_pattern`, `status`). Adding labels never changes
+ * existing labels' values; only adds new ones.
  */
 export function recordRequestMetric(
   method: string,
   path: string,
   status: number,
   durationMs: number,
+  labels?: RequestMetricLabels,
 ) {
   const m = getState().meter;
   if (!m) return;
-  const labels: Labels = { method, path: normalizePath(path), status };
-  m.counterInc(MetricNames.HTTP_REQUESTS_TOTAL, 1, labels);
-  m.histogramRecord?.(MetricNames.HTTP_REQUEST_DURATION_MS, durationMs, labels);
+  // Cardinality discipline:
+  //   - `method`: small (GET, POST, ...).
+  //   - `route_pattern`: closed set (caller-supplied) OR normalized path
+  //     (fallback). Either way bounded.
+  //   - `status`: full HTTP code (bounded ~50 values in practice).
+  //   - `status_class`: 5-element enum (2xx / 3xx / 4xx / 5xx / unknown).
+  //   - `outcome`: CF outcome enum (~7 values).
+  //   - `cache_decision`: 5-element enum.
+  //   - `cache_layer`: 3-element enum (edge / cachedLoader / vtex-swr).
+  //   - `region`: ~250 CF colo codes worldwide.
+  // Total combinations are bounded — safe for unbounded series on
+  // ClickHouse but operators should still avoid grouping by `region`
+  // unless explicitly needed.
+  const merged: Labels = {
+    method,
+    route_pattern: labels?.route_pattern ?? normalizePath(path),
+    status,
+    status_class: statusClassFor(status),
+  };
+  if (labels?.outcome) merged.outcome = labels.outcome;
+  if (labels?.cache_decision) merged.cache_decision = labels.cache_decision;
+  if (labels?.cache_layer) merged.cache_layer = labels.cache_layer;
+  if (labels?.region) merged.region = labels.region;
+  if (labels?.extra) {
+    for (const [k, v] of Object.entries(labels.extra)) merged[k] = v;
+  }
+  m.counterInc(MetricNames.HTTP_REQUESTS_TOTAL, 1, merged);
+  m.histogramRecord?.(MetricNames.HTTP_REQUEST_DURATION_MS, durationMs, merged);
   if (status >= 500) {
-    m.counterInc(MetricNames.HTTP_REQUEST_ERRORS, 1, labels);
+    m.counterInc(MetricNames.HTTP_REQUEST_ERRORS, 1, merged);
   }
 }
 
@@ -272,22 +368,45 @@ export function recordRequestMetric(
  */
 export type CacheDecision = "HIT" | "STALE-HIT" | "STALE-ERROR" | "MISS" | "BYPASS";
 
+/**
+ * Where the cache lives. Phase 2 label expansion (D-11).
+ *  - `edge`         — Cloudflare Cache API (HTML pages, server-fn responses)
+ *  - `cachedLoader` — In-memory per-isolate via `sdk/cachedLoader.ts`
+ *                     (loader-level SWR, dedup, in-flight)
+ *  - `vtex-swr`     — Apps-side in-memory cache shared by VTEX clients
+ *                     (intelligent-search, cross-selling, etc.)
+ */
+export type CacheLayer = "edge" | "cachedLoader" | "vtex-swr";
+
 /**
  * Record a cache hit/miss metric. Also stamps the decision on the active
  * trace span (when one exists) as `deco.cache.decision` / `deco.cache.profile`
  * so operators can filter ClickStack traces by cache decision directly,
  * without joining to metrics.
  *
- * `decision` is optional — when omitted, the metric still records HIT vs MISS
- * but dashboards can't distinguish SWR/SIE paths. Pass it whenever known.
+ * Backward-compatible signature:
+ *   recordCacheMetric(hit, profile?, decision?)
+ *   recordCacheMetric(hit, profile?, decision?, layer?)
+ *
+ * `decision` is optional — when omitted, the metric still records HIT
+ * vs MISS but dashboards can't distinguish SWR/SIE paths. Pass it
+ * whenever known. `layer` defaults to `edge` when called from
+ * workerEntry; cachedLoader / vtex-swr call sites should pass their
+ * value explicitly.
  */
-export function recordCacheMetric(hit: boolean, profile?: string, decision?: CacheDecision) {
+export function recordCacheMetric(
+  hit: boolean,
+  profile?: string,
+  decision?: CacheDecision,
+  layer?: CacheLayer,
+) {
   // Stamp on the active span FIRST so the attribute survives even if the
   // meter is a no-op (e.g. on tests, or in dev without DECO_METRICS).
   const active = getActiveSpan();
   if (active) {
     if (decision) active.setAttribute?.("deco.cache.decision", decision);
     if (profile) active.setAttribute?.("deco.cache.profile", profile);
+    if (layer) active.setAttribute?.("deco.cache.layer", layer);
   }
 
   const m = getState().meter;
@@ -295,9 +414,47 @@ export function recordCacheMetric(hit: boolean, profile?: string, decision?: Cac
   const labels: Labels = {};
   if (profile) labels.profile = profile;
   if (decision) labels.decision = decision;
+  if (layer) labels.layer = layer;
   m.counterInc(hit ? MetricNames.CACHE_HIT : MetricNames.CACHE_MISS, 1, labels);
 }
 
+/**
+ * Labels for `commerce_request_duration_ms`. Owned by the framework so
+ * apps-start (and any future provider package) can register operation
+ * strings without owning the histogram declaration. Phase 2 (D-11).
+ */
+export interface CommerceMetricLabels {
+  /** `vtex`, `shopify`, `wake`, ... — small closed set. */
+  provider: string;
+  /** Per-provider operation, e.g. `intelligent-search.product_search`. */
+  operation: string;
+  /** Set when known (e.g. from the HTTP response). Bounded enum. */
+  status_class?: string;
+  /** Whether the underlying fetch was served from a cache. */
+  cached?: boolean;
+}
+
+/**
+ * Record a commerce / outbound-fetch duration sample. No-op when no
+ * meter is configured. The metric name is constant
+ * (`commerce_request_duration_ms`) — providers vary by the `provider`
+ * label, not by name, so dashboards aggregate cleanly across the fleet.
+ */
+export function recordCommerceMetric(
+  durationMs: number,
+  labels: CommerceMetricLabels,
+) {
+  const m = getState().meter;
+  if (!m) return;
+  const merged: Labels = {
+    provider: labels.provider,
+    operation: labels.operation,
+  };
+  if (labels.status_class) merged.status_class = labels.status_class;
+  if (typeof labels.cached === "boolean") merged.cached = labels.cached;
+  m.histogramRecord?.(MetricNames.COMMERCE_REQUEST_DURATION_MS, durationMs, merged);
+}
+
 function normalizePath(path: string): string {
   // Collapse dynamic segments to reduce cardinality
   return path

package/src/sdk/cachedLoader.ts

@@ -99,13 +99,13 @@ export function createCachedLoader<TProps, TResult>(
     const inflight = inflightRequests.get(cacheKey);
     if (inflight) {
       // Treat in-flight dedup as a cache hit — avoided the origin call.
-      recordCacheMetric(true, name);
+      recordCacheMetric(true, name, undefined, "cachedLoader");
       return inflight as Promise<TResult>;
     }
 
     if (isDev) {
       // Dev mode: no caching, but still useful to count attempts.
-      recordCacheMetric(false, name);
+      recordCacheMetric(false, name, undefined, "cachedLoader");
       const promise = withTracing(
         "deco.cachedLoader",
         () => loaderFn(props).finally(() => inflightRequests.delete(cacheKey)),
@@ -121,20 +121,20 @@ export function createCachedLoader<TProps, TResult>(
 
     if (policy === "no-cache") {
       if (entry && !isStale) {
-        recordCacheMetric(true, name);
+        recordCacheMetric(true, name, "HIT", "cachedLoader");
         return entry.value;
       }
     }
 
     if (policy === "stale-while-revalidate") {
       if (entry && !isStale) {
-        recordCacheMetric(true, name);
+        recordCacheMetric(true, name, "HIT", "cachedLoader");
         return entry.value;
       }
 
       if (entry && isStale && !entry.refreshing) {
         // Stale-while-revalidate hit: serve stale, refresh in background.
-        recordCacheMetric(true, name);
+        recordCacheMetric(true, name, "STALE-HIT", "cachedLoader");
         entry.refreshing = true;
         loaderFn(props)
           .then((result) => {
@@ -156,14 +156,17 @@ export function createCachedLoader<TProps, TResult>(
       }
 
       if (entry) {
-        recordCacheMetric(true, name);
+        // Past SIE window — still serve the stale value once but mark
+        // the decision as STALE-ERROR so dashboards can distinguish
+        // this from healthy SWR.
+        recordCacheMetric(true, name, "STALE-ERROR", "cachedLoader");
         return entry.value;
       }
     }
 
     // Cache miss — emit metric, then run loader inside a span so individual
     // slow loaders are visible in traces.
-    recordCacheMetric(false, name);
+    recordCacheMetric(false, name, "MISS", "cachedLoader");
     const promise = withTracing("deco.cachedLoader", () => loaderFn(props), {
       "deco.loader": name,
       "deco.cache.policy": policy,