@decocms/start 5.3.0-rc.2 → 5.4.0

package/docs/o11y.md

@@ -1,602 +0,0 @@
-# First context, made at stats-lake:
-
-@decocms/start Observability — Investigation & Implementation Brief
-
-Goal
-
-Make @decocms/start’s instrumentWorker() produce rich, well-shaped OTel telemetry that lands in our existing ClickStack instance (hyperdx.clickhouse.cloud) backed by stats-lake ClickHouse. The transport is already solved (CF Destinations → deco-otel-ingest Worker → otel_traces / otel_logs). This brief is about what the framework should emit.
-
-Context — what’s already built
-
-casaevideo-tanstack (CF Worker)
-  observability.destinations in wrangler.jsonc
-        │  OTLP/HTTP JSON
-        ▼
-deco-otel-ingest  (https://deco-otel-ingest.deco-cx.workers.dev)
-  maps OTLP → ClickHouse clickhouseexporter schema
-  redacts PII (cookie, authorization, x-vtex-*)
-        ▼
-stats-lake ClickHouse
-  default.otel_traces / default.otel_logs (30-day TTL)
-        ▼
-hyperdx.clickhouse.cloud (ClickStack UI)
-End-to-end is verified. Synthetic spans + logs ingest correctly, render in the ClickStack UI under service:deco-otel-ingest-smoke.
-
-Repos involved:
-
-/Users/fernandofrizzatti/development/workspace/decocms/deco-start/ — this brief is about this repo
-/Users/fernandofrizzatti/conductor/workspaces/stats-lake/calgary/ — ingest Worker + DDL (Phase 1, PR https://github.com/decocms/stats-lake/pull/1)
-/Users/fernandofrizzatti/development/workspace/decocms/casaevideo-storefront/ — first site to wire (Phase 3)
-/Users/fernandofrizzatti/development/workspace/decocms/apps/ — @decocms/apps, has createInstrumentedFetch used by sites
-What works today in @decocms/start v5.0
-
-instrumentWorker(handler, { serviceName }) — wraps a Worker; configures structured JSON logger + per-span attribute floor via configureTracer() bridge onto @opentelemetry/api global tracer
-withTracing("name", fn, attrs) — public API, called from workerEntry.ts:981 around every request as deco.http.request
-defaultLoggerAdapter — writes { level, msg, timestamp, ...attrs } to console.*, which CF Destinations captures as OTLP logs
-AE meter wiring via DECO_METRICS binding (existing, unrelated to the OTLP path)
-Single dependency: @opentelemetry/api ^1.9.1. No SDK, no exporters. The transport is entirely CF Destinations.
-
-Confirmed gaps (ranked by impact)
-
-1. service.version is never set from CF_VERSION_METADATA
-instrumentWorker() reads decoRuntimeVersion option (defaults to hardcoded "5.0.0") and stamps it as deco.runtime.version
-It does NOT read env.CF_VERSION_METADATA?.id to set the OTel-standard service.version
-Why this matters: dashboards correlate errors / latency with specific deployments via service.version. Without it, “did the regression land in deployment X?” is unanswerable
-Fix: in bootObservability(), read env.CF_VERSION_METADATA?.id and stamp it as service.version in spanAttributeFloor AND in the logger floor. Files: src/sdk/otel.ts:168-214
-2. Per-span attribute floor only covers framework-created spans, not CF auto-instrumented ones
-spanAttributeFloor (deco.runtime.version, deployment.environment) is applied inside the configureTracer bridge in otel.ts:137-149 — only spans created through withTracing() get it
-CF runtime auto-instrumented spans (fetch/KV/R2/DO) come out without these attributes
-Why this matters: ~90% of spans in a typical request come from CF auto-instrumentation. Without the floor, dashboards can’t filter those by environment or runtime version
-Fix options:
-(a) Move stamping into ingest Worker: if a resource attribute is missing, fill from the request’s known service. Requires the ingest Worker to know per-service defaults — leaks coupling.
-(b) Accept the gap. Filter dashboards by service.name only (resource attribute, always present). deco.runtime.version becomes “best-effort for framework spans”
-(c) Use CF Tail Workers to enrich spans post-hoc (heavy)
-Recommendation: (b) for now, document explicitly. Revisit if a real dashboard need surfaces.
-3. Sampling: head-only via CF Destinations, no tail-on-error
-CF Destinations only supports head_sampling_rate — decided at trace start, before status is known
-Goal: “100% errors + 1% baseline” — tail-on-error semantics. Not achievable with CF Destinations alone
-Options:
-A (long term) — CF Tail Workers: bind a separate Worker as a tail handler that receives every invocation summary (status + logs). It decides to forward full traces to ingest based on outcome. Most flexible; requires a second Worker.
-B (pragmatic short term) — Head 100% + ingest-side filter: set traces.head_sampling_rate: 1.0 in wrangler, let everything flow. deco-otel-ingest keeps all errors (StatusCode == STATUS_CODE_ERROR or http.response.status_code >= 500) plus a deterministic % of OK traces (hash on TraceId). Costs more CF events but no missed errors.
-C — Status quo: head=0.1, logs=1.0. Miss 90% of error traces; lean on 100% logs for error context.
-Recommendation: start with C for the pilot (no extra work), measure ratio of 5xx in the first 24h of real traffic on casaevideo. If we’re losing relevant signal, move to B (1-2 hour task in the ingest Worker).
-Where the change lives: not in @decocms/start. Either the site’s wrangler.jsonc (option C → B head rate change) or deco-otel-ingest/src/index.ts (option B filter logic).
-4. Logger output is JSON-in-stringValue, not native OTel log attributes
-defaultLoggerAdapter does console.log(JSON.stringify({ level, msg, timestamp, ...attrs }))
-CF Destinations wraps that into an OTLP LogRecord with body.stringValue = "<the JSON>". The attrs object isn’t exposed as native log attributes
-Our ingest Worker maps body.stringValue → otel_logs.Body verbatim. Queries need JSONExtract(Body, 'level', 'String') to filter by level
-Why this matters: ClickStack UI auto-detects severity from body.stringValue substring matching, so “INFO” / “ERROR” show up correctly (confirmed empirically). But filtering by custom attributes (requestId, userId) requires JSON extraction at query time
-Fix options:
-(a) Have the logger emit severityText and severityNumber to a separate sidecar that’s understood by CF Destinations. Looks hard — CF only sees console output
-(b) Accept the JSON-in-body shape, document the JSONExtract pattern for queries
-(c) Change the logger to log key-value pairs as separate console.log calls, but that breaks correlation
-Recommendation: (b). Document with example queries in @decocms/start observability README.
-Open investigation points (verify empirically with casaevideo)
-
-What does CF Destinations actually emit for resource attributes? I assumed service.name = Worker name from wrangler.jsonc. Confirm by reading from otel_traces.ResourceAttributes after the first casaevideo request lands. Also check: cloud.provider, cloud.platform, faas.name, cloudflare.script_version.id.
-
-What does withTracing() in workerEntry.ts:981 actually emit? Confirm deco.http.request spans show up with http.method + url.path and inherit the spanAttributeFloor. Cross-check that the framework span and the CF root span are properly parent-child related (or whether they’re siblings — depends on how CF auto-instrumentation handles user spans).
-
-createInstrumentedFetch("vtex") from @decocms/apps — does it use withTracing from @decocms/start, or its own tracer? Either way, do VTEX calls show up as child spans of the request root in ClickStack? Trace propagation (W3C traceparent header) to VTEX would be a nice future win.
-
-TanStack-specific spans — server entry render, data fetch, route resolution. Are there withTracing() calls in the framework / @tanstack/react-start/server-entry that produce meaningful spans? If not, add them at strategic points.
-
-Error reporting — when a request throws, does withTracing() set setError on the span (workerEntry.ts:996 area)? Verify the span gets StatusCode = STATUS_CODE_ERROR and a StatusMessage. If not, fix in src/middleware/observability.ts:77-95.
-
-CF_VERSION_METADATA binding — is it declared in casaevideo’s wrangler.jsonc? (Yes, confirmed earlier.) Does instrumentWorker() receive it via env? Should: yes, since the binding is on env.
-
-Concrete deliverables for the @decocms/start PR
-
-In priority order:
-
-src/sdk/otel.ts — read env.CF_VERSION_METADATA?.id in bootObservability(). Stamp as service.version in both spanAttributeFloor and the logger floor. Boot-time log should include version. ~10 LOC.
-
-src/sdk/otel.ts — also stamp service.name in both floors (so framework-created spans carry it, even though the resource attribute already has it from CF). Defensive, lets ClickStack dashboards filter by either path. ~3 LOC.
-
-src/middleware/observability.ts:77-95 — confirm withTracing() sets span status to STATUS_CODE_ERROR on thrown exception via the bridge’s setError. Add a test.
-
-README.md / docs/observability.md — document the architecture: instrumentWorker() + CF Destinations + ingest Worker. Show the recommended wrangler.jsonc shape. Show the ClickStack query patterns (JSONExtract for log filtering).
-
-Optional: add withTracing() calls at strategic framework points — render entry, loader resolution, data fetching. Only if the empirical investigation (point 2 above) shows current coverage is too thin.
-
-Out of scope for this PR
-
-Reviving the OTLP SDK exporters that lived at v4.6.0 — not needed; CF Destinations replaces that path
-Tail-on-error sampling — lives in the ingest Worker or a CF Tail Worker, not in @decocms/start
-@microlabs/otel-cf-workers integration — rejected; CF Destinations + thin framework attributes is sufficient
-PII redaction — handled at ingest Worker, no per-site code needed
-How to validate the framework changes
-
-Branch off @decocms/start main, apply the 4 changes above
-pnpm link it into casaevideo-storefront
-Add destinations block to casaevideo/wrangler.jsonc pointing at https://deco-otel-ingest.deco-cx.workers.dev
-Wrap casaevideo/src/worker-entry.ts final export with instrumentWorker(handler, { serviceName: "casaevideo-tanstack" })
-wrangler deploy (preview URL is fine; routes are still commented out in production)
-Hit the preview URL a few times
-ClickHouse: verify service.version lands in otel_traces.ResourceAttributes['service.version'] AND otel_traces.SpanAttributes['service.version']
-ClickStack UI: filter service:casaevideo-tanstack, confirm spans show with sensible names, durations, and errors-as-errors
-Reference files
-
-src/sdk/otel.ts — instrumentWorker(), bootObservability(), attribute floor wiring
-src/sdk/logger.ts — defaultLoggerAdapter, severity routing
-src/sdk/workerEntry.ts:968-996 — createDecoWorkerEntry, RequestContext.run, root withTracing("deco.http.request", ...)
-src/middleware/observability.ts:77-95 — withTracing() implementation, error path
-package.json — only @opentelemetry/api ^1.9.1, no SDK
-src/sdk/otelAdapters/clickhouseCollector.ts — documented stub, unchanged; leave for future use
-
-
-# Second Context: made at deco-start
-
-Let's plan for all of those, straight away. we need to be able to track everything we need; cache, performance, errors, etc; 
-Separate what goes into this repo as a PLAN, and separate what goes into decocms/apps as Plan prompt for investigation and stuff; please. 
-Let's implement what goes into this repo, and I will send the prompt to apps in another thread.
-Observability completeness — full coverage for cache, performance, errors
-
-Context
-
-The brief at /tmp/attachments/pasted_text_2026-05-15_15-42-09.txt covers the next iteration of @decocms/start observability. Transport is solved end-to-end:
-
-site Worker → CF Destinations (OTLP/HTTP JSON)
-  → deco-otel-ingest (Worker)
-  → stats-lake ClickHouse (otel_traces / otel_logs)
-  → hyperdx.clickhouse.cloud (ClickStack UI)
-The framework already has scaffolding in place — instrumentWorker, withTracing, AE meter, structured logger, log severity floor — but the signal it emits is incomplete:
-
-Identity fields missing: service.version, service.name aren’t on the per-span/per-log floor.
-Error spans don’t carry STATUS_CODE_ERROR — they look successful with an exception attached.
-Two CMS hotspots are already instrumented (resolveDecoPage → deco.cms.resolvePage, runSingleSectionLoader → deco.section.loader). The rest of the framework is dark to ClickStack.
-No cache decision spans / metrics at the application layer (CF Cache API spans show storage ops but not HIT/MISS/profile).
-Admin protocol handlers (handleMeta, handleDecofile, handleRender, handleInvoke) emit no spans, so admin traffic mixes with customer traffic in the trace tree.
-Deferred section loads (loadDeferredSection server fn) have no span — on-scroll latency is invisible.
-logRequest doesn’t emit traceId, so logs and traces can’t be cross-referenced in ClickStack.
-No W3C trace-context propagation outbound — upstream services (VTEX, Shopify) can’t link their spans to ours.
-No docs/observability.md.
-Goal: ship a single coherent PR that makes the framework emit rich, queryable telemetry covering cache decisions, request shape, admin protocol, errors, and trace correlation — without changing the public API. The companion work in @decocms/apps (createInstrumentedFetch, VTEX-specific spans) is described as a standalone brief at the bottom of this document for the user to send to a separate agent thread.
-
-Plan A — Changes in @decocms/start (this repo)
-
-A1. Identity on the floor (src/core/sdk/otel.ts:168-214)
-bootObservability():
-
-Read env.CF_VERSION_METADATA?.id; if present, add service.version to floor.
-Add service.name (already computed at line 171) to floor.
-Update the boot breadcrumb (otel.ts:208-213) to log version when known.
-Resulting shape:
-
-const floor: Record<string, string> = {
-  "service.name": serviceName,
-  "deco.runtime.version": decoRuntimeVersion,
-  "deployment.environment": deploymentEnvironment,
-};
-const versionId = (env.CF_VERSION_METADATA as { id?: string } | undefined)?.id;
-if (versionId) floor["service.version"] = versionId;
-if (opts.decoAppsVersion) floor["deco.apps.version"] = opts.decoAppsVersion;
-Both spanAttributeFloor = floor and setLoggerAttributeFloor(floor) already exist (lines 185, 191) — no plumbing needed downstream.
-
-A2. Span-status on errors (src/core/sdk/otel.ts:137-149)
-In the configureTracer bridge, extend setError to also call span.setStatus({ code: SpanStatusCode.ERROR, message }). Import SpanStatusCode from @opentelemetry/api.
-
-setError: (error) => {
-  const message = error instanceof Error ? error.message : String(error);
-  if (error instanceof Error) span.recordException(error);
-  span.setStatus({ code: SpanStatusCode.ERROR, message });
-},
-withTracing() at src/core/sdk/observability.ts:101-105 already calls span.setError?.(error) in its catch — no change needed there.
-
-A3. Extend Span interface to expose trace context (src/core/sdk/observability.ts:43-47)
-Required by A7 (logger traceId) and A8 (traceparent propagation).
-
-export interface Span {
-  end(): void;
-  setError?(error: unknown): void;
-  setAttribute?(key: string, value: string | number | boolean): void;
-  spanContext?(): { traceId: string; spanId: string; traceFlags: number };
-}
-In the bridge (src/core/sdk/otel.ts:140-148), expose spanContext by delegating to the OTel API span:
-
-spanContext: () => {
-  const ctx = span.spanContext();
-  return { traceId: ctx.traceId, spanId: ctx.spanId, traceFlags: ctx.traceFlags };
-},
-This is opt-in via the optional method — adapters that don’t implement it still work; helpers that need it (logger, traceparent) just no-op gracefully.
-
-A4. Cache decision spans + metrics (src/tanstack/sdk/workerEntry.ts)
-Two cache stores, two lookup paths. Wrap each cache.match call in withTracing("deco.cache.lookup", ...) and emit recordCacheMetric(...) at the decision points already in the code.
-
-Server-function cache (POST _serverFn): workerEntry.ts:1246-1303
-
-Wrap line 1247-1251 lookup: span deco.cache.lookup, attrs { "cache.profile": "listing", "cache.kind": "serverFn" }.
-At each decision point (HIT line 1254-1264, STALE-HIT line 1267-1292, MISS line 1303+), call recordCacheMetric(hit, profile) and set span attribute cache.decision.
-Wrap cache.put at line 1334 with deco.cache.store, attrs { "cache.profile": "listing", "cache.kind": "serverFn" }.
-HTML cache (GET): workerEntry.ts:1484-1595
-
-Wrap line 1486-1489 lookup: span deco.cache.lookup, attrs { "cache.profile": profile, "cache.kind": "html" }.
-At each decision point (HIT line 1492-1501, STALE-HIT line 1502-1511, MISS line 1514+), call recordCacheMetric(...) with the right profile, set cache.decision.
-Wrap cache.put (helper storeInCache line 1446-1458) with deco.cache.store, attrs { "cache.profile": profile, "cache.kind": "html" }.
-Attributes on every cache span: cache.profile, cache.kind, cache.decision (HIT/STALE-HIT/MISS), cache.age_seconds (when known).
-
-A5. Section loader batch wrapper (src/core/cms/sectionLoaders.ts:253)
-Wrap runSectionLoaders body in withTracing("deco.section.loaders.batch", ..., { "section.count": sections.length }) so the existing per-section spans (deco.section.loader at sectionLoaders.ts:332-334) nest cleanly under one batch parent. Makes traces readable when a page has 15+ sections.
-
-A6. Admin protocol handler spans
-The map agent found 5 admin handlers. Wrap each at the call site in src/tanstack/sdk/workerEntry.ts (inside tryAdminRoute) — not inside the core handler — so the handlers stay framework-agnostic and reusable from src/next/. Names + attrs:
-
-Span name	Call site	Attributes
-deco.admin.meta	wrapper around handleMeta	(none — handler is sync, fast)
-deco.admin.decofile.read	wrapper around handleDecofileRead	(none)
-deco.admin.decofile.reload	wrapper around handleDecofileReload	decofile.block_count (set after reload)
-deco.admin.render	wrapper around handleRender	cms.component (from pathComponent), cms.resolve_chain (bool)
-deco.admin.invoke	wrapper around handleInvoke	invoke.key, invoke.kind (loader
-For the sync handlers (handleMeta, handleDecofileRead), use a thin async wrapper inside the withTracing callback. Negligible cost; uniform span shape.
-
-A7. Logger trace correlation (src/core/sdk/observability.ts:189-216)
-Inside logRequest, before the JSON-serialize step, pull the active span’s spanContext() (via the interface from A3) and add traceId / spanId to the payload. No-op when no active span. Field names match OTel conventions so ClickStack’s auto-correlation panel works (trace_id, span_id).
-
-Same enrichment for the logger floor in setLoggerAttributeFloor — actually skip that: the floor is per-boot, not per-request. Instead, plumb traceId inside the emit() function in src/core/sdk/logger.ts:228-250 so every log line (not just logRequest) carries trace correlation. Cheap: one getActiveSpan()?.spanContext?.() call per line, no allocation when no active span.
-
-A8. W3C trace-context propagation helper (src/core/sdk/http.ts or new src/core/sdk/traceContext.ts)
-Add a small helper:
-
-export function injectTraceContext(headers: Headers): void {
-  const ctx = getActiveSpan()?.spanContext?.();
-  if (!ctx) return;
-  const flags = ctx.traceFlags.toString(16).padStart(2, "0");
-  headers.set("traceparent", `00-${ctx.traceId}-${ctx.spanId}-${flags}`);
-}
-Frame as a public helper exported from @decocms/start/sdk/observability so apps repo (@decocms/apps) can call it from createInstrumentedFetch. No new dependency — we already have @opentelemetry/api.
-
-Document on the export: “Call before sending an outbound fetch from within a withTracing scope to enable upstream trace correlation.”
-
-A9. Deferred section span (src/tanstack/routes/cmsRoute.ts:167-214)
-Wrap the internal resolveDeferredSectionPure call at line 197:
-
-const result = await withTracing(
-  "deco.section.deferred.load",
-  () => resolveDeferredSectionPure(pagePath, component, matcherCtx, opts),
-  { "section.name": component, "section.index": index, "page.path": pagePath },
-);
-Lives inside the TanStack server fn handler — the span naturally parents under whatever withTracing context CF threads through (or stands alone if not).
-
-A10. Tests
-src/core/sdk/otel.test.ts:
-
-service.version lands on the logger floor when env.CF_VERSION_METADATA = { id: "abc" }.
-service.name lands on the logger floor (from serviceName option and DECO_SITE_NAME fallback).
-The bridge’s setError sets SpanStatusCode.ERROR — assert via a vi.spyOn(trace, "getTracer") stub that returns a recording span.
-spanContext() round-trips traceId/spanId from a stub OTel span.
-src/core/sdk/observability.test.ts (new file):
-
-injectTraceContext writes a well-formed traceparent when an active span exists, no-ops otherwise.
-withTracing nests getActiveSpan correctly across await.
-src/core/cms/sectionLoaders.test.ts:
-
-New runSectionLoaders batch test: with a stub tracer, verify one deco.section.loaders.batch parent appears with section.count set.
-src/tanstack/sdk/workerEntry.test.ts (new file or extend existing):
-
-Stub caches.open + a fake cache that returns a fresh Response — verify span deco.cache.lookup is emitted with cache.decision = "HIT" and recordCacheMetric(true, profile) is called.
-Same for MISS.
-Admin route hits emit the right span names.
-src/core/sdk/logger.test.ts:
-
-A log emitted inside withTracing carries trace_id / span_id fields.
-A log emitted outside any span doesn’t carry them.
-A11. Documentation — docs/observability.md (new) + README link
-Single page covering:
-
-Architecture — site Worker → CF Destinations → deco-otel-ingest → ClickHouse → ClickStack. Diagram in plain text.
-What’s instrumented — table of every span name and its attributes:
-Span	Source	Key attributes
-deco.http.request	workerEntry root	http.method, url.path
-deco.cms.resolvePage	resolveDecoPage	deco.route
-deco.section.loaders.batch	runSectionLoaders	section.count
-deco.section.loader	runSingleSectionLoader	deco.section
-deco.section.deferred.load	loadDeferredSection	section.name, section.index
-deco.cache.lookup	edge cache match	cache.profile, cache.kind, cache.decision, cache.age_seconds
-deco.cache.store	edge cache put	cache.profile, cache.kind
-deco.admin.meta / decofile.* / render / invoke	admin route wrappers	see table in A6
-What’s measured — table of metric names:
-Metric	Source	Labels
-http_requests_total (counter)	workerEntry	method, path, status
-http_request_duration_ms (histogram)	workerEntry	method, path, status
-http_request_errors_total (counter, 5xx)	workerEntry	method, path, status
-cache_hit_total / cache_miss_total (counter)	workerEntry	profile
-resolve_duration_ms (histogram)	resolveDecoPage	(none)
-Identity stamped on every span and log: service.name, service.version (when CF_VERSION_METADATA bound), deco.runtime.version, deployment.environment, deco.apps.version (optional). Each log line additionally carries trace_id / span_id when emitted inside an active span.
-
-Required wrangler.jsonc — lift the JSDoc example from src/core/sdk/otel.ts:19-30, plus the observability.logs.destinations / observability.traces.destinations blocks pointing at https://deco-otel-ingest.deco-cx.workers.dev. Include version_metadata binding and analytics_engine_datasets.
-
-Logger output shape gap — JSON-in-stringValue for log records. Show one ClickHouse query:
-
-SELECT JSONExtract(Body, 'level', 'String') as level, count()
-FROM otel_logs
-WHERE ServiceName = 'my-store' AND Timestamp > now() - INTERVAL 1 HOUR
-GROUP BY level
-Cross-referencing logs and traces in ClickStack — filter logs by trace_id to jump from a log line to its trace.
-
-Outbound trace propagation — short note on injectTraceContext(headers) for sites/apps that issue outbound fetches.
-
-Not in scope — tail-on-error sampling (lives in ingest Worker), VTEX/Shopify-specific spans (lives in @decocms/apps).
-
-Add a link line under the “Documentation” section of README.md.
-
-Critical files
-src/core/sdk/otel.ts (A1, A2, A3)
-src/core/sdk/observability.ts (A3, A7, A8)
-src/core/sdk/logger.ts (A7)
-src/core/sdk/http.ts or new src/core/sdk/traceContext.ts (A8)
-src/core/cms/sectionLoaders.ts (A5)
-src/tanstack/sdk/workerEntry.ts (A4, A6)
-src/tanstack/routes/cmsRoute.ts (A9)
-src/core/sdk/otel.test.ts, src/core/sdk/observability.test.ts (new), src/core/sdk/logger.test.ts, src/core/cms/sectionLoaders.test.ts, src/tanstack/sdk/workerEntry.test.ts (new) (A10)
-docs/observability.md (new), README.md (link) (A11)
-Existing utilities to reuse
-withTracing (src/core/sdk/observability.ts:88-106) — span lifecycle.
-recordCacheMetric / recordRequestMetric (src/core/sdk/observability.ts:145-167) — existing meter helpers; just wire them at the cache decision points.
-MetricNames (src/core/sdk/observability.ts:131-139) — already declares RESOLVE_DURATION_MS, FETCH_DURATION_MS, CACHE_HIT, CACHE_MISS. No new metric names needed.
-getActiveSpan (src/core/sdk/observability.ts:79-81) — used in A7 (logger) and A8 (traceparent helper).
-setLoggerAttributeFloor (src/core/sdk/logger.ts:118-120) — already wired from bootObservability.
-defaultLoggerAdapter (src/core/sdk/logger.ts:49-82) — wraps console.*, picked up by CF Destinations.
-Verification
-bun run typecheck — clean.
-bun test — all new + existing tests green.
-bun run build — package builds clean; dist/ exports unchanged.
-Manual smoke on a preview deployment (after the user merges):
-Deploy site Worker with version_metadata binding + observability.{logs,traces}.destinations pointing at deco-otel-ingest.
-Hit a PDP, a PLP, a cart route (so we exercise multiple cache profiles).
-ClickHouse query confirms ResourceAttributes['service.version'], SpanAttributes['service.version'], and SpanAttributes['cache.profile'] are populated.
-Force a 5xx (invalid CMS resolve), confirm the framework span shows STATUS_CODE_ERROR and the exception message in ClickStack.
-Filter ClickStack logs by trace_id from a known request, confirm logs and traces correlate.
-Release channel
-All changes are additive — no public API removed or renamed. The only behavioral change for sites that don’t have CF_VERSION_METADATA bound is that they get richer spans and per-line trace_id on logs; no breaking surface. Target main → @decocms/start@latest. Confirm with user before opening the PR.
-
-Plan B — Brief for @decocms/apps (paste into a separate agent thread)
-
-The user will copy the section below verbatim into a new agent conversation rooted at /Users/fernandofrizzatti/development/workspace/decocms/apps/. Self-contained, no shared state with this thread.
-
-Brief: instrument @decocms/apps for end-to-end observability
-Context
-
-The deco platform pipes telemetry from site Workers → CF Destinations → deco-otel-ingest (https://deco-otel-ingest.deco-cx.workers.dev) → stats-lake ClickHouse → ClickStack UI (hyperdx.clickhouse.cloud). The framework (@decocms/start) emits service identity (service.name, service.version from CF_VERSION_METADATA), a root deco.http.request span, per-section spans, cache decision spans, admin protocol spans, and W3C traceparent-injection helper injectTraceContext(headers) exported from @decocms/start/sdk/observability.
-
-What’s missing is per-app commerce instrumentation. VTEX, Shopify, Nuvemshop, etc. all issue many outbound fetches per request — search, catalog, OMS, intelligent-search, checkout. Today these are CF-auto-instrumented as flat fetch spans with no semantic grouping. Operators can’t answer “is intelligent-search slow today?” without grep-style URL filtering.
-
-Investigation tasks
-
-Audit createInstrumentedFetch — find its current implementation in this repo (likely under vtex/utils/ or utils/). For each:
-Does it use withTracing from @decocms/start? If yes, what’s the span name shape?
-Are spans named per-endpoint (e.g. vtex.intelligent-search.product_search) or just per-client (vtex.fetch)?
-Are HTTP-level attributes set (http.method, http.url, http.status_code, http.response.body_size)?
-Does it set STATUS_CODE_ERROR on 5xx responses or on thrown errors?
-Is response timing recorded as a histogram metric (e.g. vtex_request_duration_ms with endpoint label)?
-W3C trace propagation — does createInstrumentedFetch inject traceparent headers? If not, import injectTraceContext from @decocms/start/sdk/observability and call it before each outbound fetch. Confirms downstream correlation if upstream is OTel-aware.
-Per-app coverage matrix — list every commerce app (VTEX, Shopify, Wake, Nuvemshop, …) and whether each uses createInstrumentedFetch consistently across loaders, actions, and hooks. Identify any direct fetch() calls that bypass instrumentation.
-Cross-app pattern — should there be a shared createInstrumentedFetch(provider: string) factory that emits spans {provider}.{endpoint}, with provider as a label on the per-app histogram? Today each app likely has its own.
-GraphQL — VTEX/Shopify use GraphQL. Should the span attributes include graphql.operation.name so dashboards can group by GraphQL operation rather than POST endpoint?
-Deliverables (after investigation)
-
-A short report (in the conversation, no file) summarizing the current state of fetch instrumentation across apps.
-A unified PR (or per-app PRs) that:
-Standardizes createInstrumentedFetch to emit a span per outbound call with {provider}.{operation_or_endpoint} shape, full HTTP attrs, error status on failures, and traceparent propagation.
-Adds a per-provider duration histogram (vtex_request_duration_ms, shopify_request_duration_ms) with endpoint / operation labels.
-Wraps GraphQL clients to expose graphql.operation.name on the span.
-Audits direct fetch() calls and converts them to createInstrumentedFetch.
-Local smoke verification (run a site against a real VTEX account, confirm spans appear in ClickStack with the expected shape).
-Constraints
-
-Don’t add new dependencies. @opentelemetry/api is already pulled in transitively via @decocms/start.
-Don’t change public function signatures — wrap implementations, don’t break consumers.
-PII redaction is handled at the ingest Worker; no per-app code needed for that.
-Don’t introduce a custom OTLP exporter — telemetry rides the CF Destinations pipe.
-Reference files (in @decocms/start, after the upcoming PR lands)
-
-src/core/sdk/observability.ts — withTracing, injectTraceContext, getActiveSpan.
-src/core/sdk/logger.ts — logger.info|warn|error (logs carry trace_id automatically).
-docs/observability.md — full pipeline + query patterns.
-Approved
-All 11 plan items complete. Verification clean: bun run typecheck, all 553 tests pass, bun run build succeeds, biome lint clean on touched files, tier boundaries clean.
-
-Summary of changes (this repo)
-
-Identity & errors — service.name and service.version (from env.CF_VERSION_METADATA.id) now stamp on every framework span and every log line via the attribute floor. Framework spans correctly carry SpanStatusCode.ERROR with the exception message when withTracing() rethrows. Span interface gained an optional spanContext() so logger/propagation helpers can read trace IDs.
-
-New spans — deco.section.loaders.batch, deco.section.deferred.load, deco.cache.lookup (×2 paths: SFn + HTML), deco.cache.store (×2), deco.admin.{meta,decofile.read,decofile.reload,render,invoke}. Wired at workerEntry callsites and TanStack admin routes.
-
-Cache metrics — extended recordCacheMetric(hit, profile, decision) with a CacheDecision union (“HIT” | “STALE-HIT” | “STALE-ERROR” | “MISS” | “BYPASS”). Wired at every HIT / STALE-HIT / MISS / STALE-ERROR decision point in both cache paths.
-
-Trace correlation — every log emitted inside a withTracing() scope now carries trace_id / span_id for cross-referencing logs and traces in ClickStack. logRequest does the same.
-
-injectTraceContext(headers) — new public helper that writes a W3C traceparent header from the active span. Exported from @decocms/start/sdk/observability (and /middleware) so @decocms/apps’s createInstrumentedFetch can propagate context to upstream services.
-
-Tests — 11 new test cases covering: identity floor, error status, spanContext round-trip, batch-loader span, deferred span, trace-id in logs, injectTraceContext happy/no-op paths, getActiveSpan nesting.
-
-Docs — new docs/observability.md covering architecture, instrumented spans, metrics, identity floor, wrangler.jsonc shape, ClickHouse JSONExtract query patterns, log-to-trace correlation, injectTraceContext usage, sampling notes, and what’s out of scope. README links it under “Documentation”.
-
-Apps brief
-
-The standalone apps brief is in Plan B of /Users/fernandofrizzatti/.claude/plans/system-instruction-you-are-working-shimmying-badger.md — copy that section into a new agent thread rooted at /Users/fernandofrizzatti/development/workspace/decocms/apps/.
-
-PR instructions.md
-The user likes the current state of the code. Ther...
-Create a PR
-PR created: https://github.com/decocms/deco-start/pull/179
-
-The branch is pushed, the commit follows the repo’s conventional-commit style, and the PR body lays out what shipped plus the post-merge smoke steps from the plan.
-
-
-# Third Context: decocms/apps
-https://github.com/decocms/deco-start/pull/179 check that PR, check if that solves some problems, if any
-Exit code 1 accepts at most 1 arg(s), received 3
-PR 179 in decocms/deco-start directly addresses the framework gaps I found. Summary:
-
-injectTraceContext(headers: Headers) is added and exported from @decocms/start/sdk/observability — exactly the symbol the brief assumed exists. No-op when no active span. ✅
-Log↔trace correlation is automatic: any log emitted inside a withTracing(...) scope gets trace_id/span_id stamped via the logger attribute floor. Free win for @decocms/apps once we wrap call sites.
-recordCacheMetric(hit, profile, decision?) extended with decision label (HIT|STALE-HIT|STALE-ERROR|MISS|BYPASS).
-New framework spans added (cache lookup/store, loaders batch, deferred section load, admin handlers). Doesn’t affect us, but means once our outbound fetch spans nest under deco.http.request, the whole tree is contiguous.
-spanContext?() added to Span — adapters that implement it light up trace propagation; adapters that don’t degrade gracefully.
-What PR 179 does not solve (still on the @decocms/apps plate):
-
-createInstrumentedFetch is unchanged — still emits only ${name}.fetch spans, still does not call injectTraceContext (the PR’s doc claims it does, but the diff doesn’t touch src/core/sdk/instrumentedFetch.ts). So either (a) we add injectTraceContext inside our per-app fetch wrappers in this repo, or (b) we open a follow-up upstream PR to wire it into createInstrumentedFetch itself.
-Per-operation span granularity — upstream still produces vtex.fetch. To get vtex.intelligent-search.product_search, we wrap call sites in withTracing(...) inside @decocms/apps.
-GraphQL operationName on spans — needs to happen in our Shopify GraphQL client.
-~27 raw fetch( sites that bypass _fetch need to be routed through it.
-Per-provider duration histograms — need to call getMeter()?.histogramRecord(...) ourselves at the per-app fetch helper.
-Let me write the plan, then I’ll ask a couple of decisions via ExitPlanMode.
-Instrument @decocms/apps for end-to-end observability
-
-Context
-
-The deco platform pipes telemetry from site Workers → CF Destinations → deco-otel-ingest → ClickHouse stats-lake → ClickStack UI. The framework half (@decocms/start) emits a root deco.http.request span, per-section spans, cache decision spans, admin protocol spans, and exposes tracer/meter primitives. The apps half (@decocms/apps — this repo) is the integration layer where outbound commerce fetches happen (VTEX REST, Shopify GraphQL, Resend, Google Fonts, etc.). Today those fetches are essentially invisible to operators — the only span shape produced is vtex.fetch / shopify.fetch via createInstrumentedFetch(name), and most fetches bypass even that.
-
-The goal: every outbound fetch in every app produces a span named {provider}.{operation}, carries full HTTP attributes, sets error status on failures, records a per-provider duration histogram with operation label, and propagates W3C traceparent so any OTel-aware upstream joins the trace.
-
-Upstream dependency: this plan depends on decocms/deco-start#179 being released. That PR adds injectTraceContext, CacheDecision, spanContext?() on the Span interface, and automatic log/trace correlation via the logger attribute floor. Without it, we’d need to vendor a W3C propagator. Once it ships, we just import.
-
-Current state (from exploration)
-
-createInstrumentedFetch is defined in @decocms/start/sdk/instrumentedFetch, not in this repo. Span shape is fixed at ${name}.fetch. No HTTP attributes, no traceparent injection, no per-operation granularity.
-Wiring lives in this repo via setter pattern:
-vtex/client.ts:117 — setVtexFetch(fetchFn) stores into module-level _fetch. Called at startup in setup.ts.
-shopify/client.ts:24 — setShopifyFetch(fetchFn) does the same for Shopify.
-Only 4 paths actually use _fetch:
-vtex/client.ts — vtexFetchResponse:214, vtexCachedFetch:260, intelligentSearch:360
-shopify/utils/graphql.ts:32 — createGraphqlClient.query (when fetchFn arg passed)
-~27 raw fetch( sites bypass instrumentation entirely:
-VTEX actions (6): actions/analytics/sendEvent.ts:54, actions/profile.ts, actions/masterData.ts, …
-VTEX hooks (9): hooks/useCart.ts:77,96,111,131, hooks/useWishlist.ts:34,48,58, hooks/useAutocomplete.ts:49, hooks/useUser.ts:34
-VTEX utils (5): utils/authHelpers.ts:52, utils/proxy.ts:176,383, utils/fetch.ts:91
-Shopify: 1 fallback in utils/graphql.ts when fetchFn is undefined
-Resend: resend/actions/send.ts:35
-Website: website/loaders/fonts/googleFonts.ts:97,103
-Shopify GraphQL client (shopify/utils/graphql.ts) does not surface operationName anywhere — span attribute opportunity.
-No file in this repo currently imports from @decocms/start/sdk/observability.
-Approach
-
-Two layers of change in this repo:
-
-A. A small shared “traced fetch” helper
-Add a single helper that all per-app fetch utilities funnel through. It composes the three pieces we need on every outbound call:
-
-// utils/tracedFetch.ts (new file)
-import { withTracing, injectTraceContext, getMeter, setSpanAttribute } from "@decocms/start/sdk/observability";
-
-export interface TracedFetchOptions {
-  provider: string;            // e.g. "vtex"
-  operation: string;           // e.g. "intelligent-search.product_search"
-  attributes?: Record<string, string | number | boolean>;
-}
-
-export async function tracedFetch(
-  input: Request | URL | string,
-  init: RequestInit | undefined,
-  opts: TracedFetchOptions,
-  underlyingFetch: typeof fetch = fetch,
-): Promise<Response> {
-  const headers = new Headers(init?.headers);
-  injectTraceContext(headers);
-  const url = typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
-  const method = init?.method ?? "GET";
-  const start = performance.now();
-
-  return withTracing(
-    `${opts.provider}.${opts.operation}`,
-    async () => {
-      setSpanAttribute("http.method", method);
-      setSpanAttribute("http.url", redactUrl(url));   // strip query secrets like authToken
-      for (const [k, v] of Object.entries(opts.attributes ?? {})) setSpanAttribute(k, v);
-
-      let res: Response;
-      try {
-        res = await underlyingFetch(input, { ...init, headers });
-      } catch (err) {
-        recordDuration(opts, performance.now() - start, "error");
-        throw err;
-      }
-
-      setSpanAttribute("http.status_code", res.status);
-      const len = res.headers.get("content-length");
-      if (len) setSpanAttribute("http.response.body_size", Number(len));
-      recordDuration(opts, performance.now() - start, String(res.status));
-      return res;
-    },
-  );
-}
-
-function recordDuration(opts: TracedFetchOptions, ms: number, status: string) {
-  getMeter()?.histogramRecord?.("commerce_request_duration_ms", ms, {
-    provider: opts.provider,
-    operation: opts.operation,
-    status,
-  });
-}
-Notes:
-
-Error status on 5xx and on thrown errors is handled by withTracing itself (PR 179 sets SpanStatusCode.ERROR when the wrapped function rethrows). For 5xx responses we don’t throw — we just rely on the http.status_code attribute for filtering, which is the OTel-recommended approach (status codes ≠ span errors unless the client treats them as errors).
-We rely on getMeter()?.histogramRecord?.(...) from PR 179. Single generic histogram commerce_request_duration_ms with provider + operation labels is more useful for cross-provider dashboards than per-provider names; matches how http_request_duration_ms is shaped upstream.
-redactUrl strips known-sensitive query params (authToken, cookie, vtex_session, etc.) consistent with the existing sanitization in vtex/utils/fetch.ts.
-B. Funnel every raw fetch through it
-VTEX (vtex/):
-
-In vtex/client.ts, change vtexFetchResponse, vtexCachedFetch, intelligentSearch to call tracedFetch(url, init, { provider: "vtex", operation }, _fetch) instead of _fetch(url, init). operation is hardcoded per call path:
-vtex.api (catch-all in vtexFetchResponse)
-vtex.intelligent-search.{endpoint} — parsed once from the URL path inside intelligentSearch
-vtex.catalog.*, vtex.checkout.orderForm, etc. as appropriate
-Convert each raw fetch(...) in vtex/actions/, vtex/hooks/, vtex/utils/authHelpers.ts, vtex/utils/proxy.ts, vtex/utils/fetch.ts to either:
-Go through _fetch (preferred — they’re VTEX-account-aware), then through tracedFetch, with an operation derived from the call site (e.g. vtex.checkout.addItems, vtex.events.send, vtex.wishlist.list, …).
-For the proxy paths (utils/proxy.ts) where we forward arbitrary requests, use operation vtex.proxy and set proxy.path as an attribute.
-The hook fetches (hooks/useCart.ts, etc.) currently run on the browser via the invoke proxy → server function path. They still need instrumentation on the server side; if the hooks themselves run on the client they won’t have a span context, and injectTraceContext will no-op. That’s fine. The server-side _fetch call is where instrumentation lives.
-Shopify (shopify/):
-
-shopify/utils/graphql.ts — wrap the POST in tracedFetch with operation: graphqlOperationName(query) ?? "graphql" and set graphql.operation.name as a span attribute. Parse the operation name with a small regex on the GraphQL document (/^\s*(?:query|mutation|subscription)\s+(\w+)/) — no external dep needed.
-Remove the globalThis.fetch fallback path: require that callers route through setShopifyFetch at startup. (Or wrap the fallback in tracedFetch too — pick whichever is least disruptive to existing tests.)
-Resend, Website:
-
-resend/actions/send.ts:35 — wrap in tracedFetch with provider: "resend", operation: "emails.send".
-website/loaders/fonts/googleFonts.ts:97,103 — wrap with provider: "google-fonts", operation: "css". Low priority but cheap.
-Setter signature unchanged: setVtexFetch / setShopifyFetch still accept the result of createInstrumentedFetch("vtex") / ("shopify") from @decocms/start. The new tracedFetch helper sits between the call site and _fetch, so the upstream ${provider}.fetch span becomes the parent of our per-operation span. Good — gives operators both views.
-
-Files to modify
-
-utils/tracedFetch.ts (new)
-utils/tracedFetch.test.ts (new — exercise span name, attributes, traceparent header, histogram record, error status)
-vtex/client.ts — route vtexFetchResponse, vtexCachedFetch, intelligentSearch through tracedFetch
-vtex/utils/fetch.ts — route the generic wrapper through tracedFetch
-vtex/utils/proxy.ts — wrap the two fetch calls
-vtex/utils/authHelpers.ts — wrap the logout call
-vtex/actions/analytics/sendEvent.ts, vtex/actions/profile.ts, vtex/actions/masterData.ts, and the rest of the VTEX action files identified during audit
-vtex/hooks/useCart.ts, vtex/hooks/useWishlist.ts, vtex/hooks/useAutocomplete.ts, vtex/hooks/useUser.ts — convert direct fetch to the per-app wrapper
-shopify/utils/graphql.ts — wrap query in tracedFetch, surface graphql.operation.name
-resend/actions/send.ts — wrap in tracedFetch
-website/loaders/fonts/googleFonts.ts — wrap in tracedFetch
-package.json — bump @decocms/start to the version that includes PR 179 (likely a next channel release first)
-Reused helpers
-
-withTracing, injectTraceContext, setSpanAttribute, getMeter, MetricNames — all from @decocms/start/sdk/observability (post-PR-179).
-createInstrumentedFetch — still the boundary at startup, unchanged.
-RequestContext from @decocms/start/sdk/requestContext — already imported in this repo; useful inside tracedFetch if we want to stamp cookie.account or similar on spans.
-Release path
-
-Per CLAUDE.md, this is behavior-changing (new spans, new metric names, log volume). Route through @next:
-
-Open PR to next branch (not main). Title/body must avoid the canonical CI-skip token.
-Validate on a real VTEX storefront via npm install @decocms/apps@next.
-Confirm spans appear in ClickStack with expected shape (vtex.intelligent-search.product_search, full HTTP attrs, traceparent received downstream).
-Promote next → main via PR.
-Verification
-
-npm run check clean (typecheck + biome + knip).
-npm test — new unit tests on tracedFetch pass:
-Asserts span name is ${provider}.${operation}.
-Asserts http.method, http.url (redacted), http.status_code attributes set.
-Asserts traceparent header is injected when called inside withTracing (use the fake tracer pattern from PR 179’s observability.test.ts).
-Asserts histogram histogramRecord is called with commerce_request_duration_ms and the right labels.
-Asserts span has STATUS_CODE_ERROR when the underlying fetch throws.
-Local smoke against a real VTEX storefront:
-Bind to a dev site, hit PLP/PDP/cart pages.
-Tail ClickHouse / ClickStack for service.name = <site> and confirm vtex.intelligent-search.*, vtex.checkout.*, vtex.api spans appear nested under deco.http.request.
-Confirm logs from inside those fetches show trace_id/span_id matching the parent trace.
-Confirm at least one VTEX endpoint that has its own OTel collector receives our traceparent header (curl or check VTEX-side logs if available).
-Open decisions
-
-Operation-name source — explicit per-caller string vs URL-derived inside tracedFetch. The plan above assumes explicit per-caller (most accurate, no regex maintenance), but a URL→operation router inside tracedFetch would mean fewer per-call-site edits. Worth deciding before implementation starts.
-Should injectTraceContext live in upstream createInstrumentedFetch instead? PR 179’s doc claims it does, but the diff doesn’t touch instrumentedFetch.ts. If yes, file a separate upstream PR; if no, our tracedFetch is the right place.
-Per-provider vs generic histogram name — plan uses generic commerce_request_duration_ms with labels. Brief asked for vtex_request_duration_ms. Generic is more flexible; brief-literal is matchable to existing per-provider dashboards if any exist. Decide.