@voltro/cli 0.1.0

package/templates/agent-docs/observability.md

@@ -0,0 +1,350 @@
+# Observability
+
+> OpenTelemetry tracing in Voltro — the auto-emitted spans for every primitive, span attributes and nesting, the three enabling modes (console / OTLP / buffer), and adding your own spans with Effect.withSpan.
+
+
+
+---
+
+<!-- source: en/observability/overview.md -->
+## Overview
+
+_OpenTelemetry tracing in Voltro — the auto-emitted spans for every primitive, span attributes and nesting, the three enabling modes (console / OTLP / buffer), and adding your own spans with Effect.withSpan._
+
+Every framework primitive emits **OpenTelemetry spans** when tracing is enabled. You don't instrument anything by hand — the handler boundaries, the transactional scope, and the streaming deliveries are all traced automatically, and the spans nest into a single waterfall per request.
+
+## Auto-emitted spans
+
+| Span | What it marks |
+|---|---|
+| `client.mutation.<rpcTag>` / `client.subscription.<rpcTag>` | The browser-side span that **roots** the trace and sets the shared `traceId`. |
+| `mutation.<rpcTag>` | The rpc mutation-handler boundary. |
+| `action.<rpcTag>` | The unary action-handler boundary. |
+| `subscription.<rpcTag>` | The streaming-rpc handler **setup** (subscribe → first snapshot). NOT the open-duration. |
+| `subscription.<rpcTag>.snapshot` / `.delta` | Each **data delivery** to the subscriber, with its real produce→push latency. |
+| `store.transactional` | The postgres `BEGIN`/`COMMIT` (or retry) scope. Carries `db.system=postgresql` + `db.operation=transaction`. |
+| `webhook<path>` | An inbound `*.webhook.tsx` route (an incoming HTTP handler, not a query). Continues the caller's trace when the request carries a `traceparent` header. |
+
+## Span attributes
+
+Handler spans carry:
+
+- `rpc.tag` — the rpc tag of the call
+- `subject.type` — the authenticated subject type
+- `tenant.id` — the active tenant
+
+`@effect/rpc`'s own `RpcServer.<tag>` transport span is **suppressed** from the trace view — the framework spans carry the real latencies. For streaming subscriptions the transport span would just be a misleading open-duration bar, so the per-delivery `.snapshot` / `.delta` spans surface the real latency instead.
+
+## Nesting
+
+Spans nest. A mutation handler span wraps the `store.transactional` span, which wraps any `Effect.withSpan` you add inside the executor:
+
+```
+client.mutation.createOrder
+└─ mutation.createOrder           rpc.tag, subject.type, tenant.id
+   └─ store.transactional         db.system=postgresql, db.operation=transaction
+      └─ createOrder.charge-card  (your custom span)
+```
+
+## Enabling modes
+
+Three modes, **env-driven, no code changes**:
+
+```bash
+# Default — no EXPORTER. `voltro dev` still runs a buffer-only tracer
+# (powers the Traces dashboards + traceId-in-logs); nothing is shipped
+# off-box. `VOLTRO_TRACING_BUFFER=off` disables even that.
+unset FRAMEWORK_TRACING OTEL_EXPORTER_OTLP_ENDPOINT
+
+# Console (local dev — spans printed to stdout)
+FRAMEWORK_TRACING=console voltro dev .
+
+# OTLP (production — sends to any OTLP/HTTP collector: Jaeger, Tempo, Honeycomb, etc.)
+OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 voltro dev .
+# OTEL_SERVICE_NAME defaults to 'voltro-api'; override per app:
+OTEL_SERVICE_NAME=my-api OTEL_EXPORTER_OTLP_ENDPOINT=... voltro dev .
+```
+
+The framework **auto-detects**: if any standard OpenTelemetry env-var (`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`) is set, the mode resolves to `otlp` automatically. Set `FRAMEWORK_TRACING` explicitly (`console` / `otlp` / `off`) to force a mode.
+
+Even with no exporter, `voltro dev` installs a **buffer-only** tracer — that's what powers the [DevTools / cloud Traces panel](/docs/observability/distributed-tracing) and the trace id stamped on every log line. Set `VOLTRO_TRACING_BUFFER=off` to disable it entirely.
+
+## Metrics export
+
+Separate from tracing, the framework records **metrics** into Effect's global `MetricRegistry` — one source of truth (`voltro_rpc_*`, `voltro_http_*`, `voltro_plugin_hook_*`, `voltro_subscription_*` + `voltro_subscriptions_active`, plus `effect_fiber_*` and any custom metric). Three ways to get them out:
+
+```bash
+# OTLP metrics — the SAME OTEL endpoint that enables trace export also enables
+# metrics. Ships to any OTLP/HTTP collector (Prometheus OTLP, Grafana Agent,
+# the Datadog Agent's OTLP port, etc.). @effect/opentelemetry auto-bridges the
+# Effect MetricRegistry into the OTel MeterProvider — no per-metric wiring.
+OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 voltro start
+# metrics-only endpoint (no traces):
+OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://collector:4318/v1/metrics voltro start
+# console (local dev — periodic metric dump to stdout):
+FRAMEWORK_METRICS=console voltro dev .
+# export cadence (ms): OTEL_METRIC_EXPORT_INTERVAL (default 60000).
+```
+
+The framework auto-detects: `OTEL_EXPORTER_OTLP_ENDPOINT` (or the metrics-specific endpoint) → metrics export `otlp`; force with `FRAMEWORK_METRICS` (`console` / `otlp` / `off`).
+
+Two pull/push consumers read the SAME registry, so they can't disagree:
+
+- **[`@voltro/plugin-prometheus`](/docs/plugins/prometheus)** — `GET /metrics` scrape endpoint (Prometheus / Grafana).
+- **[`@voltro/plugin-datadog`](/docs/plugins/datadog)** — agentless push to Datadog's `/api/v2/series` (for setups without an Agent / OTLP collector). With an Agent, prefer pointing `OTEL_EXPORTER_OTLP_ENDPOINT` at it instead.
+- The dashboard **Metrics** panel reads the same snapshot via `GET /_voltro/inspect/metrics`.
+
+## Routing traces to a vendor
+
+The env-driven OTLP path above ships traces to any OTLP/HTTP collector. For **deep, opt-in vendor integration** — install one plugin, get traces + errors + logs all correlated by the same `traceId`, zero `OTEL_*` env — a plugin can contribute to the framework's tracer directly via `contributeObservability`:
+
+```ts
+interface ObservabilityContribution {
+  resourceAttributes?: Record<string, string>
+  spanProcessors?: ReadonlyArray<unknown>   // OTel SpanProcessor — receives every framework span
+  metricReaders?: ReadonlyArray<unknown>    // OTel MetricReader
+  sampler?: unknown                          // OTel Sampler (use with care — see below)
+}
+```
+
+The CLI gathers every plugin's contribution at boot and passes the merged span-processors / metric-readers / resource-attributes into the framework's NodeSdk tracer. The vendor SDK runs as an OTel **consumer** — it gets a `SpanProcessor` alongside the framework's own buffer sink; it never becomes the global provider. So the in-app **Traces dashboard keeps working** and the vendor receives the identical spans.
+
+This is how [`@voltro/plugin-sentry`](/docs/plugins/sentry) (`SentrySpanProcessor`) and [`@voltro/plugin-datadog`](/docs/plugins/datadog) (`OTLPTraceExporter` → DD Agent) route traces with no env fiddling.
+
+> **Don't contribute a sampler that drops spans.** The framework's buffer sink (powering the in-app Traces dashboard) is a `SpanProcessor` — it only sees *recorded* spans. A vendor sampler set as the tracer's sampler gates recording for the WHOLE tracer, blinding the dashboard. The framework stays always-on; vendors sample at their own export layer (e.g. Sentry's `tracesSampleRate` is applied by the Sentry client, not a tracer sampler).
+
+To attach an error to the exact active span, the interceptor context carries both `ctx.traceId` and `ctx.spanId` — plugins read them explicitly (the active OTel span lives in Effect's fiber context, not the AsyncLocalStorage vendor SDKs read implicitly).
+
+## Adding your own spans
+
+Inside a mutation / action / workflow executor, just use Effect:
+
+```typescript
+import { Effect } from 'effect'
+
+export const myAction = defineAction({ ... })
+
+export default (input, ctx) =>
+  Effect.gen(function* () {
+    yield* someExpensiveWork.pipe(
+      Effect.withSpan('myAction.expensive-work', {
+        attributes: { 'input.id': input.id },
+      }),
+    )
+    return { ok: true }
+  })
+```
+
+The span nests under the auto-emitted `action.myAction` span. In synchronous (non-Effect) executors, ad-hoc spans require explicitly adopting the Effect runtime — usually not worth it; rely on the auto spans the framework emits at each handler boundary.
+
+Continue to [Distributed tracing](/docs/observability/distributed-tracing) for how one `traceId` flows frontend → api → api, and [Traces & logs from the shell](/docs/observability/cli) for the `voltro traces` / `voltro logs` workflow.
+
+
+
+---
+
+<!-- source: en/observability/distributed-tracing.md -->
+## Distributed tracing
+
+_One traceId from frontend through every api hop — automatic W3C propagation, ctx.request.traceId, traceId-in-logs, the dev error-surfacing toast, and the DevTools / cloud Traces panel._
+
+Trace continuity is **automatic and on by default** in `voltro dev` — a buffer-only tracer is always installed (set `VOLTRO_TRACING_BUFFER=off` to disable). You don't wire anything. A single request gets **one `traceId`** that flows frontend → api → api, and every log line of that request carries it.
+
+## How propagation works
+
+- **Frontend → server is automatic.** `useMutation` / `useSubscription` wrap each rpc call in a client span; `@effect/rpc` propagates that span's W3C trace id to the server, so the handler span — and everything it calls — shares the **same** `traceId`. In any handler, `ctx.request.traceId` is that id.
+- **api → api is automatic over HTTP.** An action or handler that calls another api via the framework `HttpClient` sends a W3C `traceparent` header; an inbound `*.webhook.tsx` route (an incoming HTTP handler) **continues** the caller's trace. So frontend → api Y → api Z (via webhook / HTTP) is **one** trace.
+- **Every log line carries it.** Handlers stamp `fields.traceId` onto each log line they emit, so [`voltro logs --trace <id>`](/docs/observability/cli) returns the whole chain across hops, in order.
+
+```
+client.mutation.placeOrder        traceId=4bf92f35…
+└─ mutation.placeOrder            traceId=4bf92f35…   (api Y, ctx.request.traceId)
+   └─ webhook/fulfilment          traceId=4bf92f35…   (api Z, continued via traceparent)
+```
+
+## Seeing traces
+
+- **DevTools "Traces" panel** (dev) — a waterfall per request, failed hops tinted red. Backed by the in-memory ring buffer.
+- **Cloud dashboard's per-app "Traces" page** — same waterfall, with history kept in `_voltro_traces` (postgres) **when durable persistence is enabled** (dev: on by default, interesting-only; prod: off by default → use OTLP). The in-memory ring is always the live source; see *Durable trace volume* below.
+
+## Durable trace volume (postgres)
+
+The in-memory ring always keeps EVERY span for live debugging. SEPARATELY, on postgres, the framework can durably mirror a SUBSET of spans into `_voltro_traces` so the in-app Traces dashboard survives restarts. This is a **dev convenience** — at real volume durable tracing belongs in an **OTLP backend (Tempo / Honeycomb / Datadog), not your OLTP postgres** (set `OTEL_EXPORTER_OTLP_ENDPOINT`). A row-per-span firehose into postgres is what turned this table into ~85% of a production DB. So persistence is **environment-aware + fully configurable**:
+
+- **Default OFF in production, `interesting` in dev.** `voltro dev` (not `NODE_ENV=production`) persists only "interesting" spans. `voltro serve` / `voltro start` default `NODE_ENV=production` when it's unset, so they persist NOTHING and **`_voltro_traces` is not even created** — use OTLP in prod. An explicit `NODE_ENV` is never overridden.
+- **`VOLTRO_TRACING_PERSIST`** = `off` | `errors` | `interesting` | `all` (overrides the env default everywhere). `interesting` = error + slow + per-trace root spans, MINUS the high-volume subscription delivery (snapshot/delta) spans. `errors` = error spans only. `all` = every non-delivery span. `off` = no persistence and no table.
+- **`VOLTRO_TRACING_SLOW_MS`** (default `500`) — the "slow" threshold used by `interesting`.
+- **`VOLTRO_TRACING_PERSIST_DELIVERY`** (default off) — also persist the subscription delivery spans (the firehose; rarely wanted).
+- **`VOLTRO_TRACING_SAMPLE`** (`0`–`1`, default `1`) — per-*trace* sampling (a kept trace keeps all its eligible spans; errors are never sampled out). `=0` with no explicit mode means off.
+- **`VOLTRO_TRACING_TTL_HOURS`** (default `24`) — the GC prunes rows past the TTL; it loops until the whole over-TTL backlog drains each pass (batched set-based delete), so it always catches up. It WARNs if a pass errors or is shifting very high volume — a silently-failing GC is no longer silent.
+- **`VOLTRO_TRACING_BUFFER=off`** — disables the durable flusher + the in-memory tracer entirely (and drops the table).
+
+`_voltro_traces` has no foreign keys — `TRUNCATE TABLE _voltro_traces` reclaims the space instantly if it ever grew under old defaults. To get the in-app Traces *history* in prod anyway, set `VOLTRO_TRACING_PERSIST=interesting` (keeps it small) — but prefer OTLP for production-grade tracing.
+
+## Dev error-surfacing toast
+
+In dev, if any hop of a trace **you** triggered errors — even when the frontend call itself returned OK (a fire-and-forget emit, a detached workflow, a swallowed downstream error) — an auto-toast appears in the browser pointing at the failing hop and at `voltro logs --trace <id>`.
+
+This catches the class of failure that's normally invisible: the UI shows success, but a downstream hop quietly failed. The toast surfaces it immediately with the exact `traceId` to chase.
+
+## The debugging loop
+
+When you debug a failure:
+
+1. Get the `traceId` — from the dev error toast, an error line's `fields.traceId`, or `/_voltro/inspect/traces?onlyErrors=1`.
+2. Run [`voltro logs --trace <id>`](/docs/observability/cli) — that's the end-to-end causal chain, every log line of the request across hops, in order.
+
+This is the fastest way to answer "where did this error come from?" — far faster than inferring it from source or from the user's description.
+
+## Shipping traces to a vendor (Sentry / Datadog)
+
+The same trace continuity flows to a vendor APM when you install a deep-observability plugin — no `OTEL_*` env. Each contributes an OTel span-processor to the framework's tracer via `contributeObservability`, so the vendor runs as a **consumer** of the existing tracer (the in-app Traces panel keeps working) and receives the identical spans:
+
+- [`@voltro/plugin-sentry`](/docs/plugins/sentry) — every mutation/query/action error becomes a Sentry issue correlated to its `trace_id` + `span_id`, with the request's log lines as breadcrumbs; opt-in performance traces (`traces: true`).
+- [`@voltro/plugin-datadog`](/docs/plugins/datadog) — framework spans → the Datadog Agent's OTLP receiver (`traces: true`), plus log forwarding with `dd.trace_id` correlation.
+
+See [Observability › Routing traces to a vendor](/docs/observability/overview#routing-traces-to-a-vendor) for the contribution surface + the sampler caveat.
+
+
+
+---
+
+<!-- source: en/observability/cli.md -->
+## voltro traces & voltro logs
+
+_Inspect traces and logs from the shell — flags, the --trace <id> end-to-end chain workflow, --format json for programmatic analysis, and the guidance for AI coding agents._
+
+`voltro logs` and `voltro traces` give you the running app's log buffer and trace buffer from a shell — no DevTools UI needed. Both discover every running process via `~/.voltro/runtime-registry.json`, fan out to each one's inspect endpoint in parallel, merge by timestamp, and render.
+
+## `voltro logs`
+
+Every `voltro dev` / `voltro start` instance buffers the last **2000** server-side `@voltro/logger` records **and** every browser console line that the dev console bridge forwarded.
+
+```sh
+voltro logs                                # last 100 from every running process
+voltro logs --tail 50 --level error        # only errors, last 50
+voltro logs --since 30s                     # last 30 seconds
+voltro logs --scope 'orders'               # case-insensitive scope filter
+voltro logs --filter 'channels.create'     # message substring
+voltro logs --source client                # only browser-originated lines
+voltro logs --trace <traceId>              # the WHOLE causal chain for one request
+voltro logs --process voltroCloudApi       # restrict to one registry name
+voltro logs --format json | jq '.[]'       # machine-parseable
+```
+
+Flags: `--tail <n>` (default 100, cap 2000), `--since <spec>` (`30s` / `5m` / `1h` / unix-ms), `--level <l>` (`trace|debug|info|warn|error|fatal`, minimum level), `--scope <substr>`, `--source <server|client>`, `--filter <substr>`, `--trace <traceId>`, `--process <name>`, `--format <pretty|json|text>` (default `pretty`), `--no-color`.
+
+`--format json` returns one object per record: `{ ts, level, source: 'server'|'client', scope, message, fields }`.
+
+## `voltro traces`
+
+Mirror of `voltro logs` for the trace buffer — a transient ring of 2000 records, no persistence.
+
+```sh
+voltro traces                            # 20 newest traces, pretty
+voltro traces --errors                   # only traces containing an errored span
+voltro traces --since 30s --tail 50      # recent
+voltro traces --process voltroCloudApi   # restrict to one process
+voltro traces --id <traceId>             # one trace, span waterfall (detail mode)
+voltro traces --errors --format json | jq '.traces[]'
+```
+
+List-mode flags: `--tail <n>`, `--since <spec>`, `--errors`, `--process <name>`, `--format <pretty|json>` (default `pretty`), `--no-color`.
+
+Detail mode (`--id <traceId>`, alias `--trace`) renders one trace's span waterfall and additionally accepts `--min-duration <ms>` and `--status <ok|error|unset>` to filter the spans shown.
+
+**Subscriptions vs requests in the output**: traces from `useMutation` / `useSubscription` / actions / webhooks show `spans · duration` (the duration *is* the call's latency). Subscriptions show `sub · N deliveries · last Xms` — the open-duration is liveness, not latency, so the command surfaces the latency of the most recent data transfer instead (snapshot + each delta is its own span with its produce→push latency).
+
+## The `--trace <id>` end-to-end chain
+
+The whole point of the trace id is that it's **shared** across the entire frontend → api → api chain (see [Distributed tracing](/docs/observability/distributed-tracing)). Once you have a `traceId`, you can pull the complete causal chain for one request across every hop, in order:
+
+```sh
+# 1. find a failure
+voltro traces --errors --format json | jq '.traces[]'
+
+# 2. pull the full chain for that one request
+voltro logs --trace <traceId> --format json
+```
+
+That returns every log line of the request — frontend action → api Y → api Z — in order. It's the fastest answer to "where did this error come from?".
+
+## For AI coding agents
+
+**Claude Code, Cursor, GitHub Copilot, and other agents: use these whenever you debug.** Before you grep code or read the user's description of a bug, run `voltro logs --tail 100` (or `--since 30s` after a reproducible action). The buffer carries the actual error message, the actual stack, the actual request/mutation tag — faster and more accurate than inferring what happened from source.
+
+Common patterns:
+
+- **"Is the change working?"** — make the edit, run the reproducing action, then `voltro logs --since 10s`. New errors show immediately.
+- **"Why did the mutation fail?"** — `voltro logs --filter '<rpcTag>' --level warn`.
+- **"Is the browser even reaching the server?"** — `voltro logs --source client --tail 20`.
+- **"Why did this request fail end-to-end?"** — grab the traceId of a failed span (`voltro traces --errors`, or the dev error toast), then `voltro logs --trace <id> --format json` for the whole chain.
+
+Discovery uses `~/.voltro/runtime-registry.json` — every `voltro dev` / `voltro start` registers itself with a heartbeat. If nothing is running, both commands print a clear hint instead of failing. The trace buffer is transient; if you need older context than the ring holds, ask the user to reproduce the action.
+
+
+
+---
+
+<!-- source: en/observability/time-travel.md -->
+## Time-travel debugger
+
+_Record the app's ChangeEvent timeline and scrub a table's rows back and forward through it — read-only point-in-time replay, env-gated + redacted, surfaced in the DevTools / cloud Time-travel panel._
+
+The time-travel debugger records the app's **ChangeEvent stream** into a bounded,
+redacted ring and lets you **scrub any table's rows back and forward** through it
+— replaying what the data looked like at any recorded point. Read-only: replay
+reconstructs a row-set by reversing the recorded changes over the live rows; it
+never writes.
+
+## Turn it on
+
+Recording is **env-gated** and **off by default in production** (volume + PII are
+the real cost — use OTLP for prod forensics). Set `VOLTRO_TIMELINE`:
+
+| value | records |
+|---|---|
+| `off` (also `0` / `false`) | nothing |
+| `interesting` (also `on` / `1` / `true`) | user tables only (skips `_voltro_*` framework noise) |
+| `all` | every table (except the timeline's own writes) |
+
+The default is `interesting` outside production, `off` in production — the same
+environment-aware default the durable trace persistence uses.
+
+Sensitive-looking columns (`password` / `secret` / `token` / `api_key` /
+`authorization` / `cookie` / `ssn` / `credit_card` / `cvv` / …) are **redacted**
+before a row enters the ring — PII never sits in the debug buffer.
+
+## Scrub it
+
+Open the **Time-travel** panel in the DevTools dashboard (or the cloud dashboard,
+for a deployed app): pick a table, drag the scrubber across the recorded
+sequence, and the panel reconstructs that table's rows as of the selected point.
+The event log shows every recorded change; click one to scrub to just before it.
+"Jump to now" returns to live.
+
+## The endpoint
+
+The panel reads two read-only inspect endpoints:
+
+- `GET /_voltro/inspect/timeline` — the recorded events
+  (`?table=&tenantId=&from=&to=&limit=`) plus the current sequence (`currentSeq`).
+- `GET /_voltro/inspect/timeline/replay?table=<t>&seq=<n>` — `<t>`'s row-set as
+  of sequence `<n>`.
+
+```sh
+# Gated by the same inspect auth as every /_voltro/inspect/* endpoint.
+curl "http://localhost:4000/_voltro/inspect/timeline?table=todos&limit=50"
+curl "http://localhost:4000/_voltro/inspect/timeline/replay?table=todos&seq=42"
+```
+
+## Limits
+
+- **Read-only.** Time-travel never writes — restoring a past value is the
+  universal-undo primitive's job (and only for safe inverses).
+- **In-memory ring** (bounded, per-process) in v1 — it is for scrubbing recent
+  history, not durable forensics: it resets on restart, and a replay is correct
+  only as long as the events after your target sequence are still in the ring.
+- The inspect surface is a `voltro dev` / DevTools concern; out-of-band DB writes
+  (not in the app's ChangeEvent stream) aren't recorded, and production should
+  use OTLP for forensics.