autotel 3.0.0 → 3.0.4

package/skills/autotel-request-logging/SKILL.md

@@ -2,14 +2,6 @@
 name: autotel-request-logging
 description: >
   getRequestLogger(), set(), info/warn/error, emitNow(). One snapshot per request; requires active span. Use when adding request-scoped context or replacing scattered console.log.
-type: core
-library: autotel
-library_version: '2.23.0'
-requires:
-  - autotel-instrumentation
-sources:
-  - jagreehal/autotel:packages/autotel/src/request-logger.ts
-  - jagreehal/autotel:docs/AGENT-GUIDE.md
 ---
 
 # Autotel — Request Logging

package/skills/autotel-structured-errors/SKILL.md

@@ -2,13 +2,6 @@
 name: autotel-structured-errors
 description: >
   createStructuredError, parseError, recordStructuredError. API errors with message, why, fix, link; client parsing for UI. Use in API routes and client catch blocks.
-type: core
-library: autotel
-library_version: '2.23.0'
-sources:
-  - jagreehal/autotel:packages/autotel/src/structured-error.ts
-  - jagreehal/autotel:packages/autotel/src/parse-error.ts
-  - jagreehal/autotel:docs/AGENT-GUIDE.md
 ---
 
 # Autotel — Structured Errors

package/skills/build-audit-trails/SKILL.md

@@ -0,0 +1,302 @@
+---
+name: build-audit-trails
+description: >
+  Design tamper-aware audit trails on top of OpenTelemetry spans using
+  autotel. Covers what counts as auditable, the audit-only span discipline,
+  signing and tamper-detection, denial logging, redaction, retention,
+  separation of concerns from operational telemetry, and framework wiring
+  (Next.js, Nuxt, Hono, Express, Cloudflare Workers).
+license: MIT
+---
+
+# Build audit trails
+
+An _audit trail_ is a record of who did what to which resource, when, and whether it was permitted — durable, tamper-evident, and admissible. Operational telemetry (latency, errors, span shapes) is for engineers; audit trails are for compliance, security, and forensics. They overlap technically but differ on every other axis.
+
+autotel lets you express both with the same primitive — a span — but you should keep them on **separate processors** so an audit event never gets dropped by sampling, never gets redacted by a debug rule, and never goes to the same backend as your ops data.
+
+## When to use
+
+- Implementing GDPR / HIPAA / SOC2 / PCI-DSS / ISO 27001 / GxP compliance
+- Adding "who did what" trails for admin actions, access reviews, payments
+- Recording authorization decisions (allow + deny)
+- Building immutable evidence for incident response
+
+## The audit span discipline
+
+An auditable event has six required parts:
+
+| Field               | OTel attribute                                                  | Example                                       |
+| ------------------- | --------------------------------------------------------------- | --------------------------------------------- |
+| When                | (span timestamp)                                                | `2026-05-04T17:23:11.412Z`                    |
+| Who                 | `enduser.id` + `enduser.role`                                   | `usr_42`, `admin`                             |
+| Where (acting from) | `client.address`, `network.peer.address`, `user_agent.original` | `203.0.113.5`, `Chrome 121`                   |
+| What                | `audit.action`                                                  | `secret.read`, `policy.update`, `user.delete` |
+| Which resource      | `audit.resource.type` + `audit.resource.id`                     | `secret`, `sec_abc`                           |
+| Outcome             | `audit.outcome` (`allow` / `deny`) + `audit.reason`             | `deny`, `MFA required`                        |
+
+Plus useful optional fields: `audit.policy.id` (which policy made the call), `audit.evidence` (linked artefact id), `audit.actor.session.id`.
+
+## Step 1: Define a typed `audit()` helper
+
+Centralise the schema in one place so every site gets it right:
+
+```typescript
+import { trace, SpanKind } from '@opentelemetry/api';
+
+type AuditAction =
+  | 'secret.read'
+  | 'secret.write'
+  | 'secret.delete'
+  | 'policy.update'
+  | 'user.create'
+  | 'user.delete'
+  | 'data.export'
+  | 'session.assume';
+
+interface AuditPayload {
+  action: AuditAction;
+  resource: { type: string; id: string };
+  outcome: 'allow' | 'deny';
+  reason?: string;
+  actor?: { id: string; role?: string; sessionId?: string };
+  policy?: { id: string };
+  evidence?: { id: string };
+}
+
+const tracer = trace.getTracer('autotel-audit', '1.0.0');
+
+export function audit(payload: AuditPayload): void {
+  const span = tracer.startSpan(`audit.${payload.action}`, {
+    kind: SpanKind.INTERNAL,
+    attributes: {
+      audit: true,
+      'audit.action': payload.action,
+      'audit.outcome': payload.outcome,
+      'audit.resource.type': payload.resource.type,
+      'audit.resource.id': payload.resource.id,
+      ...(payload.reason && { 'audit.reason': payload.reason }),
+      ...(payload.actor?.id && { 'enduser.id': payload.actor.id }),
+      ...(payload.actor?.role && { 'enduser.role': payload.actor.role }),
+      ...(payload.actor?.sessionId && {
+        'audit.actor.session.id': payload.actor.sessionId,
+      }),
+      ...(payload.policy?.id && { 'audit.policy.id': payload.policy.id }),
+      ...(payload.evidence?.id && { 'audit.evidence.id': payload.evidence.id }),
+    },
+  });
+  span.end();
+}
+```
+
+## Step 2: Always log denials
+
+A frequent compliance failure is "we logged what users did but not what we **stopped** them doing." Wrap the authorization decision so both branches go through `audit()`:
+
+```typescript
+export async function withAuthz<T>(
+  payload: Omit<AuditPayload, 'outcome'>,
+  decide: () => Promise<{ allow: boolean; reason?: string }>,
+  body: () => Promise<T>,
+): Promise<T> {
+  const decision = await decide();
+  if (!decision.allow) {
+    audit({ ...payload, outcome: 'deny', reason: decision.reason });
+    throw createStructuredError({
+      status: 403,
+      code: 'FORBIDDEN',
+      message: 'Not allowed',
+      why: decision.reason ?? 'Insufficient permissions',
+    });
+  }
+  audit({ ...payload, outcome: 'allow' });
+  return body();
+}
+```
+
+## Step 3: Separate the audit pipeline
+
+Critical: route audit spans to a **different processor and backend** so:
+
+- They are never dropped by head or tail sampling.
+- They are not subject to development-mode debug exporters.
+- They go to a write-once / append-only store (S3 Object Lock, immutable bucket, dedicated audit DB).
+
+```typescript
+import {
+  composeSpanProcessors,
+  composeSubscribers,
+  defineConfig,
+} from 'autotel-edge';
+import { BatchSpanProcessor, FilteringSpanProcessor } from 'autotel/processors';
+
+const auditExporter = new BatchSpanProcessor(
+  new OTLPHttpJsonExporter({
+    url: process.env.AUDIT_OTLP!,
+    headers: { authorization: `Bearer ${process.env.AUDIT_TOKEN!}` },
+  }),
+);
+const opsExporter = new BatchSpanProcessor(
+  new OTLPHttpJsonExporter({ url: process.env.OPS_OTLP! }),
+);
+
+// Only audit spans reach the audit pipeline.
+const auditOnly = new FilteringSpanProcessor({
+  include: (span) => span.attributes['audit'] === true,
+  next: auditExporter,
+});
+
+// Conversely, ops never sees audit spans (avoid leaking PII to dashboards).
+const opsOnly = new FilteringSpanProcessor({
+  exclude: (span) => span.attributes['audit'] === true,
+  next: opsExporter,
+});
+
+export const otelConfig = defineConfig({
+  service: { name: 'app' },
+  spanProcessors: composeSpanProcessors([auditOnly, opsOnly]),
+});
+```
+
+## Step 4: Tamper detection
+
+For environments where audit storage is shared with the producing service (no append-only bucket), sign each span:
+
+```typescript
+import { createHmac, randomUUID } from 'node:crypto'
+
+function signAuditAttributes(attrs: Record<string, unknown>): string {
+  const key = process.env.AUDIT_HMAC_KEY!
+  const payload = JSON.stringify(Object.fromEntries(Object.entries(attrs).sort()))
+  return createHmac('sha256', key).update(payload).digest('hex')
+}
+
+export function audit(payload: AuditPayload): void {
+  const id = randomUUID()
+  const attributes = { /* … as before … */, 'audit.id': id }
+  const signature = signAuditAttributes(attributes)
+  attributes['audit.signature.alg'] = 'HMAC-SHA256'
+  attributes['audit.signature.value'] = signature
+  // … startSpan …
+}
+```
+
+Verify on the read side: recompute the HMAC over the same sorted attribute set (excluding `audit.signature.value` itself); mismatched ⇒ tampered.
+
+For multi-tenant or extra-strict (HIPAA), use Ed25519 with per-environment keys and rotate.
+
+## Step 5: Redaction — what stays and what goes
+
+| Field                    | In audit span? | Notes                                                                                 |
+| ------------------------ | -------------- | ------------------------------------------------------------------------------------- |
+| `enduser.id`             | ✅             | Internal user id; never the email                                                     |
+| `audit.resource.id`      | ✅             | Required for forensics                                                                |
+| `client.address`         | ✅             | Last-octet redaction acceptable for IPv4                                              |
+| Free-form payload bodies | ❌             | Never inline raw input — link by id (`audit.evidence.id`)                             |
+| Secret values            | ❌             | Use `audit.action=secret.read` + `audit.resource.id=sec_abc`, never the secret itself |
+| Authorization headers    | ❌             | Token names ok (`bearer.*`), values never                                             |
+
+`attributeRedactor` defaults are too aggressive for audit (you may need `enduser.id` literal, not masked). Disable redaction selectively:
+
+```typescript
+spanProcessors: composeSpanProcessors([
+  // No redactor on the audit branch — keys are already conservative
+  auditOnly,
+  // Strict redactor on ops
+  new AttributeRedactingProcessor(opsOnly, { redactor: 'strict' }),
+]);
+```
+
+## Step 6: Retention
+
+Audit retention is set by regulation, not engineering taste. Common minimums:
+
+| Regulation           | Minimum retention                            |
+| -------------------- | -------------------------------------------- |
+| GDPR                 | 6 years (financial), 12 months (operational) |
+| HIPAA                | 6 years                                      |
+| PCI-DSS              | 1 year (online), 3 months hot                |
+| SOX                  | 7 years                                      |
+| GxP / 21 CFR Part 11 | Lifetime of product + 10 years               |
+
+Express retention as a backend lifecycle policy (S3 Object Lock COMPLIANCE mode, BigQuery `--time_partitioning_expiration`), not application code.
+
+## Step 7: Framework wiring
+
+### Next.js
+
+```typescript
+// app/admin/users/[id]/route.ts
+import { withAuthz, audit } from '@/lib/audit';
+
+export async function DELETE(
+  req: Request,
+  { params }: { params: { id: string } },
+) {
+  return withAuthz(
+    {
+      action: 'user.delete',
+      resource: { type: 'user', id: params.id },
+      actor: { id: req.headers.get('x-user-id')!, role: 'admin' },
+    },
+    async () => ({ allow: await canDelete(req, params.id) }),
+    async () => {
+      await db.user.delete({ where: { id: params.id } });
+      return Response.json({ ok: true });
+    },
+  );
+}
+```
+
+### Hono
+
+```typescript
+import { audit, withAuthz } from './audit';
+app.post('/secrets/:id/read', async (c) => {
+  return withAuthz(
+    {
+      action: 'secret.read',
+      resource: { type: 'secret', id: c.req.param('id') },
+      actor: { id: c.var.user.id, role: c.var.user.role },
+    },
+    () => requireScope(c, 'secrets:read'),
+    async () => c.json({ value: await secrets.read(c.req.param('id')) }),
+  );
+});
+```
+
+### Cloudflare Workers
+
+`audit()` from inside `defineWorkerFetch` — `ctx.waitUntil` makes sure the audit span is exported before the response returns:
+
+```typescript
+export default defineWorkerFetch(
+  { service: { name: 'admin-api' } },
+  async (request, env, ctx, log) => {
+    return withAuthz(
+      {
+        action: 'data.export',
+        resource: { type: 'project', id: 'p_123' },
+        actor: { id: 'usr_42' },
+      },
+      async () => ({ allow: true }),
+      async () => Response.json({ ok: true }),
+    );
+  },
+);
+```
+
+## Anti-patterns
+
+| Anti-pattern                                   | Fix                                                            |
+| ---------------------------------------------- | -------------------------------------------------------------- |
+| Audit logs in `console.log` / unstructured     | Use `audit()` so every event has the same shape                |
+| Same backend for audit and ops                 | Separate processors, separate retention                        |
+| Audit subject to sampling                      | `FilteringSpanProcessor` with `include: span.attributes.audit` |
+| Logging only successes                         | Always log denials too                                         |
+| Putting secrets / payloads in audit attributes | Reference by id only (`audit.evidence.id`)                     |
+| No tamper detection                            | HMAC signature on critical environments                        |
+| Custom retention in code                       | Express via storage-layer lifecycle policy                     |
+| Audit on every read of harmless data           | Audit _meaningful_ events; not every list call                 |
+| Audit row tied to a specific framework         | The `audit()` function is framework-agnostic                   |
+| `enduser.id` = email                           | Use the internal id; emails go in a separate identity table    |

package/skills/debug-missing-spans/SKILL.md

@@ -0,0 +1,248 @@
+---
+name: debug-missing-spans
+description: >
+  Troubleshoot when expected OpenTelemetry spans don't reach the backend.
+  Walks the chain top-to-bottom — code → SDK init → processor → exporter →
+  network → backend ingest — with concrete tests at each step. Covers head
+  sampling, ctx.waitUntil drops on Cloudflare, init-order races, runtime
+  detection failures, propagation breaks, exporter auth errors, and
+  silent ratelimits.
+license: MIT
+---
+
+# Debug missing spans
+
+When a span you expect isn't in the backend, the cause is somewhere in this chain:
+
+```
+code → SDK init → head sampler → processor → exporter → network → backend ingest → backend index
+```
+
+This skill walks each link in order with a quick check you can run. Don't skip steps — the cause is rarely where you'd guess.
+
+## Step 0: Reproduce locally with the pretty exporter
+
+Before chasing remote backends, confirm the span exists at all:
+
+```typescript
+init({
+  service: 'my-app',
+  debug: 'pretty', // hierarchical colourised output to stdout
+});
+```
+
+If you see the span in stdout, the SDK + sampler are fine — skip to "exporter / network". If you don't, keep reading.
+
+## Step 1: Is the SDK actually initialised?
+
+Common failure: `init()` runs after the first request because of import-order.
+
+```typescript
+import { trace } from '@opentelemetry/api';
+
+const tracer = trace.getTracer('autotel-debug');
+console.log(
+  '[autotel-debug] tracer is no-op:',
+  tracer.constructor.name === 'NoopTracer',
+);
+```
+
+If `true`, `init()` ran too late. Move it to the very top of the entry file (or to `instrumentation.ts` for Next.js).
+
+## Step 2: Head sampler
+
+Print the effective head rate:
+
+```typescript
+import { getActiveConfig } from 'autotel-edge';
+console.log('[autotel-debug] sampling:', getActiveConfig()?.sampling);
+```
+
+Common gotchas:
+
+- `sampling.rates: { server: 5 }` — 5 % means 95 % of spans never start.
+- Inheriting `OTEL_TRACES_SAMPLER_ARG=0.01` from the environment via the OTel default sampler.
+- Your test happens to hit the unsampled branch — instrument with `sampling: { rates: { server: 100 } }` while reproducing.
+
+To force sampling for one request, send a `traceparent` with the sampled flag set:
+
+```
+traceparent: 00-<traceid>-<spanid>-01
+```
+
+(`-01` at the end = sampled.) autotel's parent-based sampler will respect it.
+
+## Step 3: Cloudflare Workers — `ctx.waitUntil`
+
+The single biggest cause of missing spans on the edge: **the response returned before the exporter flushed**.
+
+If you're using `addEventListener('fetch', …)` or a hand-rolled `fetch` in a module worker without wiring `ctx.waitUntil(…)` to the export call, async drains drop silently.
+
+Fix — switch to `defineWorkerFetch` or `wrapModule`, both of which wire `waitUntil` automatically:
+
+```typescript
+import { defineWorkerFetch } from 'autotel-cloudflare';
+
+export default defineWorkerFetch(
+  { service: { name: 'edge' } },
+  async (request, env, ctx, log) => {
+    // log.set / spans here all flush via ctx.waitUntil before response returns
+    return new Response('ok');
+  },
+);
+```
+
+## Step 4: Processor pipeline
+
+Print what's wired:
+
+```typescript
+import { trace } from '@opentelemetry/api';
+const provider = trace.getTracerProvider();
+console.log('[autotel-debug] provider:', provider.constructor.name);
+console.log(
+  '[autotel-debug] processors:',
+  (provider as any)._registeredSpanProcessors?.map(
+    (p: any) => p.constructor.name,
+  ),
+);
+```
+
+Common issues:
+
+- **A `FilteringSpanProcessor` excludes your span.** Check the `include` / `exclude` predicates.
+- **A `TailSamplingProcessor` dropped the trace** (no error, no slow root, no debug header).
+- **A `composePostProcessors` step returns `[]` for your span.**
+
+To bisect, temporarily strip post-processors:
+
+```typescript
+init({
+  service: 'my-app',
+  exporter: { url: process.env.OTLP_ENDPOINT! },
+  // no postProcessor, no tail sampler, no filter
+});
+```
+
+If the span shows up now, add back the processors one at a time.
+
+## Step 5: Exporter
+
+Tail the SDK's diagnostic log:
+
+```typescript
+import { diag, DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api';
+diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG);
+```
+
+Look for:
+
+```
+@opentelemetry/api: ... OTLPExporter: failed to send 4 traces, status: 401, error: ...
+```
+
+Common exporter errors:
+
+| Status        | Meaning                   | Fix                                                         |
+| ------------- | ------------------------- | ----------------------------------------------------------- |
+| `401`         | Bad / missing auth header | Check `OTLP_HEADERS` / vendor token name                    |
+| `403`         | Token has no write scope  | Issue a token with the right scope                          |
+| `404`         | Wrong endpoint URL        | Check region (`api.honeycomb.io` vs `api.eu1.honeycomb.io`) |
+| `413`         | Batch too big             | Lower `BatchSpanProcessor` `maxExportBatchSize`             |
+| `429`         | Rate-limited              | Reduce head/tail rates; honour `retry-after`                |
+| `502/503/504` | Upstream unhealthy        | Often transient; add retries; check backend status          |
+| Network error | DNS / firewall            | `curl -v <url>` from the same network                       |
+
+## Step 6: Network / TLS
+
+For self-hosted Collectors:
+
+```bash
+curl -v -X POST $OTLP_ENDPOINT \
+  -H 'content-type: application/json' \
+  -H "$AUTH_HEADER" \
+  -d '{"resourceSpans":[]}'
+```
+
+Should return `200`. If it doesn't, the problem is between you and the Collector — not autotel.
+
+For Cloudflare Workers, run `wrangler tail` and look for `OTLPExporter` errors.
+
+## Step 7: Backend ingest — silent rejection
+
+Some backends accept the request with a 200 but drop the events:
+
+- **Honeycomb**: dataset must exist _and_ the API key must have write access to it. Mismatched key/dataset → silent drop.
+- **Datadog**: check `service` is set (resource attribute `service.name`) — they ignore spans without it.
+- **Sentry**: SDK version mismatch on envelope → 200 but events disappear.
+- **Grafana Cloud Tempo**: spans without `service.name` go to a fallback service called `unknown_service`.
+
+For each backend, the dataset / index / project where you'd expect the span:
+
+| Backend       | Where the span lands                    |
+| ------------- | --------------------------------------- |
+| Honeycomb     | dataset = `service.name` (auto-created) |
+| Datadog       | `service:<name>` filter                 |
+| Grafana Tempo | search by `traceId`                     |
+| Jaeger        | service dropdown = `service.name`       |
+| Sentry        | project linked to the DSN               |
+
+## Step 8: Backend index lag
+
+After a 200, expect ingestion lag of:
+
+| Backend            | Typical lag |
+| ------------------ | ----------- |
+| Honeycomb          | < 5 s       |
+| Datadog            | 30–60 s     |
+| Grafana Tempo      | 10–30 s     |
+| Sentry             | 30–120 s    |
+| Self-hosted Jaeger | < 1 s       |
+
+Don't conclude the span is missing until you've waited > 2× the expected lag.
+
+## Step-by-step checklist
+
+```
+[ ] Span shows in `debug: 'pretty'` stdout
+[ ] `tracer.constructor.name !== 'NoopTracer'` (SDK initialised)
+[ ] Head rate is high enough to allow the request
+[ ] Workers handler uses defineWorkerFetch / wrapModule
+[ ] No post-processor / tail sampler / filter strips it
+[ ] Exporter logs no 4xx/5xx
+[ ] Curl to OTLP endpoint returns 200
+[ ] Backend has the right service.name / dataset / project
+[ ] Waited 2× expected ingest lag
+```
+
+## When the trace partially shows up
+
+Some spans land, some don't:
+
+- **Trace context broken between services** — outbound HTTP calls aren't propagating `traceparent`. Confirm autotel's global fetch instrumentation is on (`instrumentation.instrumentGlobalFetch: true`, default).
+- **Async boundary loses context** — a `setTimeout` / queue callback ran outside the AsyncLocalStorage scope. Wrap with `trace()` or use `context.with()`.
+- **Cross-runtime call** — Node service → Workers → browser; verify `traceparent` arrives at each leg via response headers / network panel.
+
+## When the SDK itself crashes
+
+```
+TypeError: Cannot read properties of undefined (reading 'startActiveSpan')
+```
+
+Usually means the API version (`@opentelemetry/api`) and SDK version (`@opentelemetry/sdk-trace-base`) drifted. Run:
+
+```bash
+pnpm why @opentelemetry/api
+```
+
+There should be exactly one resolved version. If there are two, dedup via `pnpm.overrides`.
+
+## Anti-patterns to fix as you debug
+
+| Anti-pattern                                              | Why it loses spans                                                 |
+| --------------------------------------------------------- | ------------------------------------------------------------------ |
+| `init()` after the first import that uses tracing         | Spans before `init()` are no-ops                                   |
+| `addEventListener('fetch', …)` on Workers                 | Pre-module-worker style; no `ctx.waitUntil` to wire                |
+| Single `OTLP_ENDPOINT` env var with `?` chars URL-encoded | Auth gets parsed as part of the path                               |
+| Importing both `@sentry/tracing` and `autotel`            | Double-instrumentation eats spans                                  |
+| `process.exit(0)` immediately after the work              | The exporter never flushed; call `await provider.shutdown()` first |