@autter/otlp-ingester 1.4.0 → 1.6.0

package/README.md

@@ -9,10 +9,13 @@ one per-repo signal model, fingerprints errors, and writes ClickHouse.
 | Route | Payload | Purpose |
 | --- | --- | --- |
 | `POST /v1/traces` | OTLP/JSON `ExportTraceServiceRequest` | Error spans → occurrences; all spans → `runtime_spans`; server spans → usage rollups; GenAI spans → `runtime_llm_calls` |
+| `POST /v1/logs` | OTLP `ExportLogsServiceRequest`, server key | Wide events (request/operation summaries, logs) → `runtime_logs` (+ `runtime_request_1m` rollup); logger-only error records (`autter.capture.mode=log`) → occurrences |
 | `POST /v1/metrics` | OTLP `ExportMetricsServiceRequest` | HTTP-server duration histograms → usage rollups; portable process memory/GC metrics → per-instance memory samples |
 | `POST /v1/platform-events` | JSON, server key | ECS/Kubernetes OOM kills and restarts → memory incident correlation |
 | `POST /v1/profiles` | Symbolized pprof, server key | CPU/in-use heap profile samples |
 | `POST /v1/browser` | Browser payload `version: 1` | Errors/rejections → occurrences; session pings → rollups |
+| `POST /v1/logs` | OTLP logs, server key | Structured logs and operation summaries → `runtime_logs` |
+| `GET /v1/compat` | — | Version, schema level and supported features (public) |
 | `GET /healthz` | — | Liveness + ClickHouse reachability |
 
 Auth on every ingest route: `Authorization: Bearer <ingest key>`,
@@ -45,6 +48,17 @@ AUTTER_INGEST_KEYS='[
 The validator webhook may return the same extra fields:
 `{ orgId, repositoryId, scope?, allowedOrigins? }`.
 
+### Version compatibility
+
+`GET /v1/compat` (public, no tenant data) returns this ingester's version,
+its ClickHouse schema state, and which features from
+`src/compat-manifest.json` it supports. Add
+`?features=operation_logging&sdk=@autter/runtime-node@1.3.0` to get the evaluated
+incompatibilities. Every response carries `x-autter-ingester-version`. The
+ingester records the SDK name and version each service sends with
+(`runtime_sdk_versions`) and its own version (`runtime_ingester_info`), and
+the Autter dashboard reads both. See [docs/COMPATIBILITY.md](../../docs/COMPATIBILITY.md).
+
 ## Configuration
 
 | Env | Default | Description |
@@ -61,11 +75,48 @@ The validator webhook may return the same extra fields:
 | `SINK_MAX_ATTEMPTS` | `12` | Delivery attempts per batch (1–60 s backoff) |
 | `SINK_MAX_BUFFERED_BATCHES` | `1000` | Retry buffer cap; oldest drops are logged |
 | `SINK_MAX_BUFFERED_MB` | `64` | Sink retry buffer cap (memory) |
+| `SOURCE_MAP_CACHE_MB` | `64` | Memory budget (estimated decoded size) for parsed source maps used to symbolicate browser frames at ingest; least recently used maps are evicted |
 | `MAX_BODY_BYTES` | `1048576` | Request body cap |
 | `RATE_LIMIT_PER_MINUTE` | `300` | Per-key fixed window (server keys) |
 | `CLIENT_RATE_LIMIT_PER_MINUTE` | `120` | Per-key fixed window (client keys) |
+| `PROMOTION_LOOKUPS_PER_MINUTE` | `30` | Per-tenant ClickHouse dedupe lookups for `/v1/logs` error promotion; over budget, records are promoted with in-batch dedupe only |
 | `OCCURRENCE_TTL_DAYS` / `SPAN_TTL_DAYS` / `METRICS_TTL_DAYS` | `14` / `7` / `90` | ClickHouse TTLs (applied at table creation) |
 | `LLM_CALL_TTL_DAYS` | `90` | Retention for `runtime_llm_calls` rows |
+| `LOG_TTL_DAYS` | `14` | Retention for `runtime_logs` (request summaries are always kept, never sampled, so this is the main volume knob). Unlike the other TTLs it is also applied to **existing** tables: at boot the ingester compares it with the table's TTL and runs `ALTER TABLE … MODIFY TTL` only when they differ; a failure is logged and retried next boot |
+| `AUTTER_REDACT_VALUE_PATTERNS` | — | JSON array of extra regex sources masked in stored/forwarded text and attribute values (built-in secret/PII patterns always apply) |
+| `AUTTER_REDACT_KEY_PATTERNS` | — | JSON array of extra regex sources for attribute keys whose values are masked wholesale |
+
+## Coded errors and request summaries (1.5.0)
+
+- **Code-based grouping.** An occurrence carrying a valid `autter.error.code`
+  (`^[a-z][a-z0-9_]*(\.[a-z0-9_]+){0,3}$`, ≤80 chars) is fingerprinted by
+  scheme `code-v1`: `sha256("code-v1\0" + service + "\0" + code)[:32]` —
+  independent of source, message, stack and route, so one code is one issue
+  across traces, log promotion, the browser and external connectors.
+  Uncoded errors keep the historical fingerprint (`message-v1`) byte-for-byte.
+  `fingerprint_scheme` is stored per occurrence and sent to the sink.
+- **Declared fields.** `autter.error.why|fix|link|expected` and
+  `autter.request.id` are lifted from span/exception-event attributes, log
+  attributes and browser context into `runtime_error_occurrences`
+  (`error_why`, `error_fix`, `error_link`, `expected`, `request_id`) and the
+  sink payload. why/fix are scrubbed and capped at 1000 chars; links must be
+  http(s) and ≤500 chars. Exception spans inherit the request id of their
+  server span.
+- **Request summaries.** `/v1/logs` lifts `autter.operation.kind`,
+  `autter.request.id`, `http.route` (id-normalised), `http.response.status_code`,
+  `autter.error.code` and the `autter.operation.ai` rollup (`cost_usd`, `calls`)
+  into `runtime_logs` columns. `kind = 'request'` rows feed the
+  `runtime_request_1m` materialized view (per route/method/minute: count,
+  failed, duration sum, p50/p95). The view counts every insert, so a retried
+  batch can be counted twice — route stats are approximate under exporter
+  retries; `runtime_logs` itself collapses duplicates.
+- **Log promotion.** SDKs without a trace pipeline (logger-only mode,
+  `@autter/runtime-edge`) send errors as log records with `exception.*` and
+  `autter.capture.mode = "log"`. `/v1/logs` promotes those (severity ≥ error)
+  to server occurrences — stored, counted and forwarded to the sink —
+  skipping a record when its trace id already produced an occurrence in the
+  same batch or (ClickHouse lookup) within ±60 s. If the lookup fails the
+  record is promoted anyway.
 
 ## LLM / GenAI calls
 

package/dist/breadcrumbs.d.ts

@@ -0,0 +1,41 @@
+/**
+ * Breadcrumbs — the steps before an error (sink contract §1).
+ *
+ * Browser events carry them as `breadcrumbs` (runtime-browser's opt-in
+ * `/breadcrumbs` module); server SDKs put a JSON string of the same array on
+ * the exception span event as `autter.breadcrumbs`. Either way they are
+ * untrusted client input, so this is a whitelist gate: unknown fields are
+ * dropped, every string is scrubbed, and the list is bounded before it is
+ * stored (under `autter.breadcrumbs` in the occurrence's attributes JSON)
+ * and forwarded to the sink.
+ */
+export type BreadcrumbType = "console" | "request" | "navigation" | "click" | "log" | "custom";
+export type BreadcrumbLevel = "debug" | "info" | "warning" | "error";
+export interface Breadcrumb {
+    type: BreadcrumbType;
+    /** ISO timestamp. */
+    at: string;
+    level?: BreadcrumbLevel;
+    /** ≤ 500 chars, scrubbed. */
+    message: string;
+    /** ≤ 10 keys of primitives; strings ≤ 200 chars, scrubbed. */
+    data?: Record<string, string | number | boolean | null>;
+}
+/** Span event / log attribute carrying a JSON string of `Breadcrumb[]`. */
+export declare const BREADCRUMBS_ATTRIBUTE = "autter.breadcrumbs";
+/** Key under which an occurrence's breadcrumbs are stored in its attributes JSON. */
+export declare const BREADCRUMBS_STORAGE_KEY = "autter.breadcrumbs";
+export declare const MAX_BREADCRUMBS = 30;
+/**
+ * Validate, bound and scrub breadcrumbs from any sender: an array, or a JSON
+ * string of one (the span attribute). Keeps the newest 30 valid entries,
+ * oldest first, within 16 KB of JSON; malformed entries are dropped one by
+ * one. Undefined when nothing valid remains.
+ */
+export declare function parseBreadcrumbs(raw: unknown): Breadcrumb[] | undefined;
+/** `{ breadcrumbs }` for an occurrence, or `{}` — spread into the input. */
+export declare function breadcrumbsField(...candidates: unknown[]): {
+    breadcrumbs?: Breadcrumb[];
+};
+/** Attributes JSON to store: the occurrence's attributes plus its breadcrumbs. */
+export declare function attributesWithBreadcrumbs(attributes: Record<string, unknown> | null, breadcrumbs: Breadcrumb[] | undefined): Record<string, unknown>;

package/dist/breadcrumbs.js

@@ -0,0 +1,151 @@
+import { isSensitiveKey, scrubText } from "./redact.js";
+/** Span event / log attribute carrying a JSON string of `Breadcrumb[]`. */
+export const BREADCRUMBS_ATTRIBUTE = "autter.breadcrumbs";
+/** Key under which an occurrence's breadcrumbs are stored in its attributes JSON. */
+export const BREADCRUMBS_STORAGE_KEY = "autter.breadcrumbs";
+export const MAX_BREADCRUMBS = 30;
+const MAX_MESSAGE = 500;
+const MAX_DATA_KEYS = 10;
+const MAX_DATA_STRING = 200;
+/** Stored JSON per occurrence; the oldest breadcrumbs go first. */
+const MAX_TOTAL_CHARS = 16 * 1024;
+/** Raw JSON accepted from a span attribute before parsing. */
+const MAX_RAW_CHARS = 256 * 1024;
+/** Entries examined from the end of an oversized list. */
+const MAX_SCANNED = 200;
+const TYPES = new Set(["console", "request", "navigation", "click", "log", "custom"]);
+const LEVELS = {
+    debug: "debug",
+    trace: "debug",
+    info: "info",
+    log: "info",
+    warning: "warning",
+    warn: "warning",
+    error: "error",
+    fatal: "error",
+    critical: "error",
+};
+const UNSAFE_KEY = /^(?:__proto__|constructor|prototype)$/;
+/** Drop query strings and fragments from http(s) URLs (procedural: linear). */
+function stripUrlQueries(value) {
+    return value.replace(/https?:\/\/[^\s]*/gi, (url) => {
+        const rest = url.slice(url.indexOf("://") + 3);
+        const cut = rest.search(/[?#]/);
+        return cut <= 0 ? url : url.slice(0, url.length - rest.length + cut);
+    });
+}
+function cleanText(value, max) {
+    // Cut before scrubbing (bounded work), with slack so a secret straddling
+    // the cut is masked whole.
+    return stripUrlQueries(scrubText(value.slice(0, max + 512))).slice(0, max);
+}
+function cleanData(raw) {
+    if (!raw || typeof raw !== "object" || Array.isArray(raw))
+        return undefined;
+    const out = {};
+    let count = 0;
+    for (const [rawKey, value] of Object.entries(raw).slice(0, 64)) {
+        if (count >= MAX_DATA_KEYS)
+            break;
+        const key = rawKey.slice(0, 100);
+        if (!key || UNSAFE_KEY.test(key))
+            continue;
+        if (isSensitiveKey(key)) {
+            out[key] = "[redacted]";
+        }
+        else if (typeof value === "string") {
+            const text = /(?:url|path|route|target)$/i.test(key) ? value.split(/[?#]/)[0] : value;
+            out[key] = cleanText(text, MAX_DATA_STRING);
+        }
+        else if (typeof value === "number") {
+            if (!Number.isFinite(value))
+                continue;
+            out[key] = value;
+        }
+        else if (typeof value === "boolean" || value === null) {
+            out[key] = value;
+        }
+        else {
+            continue;
+        }
+        count++;
+    }
+    return count > 0 ? out : undefined;
+}
+function cleanBreadcrumb(raw) {
+    if (!raw || typeof raw !== "object" || Array.isArray(raw))
+        return null;
+    const input = raw;
+    const at = typeof input.at === "string" || typeof input.at === "number" ? new Date(input.at) : null;
+    if (!at || Number.isNaN(at.getTime()))
+        return null;
+    const message = typeof input.message === "string"
+        ? input.message
+        : typeof input.message === "number" || typeof input.message === "boolean"
+            ? String(input.message)
+            : "";
+    const type = typeof input.type === "string" && TYPES.has(input.type) ? input.type : "custom";
+    const level = typeof input.level === "string" ? LEVELS[input.level.toLowerCase()] : undefined;
+    const data = cleanData(input.data);
+    if (!message && !data)
+        return null;
+    return {
+        type: type,
+        at: at.toISOString(),
+        ...(level ? { level } : {}),
+        message: cleanText(message, MAX_MESSAGE),
+        ...(data ? { data } : {}),
+    };
+}
+/**
+ * Validate, bound and scrub breadcrumbs from any sender: an array, or a JSON
+ * string of one (the span attribute). Keeps the newest 30 valid entries,
+ * oldest first, within 16 KB of JSON; malformed entries are dropped one by
+ * one. Undefined when nothing valid remains.
+ */
+export function parseBreadcrumbs(raw) {
+    let list = raw;
+    if (typeof list === "string") {
+        if (list.length > MAX_RAW_CHARS)
+            return undefined;
+        try {
+            list = JSON.parse(list);
+        }
+        catch {
+            return undefined;
+        }
+    }
+    if (!Array.isArray(list) || list.length === 0)
+        return undefined;
+    const out = [];
+    let chars = 2;
+    const first = Math.max(0, list.length - MAX_SCANNED);
+    for (let i = list.length - 1; i >= first && out.length < MAX_BREADCRUMBS; i--) {
+        const crumb = cleanBreadcrumb(list[i]);
+        if (!crumb)
+            continue;
+        const size = JSON.stringify(crumb).length + 1;
+        if (chars + size > MAX_TOTAL_CHARS)
+            break;
+        chars += size;
+        out.push(crumb);
+    }
+    return out.length > 0 ? out.reverse() : undefined;
+}
+/** `{ breadcrumbs }` for an occurrence, or `{}` — spread into the input. */
+export function breadcrumbsField(...candidates) {
+    for (const candidate of candidates) {
+        if (candidate === undefined || candidate === null)
+            continue;
+        const breadcrumbs = parseBreadcrumbs(candidate);
+        if (breadcrumbs)
+            return { breadcrumbs };
+    }
+    return {};
+}
+/** Attributes JSON to store: the occurrence's attributes plus its breadcrumbs. */
+export function attributesWithBreadcrumbs(attributes, breadcrumbs) {
+    return breadcrumbs?.length
+        ? { ...(attributes ?? {}), [BREADCRUMBS_STORAGE_KEY]: breadcrumbs }
+        : (attributes ?? {});
+}

package/dist/clickhouse.d.ts

@@ -2,13 +2,19 @@ import type { IngesterConfig } from "./config.js";
 import { type LatencyHistogram } from "./latency.js";
 import { type ProfileSample } from "./profiles.js";
 import { type MemorySample, type PlatformEvent } from "./memory.js";
+import { type SourceMapCheck, type ValidSourceMap } from "./source-maps.js";
+import { type SourceMapKey } from "./source-maps-lookup.js";
 import { type RuntimeLogRecord } from "./logs.js";
+import { type SdkVersionRow } from "./sdk-versions.js";
+import type { SchemaStatus } from "./compat.js";
 import { type Migration } from "./migrations.js";
 import type { IngestContext, RuntimeLlmCall, RuntimeMetricPoint, RuntimeOccurrence, RuntimeSpanRow } from "./types.js";
 export declare class ClickHouseStore {
     private readonly config;
     private client;
     private ensurePromise;
+    private schemaState;
+    private appliedIds;
     constructor(config: IngesterConfig);
     get configured(): boolean;
     private getClient;
@@ -23,6 +29,37 @@ export declare class ClickHouseStore {
      * next request retries.
      */
     ensureSchema(migrations?: Migration[]): Promise<void>;
+    /**
+     * Bring runtime_logs' TTL in line with LOG_TTL_DAYS. Idempotent: reads the
+     * current TTL from system.tables and only issues `MODIFY TTL` when it
+     * differs (MODIFY TTL rewrites TTL info for existing parts, so it must not
+     * run on every boot). Best-effort — a failure is logged and never blocks
+     * ingest; the next boot retries.
+     */
+    applyLogTtl(): Promise<void>;
+    /**
+     * Trace ids (from `traceIds`) that already produced an occurrence for
+     * this tenant in [from, to] — the cross-batch half of the log-promotion
+     * dedupe. `excludeIds` are the candidate occurrences' own deterministic
+     * ids, so a retried batch does not dedupe against its own earlier,
+     * partially-committed write. Browser occurrences share the server's trace
+     * id (runtime-browser traceparent) but are a different signal, so they
+     * never suppress a server error. Throws on failure; the caller decides.
+     */
+    recentOccurrenceTraceIds(ctx: IngestContext, traceIds: string[], from: Date, to: Date, excludeIds: string[]): Promise<Set<string>>;
+    /** Schema state as of the last ensureSchema attempt — never triggers DDL
+     * (it backs the unauthenticated /v1/compat endpoint). */
+    schemaStatus(): {
+        status: SchemaStatus;
+        applied: string[];
+    };
+    insertSdkVersions(ctx: IngestContext, rows: SdkVersionRow[]): Promise<void>;
+    /** Record this ingester's version and schema level (global, not tenant data). */
+    insertIngesterInfo(info: {
+        version: string;
+        schemaLevel: string;
+        report: unknown;
+    }): Promise<void>;
     ping(): Promise<boolean>;
     insertOccurrences(ctx: IngestContext, occurrences: RuntimeOccurrence[]): Promise<void>;
     insertSpans(ctx: IngestContext, spans: RuntimeSpanRow[]): Promise<void>;
@@ -30,11 +67,11 @@ export declare class ClickHouseStore {
     insertProfileSamples(ctx: IngestContext, samples: ProfileSample[]): Promise<void>;
     insertMemorySamples(ctx: IngestContext, samples: MemorySample[]): Promise<void>;
     insertPlatformEvent(ctx: IngestContext, event: PlatformEvent): Promise<void>;
-    insertSourceMap(ctx: IngestContext, sourceMap: {
-        release: string;
-        filename: string;
-        map: string;
-    }): Promise<void>;
+    insertSourceMap(ctx: IngestContext, sourceMap: ValidSourceMap): Promise<void>;
+    /** The uploaded source map for a minified file, or null (source-maps-lookup.ts). */
+    lookupSourceMap(key: SourceMapKey): Promise<string | null>;
+    /** Indexes of `check.entries` this tenant uploaded recently (see presentSourceMaps). */
+    sourceMapsPresent(ctx: IngestContext, check: SourceMapCheck): Promise<number[]>;
     insertLlmCalls(ctx: IngestContext, calls: RuntimeLlmCall[]): Promise<void>;
     insertMetricPoints(ctx: IngestContext, points: RuntimeMetricPoint[]): Promise<void>;
     insertLatencyHistograms(ctx: IngestContext, points: LatencyHistogram[]): Promise<void>;

package/dist/clickhouse.js

@@ -3,8 +3,11 @@ import { latencyTableDDL } from "./latency.js";
 import { profileTableDDL } from "./profiles.js";
 import { memoryTableDDL, platformEventTableDDL } from "./memory.js";
 import { sourceMapTableDDL } from "./source-maps.js";
-import { logTableDDL } from "./logs.js";
-import { MIGRATIONS, migrationsTableDDL, } from "./migrations.js";
+import { attributesWithBreadcrumbs } from "./breadcrumbs.js";
+import { lookupSourceMap, presentSourceMaps } from "./source-maps-lookup.js";
+import { logTableDDL, requestRollupTableDDL, } from "./logs.js";
+import { ingesterInfoTableDDL, sdkVersionTableDDL, } from "./sdk-versions.js";
+import { MIGRATIONS, OCCURRENCE_CODE_COLUMNS, OCCURRENCE_GROUPING_COLUMNS, OCCURRENCE_REQUEST_ID_INDEX, migrationsTableDDL, } from "./migrations.js";
 /**
  * ClickHouse persistence. The schema is idempotent and applied lazily once
  * per process. Every row is keyed by (org_id, repository_id) — tenant
@@ -16,10 +19,16 @@ const INSERT_SETTINGS = {
     async_insert: 1,
     wait_for_async_insert: 1,
 };
+/** "YYYY-MM-DD hh:mm:ss.sss" (UTC) — the DateTime64 query-parameter form. */
+function chDateTime(date) {
+    return date.toISOString().replace("T", " ").replace("Z", "");
+}
 export class ClickHouseStore {
     config;
     client = null;
     ensurePromise = null;
+    schemaState = "pending";
+    appliedIds = [];
     constructor(config) {
         this.config = config;
     }
@@ -47,6 +56,7 @@ export class ClickHouseStore {
     schemaStatements() {
         const db = this.config.clickhouseDatabase;
         const { occurrenceTtlDays, spanTtlDays, metricsTtlDays, llmCallTtlDays } = this.config;
+        const logTtlDays = this.config.logTtlDays ?? 14;
         return [
             `CREATE DATABASE IF NOT EXISTS ${db}`,
             latencyTableDDL(db, metricsTtlDays),
@@ -54,7 +64,13 @@ export class ClickHouseStore {
             memoryTableDDL(db),
             platformEventTableDDL(db),
             sourceMapTableDDL(db),
-            logTableDDL(db),
+            logTableDDL(db, logTtlDays),
+            // The table only: its materialized view reads 0012 columns that an
+            // existing runtime_logs lacks until migrations run, so the view is
+            // created by migration 0014 alone (migrations also run on fresh DBs).
+            requestRollupTableDDL(db),
+            sdkVersionTableDDL(db, metricsTtlDays),
+            ingesterInfoTableDDL(db),
             `CREATE TABLE IF NOT EXISTS ${db}.runtime_error_occurrences (
 				org_id             String,
 				repository_id      String,
@@ -79,7 +95,10 @@ export class ClickHouseStore {
 				session_id         String DEFAULT '',
 				attributes         String DEFAULT '{}' CODEC(ZSTD(1)),
 				occurred_at        DateTime64(3, 'UTC'),
-				ingested_at        DateTime64(3, 'UTC') DEFAULT now64(3)
+				ingested_at        DateTime64(3, 'UTC') DEFAULT now64(3),
+				${OCCURRENCE_CODE_COLUMNS.join(",\n\t\t\t\t")},
+				${OCCURRENCE_GROUPING_COLUMNS.join(",\n\t\t\t\t")},
+				${OCCURRENCE_REQUEST_ID_INDEX}
 			)
 			ENGINE = MergeTree
 			PARTITION BY toDate(occurred_at)
@@ -192,12 +211,129 @@ export class ClickHouseStore {
                 });
                 console.log(`clickhouse migration applied: ${migration.id}`);
             }
+            await this.applyLogTtl();
+            this.appliedIds = migrations.map((migration) => migration.id);
+            this.schemaState = "ready";
         })().catch((err) => {
             this.ensurePromise = null;
+            this.schemaState = "failed";
             throw err;
         });
         return this.ensurePromise;
     }
+    /**
+     * Bring runtime_logs' TTL in line with LOG_TTL_DAYS. Idempotent: reads the
+     * current TTL from system.tables and only issues `MODIFY TTL` when it
+     * differs (MODIFY TTL rewrites TTL info for existing parts, so it must not
+     * run on every boot). Best-effort — a failure is logged and never blocks
+     * ingest; the next boot retries.
+     */
+    async applyLogTtl() {
+        const days = this.config.logTtlDays ?? 14;
+        const db = this.config.clickhouseDatabase;
+        try {
+            const client = this.getClient();
+            const rows = await client.query({
+                query: `SELECT engine_full FROM system.tables WHERE database = {db:String} AND name = 'runtime_logs'`,
+                query_params: { db },
+                format: "JSONEachRow",
+            });
+            const [table] = await rows.json();
+            if (!table)
+                return;
+            const current = /\bTTL\b[^]*?toIntervalDay\((\d+)\)/.exec(table.engine_full);
+            if (!current) {
+                console.warn("runtime_logs TTL not recognised — LOG_TTL_DAYS not applied");
+                return;
+            }
+            if (Number(current[1]) === days)
+                return;
+            await client.command({
+                query: `ALTER TABLE ${db}.runtime_logs MODIFY TTL toDateTime(occurred_at) + INTERVAL ${days} DAY`,
+            });
+            console.log(`runtime_logs TTL changed from ${current[1]} to ${days} days (LOG_TTL_DAYS)`);
+        }
+        catch (err) {
+            console.warn("applying LOG_TTL_DAYS to runtime_logs failed (will retry next boot):", err instanceof Error ? err.message : err);
+        }
+    }
+    /**
+     * Trace ids (from `traceIds`) that already produced an occurrence for
+     * this tenant in [from, to] — the cross-batch half of the log-promotion
+     * dedupe. `excludeIds` are the candidate occurrences' own deterministic
+     * ids, so a retried batch does not dedupe against its own earlier,
+     * partially-committed write. Browser occurrences share the server's trace
+     * id (runtime-browser traceparent) but are a different signal, so they
+     * never suppress a server error. Throws on failure; the caller decides.
+     */
+    async recentOccurrenceTraceIds(ctx, traceIds, from, to, excludeIds) {
+        if (traceIds.length === 0 || !this.configured)
+            return new Set();
+        await this.ensureSchema();
+        const rows = await this.getClient().query({
+            query: `SELECT DISTINCT trace_id FROM ${this.table("runtime_error_occurrences")}
+				WHERE org_id = {org:String} AND repository_id = {repo:String}
+					AND occurred_at >= {from:DateTime64(3, 'UTC')} AND occurred_at <= {to:DateTime64(3, 'UTC')}
+					AND source != 'browser'
+					AND has({traces:Array(String)}, trace_id)
+					AND NOT has({exclude:Array(String)}, occurrence_id)`,
+            query_params: {
+                org: ctx.orgId,
+                repo: ctx.repositoryId,
+                from: chDateTime(from),
+                to: chDateTime(to),
+                traces: traceIds,
+                exclude: excludeIds,
+            },
+            format: "JSONEachRow",
+        });
+        return new Set((await rows.json()).map((row) => row.trace_id));
+    }
+    /** Schema state as of the last ensureSchema attempt — never triggers DDL
+     * (it backs the unauthenticated /v1/compat endpoint). */
+    schemaStatus() {
+        if (!this.configured)
+            return { status: "unconfigured", applied: [] };
+        return { status: this.schemaState, applied: [...this.appliedIds] };
+    }
+    async insertSdkVersions(ctx, rows) {
+        if (!rows.length || !this.configured)
+            return;
+        await this.ensureSchema();
+        await this.getClient().insert({
+            table: this.table("runtime_sdk_versions"),
+            format: "JSONEachRow",
+            clickhouse_settings: INSERT_SETTINGS,
+            values: rows.map((row) => ({
+                org_id: ctx.orgId,
+                repository_id: ctx.repositoryId,
+                service: row.service,
+                environment: row.environment,
+                sdk_name: row.sdkName,
+                sdk_version: row.sdkVersion,
+                sdk_language: row.sdkLanguage,
+                ingester_version: row.ingesterVersion,
+                last_seen: row.seenAt.toISOString(),
+            })),
+        });
+    }
+    /** Record this ingester's version and schema level (global, not tenant data). */
+    async insertIngesterInfo(info) {
+        if (!this.configured)
+            return;
+        await this.ensureSchema();
+        await this.getClient().insert({
+            table: this.table("runtime_ingester_info"),
+            format: "JSONEachRow",
+            clickhouse_settings: INSERT_SETTINGS,
+            values: [{
+                    ingester_version: info.version,
+                    schema_level: info.schemaLevel,
+                    report: JSON.stringify(info.report),
+                    reported_at: new Date().toISOString(),
+                }],
+        });
+    }
     async ping() {
         if (!this.configured)
             return false;
@@ -234,8 +370,17 @@ export class ClickHouseStore {
                 status_code: o.statusCode ?? 0,
                 trace_id: o.traceId ?? "",
                 session_id: o.sessionId ?? "",
-                attributes: JSON.stringify(o.attributes ?? {}),
+                attributes: JSON.stringify(attributesWithBreadcrumbs(o.attributes, o.breadcrumbs)),
                 occurred_at: o.occurredAt.toISOString(),
+                error_code: o.errorCode ?? "",
+                error_why: o.why ?? "",
+                error_fix: o.fix ?? "",
+                error_link: o.link ?? "",
+                expected: o.expected ? 1 : 0,
+                request_id: o.requestId ?? "",
+                fingerprint_scheme: o.fingerprintScheme,
+                frames: o.frames?.length ? JSON.stringify(o.frames) : "",
+                fingerprints: (o.fingerprints ?? []).map((candidate) => candidate.fingerprint),
             })),
         });
     }
@@ -278,7 +423,9 @@ export class ClickHouseStore {
                 service: row.service, environment: row.environment, release: row.release, trace_id: row.traceId,
                 span_id: row.spanId, operation_id: row.operationId, operation: row.operation, event_type: row.type,
                 severity: row.severity, message: row.message, outcome: row.outcome, duration_ms: row.durationMs,
-                attributes: JSON.stringify(row.attributes), occurred_at: row.occurredAt.toISOString() })) });
+                attributes: JSON.stringify(row.attributes), occurred_at: row.occurredAt.toISOString(),
+                kind: row.kind, request_id: row.requestId, route: row.route, status_code: row.statusCode,
+                error_code: row.errorCode, ai_cost_usd: row.aiCostUsd, ai_calls: row.aiCalls })) });
     }
     async insertProfileSamples(ctx, samples) {
         if (!samples.length)
@@ -328,9 +475,23 @@ export class ClickHouseStore {
         await this.getClient().insert({
             table: this.table("runtime_source_maps"), format: "JSONEachRow",
             clickhouse_settings: INSERT_SETTINGS,
-            values: [{ org_id: ctx.orgId, repository_id: ctx.repositoryId, ...sourceMap }],
+            values: [{ org_id: ctx.orgId, repository_id: ctx.repositoryId, release: sourceMap.release,
+                    filename: sourceMap.filename, map: sourceMap.map, debug_id: sourceMap.debugId,
+                    content_hash: sourceMap.contentHash }],
         });
     }
+    /** The uploaded source map for a minified file, or null (source-maps-lookup.ts). */
+    async lookupSourceMap(key) {
+        if (!this.configured)
+            return null;
+        await this.ensureSchema();
+        return lookupSourceMap(this.getClient(), this.config.clickhouseDatabase, key);
+    }
+    /** Indexes of `check.entries` this tenant uploaded recently (see presentSourceMaps). */
+    async sourceMapsPresent(ctx, check) {
+        await this.ensureSchema();
+        return presentSourceMaps(this.getClient(), this.config.clickhouseDatabase, ctx, check);
+    }
     async insertLlmCalls(ctx, calls) {
         if (calls.length === 0 || !this.configured)
             return;