@aryasaatvik/motel 0.3.5 → 0.4.0

package/AGENTS.md

@@ -97,7 +97,7 @@ The repo is wired up with `@effect/language-service` as a `tsconfig.json` `plugi
 - `src/server.ts` runs the local server without the TUI.
 - `src/instructions.ts` contains the copied setup instructions for other Effect apps.
 - `src/services/TelemetryStore.ts` persists traces and logs in SQLite and exposes indexed queries through writer and read-only service identifiers.
-- `src/services/TelemetryQuery.ts` proxies read-only store calls to `src/services/telemetryQueryWorker.ts`, keeping synchronous Bun SQLite queries off the HTTP event loop.
+- `src/services/TelemetryQuery.ts` schedules read-only store calls through a bounded queue and a disposable subprocess containing `src/services/telemetryQueryWorker.ts`. Deadlines and cancellation terminate that process to release synchronous SQLite readers; the HTTP server and TUI share this architecture.
 - `src/config.ts` is the source of truth for ports and env-driven OTEL settings.
 - `web/` is a Vite + React SPA for the browser-based UI (Tailwind CSS, `@effect/atom-react`, `AtomHttpApi`).
 - `web/src/api.ts` creates the typed `AtomHttpApi.Service` client from `src/httpApi.ts`.
@@ -147,6 +147,9 @@ The repo is wired up with `@effect/language-service` as a `tsconfig.json` `plugi
 - `MOTEL_OTEL_MAX_DB_SIZE_MB`: defaults to `1024` (size-based retention cap)
 - `MOTEL_OTEL_RETENTION_TRACE_BATCH`: defaults to `100` completed traces per cleanup pass
 - `MOTEL_OTEL_RETENTION_LOG_BATCH`: defaults to `5000` logs per cleanup pass
+- `MOTEL_OTEL_QUERY_CAPACITY`: defaults to `8` outstanding queries per runtime (maximum `64`)
+- `MOTEL_OTEL_QUERY_DEADLINE_MS`: defaults to `5000` including queue and startup time (maximum `30000`)
+- `MOTEL_OTEL_RETENTION_ROW_BATCH`: defaults to `1000` physical rows per cleanup pass
 - `MOTEL_OTEL_RETENTION_INTERVAL_SECONDS`: defaults to `10`
 
 ## TUI Keys

package/README.md

@@ -135,3 +135,54 @@ telemetry. Correlated logs may include secrets, tokens, or PII if your
 app logs them; AI call traces may include full prompt content and
 response text. Treat the local SQLite store as sensitive development
 data when pointing motel at real workloads.
+
+### Ingestion readiness
+
+`GET /api/health` reports process identity and liveness. `GET /api/readiness` reports cached
+writer diagnostics without querying SQLite or joining the ingestion queue. It returns HTTP
+200 for `ready`, and 503 for `starting`, `overloaded`, or `failed`. `motel status` includes this
+snapshot when the daemon supports it; older servers remain compatible.
+
+`overloaded` means an awaiting request is at least five seconds old or at least 64 requests
+are outstanding. This is a pressure signal, not an instruction to restart a shared daemon.
+Idle time does not imply failure. `failed` records worker exit/error and requires operator
+inspection. Outstanding counts/bytes describe awaiting callers, not durable queued data;
+client cancellation cannot undo a SQLite commit. Empty probes do not advance `lastCommitAt`.
+Maintenance reports its latest operation, duration, and outcome without telemetry payloads.
+
+Run `bun scripts/bench-shared-daemon.ts` for an isolated reduced-cap mixed workload, or add
+`--1gib` for the representative capacity profile. Both create and remove their own database,
+include a held reader, and report request latency percentiles, failures, and database/WAL size.
+They never benchmark the shared machine-global database.
+
+Concurrent starts publish a complete shared lock atomically. Ensure operations preserve a live
+daemon when ingestion is slow; explicit operator recovery remains available through `motel restart`.
+An unreadable lock or interrupted stale-lock recovery fails closed and reports its path for inspection.
+When maintenance operations overlap, readiness shows the oldest active operation before a completed one.
+
+Retention marks completed traces and expired logs before removing dependent rows. Public query
+results hide the whole marked trace, including correlated logs, while bounded cleanup proceeds.
+Markers persist across restarts. Arrivals for a marked trace are discarded as part of its eviction
+and do not count as inserted records; once cleanup finishes, later exports may recreate that trace.
+Active traces are not selected. SQLite tables are internal storage; direct SQL consumers should
+use the `retained_spans`, `retained_logs`, and `retained_trace_summaries` views for this visibility rule.
+
+`MOTEL_OTEL_RETENTION_ROW_BATCH` defaults to 1000 logical rows per cleanup batch, split across
+traces and logs. It bounds row cardinality, not the size of one telemetry field. Cleanup yields
+100 ms between batches while making progress, then returns to the configured idle interval.
+Routine checkpoints use PASSIVE and report busy/frame counts in readiness; physical reclamation
+is deferred for held readers. The configured database target and WAL size limit are not hard caps.
+Existing orphan attributes/FTS entries are repaired with bounded keyset scans; normal ingests
+maintain reverse FTS mappings for indexed cleanup.
+
+Queries use one readonly subprocess at a time, with FIFO admission capped at eight outstanding
+requests by default (`MOTEL_OTEL_QUERY_CAPACITY`, maximum 64). The five-second deadline includes
+queue and startup time (`MOTEL_OTEL_QUERY_DEADLINE_MS`, maximum 30000). Overload returns HTTP 503
+with `code: "QUERY_OVERLOADED"`; expiry returns 504 with `code: "QUERY_DEADLINE"`; process failure
+returns 503 with `code: "QUERY_UNAVAILABLE"`. Successful response shapes are unchanged.
+
+Cancelling or expiring an executing query terminates its readonly process and waits for exit
+before dispatching the next query. SQLite runs in a thread inside that process so an IPC-owner
+disconnect can also terminate it if the daemon or TUI dies. Ingestion owns a separate worker.
+The HTTP idle timeout exceeds the query deadline by five seconds. TUI refreshes share in-flight
+loads, obsolete selections discard results, and facet values prefetch sequentially for two keys.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aryasaatvik/motel",
-  "version": "0.3.5",
+  "version": "0.4.0",
   "description": "A local OpenTelemetry ingest + TUI viewer for development, backed by SQLite.",
   "type": "module",
   "license": "MIT",

package/src/config.ts

@@ -39,9 +39,12 @@ export const config = {
 		traceLookbackMinutes: parsePositiveInt(process.env.MOTEL_OTEL_TRACE_LOOKBACK_MINUTES, 1440),
 		traceFetchLimit: parsePositiveInt(process.env.MOTEL_OTEL_TRACE_LIMIT, 100),
 		logFetchLimit: parsePositiveInt(process.env.MOTEL_OTEL_LOG_LIMIT, 80),
+		queryCapacity: Math.min(parsePositiveInt(process.env.MOTEL_OTEL_QUERY_CAPACITY, 8), 64),
+		queryDeadlineMs: Math.min(parsePositiveInt(process.env.MOTEL_OTEL_QUERY_DEADLINE_MS, 5_000), 30_000),
 		retentionHours: parsePositiveInt(process.env.MOTEL_OTEL_RETENTION_HOURS, 168),
 		maxDbSizeMb: parsePositiveInt(process.env.MOTEL_OTEL_MAX_DB_SIZE_MB, 1024),
 		retentionTraceBatch: parsePositiveInt(process.env.MOTEL_OTEL_RETENTION_TRACE_BATCH, 100),
+		retentionRowBatch: parsePositiveInt(process.env.MOTEL_OTEL_RETENTION_ROW_BATCH, 1_000),
 		retentionLogBatch: parsePositiveInt(process.env.MOTEL_OTEL_RETENTION_LOG_BATCH, 5_000),
 		retentionIntervalSeconds: parsePositiveInt(process.env.MOTEL_OTEL_RETENTION_INTERVAL_SECONDS, 10),
 	},

package/src/daemon.ts

@@ -1,7 +1,8 @@
 import * as fs from "node:fs"
 import { promises as fsp } from "node:fs"
 import * as path from "node:path"
-import { Effect } from "effect"
+import { Effect, Schema } from "effect"
+import { IngestReadiness } from "./ingestReadiness.ts"
 import { isAlive, isManagedDaemonProcess, listAliveEntries, motelStateDir, MOTEL_SERVICE_ID, processIdentity, removeRegistryEntry, type RegistryEntry } from "./registry.js"
 
 const DEFAULT_REPO_ROOT = path.resolve(import.meta.dir, "..")
@@ -58,6 +59,7 @@ type DaemonConfig = {
 }
 
 export type DaemonStatus = {
+	readonly readiness?: IngestReadiness
 	readonly running: boolean
 	readonly managed: boolean
 	readonly service: string | null
@@ -170,13 +172,13 @@ export const createDaemonManager = (options: DaemonOptions = {}): DaemonManager
 		}
 	}
 
-	const fetchIngestProbe = async () => {
+	const fetchIngestProbe = async (timeoutMs = INGEST_PROBE_TIMEOUT_MS) => {
 		try {
 			const postEmpty = (path: string) => fetch(`${config.baseUrl}${path}`, {
 				method: "POST",
 				headers: { "content-type": "application/json" },
 				body: "{}",
-				signal: AbortSignal.timeout(INGEST_PROBE_TIMEOUT_MS),
+				signal: AbortSignal.timeout(timeoutMs),
 			})
 			const [traces, logs] = await Promise.all([postEmpty("/v1/traces"), postEmpty("/v1/logs")])
 			return traces.ok && logs.ok
@@ -198,47 +200,55 @@ export const createDaemonManager = (options: DaemonOptions = {}): DaemonManager
 	const readLock = async (): Promise<LockShape | null> => {
 		try {
 			const raw = await fsp.readFile(config.lockPath, "utf8")
-			return JSON.parse(raw) as LockShape
+			const lock = JSON.parse(raw) as Partial<LockShape>
+			if (!Number.isInteger(lock.pid) || (lock.pid ?? 0) <= 0 || typeof lock.createdAt !== "string" ||
+				(lock.processIdentity !== undefined && typeof lock.processIdentity !== "string")) return null
+			return lock as LockShape
 		} catch {
 			return null
 		}
 	}
 
 	const removeStaleLock = async () => {
-		const current = await readLock()
-		if (!current) {
+		// Serialize stale-owner cleanup so two contenders cannot unlink a fresh
+		// replacement after both observed the same dead owner. Unknown lock contents
+		// fail closed: they may belong to an older writer still publishing its lock.
+		const recoveryPath = `${config.lockPath}.recovery`
+		try { await fsp.mkdir(recoveryPath) }
+		catch (error) {
+			if ((error as NodeJS.ErrnoException).code === "EEXIST") return false
+			throw error
+		}
+		try {
+			const current = await readLock()
+			if (!current) return false
+			if (current.processIdentity ? processIdentity(current.pid) === current.processIdentity : isAlive(current.pid)) return false
 			await fsp.rm(config.lockPath, { force: true })
 			return true
-		}
-		if (current.processIdentity ? processIdentity(current.pid) === current.processIdentity : isAlive(current.pid)) return false
-		await fsp.rm(config.lockPath, { force: true })
-		return true
+		} finally { await fsp.rmdir(recoveryPath) }
 	}
 
 	const acquireStartupLock = async () => {
 		const deadline = Date.now() + LOCK_TIMEOUT_MS
 		await fsp.mkdir(config.runtimeDir, { recursive: true })
-
-		while (Date.now() < deadline) {
-			try {
-				const handle = await fsp.open(config.lockPath, "wx")
-				const contents = JSON.stringify({ pid: process.pid, createdAt: new Date().toISOString(), processIdentity: processIdentity(process.pid) ?? undefined } satisfies LockShape)
-				await handle.writeFile(contents, "utf8")
-				return {
-					release: async () => {
-						await handle.close().catch(() => undefined)
-						await fsp.rm(config.lockPath, { force: true }).catch(() => undefined)
-					},
+		const candidate = `${config.lockPath}.${process.pid}.${crypto.randomUUID()}`
+		const contents = JSON.stringify({ pid: process.pid, createdAt: new Date().toISOString(), processIdentity: processIdentity(process.pid) ?? undefined } satisfies LockShape)
+		await fsp.writeFile(candidate, contents, { flag: "wx" })
+		try {
+			while (Date.now() < deadline) {
+				try {
+					// link is exclusive and publishes a complete file atomically. Unlike
+					// open("wx") followed by writeFile, there is no observable empty lock.
+					await fsp.link(candidate, config.lockPath)
+					return { release: () => fsp.rm(config.lockPath, { force: true }) }
+				} catch (error) {
+					if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error
+					if (await removeStaleLock()) continue
+					await sleep(POLL_INTERVAL_MS)
 				}
-			} catch (error) {
-				const errno = error as NodeJS.ErrnoException
-				if (errno.code !== "EEXIST") throw error
-				if (await removeStaleLock()) continue
-				await sleep(POLL_INTERVAL_MS)
 			}
-		}
-
-		throw new Error(`Timed out waiting for daemon startup lock at ${config.lockPath}`)
+			throw new Error(`Timed out waiting for daemon startup lock at ${config.lockPath}. Inspect its owner and ${config.lockPath}.recovery before recovery.`)
+		} finally { await fsp.rm(candidate, { force: true }) }
 	}
 
 	const openLogFile = async () => {
@@ -321,8 +331,18 @@ export const createDaemonManager = (options: DaemonOptions = {}): DaemonManager
 		}
 
 		const mismatch = describeManagedMismatch(health)
+		let readiness: IngestReadiness | undefined
+		if (mismatch === null) {
+			try {
+				const response = await fetch(`${config.baseUrl}/api/readiness`, { signal: AbortSignal.timeout(timeoutMs) })
+				if (response.status === 200 || response.status === 503) {
+					readiness = Schema.decodeUnknownSync(IngestReadiness)(await response.json())
+				}
+			} catch { /* Older daemons can still answer the identity handshake. */ }
+		}
 		const managed = mismatch === null && registry?.pid === health.pid && registry.instanceId === health.instanceId && isManagedDaemonProcess(registry)
 		return {
+			readiness,
 			running: mismatch === null,
 			managed,
 			service: health.service,
@@ -340,6 +360,21 @@ export const createDaemonManager = (options: DaemonOptions = {}): DaemonManager
 		}
 	}
 
+	const awaitExistingIngest = async (existing: DaemonStatus): Promise<DaemonStatus> => {
+		if (existing.pid === process.pid) return existing
+		const deadline = Date.now() + startTimeoutMs
+		while (Date.now() < deadline) {
+			if (await fetchIngestProbe(Math.max(1, Math.min(INGEST_PROBE_TIMEOUT_MS, deadline - Date.now())))) {
+				const current = await getStatus()
+				if (current.running && current.managed) return current
+				break
+			}
+			if (existing.readiness?.state === "failed") break
+			await sleep(Math.min(POLL_INTERVAL_MS, Math.max(0, deadline - Date.now())))
+		}
+		throw new Error("Existing managed Motel daemon is not ingest-ready. Its process was preserved; inspect `motel status` and recover explicitly.")
+	}
+
 	const ensure = async (): Promise<DaemonStatus> => {
 		// Use the patient timeout for the initial probe — this is the
 		// critical "is there already a daemon here?" check. A false
@@ -351,10 +386,11 @@ export const createDaemonManager = (options: DaemonOptions = {}): DaemonManager
 			// /api/health can stay healthy after the lazy ingest worker/RPC path
 			// has been poisoned by an interrupted request. Empty OTLP posts are
 			// side-effect free and exercise the same path real exporters need.
-			if (existing.pid === process.pid || await fetchIngestProbe()) return existing
-			if (existingEntry) await stopPid(existingEntry)
+			return awaitExistingIngest(existing)
+		}
+		if (!existing.running && existingEntry && isManagedDaemonProcess(existingEntry)) {
+			throw new Error("Registered Motel process is alive but health is unavailable. Its process was preserved; inspect `motel status` before recovery.")
 		}
-		if (!existing.running && existingEntry && isManagedDaemonProcess(existingEntry)) await stopPid(existingEntry)
 		if (existing.service !== null && existing.reason) {
 			throw new Error(existing.reason)
 		}
@@ -369,9 +405,7 @@ export const createDaemonManager = (options: DaemonOptions = {}): DaemonManager
 			// while the runtime warms up.
 			const rechecked = await getStatus(HEALTH_PATIENT_TIMEOUT_MS)
 			if (rechecked.managed && rechecked.running) {
-				if (rechecked.pid === process.pid || await fetchIngestProbe()) return rechecked
-				const recheckedEntry = readRegistryEntry()
-				if (recheckedEntry) await stopPid(recheckedEntry)
+				return awaitExistingIngest(rechecked)
 			}
 			if (rechecked.service !== null && rechecked.reason) {
 				throw new Error(rechecked.reason)

package/src/httpApi.ts

@@ -6,7 +6,14 @@ import {
 	TraceItem, TraceSummaryItem, SpanItem, LogItem,
 } from "./domain.js"
 
+import { IngestReadiness } from "./ingestReadiness.js"
+
 const ErrorResponse = Schema.Struct({ error: Schema.String })
+const QueryFailures = [
+	Schema.Struct({ error: Schema.String, code: Schema.Literals(["QUERY_OVERLOADED", "QUERY_UNAVAILABLE"]) }).pipe(HttpApiSchema.status(503)),
+	Schema.Struct({ error: Schema.String, code: Schema.Literal("QUERY_DEADLINE") }).pipe(HttpApiSchema.status(504)),
+] as const
+
 const Meta = Schema.Struct({
 	limit: Schema.Number,
 	lookback: Schema.String,
@@ -80,6 +87,9 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 					.annotate(OpenApi.Summary, "Health check and identity handshake")
 					.annotate(OpenApi.Description, "Returns liveness plus identity fields (pid, url, workdir, startedAt, version). Doubles as the MCP discovery handshake: clients compare the returned pid against a registry entry to detect stale registrations that now point at an impostor process on the same port."),
 
+				HttpApiEndpoint.get("readiness", "/api/readiness", { success: IngestReadiness, error: IngestReadiness.pipe(HttpApiSchema.status(503)) })
+					.annotate(OpenApi.Summary, "Cached writer readiness and progress; does not enqueue database work"),
+
 				HttpApiEndpoint.post("ingestTraces", "/v1/traces", {
 					payload: Schema.Unknown,
 					success: IngestTraceResponse,
@@ -94,7 +104,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 					.annotate(OpenApi.Summary, "Ingest OTLP logs")
 					.annotate(OpenApi.Description, "Accepts OTLP HTTP log export requests and stores them in the local SQLite telemetry store."),
 
-				HttpApiEndpoint.get("services", "/api/services", { success: ServiceList })
+				HttpApiEndpoint.get("services", "/api/services", { success: ServiceList, error: QueryFailures })
 					.annotate(OpenApi.Summary, "List active services")
 					.annotate(OpenApi.Description, "Returns service names that have emitted spans or logs within the default lookback window. Use this to discover what services are reporting, then query their traces or logs."),
 
@@ -106,6 +116,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						cursor: CursorParam,
 					},
 					success: TraceSummaryList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "List recent traces")
 					.annotate(OpenApi.Description, "Returns compact trace summaries ordered by start time descending. Use /api/traces/{traceId} for the full span tree. Supports cursor pagination and applies default/max limit and lookback bounds."),
@@ -130,6 +141,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						cursor: CursorParam,
 					},
 					success: TraceSummaryList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "Search traces with filters")
 					.annotate(OpenApi.Description, "Search compact trace summaries with filters. Use /api/traces/{traceId} for full details. Supports cursor pagination, attr.<key> filters in the query string, and aiText for full-text search across LLM prompt/response content."),
@@ -146,7 +158,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						limit: LimitParam,
 					},
 					success: StatList,
-					error: ErrorResponse,
+					error: [ErrorResponse, ...QueryFailures],
 				})
 					.annotate(OpenApi.Summary, "Aggregate trace statistics")
 					.annotate(OpenApi.Description, "Returns grouped trace aggregates such as count, average duration, p95 duration, or error rate. Supports the same core filters as trace search plus groupBy dimensions like service, operation, status, and attr.<key>."),
@@ -156,7 +168,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						traceId: Schema.String.pipe(Schema.annotateKey({ description: "Full 32-character hex trace ID" })),
 					},
 					success: TraceResponse,
-					error: ErrorResponse,
+					error: [ErrorResponse, ...QueryFailures],
 				})
 					.annotate(OpenApi.Summary, "Get a single trace")
 					.annotate(OpenApi.Description, "Returns the full trace with all spans ordered by parent-child hierarchy. Returns 404 if the trace ID is not found or has expired."),
@@ -166,7 +178,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						traceId: Schema.String.pipe(Schema.annotateKey({ description: "Full 32-character hex trace ID" })),
 					},
 					success: HtmlText,
-					error: ErrorResponse,
+					error: [ErrorResponse, ...QueryFailures],
 				})
 					.annotate(OpenApi.Summary, "Render a browser trace page")
 					.annotate(OpenApi.Description, "Renders a simple HTML waterfall/log view for one trace, suitable for opening from the TUI or browser."),
@@ -181,6 +193,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						cursor: CursorParam,
 					},
 					success: LogList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "Get logs for a trace")
 					.annotate(OpenApi.Description, "Returns log records correlated with the given trace, across all spans. Ordered by timestamp descending. Supports cursor pagination and bounded lookback/limit defaults."),
@@ -190,6 +203,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						traceId: Schema.String.pipe(Schema.annotateKey({ description: "Full 32-character hex trace ID" })),
 					},
 					success: SpanList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "List spans for a trace")
 					.annotate(OpenApi.Description, "Returns the flat list of spans for one trace, preserving trace context on each row. Useful for span-level filtering and sorting without traversing the full tree shape."),
@@ -199,7 +213,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						spanId: Schema.String.pipe(Schema.annotateKey({ description: "Full 16-character hex span ID" })),
 					},
 					success: SpanResponse,
-					error: ErrorResponse,
+					error: [ErrorResponse, ...QueryFailures],
 				})
 					.annotate(OpenApi.Summary, "Get a single span")
 					.annotate(OpenApi.Description, "Returns a span by its ID, including the parent trace ID and root operation name for context. Returns 404 if the span is not found."),
@@ -214,6 +228,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						cursor: CursorParam,
 					},
 					success: LogList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "Get logs for a span")
 					.annotate(OpenApi.Description, "Returns log records correlated with the given span. Ordered by timestamp descending. Supports cursor pagination and bounded lookback/limit defaults."),
@@ -231,6 +246,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						limit: LimitParam,
 					},
 					success: PaginatedSpanList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "Search spans directly")
 					.annotate(OpenApi.Description, "Search spans directly instead of root traces. Supports service, traceId, operation, parentOperation, status, lookback, limit, attr.<key>=<value> (exact match), and attrContains.<key>=<substring> (case-insensitive substring search inside attribute values) in the query string."),
@@ -255,6 +271,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						cursor: CursorParam,
 					},
 					success: LogList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "Search logs")
 					.annotate(OpenApi.Description, "Search log records by service, severity, trace/span correlation, or body text (case-insensitive). Supports attr.<key>=<value> (exact match) and attrContains.<key>=<substring> (case-insensitive substring) in the query string. Cursor pagination and bounded lookback/limit defaults. Ordered by timestamp descending."),
@@ -273,6 +290,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						cursor: CursorParam,
 					},
 					success: LogList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "Alias for log search")
 					.annotate(OpenApi.Description, "Same behavior as GET /api/logs. Exists as an explicit search endpoint for agents and scripts that distinguish list vs search routes."),
@@ -289,7 +307,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						limit: LimitParam,
 					},
 					success: StatList,
-					error: ErrorResponse,
+					error: [ErrorResponse, ...QueryFailures],
 				})
 					.annotate(OpenApi.Summary, "Aggregate log statistics")
 					.annotate(OpenApi.Description, "Returns grouped log counts by fields like severity, service, scope, or attr.<key>. Useful for quickly understanding log distribution before drilling into raw entries."),
@@ -324,7 +342,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						limit: LimitParam,
 					},
 					success: FacetList,
-					error: ErrorResponse,
+					error: [ErrorResponse, ...QueryFailures],
 				})
 					.annotate(OpenApi.Summary, "Get facet value counts")
 					.annotate(OpenApi.Description, "Returns distinct values and their counts for a given field, useful for discovering what data exists before querying. Examples: ?type=logs&field=severity returns log level distribution; ?type=traces&field=attribute_keys&service=opencode lists top span attribute keys; ?type=traces&field=attribute_values&key=ai.model.id lists values seen for that key."),
@@ -346,6 +364,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						limit: LimitParam,
 					},
 					success: AiCallList,
+					error: QueryFailures,
 				})
 					.annotate(OpenApi.Summary, "Search AI calls")
 					.annotate(OpenApi.Description, "Search AI SDK calls (streamText, generateText, etc.) with normalized fields. Returns compact summaries with previews — use /api/ai/calls/{spanId} for full prompt/response payloads. Supports filtering by service, session, model, provider, function, operation, status, duration, and free-text search across prompt/response content."),
@@ -355,7 +374,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						spanId: Schema.String.pipe(Schema.annotateKey({ description: "The span ID of the AI call" })),
 					},
 					success: AiCallDetailResponse,
-					error: ErrorResponse,
+					error: [ErrorResponse, ...QueryFailures],
 				})
 					.annotate(OpenApi.Summary, "Get AI call detail")
 					.annotate(OpenApi.Description, "Returns the full detail of a single AI call including complete prompt messages, response text, tool calls with args, token usage, timing, provider metadata, and correlated logs."),
@@ -381,7 +400,7 @@ export const MotelHttpApi = HttpApi.make("MotelTelemetry")
 						limit: LimitParam,
 					},
 					success: StatList,
-					error: ErrorResponse,
+					error: [ErrorResponse, ...QueryFailures],
 				})
 					.annotate(OpenApi.Summary, "Aggregate AI call statistics")
 					.annotate(OpenApi.Description, "Returns grouped statistics for AI calls. Supports grouping by provider, model, functionId, sessionId, or status with aggregations: count, avg_duration, p95_duration, total_input_tokens, total_output_tokens."),

package/src/ingestReadiness.test.ts

@@ -0,0 +1,160 @@
+import { Database } from "bun:sqlite"
+import { expect, test } from "bun:test"
+import { mkdtempSync, mkdirSync, rmSync } from "node:fs"
+import { tmpdir } from "node:os"
+import { join } from "node:path"
+import { Effect } from "effect"
+import { createDaemonManager } from "./daemon.ts"
+import { IngestProgress, type IngestReadiness } from "./ingestReadiness.ts"
+
+const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms))
+
+test("idle readiness, overload age, empty commits, and terminal failure are distinct", () => {
+	const progress = new IngestProgress()
+	expect(progress.snapshot().state).toBe("starting")
+	progress.receive({ _tag: "ready" })
+	expect(progress.snapshot(Date.now() + 86_400_000).state).toBe("ready")
+	const finish = progress.begin(42, 100)
+	expect(progress.snapshot(5100)).toMatchObject({ state: "overloaded", outstandingBytes: 42, outstandingRequests: 1 })
+	progress.receive({ _tag: "commit", records: 0, at: 500 })
+	expect(progress.snapshot().lastCommitAt).toBeNull()
+	finish()
+	finish()
+	expect(progress.snapshot()).toMatchObject({ state: "ready", outstandingBytes: 0, outstandingRequests: 0 })
+	progress.fail()
+	progress.receive({ _tag: "ready" })
+	expect(progress.snapshot().state).toBe("failed")
+})
+
+test("cached readiness bypasses a blocked writer and acknowledgement follows commit", async () => {
+	const runtimeDir = mkdtempSync(join(tmpdir(), "motel-readiness-"))
+	const databasePath = join(runtimeDir, "telemetry.sqlite")
+	const port = 31000 + Math.floor(Math.random() * 2000)
+	const manager = createDaemonManager({ runtimeDir, databasePath, port })
+	let lock: Database | undefined
+	try {
+		await Effect.runPromise(manager.ensure)
+		const base = `http://127.0.0.1:${port}`
+		const readiness = async () => {
+			const response = await fetch(`${base}/api/readiness`, { signal: AbortSignal.timeout(1500) })
+			return await response.json() as IngestReadiness
+		}
+		expect((await readiness()).state).toBe("ready")
+		lock = new Database(databasePath)
+		lock.exec("BEGIN IMMEDIATE")
+		const body = JSON.stringify({ resourceLogs: [{ resource: { attributes: [{ key: "service.name", value: { stringValue: "readiness-test" } }] }, scopeLogs: [{ logRecords: [{ timeUnixNano: String(BigInt(Date.now()) * 1_000_000n), body: { stringValue: "commit barrier" } }] }] }] })
+		let acknowledged = false
+		const ingestion = fetch(`${base}/v1/logs`, { method: "POST", headers: { "content-type": "application/json" }, body }).then(async (response) => {
+			expect(response.status).toBe(200)
+			acknowledged = true
+			return response.json()
+		})
+		await sleep(150)
+		const snapshot = await readiness()
+		expect(snapshot.outstandingRequests).toBe(1)
+		expect(snapshot.outstandingBytes).toBe(Buffer.byteLength(body))
+		expect(acknowledged).toBe(false)
+		expect(lock.query("SELECT count(*) AS n FROM logs WHERE body = 'commit barrier'").get()).toEqual({ n: 0 })
+		lock.exec("COMMIT")
+		expect(await ingestion).toEqual({ insertedLogs: 1 })
+		for (let i = 0; i < 50 && (await readiness()).committedRecords === 0; i++) await sleep(10)
+		expect(await readiness()).toMatchObject({ state: "ready", outstandingRequests: 0, outstandingBytes: 0, committedRecords: 1 })
+		expect(lock.query("SELECT count(*) AS n FROM logs WHERE body = 'commit barrier'").get()).toEqual({ n: 1 })
+		expect((await Effect.runPromise(manager.getStatus)).readiness?.state).toBe("ready")
+	} finally {
+		try { lock?.exec("ROLLBACK") } catch { /* The successful path committed. */ }
+		lock?.close()
+		await Effect.runPromise(manager.stop)
+		rmSync(runtimeDir, { recursive: true, force: true })
+	}
+}, 30000)
+
+test("a writer bootstrap failure is reported without mistaking liveness for readiness", async () => {
+	const root = mkdtempSync(join(tmpdir(), "motel-writer-failure-"))
+	const port = 35000 + Math.floor(Math.random() * 1000)
+	const child = Bun.spawn([process.execPath, "src/server.ts"], {
+		cwd: join(import.meta.dir, ".."),
+		env: { ...process.env, MOTEL_OTEL_DB_PATH: root, MOTEL_RUNTIME_DIR: root, MOTEL_OTEL_HOST: "127.0.0.1", MOTEL_OTEL_PORT: String(port), MOTEL_OTEL_BASE_URL: `http://127.0.0.1:${port}`, MOTEL_OTEL_ENABLED: "false" },
+		stdout: "ignore", stderr: "ignore",
+	})
+	try {
+		let state: string | undefined
+		const until = Date.now() + 10000
+		while (Date.now() < until) {
+			try {
+				const response = await fetch(`http://127.0.0.1:${port}/api/readiness`, { signal: AbortSignal.timeout(500) })
+				state = (await response.json() as IngestReadiness).state
+				if (state === "failed") {
+					expect(response.status).toBe(503)
+					break
+				}
+			} catch { /* Wait for the HTTP listener independently of writer bootstrap. */ }
+			await sleep(25)
+		}
+		expect(state).toBe("failed")
+		expect((await fetch(`http://127.0.0.1:${port}/api/health`)).status).toBe(200)
+	} finally {
+		child.kill("SIGTERM")
+		await Promise.race([child.exited, sleep(1000)])
+		if (child.exitCode === null) child.kill("SIGKILL")
+		await child.exited
+		rmSync(root, { recursive: true, force: true })
+	}
+}, 15000)
+
+
+test("completed maintenance cannot hide another operation that is still running", () => {
+	const progress = new IngestProgress()
+	progress.receive({ _tag: "maintenance", value: { operation: "retention", startedAt: 10, durationMs: null, outcome: "running" } })
+	progress.receive({ _tag: "maintenance", value: { operation: "reclaim", startedAt: 20, durationMs: null, outcome: "running" } })
+	progress.receive({ _tag: "maintenance", value: { operation: "reclaim", startedAt: 20, durationMs: 5, outcome: "completed" } })
+	expect(progress.snapshot().maintenance).toMatchObject({ operation: "retention", outcome: "running" })
+	progress.receive({ _tag: "maintenance", value: { operation: "retention", startedAt: 10, durationMs: 30, outcome: "completed" } })
+	expect(progress.snapshot().maintenance).toMatchObject({ operation: "retention", outcome: "completed", durationMs: 30 })
+})
+
+test("simultaneous cold starts from different workdirs converge on one managed daemon", async () => {
+	const root = mkdtempSync(join(tmpdir(), "motel-concurrent-start-"))
+	const workdirs = [join(root, "a"), join(root, "b")]
+	for (const workdir of workdirs) mkdirSync(workdir)
+	const port = 36000 + Math.floor(Math.random() * 1000)
+	const managers = workdirs.map((workdir) => createDaemonManager({ runtimeDir: root, databasePath: join(root, "db.sqlite"), port, workdir, startTimeoutMs: 5000, gracefulStopTimeoutMs: 500, forceStopTimeoutMs: 500 }))
+	try {
+		const states = await Promise.all(managers.map((manager) => Effect.runPromise(manager.ensure)))
+		expect(states.every((state) => state.running && state.managed)).toBe(true)
+		expect(states[0]!.pid).toBe(states[1]!.pid)
+	} finally {
+		await Effect.runPromise(managers[0]!.stop)
+		rmSync(root, { recursive: true, force: true })
+	}
+}, 30000)
+
+test("ensure preserves a live daemon when its ingest readiness budget expires", async () => {
+	const root = mkdtempSync(join(tmpdir(), "motel-busy-ensure-"))
+	const databasePath = join(root, "db.sqlite")
+	const port = 37000 + Math.floor(Math.random() * 1000)
+	const manager = createDaemonManager({ runtimeDir: root, databasePath, port })
+	const impatient = createDaemonManager({ runtimeDir: root, databasePath, port, startTimeoutMs: 150 })
+	let lock: Database | undefined
+	try {
+		const started = await Effect.runPromise(manager.ensure)
+		lock = new Database(databasePath)
+		lock.exec("BEGIN IMMEDIATE")
+		const write = fetch(`http://127.0.0.1:${port}/v1/logs`, {
+			method: "POST", headers: { "content-type": "application/json" },
+			body: JSON.stringify({ resourceLogs: [{ scopeLogs: [{ logRecords: [{ body: { stringValue: "blocked writer" } }] }] }] }),
+		})
+		await sleep(30)
+		await expect(Effect.runPromise(impatient.ensure)).rejects.toThrow("process was preserved")
+		const health = await (await fetch(`http://127.0.0.1:${port}/api/health`)).json() as { pid: number }
+		expect(started.pid).toBe(health.pid)
+		lock.exec("ROLLBACK")
+		expect((await write).status).toBe(200)
+		expect((await Effect.runPromise(manager.ensure)).pid).toBe(started.pid)
+	} finally {
+		try { lock?.exec("ROLLBACK") } catch {}
+		lock?.close()
+		await Effect.runPromise(manager.stop)
+		rmSync(root, { recursive: true, force: true })
+	}
+}, 15000)