doc2vec 2.11.0 → 2.13.0

package/README.md

@@ -6,7 +6,12 @@ This project provides a configurable tool (`doc2vec`) to crawl specified website
 
 The primary goal is to prepare documentation content for Retrieval-Augmented Generation (RAG) systems or semantic search applications.
 
-> **⚠️ Version 2.0.0 Breaking Change:** Version 2.0.0 introduced enhanced chunking with new metadata fields (`chunk_index` and `total_chunks`) that enable page reconstruction and improved chunk ordering. The database schema has changed, and databases created with versions prior to 2.0.0 use a different format. **If you're upgrading to version 2.0.0 or later, you should start with fresh databases** to take advantage of the new features. While the MCP server maintains backward compatibility for querying old databases, doc2vec itself will create databases in the new format. If you need to migrate existing data, consider re-running doc2vec on your sources to regenerate the databases with the enhanced chunking format.
+Run it two ways:
+
+*   **One-shot sync** — `doc2vec run config.yaml` processes every source in a config file and exits. Ideal for a cron job or CI step.
+*   **[Controller mode](#controller-mode)** — `doc2vec controller ./configs/` stays running: it schedules each config on its own cron expression, keeps run history and per-source statistics in Postgres, and serves a web UI with live log streaming, searchable run logs, and chunk inspection. Deploy it once and manage all your sources from the browser.
+
+[![doc2vec controller dashboard](docs/images/controller-dashboard.png)](#controller-mode)
 
 ## Key Features
 
@@ -52,7 +57,7 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
 *   **Incremental Updates:** For GitHub and Zendesk sources, tracks the last run date to only fetch new or updated issues/tickets.
 *   **Cleanup:** Removes obsolete chunks from the database corresponding to pages or files that are no longer found during processing.
 *   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, Zendesk instances, database types, metadata, and other parameters.
-*   **[Controller Mode](#controller-mode):** Optionally runs as a long-lived controller that schedules sync jobs from multiple config files (cron `schedule` field), persists run history and statistics in Postgres, and serves a web UI with live log streaming — read-only (configs from files/ConfigMap) or read-write (create/edit configs from the UI).
+*   **[Controller Mode](#controller-mode):** Optionally runs as a long-lived controller that schedules sync jobs from multiple config files (cron `schedule` field), persists run history and statistics in Postgres, and serves a web UI with live log streaming, whole-run log search, and chunk inspection — read-only (configs from files/ConfigMap) or read-write (create/edit configs from the UI).
 *   **Structured Logging:** Uses a custom logger (`logger.ts`) with levels, timestamps, colors, progress bars, and child loggers for clear execution monitoring.
 
 ## Chunk Metadata & Page Reconstruction
@@ -446,7 +451,19 @@ Besides the one-shot sync, doc2vec can run as a **long-lived controller** that s
 doc2vec controller --database-url postgres://user:pass@host:5432/doc2vec ./configs/
 ```
 
-Open http://localhost:8080 to see the dashboard: configs with their schedule and next run, run history with per-source statistics, and live log streaming for running jobs.
+Open http://localhost:8080 to see the dashboard ([screenshot above](#doc2vec)): every config with its schedule, next run, and the outcome of its last run, plus a **Run now** button.
+
+Each config gets its own page with the full run history, per-source results, and the raw YAML:
+
+![Config run history](docs/images/controller-runs.png)
+
+…and a **Stats** tab with runs per day, success rate, and duration trend over the last 7/30/90 days:
+
+![Config statistics](docs/images/controller-stats.png)
+
+A run page shows what the sync did — duration, exit code, warning and error counts, per-source chunk counts and errors — above the run's logs.
+
+![Run detail with filtered logs](docs/images/controller-run-logs.png)
 
 ### Scheduling
 
@@ -462,6 +479,20 @@ sources:
 
 Configs without a `schedule` are still listed in the UI and can be triggered manually with **Run now**. Each run executes `doc2vec run <config.yaml>` as a child process; overlapping runs of the same config are skipped, and `--max-parallel` (default 1) caps how many sync jobs run at once — keep it low, since website sources launch a headless Chromium.
 
+### Logs
+
+Every line a sync job writes is stored in Postgres and streamed to the run page as it happens (`● live`), with follow-on-scroll and a `↓ Follow` button when you scroll away.
+
+Long runs produce a lot of output, so the browser only keeps the last 10,000 lines in memory — the header tells you when that is the case (`30,000 lines (showing last 10,000)`). Everything else stays one query away:
+
+*   **Level chips** (`ERROR 4`, `WARN 93`, …) count the whole run, not the visible window, and clicking one filters **server-side** — so the 4 errors from the first minute of a 50-minute run are one click away. Results page in with **Load 2,000 more**.
+*   **Keyword filter** matches the message or the module, case-insensitively, across the whole run.
+*   **↓ download log** streams the complete log as plain text (`run-<id>.log`) for grepping locally or attaching to an issue. With a filter active it becomes **↓ download matches** and applies the same filter.
+
+While a run is live, new lines matching the active filter keep arriving in the filtered view.
+
+Run logs are pruned after `--log-retention-days` (default 14); run history and statistics are kept indefinitely.
+
 ### Flags
 
 | Flag | Default | Description |
@@ -533,7 +564,22 @@ The controller detects ConfigMap updates without a restart. Health endpoint: `GE
 
 ### API
 
-The UI is backed by a small REST API you can also use directly: `GET /api/configs`, `POST /api/configs/:id/run`, `GET /api/runs?configId=`, `GET /api/runs/:id/logs`, `POST /api/runs/:id/cancel`, `GET /api/configs/:id/stats?days=30`, plus server-sent events at `GET /api/events` and `GET /api/runs/:id/logs/stream`.
+The UI is backed by a small REST API you can also use directly:
+
+| Endpoint | Description |
+|----------|-------------|
+| `GET /api/configs` | Configs with schedule, next run, and last run |
+| `POST /api/configs/:id/run` | Trigger a run now |
+| `GET /api/configs/:id/stats?days=30` | Run counts, success rate, duration history |
+| `GET /api/configs/:id/chunks?product_name=&url=` | Inspect the stored chunks for a URL |
+| `GET /api/runs?configId=&status=&limit=&before=` | Run history |
+| `GET /api/runs/:id/logs?afterSeq=&limit=&levels=&q=` | Log lines, filtered by level (csv) and/or keyword |
+| `GET /api/runs/:id/logs/counts` | Line totals per level for the whole run |
+| `GET /api/runs/:id/logs/download?levels=&q=` | Whole log (or the matching lines) as plain text |
+| `POST /api/runs/:id/cancel` | Cancel a running job |
+| `GET /api/health` | Status, mode, version |
+
+Server-sent events: `GET /api/events` (run and config updates) and `GET /api/runs/:id/logs/stream?tail=10000` (replays the last `tail` stored lines, then tails live output).
 
 ## Database Options
 
@@ -721,6 +767,13 @@ If you don't specify a config path, it will look for config.yaml in the current
 
 ## Recent Changes
 
+### Searchable Controller Run Logs
+- **Server-side log filtering:** Level chips and the keyword filter now query Postgres instead of the browser's in-memory buffer, so errors and warnings from the start of a long run are reachable (`GET /api/runs/:id/logs?levels=error,warn&q=...`, paged with **Load 2,000 more**)
+- **Whole-run level counts:** New `GET /api/runs/:id/logs/counts` endpoint backs the level chips, which previously only counted the lines still held in the browser
+- **Plain-text download:** `GET /api/runs/:id/logs/download` streams the full log (or just the matching lines) as `run-<id>.log`
+- **Bounded replay:** The log stream accepts `?tail=N` so opening a 30,000-line run replays only the trailing window instead of the entire history
+- **Index:** New `d2v_run_logs (run_id, level, seq)` index keeps level filtering fast on long runs
+
 ### Postgres Markdown Store
 - **New feature:** Optionally store the generated markdown for each crawled website URL in a Postgres table (`url`, `product_name`, `markdown`, `updated_at`)
 - **Top-level configuration:** Configure Postgres connection once at the top level via `connection_string` or individual `host`/`port`/`database`/`user`/`password` fields, with environment variable substitution support

package/dist/content-processor.js

@@ -1841,7 +1841,9 @@ class ContentProcessor {
             if (!content) {
                 continue;
             }
-            const searchableText = contextPrefix + content;
+            // See chunkMarkdown: an unpaired surrogate makes the store's JSON
+            // body invalid, so sanitize before the id is derived from the text.
+            const searchableText = utils_1.Utils.stripLoneSurrogates(contextPrefix + content);
             const chunkId = utils_1.Utils.generateHash(`${url}::${searchableText}`);
             documentChunks.push({
                 content: searchableText,
@@ -1913,7 +1915,10 @@ class ContentProcessor {
             // to searches for parent topics even if the body doesn't mention them.
             const breadcrumbs = hierarchy.filter(h => h).join(" > ");
             const contextPrefix = breadcrumbs ? `[Topic: ${breadcrumbs}]\n` : "";
-            const searchableText = contextPrefix + content.trim();
+            // Sanitize before hashing so the id and the stored text always agree:
+            // an unpaired surrogate (from a split pair, or straight from the
+            // source) makes the JSON body invalid and the write fails.
+            const searchableText = utils_1.Utils.stripLoneSurrogates(contextPrefix + content.trim());
             const chunkId = utils_1.Utils.generateHash(searchableText);
             const chunk = {
                 content: searchableText,
@@ -1954,7 +1959,11 @@ class ContentProcessor {
                 // If the section is a massive guide, split it but keep headers on every sub-piece.
                 const overlapSize = Math.floor(MAX_CHARS * OVERLAP_PERCENT);
                 for (let i = 0; i < trimmedBuffer.length; i += (MAX_CHARS - overlapSize)) {
-                    const subContent = trimmedBuffer.slice(i, i + MAX_CHARS);
+                    // Slice on code points, not code units: a boundary falling
+                    // inside a surrogate pair (emoji, some CJK) would leave a
+                    // lone surrogate, which Qdrant's JSON parser rejects — the
+                    // chunk then fails to store and is silently lost.
+                    const subContent = utils_1.Utils.sliceSafe(trimmedBuffer, i, i + MAX_CHARS);
                     chunks.push(createDocumentChunk(subContent, topicHierarchy));
                 }
             }

package/dist/controller/chunk-inspector.js

@@ -0,0 +1,209 @@
+"use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || (function () {
+    var ownKeys = function(o) {
+        ownKeys = Object.getOwnPropertyNames || function (o) {
+            var ar = [];
+            for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
+            return ar;
+        };
+        return ownKeys(o);
+    };
+    return function (mod) {
+        if (mod && mod.__esModule) return mod;
+        var result = {};
+        if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
+        __setModuleDefault(result, mod);
+        return result;
+    };
+})();
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.lookupChunks = lookupChunks;
+const better_sqlite3_1 = __importDefault(require("better-sqlite3"));
+const path = __importStar(require("path"));
+const sqliteVec = __importStar(require("sqlite-vec"));
+const yaml = __importStar(require("js-yaml"));
+const js_client_rest_1 = require("@qdrant/js-client-rest");
+const types_1 = require("./types");
+const MAX_CHUNKS = 500;
+/**
+ * Read-only lookup of the chunks stored for a URL in a source's vector store.
+ * Resolves the store the same way a sync job would (same defaults as
+ * DatabaseManager.initDatabase, same ${ENV_VAR} substitution as Doc2Vec's
+ * config loader), but never creates tables or collections.
+ */
+async function lookupChunks(configContent, selector, url) {
+    const source = findSource(configContent, selector);
+    const dbConfig = source.database_config ?? {};
+    const params = dbConfig.params ?? {};
+    if (dbConfig.type === 'sqlite') {
+        const dbPath = params.db_path
+            || path.join(process.cwd(), `${String(source.product_name).replace(/\s+/g, '_')}-${source.version}.db`);
+        return {
+            source: { product_name: source.product_name, type: source.type, version: source.version },
+            database: { type: 'sqlite', path: dbPath },
+            chunks: lookupSqlite(dbPath, url),
+        };
+    }
+    if (dbConfig.type === 'qdrant') {
+        const qdrantUrl = params.qdrant_url || 'http://localhost:6333';
+        const qdrantPort = params.qdrant_port || 443;
+        const collectionName = params.collection_name
+            || `${String(source.product_name).toLowerCase().replace(/\s+/g, '_')}_${source.version}`;
+        const client = new js_client_rest_1.QdrantClient({ url: qdrantUrl, apiKey: process.env.QDRANT_API_KEY, port: qdrantPort });
+        return {
+            source: { product_name: source.product_name, type: source.type, version: source.version },
+            database: { type: 'qdrant', url: qdrantUrl, collection: collectionName },
+            chunks: await lookupQdrant(client, collectionName, url),
+        };
+    }
+    throw new types_1.ValidationError(`source has unsupported database type: ${dbConfig.type}`);
+}
+function findSource(configContent, selector) {
+    // Same ${VAR} substitution the sync job applies when it loads the config
+    const substituted = configContent.replace(/\$\{([^}]+)\}/g, (match, varName) => process.env[varName] !== undefined ? process.env[varName] : match);
+    let parsed;
+    try {
+        parsed = yaml.load(substituted);
+    }
+    catch (err) {
+        throw new types_1.ValidationError(`config does not parse: ${err instanceof Error ? err.message : String(err)}`);
+    }
+    const sources = Array.isArray(parsed?.sources) ? parsed.sources : [];
+    const source = sources.find(s => s?.product_name === selector.product_name
+        && (selector.type === undefined || s?.type === selector.type)
+        && (selector.version === undefined || String(effectiveVersion(s)) === selector.version));
+    if (!source) {
+        throw new types_1.NotFoundError(`source '${selector.product_name}' not found in config`);
+    }
+    return { ...source, version: effectiveVersion(source) };
+}
+// Mirrors the version defaulting in Doc2Vec.loadConfig for code sources
+function effectiveVersion(source) {
+    if (source?.version && String(source.version).trim().length > 0)
+        return String(source.version);
+    if (source?.type === 'code') {
+        if (source.branch && String(source.branch).trim().length > 0)
+            return String(source.branch);
+        return 'local';
+    }
+    return String(source?.version ?? '');
+}
+function lookupSqlite(dbPath, url) {
+    let db;
+    try {
+        db = new better_sqlite3_1.default(dbPath, { readonly: true, fileMustExist: true, allowExtension: true });
+    }
+    catch {
+        throw new types_1.NotFoundError(`SQLite database not found at ${dbPath} (has this source ever been synced?)`);
+    }
+    try {
+        sqliteVec.load(db);
+        const hasDates = db
+            .prepare(`SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'vec_chunk_dates'`)
+            .get() !== undefined;
+        const rows = db.prepare(`
+            SELECT v.chunk_id, v.url, v.product_name, v.version, v.section, v.heading_hierarchy,
+                   v.chunk_index, v.total_chunks, v.hash, v.content
+                   ${hasDates ? ', d.created_at AS created_at' : ', NULL AS created_at'}
+            FROM vec_items v
+            ${hasDates ? 'LEFT JOIN vec_chunk_dates d ON d.chunk_id = v.chunk_id' : ''}
+            WHERE v.url = ?
+            LIMIT ${MAX_CHUNKS}
+        `).all(url);
+        return sortChunks(rows.map(row => ({
+            chunk_id: String(row.chunk_id),
+            url: String(row.url),
+            product_name: row.product_name ?? null,
+            version: row.version ?? null,
+            section: row.section ?? null,
+            heading_hierarchy: parseHierarchy(row.heading_hierarchy),
+            chunk_index: row.chunk_index === null || row.chunk_index === undefined ? null : Number(row.chunk_index),
+            total_chunks: row.total_chunks === null || row.total_chunks === undefined ? null : Number(row.total_chunks),
+            hash: row.hash ?? null,
+            created_at: row.created_at ?? null,
+            content: String(row.content ?? ''),
+        })));
+    }
+    finally {
+        db.close();
+    }
+}
+async function lookupQdrant(client, collectionName, url) {
+    const chunks = [];
+    let offset = undefined;
+    do {
+        const response = await client.scroll(collectionName, {
+            limit: 200,
+            offset,
+            with_payload: true,
+            with_vector: false,
+            filter: {
+                must: [{ key: 'url', match: { value: url } }],
+                must_not: [{ key: 'is_metadata', match: { value: true } }],
+            },
+        });
+        for (const point of response.points) {
+            const p = point.payload ?? {};
+            chunks.push({
+                chunk_id: String(p.original_chunk_id ?? point.id),
+                url: String(p.url ?? url),
+                product_name: p.product_name ?? null,
+                version: p.version ?? null,
+                section: p.section ?? null,
+                heading_hierarchy: parseHierarchy(p.heading_hierarchy),
+                chunk_index: p.chunk_index === null || p.chunk_index === undefined ? null : Number(p.chunk_index),
+                total_chunks: p.total_chunks === null || p.total_chunks === undefined ? null : Number(p.total_chunks),
+                hash: p.hash ?? null,
+                created_at: p.created_at ?? null,
+                content: String(p.content ?? ''),
+            });
+            if (chunks.length >= MAX_CHUNKS)
+                return sortChunks(chunks);
+        }
+        offset = response.next_page_offset;
+    } while (offset !== null && offset !== undefined);
+    return sortChunks(chunks);
+}
+function parseHierarchy(value) {
+    if (Array.isArray(value))
+        return value.map(String);
+    if (typeof value === 'string' && value.trim().startsWith('[')) {
+        try {
+            const parsed = JSON.parse(value);
+            if (Array.isArray(parsed))
+                return parsed.map(String);
+        }
+        catch { /* fall through */ }
+    }
+    return value ? [String(value)] : [];
+}
+// Default order: position within the page. The UI offers per-column sorting
+// (including by creation date) on top of this.
+function sortChunks(chunks) {
+    return chunks.sort((a, b) => {
+        const ai = a.chunk_index ?? Number.MAX_SAFE_INTEGER;
+        const bi = b.chunk_index ?? Number.MAX_SAFE_INTEGER;
+        if (ai !== bi)
+            return ai - bi;
+        return a.chunk_id.localeCompare(b.chunk_id);
+    });
+}

package/dist/controller/migrations.js

@@ -52,4 +52,9 @@ exports.MIGRATIONS = [
         PRIMARY KEY (run_id, seq)
     );
     `,
+    // 2: index for server-side log filtering by level (error/warn lookups on
+    // long runs, where the client only keeps a trailing window in memory)
+    `
+    CREATE INDEX IF NOT EXISTS d2v_run_logs_level_idx ON d2v_run_logs (run_id, level, seq);
+    `,
 ];

package/dist/controller/server.js

@@ -40,9 +40,20 @@ exports.createServer = createServer;
 const express_1 = __importDefault(require("express"));
 const fs = __importStar(require("fs"));
 const path = __importStar(require("path"));
+const chunk_inspector_1 = require("./chunk-inspector");
 const config_registry_1 = require("./config-registry");
 const types_1 = require("./types");
 const SSE_HEARTBEAT_MS = 15000;
+const LOG_PAGE_SIZE = 5000;
+const LOG_LEVELS = new Set(['error', 'warn', 'info', 'debug']);
+/** Shared parsing for the `levels` (csv) and `q` log query parameters. */
+function parseLogFilter(req) {
+    const levels = typeof req.query.levels === 'string'
+        ? req.query.levels.split(',').map(l => l.trim().toLowerCase()).filter(l => LOG_LEVELS.has(l))
+        : [];
+    const keyword = typeof req.query.q === 'string' ? req.query.q.trim() : '';
+    return { levels, keyword: keyword || undefined };
+}
 function getVersion() {
     try {
         return require('../../package.json').version;
@@ -151,6 +162,34 @@ function createServer(deps) {
         const days = Number(req.query.days) || 30;
         res.json(await store.getConfigStats(id, days));
     });
+    // Inspect the chunks currently stored for a URL in one source's vector store.
+    // This reads the live store, so results reflect the present state, not a
+    // historical run.
+    app.get('/api/configs/:id/chunks', async (req, res) => {
+        const config = registry.get(parseId(String(req.params.id)));
+        if (!config)
+            throw new types_1.NotFoundError('config not found');
+        const url = typeof req.query.url === 'string' ? req.query.url.trim() : '';
+        const productName = typeof req.query.product_name === 'string' ? req.query.product_name : '';
+        if (!url)
+            throw new types_1.ValidationError('url query parameter is required');
+        if (!productName)
+            throw new types_1.ValidationError('product_name query parameter is required');
+        try {
+            res.json(await (0, chunk_inspector_1.lookupChunks)(config.content, {
+                product_name: productName,
+                type: typeof req.query.type === 'string' && req.query.type ? req.query.type : undefined,
+                version: typeof req.query.version === 'string' && req.query.version ? req.query.version : undefined,
+            }, url));
+        }
+        catch (err) {
+            if (err instanceof types_1.NotFoundError || err instanceof types_1.ValidationError)
+                throw err;
+            // Vector store unreachable (e.g. Qdrant down) — surface the reason
+            logger.error('Chunk lookup failed:', err);
+            res.status(502).json({ error: `chunk lookup failed: ${err instanceof Error ? err.message : String(err)}` });
+        }
+    });
     // ------------------------------------------------------------------ runs
     app.get('/api/runs', async (req, res) => {
         res.json(await store.listRuns({
@@ -173,7 +212,39 @@ function createServer(deps) {
             throw new types_1.NotFoundError('run not found');
         const afterSeq = Number(req.query.afterSeq) || 0;
         const limit = Number(req.query.limit) || 1000;
-        res.json(await store.getLogs(id, afterSeq, limit));
+        res.json(await store.getLogs(id, afterSeq, limit, parseLogFilter(req)));
+    });
+    /** Whole-run line totals per level — the UI can only hold a trailing window. */
+    app.get('/api/runs/:id/logs/counts', async (req, res) => {
+        const id = parseId(String(req.params.id));
+        if (!await store.getRun(id))
+            throw new types_1.NotFoundError('run not found');
+        res.json(await store.countLogsByLevel(id));
+    });
+    // Full log as a plain-text download: the escape hatch when the viewer's
+    // in-memory window or filters aren't enough (grep it locally, attach it).
+    app.get('/api/runs/:id/logs/download', async (req, res) => {
+        const id = parseId(String(req.params.id));
+        if (!await store.getRun(id))
+            throw new types_1.NotFoundError('run not found');
+        const filter = parseLogFilter(req);
+        res.setHeader('Content-Type', 'text/plain; charset=utf-8');
+        res.setHeader('Content-Disposition', `attachment; filename="run-${id}.log"`);
+        let afterSeq = 0;
+        let batch;
+        do {
+            batch = await store.getLogs(id, afterSeq, LOG_PAGE_SIZE, filter);
+            for (const row of batch) {
+                const module = row.module ? ` [${row.module}]` : '';
+                // pg hands back TIMESTAMPTZ as a Date; JSON responses serialise
+                // it for us, a text body has to do it explicitly
+                const ts = new Date(row.ts).toISOString();
+                res.write(`${ts} ${row.level.toUpperCase().padEnd(5)}${module} ${row.message}\n`);
+            }
+            if (batch.length > 0)
+                afterSeq = batch[batch.length - 1].seq;
+        } while (batch.length === LOG_PAGE_SIZE);
+        res.end();
     });
     app.post('/api/runs/:id/cancel', async (req, res) => {
         res.json(await runner.cancel(parseId(String(req.params.id))));
@@ -198,7 +269,12 @@ function createServer(deps) {
         if (!run)
             throw new types_1.NotFoundError('run not found');
         sseInit(res);
+        // `tail` keeps the replay to the trailing window the viewer can hold —
+        // older lines stay reachable through the filter/download endpoints.
+        const tail = Number(req.query.tail) || 0;
         let lastSeq = Number(req.query.afterSeq) || 0;
+        if (!lastSeq && tail > 0)
+            lastSeq = await store.getTailStartSeq(id, tail);
         let replayDone = false;
         const pending = [];
         const sendRow = (row) => {
@@ -230,9 +306,9 @@ function createServer(deps) {
         try {
             let batch;
             do {
-                batch = await store.getLogs(id, lastSeq, 5000);
+                batch = await store.getLogs(id, lastSeq, LOG_PAGE_SIZE);
                 batch.forEach(sendRow);
-            } while (batch.length === 5000);
+            } while (batch.length === LOG_PAGE_SIZE);
         }
         finally {
             replayDone = true;

package/dist/controller/store.js

@@ -3,6 +3,10 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.ControllerStore = void 0;
 const pg_1 = require("pg");
 const migrations_1 = require("./migrations");
+/** Neutralise LIKE wildcards so a keyword search stays a literal substring match. */
+function escapeLike(value) {
+    return value.replace(/[\\%_]/g, ch => `\\${ch}`);
+}
 /**
  * All Postgres access for controller mode: configs, runs, and run logs.
  */
@@ -143,11 +147,44 @@ class ControllerStore {
         await this.pool.query(`INSERT INTO d2v_run_logs (run_id, seq, ts, level, module, message) VALUES ${placeholders.join(', ')}
              ON CONFLICT DO NOTHING`, values);
     }
-    async getLogs(runId, afterSeq = 0, limit = 1000) {
+    /**
+     * Page through a run's logs. Filtering happens here rather than in the
+     * browser so that a search covers the whole run: the UI only holds a
+     * trailing window of a long run's output in memory.
+     */
+    async getLogs(runId, afterSeq = 0, limit = 1000, filter = {}) {
+        const values = [runId, afterSeq];
+        let where = 'run_id = $1 AND seq > $2';
+        if (filter.levels?.length) {
+            values.push(filter.levels);
+            where += ` AND level = ANY($${values.length})`;
+        }
+        if (filter.keyword) {
+            values.push(`%${escapeLike(filter.keyword)}%`);
+            where += ` AND (message ILIKE $${values.length} ESCAPE '\\' OR module ILIKE $${values.length} ESCAPE '\\')`;
+        }
+        values.push(Math.min(limit, 5000));
         const { rows } = await this.pool.query(`SELECT seq, ts, level, module, message FROM d2v_run_logs
-             WHERE run_id = $1 AND seq > $2 ORDER BY seq LIMIT $3`, [runId, afterSeq, Math.min(limit, 5000)]);
+             WHERE ${where} ORDER BY seq LIMIT $${values.length}`, values);
         return rows.map((r) => ({ ...r, seq: Number(r.seq) }));
     }
+    /**
+     * Seq to replay from so that only the last `tail` lines follow, i.e. the
+     * seq of the (tail + 1)-th newest line — 0 when the run is shorter.
+     */
+    async getTailStartSeq(runId, tail) {
+        const { rows } = await this.pool.query(`SELECT seq FROM d2v_run_logs WHERE run_id = $1 ORDER BY seq DESC OFFSET $2 LIMIT 1`, [runId, Math.max(0, tail)]);
+        return rows.length > 0 ? Number(rows[0].seq) : 0;
+    }
+    /** Per-level line totals for a whole run, for the log viewer's filter chips. */
+    async countLogsByLevel(runId) {
+        const { rows } = await this.pool.query(`SELECT level, COUNT(*)::bigint AS count FROM d2v_run_logs
+             WHERE run_id = $1 GROUP BY level`, [runId]);
+        const counts = {};
+        for (const row of rows)
+            counts[row.level] = Number(row.count);
+        return counts;
+    }
     async pruneOldLogs(retentionDays) {
         const { rowCount } = await this.pool.query(`DELETE FROM d2v_run_logs USING d2v_runs
              WHERE d2v_run_logs.run_id = d2v_runs.id