doc2vec 2.10.6 → 2.11.0

package/README.md

@@ -52,6 +52,7 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
 *   **Incremental Updates:** For GitHub and Zendesk sources, tracks the last run date to only fetch new or updated issues/tickets.
 *   **Cleanup:** Removes obsolete chunks from the database corresponding to pages or files that are no longer found during processing.
 *   **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, Zendesk instances, database types, metadata, and other parameters.
+*   **[Controller Mode](#controller-mode):** Optionally runs as a long-lived controller that schedules sync jobs from multiple config files (cron `schedule` field), persists run history and statistics in Postgres, and serves a web UI with live log streaming — read-only (configs from files/ConfigMap) or read-write (create/edit configs from the UI).
 *   **Structured Logging:** Uses a custom logger (`logger.ts`) with levels, timestamps, colors, progress bars, and child loggers for clear execution monitoring.
 
 ## Chunk Metadata & Page Reconstruction
@@ -437,6 +438,103 @@ The script will then:
 7.  Cleanup obsolete chunks.
 8.  Output detailed logs.
 
+## Controller Mode
+
+Besides the one-shot sync, doc2vec can run as a **long-lived controller** that schedules sync jobs, records run history and statistics in Postgres, and serves a web UI for monitoring and managing configs:
+
+```bash
+doc2vec controller --database-url postgres://user:pass@host:5432/doc2vec ./configs/
+```
+
+Open http://localhost:8080 to see the dashboard: configs with their schedule and next run, run history with per-source statistics, and live log streaming for running jobs.
+
+### Scheduling
+
+Add a top-level cron `schedule` (and optionally a display `name`) to a config file, and the controller runs it automatically:
+
+```yaml
+name: product-docs
+schedule: "0 2 * * *"   # every day at 02:00
+sources:
+  - type: website
+    ...
+```
+
+Configs without a `schedule` are still listed in the UI and can be triggered manually with **Run now**. Each run executes `doc2vec run <config.yaml>` as a child process; overlapping runs of the same config are skipped, and `--max-parallel` (default 1) caps how many sync jobs run at once — keep it low, since website sources launch a headless Chromium.
+
+### Flags
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--database-url <url>` | `DATABASE_URL` env | Postgres connection string (required) |
+| `--port <n>` | `8080` (or `PORT`) | HTTP port for the API and UI |
+| `--read-write` | off (read-only) | Allow creating/editing/deleting configs from the UI |
+| `--config-dir <dir>` | — | Where UI-created configs are written (required with `--read-write`) |
+| `--max-parallel <n>` | `1` | Max concurrent sync jobs |
+| `--reload-interval <s>` | `30` | How often config files are re-read for changes |
+| `--log-retention-days <n>` | `14` | Run logs older than this are pruned (run history is kept) |
+| `--slack-webhook-url <url>` | `SLACK_WEBHOOK_URL` env | Slack incoming webhook — post a message when a run finishes |
+| `--slack-notify <mode>` | `all` | `all` or `failures` (failures also covers canceled runs) |
+| `--public-url <url>` | `PUBLIC_URL` env | Externally reachable base URL, used for "view run" links in notifications |
+
+Positional arguments are config files and/or directories (directories are scanned for `*.yaml`/`*.yml`, and re-scanned on every reload).
+
+### Slack notifications
+
+With `--slack-webhook-url` (or `SLACK_WEBHOOK_URL`) set, the controller posts to a [Slack incoming webhook](https://api.slack.com/messaging/webhooks) whenever a run reaches a terminal state — ✅ succeeded, ❌ failed (naming the failing sources and their errors), or ⚠️ canceled. Overlap-`skipped` runs are not notified. Set `--slack-notify failures` to silence successes, and `--public-url https://doc2vec.example.com` to get clickable "view run" links.
+
+### Read-only vs read-write
+
+- **Read-only** (default): configs come solely from the files passed on the command line — ideal for Kubernetes, where they live in a ConfigMap. The UI can view configs, trigger runs, and browse history/stats, but not modify anything. ConfigMap updates are picked up automatically via content-hash polling.
+- **Read-write** (`--read-write --config-dir ./configs`): the UI can also create, edit, and delete config YAML files. Files stay the source of truth on disk; concurrent edits are protected by a content-hash check.
+
+In both modes the UI always shows raw YAML: `${ENV_VAR}` secret placeholders are **never** resolved outside the sync job itself.
+
+### Kubernetes example
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: doc2vec-controller
+spec:
+  replicas: 1        # keep a single replica: the scheduler is not distributed
+  selector:
+    matchLabels: { app: doc2vec-controller }
+  template:
+    metadata:
+      labels: { app: doc2vec-controller }
+    spec:
+      containers:
+        - name: controller
+          image: ghcr.io/kagent-dev/doc2vec:latest
+          command: ["node", "dist/doc2vec.js", "controller", "/etc/doc2vec"]
+          ports:
+            - containerPort: 8080
+          env:
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef: { name: doc2vec-secrets, key: database-url }
+            - name: OPENAI_API_KEY
+              valueFrom:
+                secretKeyRef: { name: doc2vec-secrets, key: openai-api-key }
+          volumeMounts:
+            - name: configs
+              mountPath: /etc/doc2vec
+          resources:
+            requests: { memory: 1Gi }
+            limits: { memory: 4Gi }   # website sources launch headless Chromium
+      volumes:
+        - name: configs
+          configMap: { name: doc2vec-configs }
+```
+
+The controller detects ConfigMap updates without a restart. Health endpoint: `GET /api/health`.
+
+### API
+
+The UI is backed by a small REST API you can also use directly: `GET /api/configs`, `POST /api/configs/:id/run`, `GET /api/runs?configId=`, `GET /api/runs/:id/logs`, `POST /api/runs/:id/cancel`, `GET /api/configs/:id/stats?days=30`, plus server-sent events at `GET /api/events` and `GET /api/runs/:id/logs/stream`.
+
 ## Database Options
 
 ### SQLite (`database_config.type: 'sqlite'`)

package/dist/content-processor.js

@@ -36,7 +36,8 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
     return (mod && mod.__esModule) ? mod : { "default": mod };
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.ContentProcessor = void 0;
+exports.ContentProcessor = exports.BrowserLaunchError = void 0;
+exports.resolveBrowserExecutablePath = resolveBrowserExecutablePath;
 const readability_1 = require("@mozilla/readability");
 const axios_1 = __importDefault(require("axios"));
 const cheerio_1 = require("cheerio");
@@ -48,6 +49,97 @@ const fs = __importStar(require("fs"));
 const path = __importStar(require("path"));
 const utils_1 = require("./utils");
 const code_chunker_1 = require("./code-chunker");
+/**
+ * Thrown when the browser cannot be launched at all (even after a retry).
+ * Without a browser no page in the crawl can be processed, so this aborts the
+ * whole website source — the run is then reported as failed instead of
+ * "succeeding" with an error on every URL.
+ */
+class BrowserLaunchError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = 'BrowserLaunchError';
+    }
+}
+exports.BrowserLaunchError = BrowserLaunchError;
+/**
+ * Resolve which browser binary to launch, in order of preference:
+ *  1. PUPPETEER_EXECUTABLE_PATH (explicit override)
+ *  2. Puppeteer's own downloaded Chrome for Testing (version-matched, the
+ *     reliable choice — Debian's chromium package has shipped builds that
+ *     crash at startup in containers, e.g. 150.0.7871 SIGTRAPs immediately)
+ *  3. A system chromium, as the fallback (also covers linux/arm64, where
+ *     Chrome for Testing is not published)
+ *  4. undefined — let puppeteer decide
+ */
+function resolveBrowserExecutablePath() {
+    const override = process.env.PUPPETEER_EXECUTABLE_PATH;
+    if (override)
+        return override;
+    try {
+        const bundled = puppeteer_1.default.executablePath();
+        if (bundled && fs.existsSync(bundled))
+            return bundled;
+    }
+    catch {
+        // no browser downloaded for this platform
+    }
+    if (fs.existsSync('/usr/bin/chromium'))
+        return '/usr/bin/chromium';
+    if (fs.existsSync('/usr/bin/chromium-browser'))
+        return '/usr/bin/chromium-browser';
+    return undefined;
+}
+/**
+ * Best-effort snapshot of process + container memory, for diagnosing browser
+ * launch failures (the usual cause in Kubernetes is the pod's memory cgroup —
+ * Chromium forks, then its renderers are immediately OOM-killed).
+ */
+function memoryDiagnostics() {
+    const gib = (bytes) => `${(bytes / 1024 ** 3).toFixed(2)}GiB`;
+    const parts = [`node rss ${gib(process.memoryUsage().rss)}`];
+    try {
+        // cgroup v2, else v1 — absent entirely outside Linux/containers
+        let current;
+        let max;
+        if (fs.existsSync('/sys/fs/cgroup/memory.current')) {
+            current = fs.readFileSync('/sys/fs/cgroup/memory.current', 'utf8').trim();
+            max = fs.readFileSync('/sys/fs/cgroup/memory.max', 'utf8').trim();
+        }
+        else if (fs.existsSync('/sys/fs/cgroup/memory/memory.usage_in_bytes')) {
+            current = fs.readFileSync('/sys/fs/cgroup/memory/memory.usage_in_bytes', 'utf8').trim();
+            max = fs.readFileSync('/sys/fs/cgroup/memory/memory.limit_in_bytes', 'utf8').trim();
+        }
+        if (current !== undefined && max !== undefined) {
+            const unlimited = max === 'max' || Number(max) > 2 ** 60;
+            parts.push(`cgroup ${gib(Number(current))} used / ${unlimited ? 'no limit' : gib(Number(max))}`);
+        }
+    }
+    catch {
+        // diagnostics only — never fail because of them
+    }
+    try {
+        // /proc/meminfo reflects the NODE in Kubernetes — catches host-level pressure
+        const meminfo = fs.readFileSync('/proc/meminfo', 'utf8');
+        const available = meminfo.match(/^MemAvailable:\s+(\d+) kB/m);
+        if (available) {
+            parts.push(`node available ${gib(Number(available[1]) * 1024)}`);
+        }
+    }
+    catch {
+        // not Linux
+    }
+    try {
+        // Chromium writes its profile and (with --disable-dev-shm-usage) its
+        // shared memory under /tmp — a full disk breaks launches
+        const stat = fs.statfsSync('/tmp');
+        parts.push(`/tmp free ${gib(stat.bavail * stat.bsize)}`);
+    }
+    catch {
+        // statfs unavailable
+    }
+    return parts.join(', ');
+}
 class ContentProcessor {
     constructor(logger) {
         this.logger = logger;
@@ -329,16 +421,8 @@ class ContentProcessor {
         let browser = null;
         let page = null;
         const launchBrowser = async () => {
-            let executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
-            if (!executablePath) {
-                if (fs.existsSync('/usr/bin/chromium')) {
-                    executablePath = '/usr/bin/chromium';
-                }
-                else if (fs.existsSync('/usr/bin/chromium-browser')) {
-                    executablePath = '/usr/bin/chromium-browser';
-                }
-            }
-            const b = await puppeteer_1.default.launch({
+            const executablePath = resolveBrowserExecutablePath();
+            const launchOptions = {
                 executablePath,
                 args: [
                     '--no-sandbox',
@@ -348,7 +432,25 @@ class ContentProcessor {
                     '--disable-extensions',
                 ],
                 protocolTimeout: 60000,
-            });
+            };
+            let b;
+            try {
+                b = await puppeteer_1.default.launch(launchOptions);
+            }
+            catch (firstError) {
+                // One retry after a pause — the failure may be transient (e.g.
+                // memory pressure from a page that just closed). If the browser
+                // cannot launch at all, no page in this crawl can be processed,
+                // so escalate as fatal instead of failing every URL one by one.
+                logger.warn(`Browser launch failed (${memoryDiagnostics()}), retrying once in 5s...`, firstError);
+                await new Promise(resolve => setTimeout(resolve, 5000));
+                try {
+                    b = await puppeteer_1.default.launch(launchOptions);
+                }
+                catch (secondError) {
+                    throw new BrowserLaunchError(`browser failed to launch twice (${memoryDiagnostics()}): ${secondError instanceof Error ? secondError.message : String(secondError)}`);
+                }
+            }
             const p = await b.newPage();
             return { browser: b, page: p };
         };
@@ -682,6 +784,13 @@ class ContentProcessor {
                     }
                 }
                 catch (error) {
+                    // A browser that cannot launch is fatal for the whole crawl —
+                    // rethrow so the source (and the run) is reported as failed
+                    // rather than logging an error per URL and finishing "green".
+                    if (error instanceof BrowserLaunchError) {
+                        logger.error(`Aborting crawl of ${baseUrl}: ${error.message}`);
+                        throw error;
+                    }
                     const status = this.getHttpStatus(error);
                     // ── Handle 429 (Too Many Requests) with retry ──
                     if (status === 429) {
@@ -846,17 +955,8 @@ class ContentProcessor {
             }
             else {
                 // Standalone mode: launch a browser for this single page
-                let executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
-                if (!executablePath) {
-                    if (fs.existsSync('/usr/bin/chromium')) {
-                        executablePath = '/usr/bin/chromium';
-                    }
-                    else if (fs.existsSync('/usr/bin/chromium-browser')) {
-                        executablePath = '/usr/bin/chromium-browser';
-                    }
-                }
                 browser = await puppeteer_1.default.launch({
-                    executablePath,
+                    executablePath: resolveBrowserExecutablePath(),
                     args: [
                         '--no-sandbox',
                         '--disable-setuid-sandbox',

package/dist/doc2vec.js

@@ -176,36 +176,59 @@ class Doc2Vec {
             await this.markdownStore.init();
         }
         this.logger.section('PROCESSING SOURCES');
+        const runStats = [];
         for (const sourceConfig of this.config.sources) {
             const sourceLogger = this.logger.child(`source:${sourceConfig.product_name}`);
             sourceLogger.info(`Processing ${sourceConfig.type} source for ${sourceConfig.product_name}@${sourceConfig.version}`);
-            if (sourceConfig.type === 'github') {
-                await this.processGithubRepo(sourceConfig, sourceLogger);
-            }
-            else if (sourceConfig.type === 'website') {
-                await this.processWebsite(sourceConfig, sourceLogger);
-            }
-            else if (sourceConfig.type === 'local_directory') {
-                await this.processLocalDirectory(sourceConfig, sourceLogger);
-            }
-            else if (sourceConfig.type === 'code') {
-                await this.processCodeSource(sourceConfig, sourceLogger);
-            }
-            else if (sourceConfig.type === 'zendesk') {
-                await this.processZendesk(sourceConfig, sourceLogger);
-            }
-            else if (sourceConfig.type === 's3') {
-                await this.processS3(sourceConfig, sourceLogger);
-            }
-            else {
-                sourceLogger.error(`Unknown source type: ${sourceConfig.type}`);
+            const startTime = Date.now();
+            let ok = true;
+            let errorMessage;
+            try {
+                if (sourceConfig.type === 'github') {
+                    await this.processGithubRepo(sourceConfig, sourceLogger);
+                }
+                else if (sourceConfig.type === 'website') {
+                    await this.processWebsite(sourceConfig, sourceLogger);
+                }
+                else if (sourceConfig.type === 'local_directory') {
+                    await this.processLocalDirectory(sourceConfig, sourceLogger);
+                }
+                else if (sourceConfig.type === 'code') {
+                    await this.processCodeSource(sourceConfig, sourceLogger);
+                }
+                else if (sourceConfig.type === 'zendesk') {
+                    await this.processZendesk(sourceConfig, sourceLogger);
+                }
+                else if (sourceConfig.type === 's3') {
+                    await this.processS3(sourceConfig, sourceLogger);
+                }
+                else {
+                    ok = false;
+                    errorMessage = `Unknown source type: ${sourceConfig.type}`;
+                    sourceLogger.error(errorMessage);
+                }
             }
+            catch (error) {
+                ok = false;
+                errorMessage = error instanceof Error ? error.message : String(error);
+                sourceLogger.error(`Failed to process ${sourceConfig.type} source for ${sourceConfig.product_name}:`, error);
+            }
+            runStats.push({
+                product_name: sourceConfig.product_name,
+                type: sourceConfig.type,
+                version: sourceConfig.version,
+                duration_ms: Date.now() - startTime,
+                ok,
+                ...(errorMessage && { error: errorMessage })
+            });
         }
         // Close the Postgres markdown store connection pool
         if (this.markdownStore) {
             await this.markdownStore.close();
         }
         this.logger.section('PROCESSING COMPLETE');
+        this.logger.event('run-summary', { sources: runStats });
+        return runStats;
     }
     async fetchAndProcessGitHubIssues(repo, sourceConfig, dbConnection, logger) {
         const [owner, repoName] = repo.split('/');
@@ -608,7 +631,11 @@ class Doc2Vec {
         this.brokenLinksByWebsite[baseUrl] = unique;
     }
     writeBrokenLinksReport() {
-        const reportPath = path.join(this.configDir, '404.yaml');
+        // The default (next to the config file) breaks when the config lives on
+        // a read-only mount (e.g. a Kubernetes ConfigMap in controller mode) —
+        // DOC2VEC_REPORT_DIR points the report somewhere writable instead.
+        const reportDir = process.env.DOC2VEC_REPORT_DIR || this.configDir;
+        const reportPath = path.join(reportDir, '404.yaml');
         const orderedEntries = Object.entries(this.brokenLinksByWebsite)
             .sort(([a], [b]) => a.localeCompare(b));
         const reportPayload = orderedEntries.map(([website, links]) => ({
@@ -1651,14 +1678,64 @@ exports.Doc2Vec = Doc2Vec;
 Doc2Vec.MAX_EMBEDDING_TOKENS = 8191;
 Doc2Vec.CHARS_PER_TOKEN = 4;
 Doc2Vec.MAX_EMBEDDING_CHARS = Doc2Vec.MAX_EMBEDDING_TOKENS * Doc2Vec.CHARS_PER_TOKEN;
-if (require.main === module) {
-    const configPath = process.argv[2] || 'config.yaml';
+function runOneShot(configPath) {
     if (!fs.existsSync(configPath)) {
         console.error('Please provide a valid path to a YAML config file.');
         process.exit(1);
     }
     const doc2Vec = new Doc2Vec(configPath);
     doc2Vec.run()
-        .then(() => process.exit(0))
+        .then((stats) => process.exit(stats.some(s => !s.ok) ? 1 : 0))
         .catch((err) => { console.error(err); process.exit(1); });
 }
+if (require.main === module) {
+    const { Command } = require('commander');
+    const program = new Command();
+    program
+        .name('doc2vec')
+        .description('Crawl documentation sources and store embeddings in vector databases')
+        // Legacy invocation: `doc2vec [config.yaml]` runs a one-shot sync
+        .argument('[config]', 'path to a YAML config file', 'config.yaml')
+        .action((configPath) => runOneShot(configPath));
+    program
+        .command('run <config>')
+        .description('Run a one-shot sync for a config file (what the controller spawns)')
+        .action((configPath) => runOneShot(configPath));
+    program
+        .command('controller [configs...]')
+        .description('Run as a long-lived controller: schedule sync jobs per config file, persist runs in Postgres, serve a web UI')
+        .option('--database-url <url>', 'Postgres connection string (or DATABASE_URL env var)')
+        .option('--port <port>', 'HTTP port for the API/UI (or PORT env var)', (v) => parseInt(v, 10))
+        .option('--read-write', 'allow creating/editing configs from the UI (default: read-only)', false)
+        .option('--config-dir <dir>', 'directory where UI-created configs are written (required with --read-write)')
+        .option('--max-parallel <n>', 'maximum number of sync jobs running at once', (v) => parseInt(v, 10), 1)
+        .option('--reload-interval <seconds>', 'how often to re-poll config files for changes', (v) => parseInt(v, 10), 30)
+        .option('--log-retention-days <days>', 'delete run logs older than this many days', (v) => parseInt(v, 10), 14)
+        .option('--slack-webhook-url <url>', 'Slack incoming webhook for run notifications (or SLACK_WEBHOOK_URL env var)')
+        .option('--slack-notify <mode>', "which runs to notify about: 'all' or 'failures'", 'all')
+        .option('--public-url <url>', 'externally reachable base URL, used for links in notifications (or PUBLIC_URL env var)')
+        .action(async (configs, options) => {
+        // Lazy-require so the one-shot path doesn't load express/pg
+        const { startController } = require('./controller');
+        try {
+            await startController({
+                configArgs: configs,
+                databaseUrl: options.databaseUrl || process.env.DATABASE_URL,
+                port: options.port || (process.env.PORT ? parseInt(process.env.PORT, 10) : 8080),
+                readWrite: options.readWrite,
+                configDir: options.configDir,
+                maxParallel: options.maxParallel,
+                reloadIntervalSec: options.reloadInterval,
+                logRetentionDays: options.logRetentionDays,
+                slackWebhookUrl: options.slackWebhookUrl || process.env.SLACK_WEBHOOK_URL,
+                slackNotify: options.slackNotify === 'failures' ? 'failures' : 'all',
+                publicUrl: options.publicUrl || process.env.PUBLIC_URL,
+            });
+        }
+        catch (err) {
+            console.error(err instanceof Error ? err.message : err);
+            process.exit(1);
+        }
+    });
+    program.parse();
+}

package/dist/logger.js

@@ -16,6 +16,35 @@ var LogLevel;
     LogLevel[LogLevel["ERROR"] = 3] = "ERROR";
     LogLevel[LogLevel["NONE"] = 100] = "NONE";
 })(LogLevel || (exports.LogLevel = LogLevel = {}));
+/**
+ * When DOC2VEC_STRUCTURED_LOGS=1 (set by the controller when spawning sync jobs),
+ * every log line is emitted as a single JSON object so a parent process can parse
+ * the stream reliably. Human-formatted output is suppressed in this mode.
+ */
+const STRUCTURED = process.env.DOC2VEC_STRUCTURED_LOGS === '1';
+function emitStructured(payload, useStderr = false) {
+    const line = JSON.stringify(payload);
+    if (useStderr) {
+        process.stderr.write(line + '\n');
+    }
+    else {
+        process.stdout.write(line + '\n');
+    }
+}
+function stringifyArg(arg) {
+    if (arg instanceof Error) {
+        return `${arg.message}${arg.stack ? `\n${arg.stack}` : ''}`;
+    }
+    if (typeof arg === 'object' && arg !== null) {
+        try {
+            return JSON.stringify(arg);
+        }
+        catch (e) {
+            return '[Unserializable Object]';
+        }
+    }
+    return String(arg);
+}
 /**
  * Basic color functions that don't rely on external packages
  */
@@ -117,8 +146,23 @@ class Logger {
      * @param message Message to log
      * @param args Additional arguments
      */
+    emitJson(level, message, args, useStderr = false) {
+        const msg = args.length > 0
+            ? `${message} ${args.map(stringifyArg).join(' ')}`
+            : message;
+        emitStructured({
+            ts: new Date().toISOString(),
+            level,
+            module: this.moduleName,
+            msg
+        }, useStderr);
+    }
     debug(message, ...args) {
         if (this.config.level <= LogLevel.DEBUG) {
+            if (STRUCTURED) {
+                this.emitJson('debug', message, args);
+                return;
+            }
             const formattedMessage = this.formatMessage('DEBUG', message, args);
             console.log(this.colorize(LogLevel.DEBUG, formattedMessage));
         }
@@ -131,6 +175,10 @@ class Logger {
      */
     info(message, ...args) {
         if (this.config.level <= LogLevel.INFO) {
+            if (STRUCTURED) {
+                this.emitJson('info', message, args);
+                return;
+            }
             const formattedMessage = this.formatMessage('INFO', message, args);
             console.log(this.colorize(LogLevel.INFO, formattedMessage));
         }
@@ -143,6 +191,10 @@ class Logger {
      */
     warn(message, ...args) {
         if (this.config.level <= LogLevel.WARN) {
+            if (STRUCTURED) {
+                this.emitJson('warn', message, args, true);
+                return;
+            }
             const formattedMessage = this.formatMessage('WARN', message, args);
             console.warn(this.colorize(LogLevel.WARN, formattedMessage));
         }
@@ -155,10 +207,28 @@ class Logger {
      */
     error(message, ...args) {
         if (this.config.level <= LogLevel.ERROR) {
+            if (STRUCTURED) {
+                this.emitJson('error', message, args, true);
+                return;
+            }
             const formattedMessage = this.formatMessage('ERROR', message, args);
             console.error(this.colorize(LogLevel.ERROR, formattedMessage));
         }
     }
+    /**
+     * Emit a structured event (e.g. run-summary). In structured mode this prints a
+     * machine-parseable JSON line; otherwise it logs the payload at info level.
+     *
+     * @param name Event name
+     * @param data Event payload
+     */
+    event(name, data = {}) {
+        if (STRUCTURED) {
+            emitStructured({ event: name, ts: new Date().toISOString(), ...data });
+            return;
+        }
+        this.info(`[event:${name}]`, data);
+    }
     /**
      * Create a child logger with a more specific module name
      *
@@ -176,6 +246,10 @@ class Logger {
      */
     section(title) {
         if (this.config.level <= LogLevel.INFO) {
+            if (STRUCTURED) {
+                emitStructured({ event: 'section', ts: new Date().toISOString(), module: this.moduleName, title });
+                return this;
+            }
             const separator = '='.repeat(Math.max(80 - title.length - 4, 10));
             const message = `${separator} ${title} ${separator}`;
             console.log(this.colorize(LogLevel.INFO, message));
@@ -192,10 +266,21 @@ class Logger {
     progress(title, total) {
         let current = 0;
         const startTime = Date.now();
+        // In structured mode, throttle progress lines to at most one per second so a
+        // large crawl doesn't flood the parent process / database with log rows.
+        let lastStructuredEmit = 0;
         const update = (increment = 1, message) => {
             if (this.config.level > LogLevel.INFO)
                 return;
             current += increment;
+            if (STRUCTURED) {
+                const now = Date.now();
+                if (now - lastStructuredEmit >= 1000 || current === total) {
+                    lastStructuredEmit = now;
+                    emitStructured({ event: 'progress', ts: new Date().toISOString(), module: this.moduleName, title, current, total, message });
+                }
+                return;
+            }
             const percentage = Math.min(Math.floor((current / total) * 100), 100);
             const elapsed = (Date.now() - startTime) / 1000;
             let rate = current / elapsed;
@@ -213,6 +298,10 @@ class Logger {
                 return;
             const elapsed = (Date.now() - startTime) / 1000;
             const rate = total / elapsed;
+            if (STRUCTURED) {
+                emitStructured({ event: 'progress', ts: new Date().toISOString(), module: this.moduleName, title, current: total, total, message: `${message} in ${elapsed.toFixed(2)}s`, done: true });
+                return;
+            }
             console.log(this.colorize(LogLevel.INFO, this.formatMessage('INFO', `${title}: ${this.createProgressBar(100)} 100% (${total}/${total}) - ${message} in ${elapsed.toFixed(2)}s (${rate.toFixed(2)} items/sec)`)));
         };
         return { update, complete };

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "doc2vec",
-    "version": "2.10.6",
+    "version": "2.11.0",
     "type": "commonjs",
     "description": "",
     "main": "dist/doc2vec.js",
@@ -18,11 +18,12 @@
     ],
     "scripts": {
         "build": "tsc && chmod 755 dist/doc2vec.js",
+        "build:ui": "npm --prefix ui ci && npm --prefix ui run build",
         "start": "node dist/doc2vec.js",
         "test": "vitest run",
         "test:watch": "vitest",
         "test:coverage": "vitest run --coverage",
-        "prepublishOnly": "npm run build",
+        "prepublishOnly": "npm run build && npm run build:ui",
         "prepare": "npm run build"
     },
     "keywords": [],
@@ -38,7 +39,10 @@
         "better-sqlite3": "^11.9.1",
         "chalk": "^5.4.1",
         "cheerio": "^1.0.0-rc.12",
+        "commander": "^15.0.0",
+        "croner": "^10.0.1",
         "dotenv": "^16.3.1",
+        "express": "^5.2.1",
         "js-yaml": "^4.1.0",
         "jsdom": "^26.0.0",
         "mammoth": "^1.11.0",
@@ -55,6 +59,7 @@
     },
     "devDependencies": {
         "@types/better-sqlite3": "^7.6.12",
+        "@types/express": "^5.0.6",
         "@types/js-yaml": "^4.0.9",
         "@types/jsdom": "^21.1.7",
         "@types/node": "^20.10.0",