doc2vec 2.10.5 → 2.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -52,6 +52,7 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
52
52
  * **Incremental Updates:** For GitHub and Zendesk sources, tracks the last run date to only fetch new or updated issues/tickets.
53
53
  * **Cleanup:** Removes obsolete chunks from the database corresponding to pages or files that are no longer found during processing.
54
54
  * **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, Zendesk instances, database types, metadata, and other parameters.
55
+ * **[Controller Mode](#controller-mode):** Optionally runs as a long-lived controller that schedules sync jobs from multiple config files (cron `schedule` field), persists run history and statistics in Postgres, and serves a web UI with live log streaming — read-only (configs from files/ConfigMap) or read-write (create/edit configs from the UI).
55
56
  * **Structured Logging:** Uses a custom logger (`logger.ts`) with levels, timestamps, colors, progress bars, and child loggers for clear execution monitoring.
56
57
 
57
58
  ## Chunk Metadata & Page Reconstruction
@@ -437,6 +438,103 @@ The script will then:
437
438
  7. Cleanup obsolete chunks.
438
439
  8. Output detailed logs.
439
440
 
441
+ ## Controller Mode
442
+
443
+ Besides the one-shot sync, doc2vec can run as a **long-lived controller** that schedules sync jobs, records run history and statistics in Postgres, and serves a web UI for monitoring and managing configs:
444
+
445
+ ```bash
446
+ doc2vec controller --database-url postgres://user:pass@host:5432/doc2vec ./configs/
447
+ ```
448
+
449
+ Open http://localhost:8080 to see the dashboard: configs with their schedule and next run, run history with per-source statistics, and live log streaming for running jobs.
450
+
451
+ ### Scheduling
452
+
453
+ Add a top-level cron `schedule` (and optionally a display `name`) to a config file, and the controller runs it automatically:
454
+
455
+ ```yaml
456
+ name: product-docs
457
+ schedule: "0 2 * * *" # every day at 02:00
458
+ sources:
459
+ - type: website
460
+ ...
461
+ ```
462
+
463
+ Configs without a `schedule` are still listed in the UI and can be triggered manually with **Run now**. Each run executes `doc2vec run <config.yaml>` as a child process; overlapping runs of the same config are skipped, and `--max-parallel` (default 1) caps how many sync jobs run at once — keep it low, since website sources launch a headless Chromium.
464
+
465
+ ### Flags
466
+
467
+ | Flag | Default | Description |
468
+ |------|---------|-------------|
469
+ | `--database-url <url>` | `DATABASE_URL` env | Postgres connection string (required) |
470
+ | `--port <n>` | `8080` (or `PORT`) | HTTP port for the API and UI |
471
+ | `--read-write` | off (read-only) | Allow creating/editing/deleting configs from the UI |
472
+ | `--config-dir <dir>` | — | Where UI-created configs are written (required with `--read-write`) |
473
+ | `--max-parallel <n>` | `1` | Max concurrent sync jobs |
474
+ | `--reload-interval <s>` | `30` | How often config files are re-read for changes |
475
+ | `--log-retention-days <n>` | `14` | Run logs older than this are pruned (run history is kept) |
476
+ | `--slack-webhook-url <url>` | `SLACK_WEBHOOK_URL` env | Slack incoming webhook — post a message when a run finishes |
477
+ | `--slack-notify <mode>` | `all` | `all` or `failures` (failures also covers canceled runs) |
478
+ | `--public-url <url>` | `PUBLIC_URL` env | Externally reachable base URL, used for "view run" links in notifications |
479
+
480
+ Positional arguments are config files and/or directories (directories are scanned for `*.yaml`/`*.yml`, and re-scanned on every reload).
481
+
482
+ ### Slack notifications
483
+
484
+ With `--slack-webhook-url` (or `SLACK_WEBHOOK_URL`) set, the controller posts to a [Slack incoming webhook](https://api.slack.com/messaging/webhooks) whenever a run reaches a terminal state — ✅ succeeded, ❌ failed (naming the failing sources and their errors), or ⚠️ canceled. Overlap-`skipped` runs are not notified. Set `--slack-notify failures` to silence successes, and `--public-url https://doc2vec.example.com` to get clickable "view run" links.
485
+
486
+ ### Read-only vs read-write
487
+
488
+ - **Read-only** (default): configs come solely from the files passed on the command line — ideal for Kubernetes, where they live in a ConfigMap. The UI can view configs, trigger runs, and browse history/stats, but not modify anything. ConfigMap updates are picked up automatically via content-hash polling.
489
+ - **Read-write** (`--read-write --config-dir ./configs`): the UI can also create, edit, and delete config YAML files. Files stay the source of truth on disk; concurrent edits are protected by a content-hash check.
490
+
491
+ In both modes the UI always shows raw YAML: `${ENV_VAR}` secret placeholders are **never** resolved outside the sync job itself.
492
+
493
+ ### Kubernetes example
494
+
495
+ ```yaml
496
+ apiVersion: apps/v1
497
+ kind: Deployment
498
+ metadata:
499
+ name: doc2vec-controller
500
+ spec:
501
+ replicas: 1 # keep a single replica: the scheduler is not distributed
502
+ selector:
503
+ matchLabels: { app: doc2vec-controller }
504
+ template:
505
+ metadata:
506
+ labels: { app: doc2vec-controller }
507
+ spec:
508
+ containers:
509
+ - name: controller
510
+ image: ghcr.io/kagent-dev/doc2vec:latest
511
+ command: ["node", "dist/doc2vec.js", "controller", "/etc/doc2vec"]
512
+ ports:
513
+ - containerPort: 8080
514
+ env:
515
+ - name: DATABASE_URL
516
+ valueFrom:
517
+ secretKeyRef: { name: doc2vec-secrets, key: database-url }
518
+ - name: OPENAI_API_KEY
519
+ valueFrom:
520
+ secretKeyRef: { name: doc2vec-secrets, key: openai-api-key }
521
+ volumeMounts:
522
+ - name: configs
523
+ mountPath: /etc/doc2vec
524
+ resources:
525
+ requests: { memory: 1Gi }
526
+ limits: { memory: 4Gi } # website sources launch headless Chromium
527
+ volumes:
528
+ - name: configs
529
+ configMap: { name: doc2vec-configs }
530
+ ```
531
+
532
+ The controller detects ConfigMap updates without a restart. Health endpoint: `GET /api/health`.
533
+
534
+ ### API
535
+
536
+ The UI is backed by a small REST API you can also use directly: `GET /api/configs`, `POST /api/configs/:id/run`, `GET /api/runs?configId=`, `GET /api/runs/:id/logs`, `POST /api/runs/:id/cancel`, `GET /api/configs/:id/stats?days=30`, plus server-sent events at `GET /api/events` and `GET /api/runs/:id/logs/stream`.
537
+
440
538
  ## Database Options
441
539
 
442
540
  ### SQLite (`database_config.type: 'sqlite'`)
@@ -36,7 +36,8 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
36
36
  return (mod && mod.__esModule) ? mod : { "default": mod };
37
37
  };
38
38
  Object.defineProperty(exports, "__esModule", { value: true });
39
- exports.ContentProcessor = void 0;
39
+ exports.ContentProcessor = exports.BrowserLaunchError = void 0;
40
+ exports.resolveBrowserExecutablePath = resolveBrowserExecutablePath;
40
41
  const readability_1 = require("@mozilla/readability");
41
42
  const axios_1 = __importDefault(require("axios"));
42
43
  const cheerio_1 = require("cheerio");
@@ -48,6 +49,97 @@ const fs = __importStar(require("fs"));
48
49
  const path = __importStar(require("path"));
49
50
  const utils_1 = require("./utils");
50
51
  const code_chunker_1 = require("./code-chunker");
52
+ /**
53
+ * Thrown when the browser cannot be launched at all (even after a retry).
54
+ * Without a browser no page in the crawl can be processed, so this aborts the
55
+ * whole website source — the run is then reported as failed instead of
56
+ * "succeeding" with an error on every URL.
57
+ */
58
+ class BrowserLaunchError extends Error {
59
+ constructor(message) {
60
+ super(message);
61
+ this.name = 'BrowserLaunchError';
62
+ }
63
+ }
64
+ exports.BrowserLaunchError = BrowserLaunchError;
65
+ /**
66
+ * Resolve which browser binary to launch, in order of preference:
67
+ * 1. PUPPETEER_EXECUTABLE_PATH (explicit override)
68
+ * 2. Puppeteer's own downloaded Chrome for Testing (version-matched, the
69
+ * reliable choice — Debian's chromium package has shipped builds that
70
+ * crash at startup in containers, e.g. 150.0.7871 SIGTRAPs immediately)
71
+ * 3. A system chromium, as the fallback (also covers linux/arm64, where
72
+ * Chrome for Testing is not published)
73
+ * 4. undefined — let puppeteer decide
74
+ */
75
+ function resolveBrowserExecutablePath() {
76
+ const override = process.env.PUPPETEER_EXECUTABLE_PATH;
77
+ if (override)
78
+ return override;
79
+ try {
80
+ const bundled = puppeteer_1.default.executablePath();
81
+ if (bundled && fs.existsSync(bundled))
82
+ return bundled;
83
+ }
84
+ catch {
85
+ // no browser downloaded for this platform
86
+ }
87
+ if (fs.existsSync('/usr/bin/chromium'))
88
+ return '/usr/bin/chromium';
89
+ if (fs.existsSync('/usr/bin/chromium-browser'))
90
+ return '/usr/bin/chromium-browser';
91
+ return undefined;
92
+ }
93
+ /**
94
+ * Best-effort snapshot of process + container memory, for diagnosing browser
95
+ * launch failures (the usual cause in Kubernetes is the pod's memory cgroup —
96
+ * Chromium forks, then its renderers are immediately OOM-killed).
97
+ */
98
+ function memoryDiagnostics() {
99
+ const gib = (bytes) => `${(bytes / 1024 ** 3).toFixed(2)}GiB`;
100
+ const parts = [`node rss ${gib(process.memoryUsage().rss)}`];
101
+ try {
102
+ // cgroup v2, else v1 — absent entirely outside Linux/containers
103
+ let current;
104
+ let max;
105
+ if (fs.existsSync('/sys/fs/cgroup/memory.current')) {
106
+ current = fs.readFileSync('/sys/fs/cgroup/memory.current', 'utf8').trim();
107
+ max = fs.readFileSync('/sys/fs/cgroup/memory.max', 'utf8').trim();
108
+ }
109
+ else if (fs.existsSync('/sys/fs/cgroup/memory/memory.usage_in_bytes')) {
110
+ current = fs.readFileSync('/sys/fs/cgroup/memory/memory.usage_in_bytes', 'utf8').trim();
111
+ max = fs.readFileSync('/sys/fs/cgroup/memory/memory.limit_in_bytes', 'utf8').trim();
112
+ }
113
+ if (current !== undefined && max !== undefined) {
114
+ const unlimited = max === 'max' || Number(max) > 2 ** 60;
115
+ parts.push(`cgroup ${gib(Number(current))} used / ${unlimited ? 'no limit' : gib(Number(max))}`);
116
+ }
117
+ }
118
+ catch {
119
+ // diagnostics only — never fail because of them
120
+ }
121
+ try {
122
+ // /proc/meminfo reflects the NODE in Kubernetes — catches host-level pressure
123
+ const meminfo = fs.readFileSync('/proc/meminfo', 'utf8');
124
+ const available = meminfo.match(/^MemAvailable:\s+(\d+) kB/m);
125
+ if (available) {
126
+ parts.push(`node available ${gib(Number(available[1]) * 1024)}`);
127
+ }
128
+ }
129
+ catch {
130
+ // not Linux
131
+ }
132
+ try {
133
+ // Chromium writes its profile and (with --disable-dev-shm-usage) its
134
+ // shared memory under /tmp — a full disk breaks launches
135
+ const stat = fs.statfsSync('/tmp');
136
+ parts.push(`/tmp free ${gib(stat.bavail * stat.bsize)}`);
137
+ }
138
+ catch {
139
+ // statfs unavailable
140
+ }
141
+ return parts.join(', ');
142
+ }
51
143
  class ContentProcessor {
52
144
  constructor(logger) {
53
145
  this.logger = logger;
@@ -329,16 +421,8 @@ class ContentProcessor {
329
421
  let browser = null;
330
422
  let page = null;
331
423
  const launchBrowser = async () => {
332
- let executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
333
- if (!executablePath) {
334
- if (fs.existsSync('/usr/bin/chromium')) {
335
- executablePath = '/usr/bin/chromium';
336
- }
337
- else if (fs.existsSync('/usr/bin/chromium-browser')) {
338
- executablePath = '/usr/bin/chromium-browser';
339
- }
340
- }
341
- const b = await puppeteer_1.default.launch({
424
+ const executablePath = resolveBrowserExecutablePath();
425
+ const launchOptions = {
342
426
  executablePath,
343
427
  args: [
344
428
  '--no-sandbox',
@@ -348,7 +432,25 @@ class ContentProcessor {
348
432
  '--disable-extensions',
349
433
  ],
350
434
  protocolTimeout: 60000,
351
- });
435
+ };
436
+ let b;
437
+ try {
438
+ b = await puppeteer_1.default.launch(launchOptions);
439
+ }
440
+ catch (firstError) {
441
+ // One retry after a pause — the failure may be transient (e.g.
442
+ // memory pressure from a page that just closed). If the browser
443
+ // cannot launch at all, no page in this crawl can be processed,
444
+ // so escalate as fatal instead of failing every URL one by one.
445
+ logger.warn(`Browser launch failed (${memoryDiagnostics()}), retrying once in 5s...`, firstError);
446
+ await new Promise(resolve => setTimeout(resolve, 5000));
447
+ try {
448
+ b = await puppeteer_1.default.launch(launchOptions);
449
+ }
450
+ catch (secondError) {
451
+ throw new BrowserLaunchError(`browser failed to launch twice (${memoryDiagnostics()}): ${secondError instanceof Error ? secondError.message : String(secondError)}`);
452
+ }
453
+ }
352
454
  const p = await b.newPage();
353
455
  return { browser: b, page: p };
354
456
  };
@@ -682,6 +784,13 @@ class ContentProcessor {
682
784
  }
683
785
  }
684
786
  catch (error) {
787
+ // A browser that cannot launch is fatal for the whole crawl —
788
+ // rethrow so the source (and the run) is reported as failed
789
+ // rather than logging an error per URL and finishing "green".
790
+ if (error instanceof BrowserLaunchError) {
791
+ logger.error(`Aborting crawl of ${baseUrl}: ${error.message}`);
792
+ throw error;
793
+ }
685
794
  const status = this.getHttpStatus(error);
686
795
  // ── Handle 429 (Too Many Requests) with retry ──
687
796
  if (status === 429) {
@@ -846,17 +955,8 @@ class ContentProcessor {
846
955
  }
847
956
  else {
848
957
  // Standalone mode: launch a browser for this single page
849
- let executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
850
- if (!executablePath) {
851
- if (fs.existsSync('/usr/bin/chromium')) {
852
- executablePath = '/usr/bin/chromium';
853
- }
854
- else if (fs.existsSync('/usr/bin/chromium-browser')) {
855
- executablePath = '/usr/bin/chromium-browser';
856
- }
857
- }
858
958
  browser = await puppeteer_1.default.launch({
859
- executablePath,
959
+ executablePath: resolveBrowserExecutablePath(),
860
960
  args: [
861
961
  '--no-sandbox',
862
962
  '--disable-setuid-sandbox',
@@ -0,0 +1,277 @@
1
+ "use strict";
2
+ var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
3
+ if (k2 === undefined) k2 = k;
4
+ var desc = Object.getOwnPropertyDescriptor(m, k);
5
+ if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
6
+ desc = { enumerable: true, get: function() { return m[k]; } };
7
+ }
8
+ Object.defineProperty(o, k2, desc);
9
+ }) : (function(o, m, k, k2) {
10
+ if (k2 === undefined) k2 = k;
11
+ o[k2] = m[k];
12
+ }));
13
+ var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
14
+ Object.defineProperty(o, "default", { enumerable: true, value: v });
15
+ }) : function(o, v) {
16
+ o["default"] = v;
17
+ });
18
+ var __importStar = (this && this.__importStar) || (function () {
19
+ var ownKeys = function(o) {
20
+ ownKeys = Object.getOwnPropertyNames || function (o) {
21
+ var ar = [];
22
+ for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
23
+ return ar;
24
+ };
25
+ return ownKeys(o);
26
+ };
27
+ return function (mod) {
28
+ if (mod && mod.__esModule) return mod;
29
+ var result = {};
30
+ if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
31
+ __setModuleDefault(result, mod);
32
+ return result;
33
+ };
34
+ })();
35
+ Object.defineProperty(exports, "__esModule", { value: true });
36
+ exports.ConfigRegistry = void 0;
37
+ exports.parseConfigMeta = parseConfigMeta;
38
+ exports.isValidCron = isValidCron;
39
+ exports.hashContent = hashContent;
40
+ const crypto = __importStar(require("crypto"));
41
+ const fs = __importStar(require("fs"));
42
+ const path = __importStar(require("path"));
43
+ const yaml = __importStar(require("js-yaml"));
44
+ const croner_1 = require("croner");
45
+ const types_1 = require("./types");
46
+ /**
47
+ * Parse raw config YAML WITHOUT ${ENV} substitution (secrets stay as placeholders)
48
+ * and without Doc2Vec.loadConfig()'s process.exit() behavior — invalid files are
49
+ * reported, never fatal.
50
+ */
51
+ function parseConfigMeta(content, filePath) {
52
+ const fallbackName = path.basename(filePath).replace(/\.ya?ml$/i, '');
53
+ try {
54
+ const parsed = yaml.load(content);
55
+ if (!parsed || typeof parsed !== 'object') {
56
+ return { name: fallbackName, schedule: null, sourceSummary: [], parseError: 'config is empty or not a YAML mapping' };
57
+ }
58
+ if (!Array.isArray(parsed.sources) || parsed.sources.length === 0) {
59
+ return { name: parsed.name || fallbackName, schedule: null, sourceSummary: [], parseError: 'config has no sources' };
60
+ }
61
+ const sourceSummary = parsed.sources.map((s) => ({
62
+ type: String(s?.type ?? 'unknown'),
63
+ product_name: String(s?.product_name ?? 'unknown'),
64
+ ...(s?.version !== undefined && { version: String(s.version) }),
65
+ }));
66
+ let schedule = null;
67
+ let parseError = null;
68
+ if (parsed.schedule !== undefined && parsed.schedule !== null) {
69
+ schedule = String(parsed.schedule);
70
+ if (!isValidCron(schedule)) {
71
+ parseError = `invalid cron expression: ${schedule}`;
72
+ schedule = null;
73
+ }
74
+ }
75
+ return { name: String(parsed.name || fallbackName), schedule, sourceSummary, parseError };
76
+ }
77
+ catch (err) {
78
+ return {
79
+ name: fallbackName,
80
+ schedule: null,
81
+ sourceSummary: [],
82
+ parseError: err instanceof Error ? err.message : String(err),
83
+ };
84
+ }
85
+ }
86
+ function isValidCron(expr) {
87
+ try {
88
+ new croner_1.Cron(expr, { paused: true }).stop();
89
+ return true;
90
+ }
91
+ catch {
92
+ return false;
93
+ }
94
+ }
95
+ function hashContent(content) {
96
+ return crypto.createHash('sha256').update(content).digest('hex');
97
+ }
98
+ /**
99
+ * Watches the config files/directories passed on the CLI (plus --config-dir in
100
+ * read-write mode), keeps the d2v_configs table in sync, and — in read-write mode —
101
+ * persists UI edits back to YAML files on disk. Disk is the source of truth.
102
+ *
103
+ * Change detection is content-hash polling rather than fs.watch: Kubernetes
104
+ * ConfigMap updates arrive as atomic symlink swaps that fs.watch misses.
105
+ */
106
+ class ConfigRegistry {
107
+ constructor(opts, store, events, logger) {
108
+ this.opts = opts;
109
+ this.store = store;
110
+ this.events = events;
111
+ this.logger = logger;
112
+ this.byPath = new Map();
113
+ this.timer = null;
114
+ this.syncing = false;
115
+ }
116
+ async start() {
117
+ await this.sync();
118
+ this.timer = setInterval(() => {
119
+ this.sync().catch(err => this.logger.error('Config re-sync failed:', err));
120
+ }, Math.max(this.opts.reloadIntervalSec, 5) * 1000);
121
+ this.timer.unref();
122
+ }
123
+ stop() {
124
+ if (this.timer)
125
+ clearInterval(this.timer);
126
+ this.timer = null;
127
+ }
128
+ list() {
129
+ return [...this.byPath.values()].sort((a, b) => a.name.localeCompare(b.name));
130
+ }
131
+ get(id) {
132
+ return [...this.byPath.values()].find(c => c.id === id);
133
+ }
134
+ /** Expand CLI args (files and directories) into the current set of config files. */
135
+ discoverFiles() {
136
+ const files = new Set();
137
+ const addDir = (dir) => {
138
+ for (const entry of fs.readdirSync(dir)) {
139
+ if (/\.ya?ml$/i.test(entry) && !entry.startsWith('.')) {
140
+ const full = path.join(dir, entry);
141
+ if (fs.statSync(full).isFile())
142
+ files.add(full);
143
+ }
144
+ }
145
+ };
146
+ for (const arg of this.opts.configArgs) {
147
+ const abs = path.resolve(arg);
148
+ if (!fs.existsSync(abs)) {
149
+ this.logger.warn(`Config path does not exist: ${abs}`);
150
+ continue;
151
+ }
152
+ if (fs.statSync(abs).isDirectory()) {
153
+ addDir(abs);
154
+ }
155
+ else {
156
+ files.add(abs);
157
+ }
158
+ }
159
+ if (this.opts.readWrite && this.opts.configDir && fs.existsSync(this.opts.configDir)) {
160
+ addDir(path.resolve(this.opts.configDir));
161
+ }
162
+ return [...files];
163
+ }
164
+ async sync() {
165
+ if (this.syncing)
166
+ return;
167
+ this.syncing = true;
168
+ try {
169
+ const files = this.discoverFiles();
170
+ const seen = new Set();
171
+ let changed = false;
172
+ for (const file of files) {
173
+ seen.add(file);
174
+ let content;
175
+ try {
176
+ content = fs.readFileSync(file, 'utf8');
177
+ }
178
+ catch (err) {
179
+ this.logger.warn(`Failed to read config ${file}:`, err);
180
+ continue;
181
+ }
182
+ const contentHash = hashContent(content);
183
+ const existing = this.byPath.get(file);
184
+ if (existing && existing.content_hash === contentHash)
185
+ continue;
186
+ const meta = parseConfigMeta(content, file);
187
+ if (meta.parseError) {
188
+ this.logger.warn(`Config ${file} has a problem: ${meta.parseError}`);
189
+ }
190
+ const record = await this.store.upsertConfig({
191
+ path: file,
192
+ name: meta.name,
193
+ content,
194
+ contentHash,
195
+ schedule: meta.schedule,
196
+ sourceSummary: meta.sourceSummary,
197
+ parseError: meta.parseError,
198
+ });
199
+ this.byPath.set(file, record);
200
+ changed = true;
201
+ this.logger.info(`${existing ? 'Reloaded' : 'Loaded'} config '${record.name}' from ${file}` +
202
+ (record.schedule ? ` (schedule: ${record.schedule})` : ' (manual runs only)'));
203
+ }
204
+ for (const [file] of this.byPath) {
205
+ if (!seen.has(file)) {
206
+ this.logger.info(`Config file removed: ${file}`);
207
+ await this.store.markConfigDeleted(file);
208
+ this.byPath.delete(file);
209
+ changed = true;
210
+ }
211
+ }
212
+ if (changed)
213
+ this.events.emitConfigUpdate();
214
+ }
215
+ finally {
216
+ this.syncing = false;
217
+ }
218
+ }
219
+ // ------------------------------------------------------------- read-write mode
220
+ /** Validate content for create/update: must parse and declare sources; cron must be valid. */
221
+ validateForWrite(content, filePath) {
222
+ const meta = parseConfigMeta(content, filePath);
223
+ if (meta.parseError) {
224
+ throw new types_1.ValidationError(meta.parseError);
225
+ }
226
+ return meta;
227
+ }
228
+ atomicWrite(target, content) {
229
+ const tmp = `${target}.tmp-${process.pid}`;
230
+ fs.writeFileSync(tmp, content, 'utf8');
231
+ fs.renameSync(tmp, target);
232
+ }
233
+ async create(filename, content) {
234
+ if (!this.opts.configDir) {
235
+ throw new types_1.ValidationError('no --config-dir configured');
236
+ }
237
+ if (!/^[A-Za-z0-9][A-Za-z0-9._-]*\.ya?ml$/.test(filename)) {
238
+ throw new types_1.ValidationError('filename must be a plain name ending in .yaml or .yml');
239
+ }
240
+ const target = path.join(path.resolve(this.opts.configDir), filename);
241
+ if (fs.existsSync(target)) {
242
+ throw new types_1.ConflictError(`config file ${filename} already exists`);
243
+ }
244
+ this.validateForWrite(content, target);
245
+ this.atomicWrite(target, content);
246
+ await this.sync();
247
+ const record = this.byPath.get(target);
248
+ if (!record)
249
+ throw new Error('config was written but failed to load');
250
+ return record;
251
+ }
252
+ async update(id, content, baseHash) {
253
+ const existing = this.get(id);
254
+ if (!existing)
255
+ throw new types_1.NotFoundError(`config ${id} not found`);
256
+ if (existing.content_hash !== baseHash) {
257
+ throw new types_1.ConflictError('config changed on disk since you loaded it — reload and reapply your edits');
258
+ }
259
+ this.validateForWrite(content, existing.path);
260
+ this.atomicWrite(existing.path, content);
261
+ await this.sync();
262
+ const record = this.byPath.get(existing.path);
263
+ if (!record)
264
+ throw new Error('config was written but failed to load');
265
+ return record;
266
+ }
267
+ async delete(id) {
268
+ const existing = this.get(id);
269
+ if (!existing)
270
+ throw new types_1.NotFoundError(`config ${id} not found`);
271
+ fs.unlinkSync(existing.path);
272
+ await this.store.markConfigDeleted(existing.path);
273
+ this.byPath.delete(existing.path);
274
+ this.events.emitConfigUpdate();
275
+ }
276
+ }
277
+ exports.ConfigRegistry = ConfigRegistry;
@@ -0,0 +1,29 @@
1
+ "use strict";
2
+ Object.defineProperty(exports, "__esModule", { value: true });
3
+ exports.ControllerEvents = void 0;
4
+ const events_1 = require("events");
5
+ /**
6
+ * In-process event bus connecting the job runner / config registry to SSE clients.
7
+ *
8
+ * Events:
9
+ * - 'run:update' (run: RunRecord) — a run was created or changed status
10
+ * - 'run:log' ({ runId: number, lines: LogRow[] }) — new log lines for a running job
11
+ * - 'config:update' () — the set of configs changed on disk
12
+ */
13
+ class ControllerEvents extends events_1.EventEmitter {
14
+ constructor() {
15
+ super();
16
+ // Every SSE client adds a listener per event; don't warn at the default 10
17
+ this.setMaxListeners(1000);
18
+ }
19
+ emitRunUpdate(run) {
20
+ this.emit('run:update', run);
21
+ }
22
+ emitRunLogs(runId, lines) {
23
+ this.emit('run:log', { runId, lines });
24
+ }
25
+ emitConfigUpdate() {
26
+ this.emit('config:update');
27
+ }
28
+ }
29
+ exports.ControllerEvents = ControllerEvents;