doc2vec 2.10.5 → 2.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +98 -0
- package/dist/content-processor.js +122 -22
- package/dist/controller/config-registry.js +277 -0
- package/dist/controller/events.js +29 -0
- package/dist/controller/index.js +139 -0
- package/dist/controller/job-runner.js +314 -0
- package/dist/controller/migrations.js +55 -0
- package/dist/controller/notifier.js +106 -0
- package/dist/controller/scheduler.js +54 -0
- package/dist/controller/server.js +288 -0
- package/dist/controller/store.js +186 -0
- package/dist/controller/types.js +25 -0
- package/dist/doc2vec.js +121 -28
- package/dist/logger.js +89 -0
- package/dist/ui/assets/index-BVg6n7Si.css +1 -0
- package/dist/ui/assets/index-D-QOw3rk.js +126 -0
- package/dist/ui/index.html +14 -0
- package/package.json +7 -2
package/README.md
CHANGED
|
@@ -52,6 +52,7 @@ The primary goal is to prepare documentation content for Retrieval-Augmented Gen
|
|
|
52
52
|
* **Incremental Updates:** For GitHub and Zendesk sources, tracks the last run date to only fetch new or updated issues/tickets.
|
|
53
53
|
* **Cleanup:** Removes obsolete chunks from the database corresponding to pages or files that are no longer found during processing.
|
|
54
54
|
* **Configuration:** Driven by a YAML configuration file (`config.yaml`) specifying sites, repositories, local directories, Zendesk instances, database types, metadata, and other parameters.
|
|
55
|
+
* **[Controller Mode](#controller-mode):** Optionally runs as a long-lived controller that schedules sync jobs from multiple config files (cron `schedule` field), persists run history and statistics in Postgres, and serves a web UI with live log streaming — read-only (configs from files/ConfigMap) or read-write (create/edit configs from the UI).
|
|
55
56
|
* **Structured Logging:** Uses a custom logger (`logger.ts`) with levels, timestamps, colors, progress bars, and child loggers for clear execution monitoring.
|
|
56
57
|
|
|
57
58
|
## Chunk Metadata & Page Reconstruction
|
|
@@ -437,6 +438,103 @@ The script will then:
|
|
|
437
438
|
7. Cleanup obsolete chunks.
|
|
438
439
|
8. Output detailed logs.
|
|
439
440
|
|
|
441
|
+
## Controller Mode
|
|
442
|
+
|
|
443
|
+
Besides the one-shot sync, doc2vec can run as a **long-lived controller** that schedules sync jobs, records run history and statistics in Postgres, and serves a web UI for monitoring and managing configs:
|
|
444
|
+
|
|
445
|
+
```bash
|
|
446
|
+
doc2vec controller --database-url postgres://user:pass@host:5432/doc2vec ./configs/
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
Open http://localhost:8080 to see the dashboard: configs with their schedule and next run, run history with per-source statistics, and live log streaming for running jobs.
|
|
450
|
+
|
|
451
|
+
### Scheduling
|
|
452
|
+
|
|
453
|
+
Add a top-level cron `schedule` (and optionally a display `name`) to a config file, and the controller runs it automatically:
|
|
454
|
+
|
|
455
|
+
```yaml
|
|
456
|
+
name: product-docs
|
|
457
|
+
schedule: "0 2 * * *" # every day at 02:00
|
|
458
|
+
sources:
|
|
459
|
+
- type: website
|
|
460
|
+
...
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
Configs without a `schedule` are still listed in the UI and can be triggered manually with **Run now**. Each run executes `doc2vec run <config.yaml>` as a child process; overlapping runs of the same config are skipped, and `--max-parallel` (default 1) caps how many sync jobs run at once — keep it low, since website sources launch a headless Chromium.
|
|
464
|
+
|
|
465
|
+
### Flags
|
|
466
|
+
|
|
467
|
+
| Flag | Default | Description |
|
|
468
|
+
|------|---------|-------------|
|
|
469
|
+
| `--database-url <url>` | `DATABASE_URL` env | Postgres connection string (required) |
|
|
470
|
+
| `--port <n>` | `8080` (or `PORT`) | HTTP port for the API and UI |
|
|
471
|
+
| `--read-write` | off (read-only) | Allow creating/editing/deleting configs from the UI |
|
|
472
|
+
| `--config-dir <dir>` | — | Where UI-created configs are written (required with `--read-write`) |
|
|
473
|
+
| `--max-parallel <n>` | `1` | Max concurrent sync jobs |
|
|
474
|
+
| `--reload-interval <s>` | `30` | How often config files are re-read for changes |
|
|
475
|
+
| `--log-retention-days <n>` | `14` | Run logs older than this are pruned (run history is kept) |
|
|
476
|
+
| `--slack-webhook-url <url>` | `SLACK_WEBHOOK_URL` env | Slack incoming webhook — post a message when a run finishes |
|
|
477
|
+
| `--slack-notify <mode>` | `all` | `all` or `failures` (failures also covers canceled runs) |
|
|
478
|
+
| `--public-url <url>` | `PUBLIC_URL` env | Externally reachable base URL, used for "view run" links in notifications |
|
|
479
|
+
|
|
480
|
+
Positional arguments are config files and/or directories (directories are scanned for `*.yaml`/`*.yml`, and re-scanned on every reload).
|
|
481
|
+
|
|
482
|
+
### Slack notifications
|
|
483
|
+
|
|
484
|
+
With `--slack-webhook-url` (or `SLACK_WEBHOOK_URL`) set, the controller posts to a [Slack incoming webhook](https://api.slack.com/messaging/webhooks) whenever a run reaches a terminal state — ✅ succeeded, ❌ failed (naming the failing sources and their errors), or ⚠️ canceled. Overlap-`skipped` runs are not notified. Set `--slack-notify failures` to silence successes, and `--public-url https://doc2vec.example.com` to get clickable "view run" links.
|
|
485
|
+
|
|
486
|
+
### Read-only vs read-write
|
|
487
|
+
|
|
488
|
+
- **Read-only** (default): configs come solely from the files passed on the command line — ideal for Kubernetes, where they live in a ConfigMap. The UI can view configs, trigger runs, and browse history/stats, but not modify anything. ConfigMap updates are picked up automatically via content-hash polling.
|
|
489
|
+
- **Read-write** (`--read-write --config-dir ./configs`): the UI can also create, edit, and delete config YAML files. Files stay the source of truth on disk; concurrent edits are protected by a content-hash check.
|
|
490
|
+
|
|
491
|
+
In both modes the UI always shows raw YAML: `${ENV_VAR}` secret placeholders are **never** resolved outside the sync job itself.
|
|
492
|
+
|
|
493
|
+
### Kubernetes example
|
|
494
|
+
|
|
495
|
+
```yaml
|
|
496
|
+
apiVersion: apps/v1
|
|
497
|
+
kind: Deployment
|
|
498
|
+
metadata:
|
|
499
|
+
name: doc2vec-controller
|
|
500
|
+
spec:
|
|
501
|
+
replicas: 1 # keep a single replica: the scheduler is not distributed
|
|
502
|
+
selector:
|
|
503
|
+
matchLabels: { app: doc2vec-controller }
|
|
504
|
+
template:
|
|
505
|
+
metadata:
|
|
506
|
+
labels: { app: doc2vec-controller }
|
|
507
|
+
spec:
|
|
508
|
+
containers:
|
|
509
|
+
- name: controller
|
|
510
|
+
image: ghcr.io/kagent-dev/doc2vec:latest
|
|
511
|
+
command: ["node", "dist/doc2vec.js", "controller", "/etc/doc2vec"]
|
|
512
|
+
ports:
|
|
513
|
+
- containerPort: 8080
|
|
514
|
+
env:
|
|
515
|
+
- name: DATABASE_URL
|
|
516
|
+
valueFrom:
|
|
517
|
+
secretKeyRef: { name: doc2vec-secrets, key: database-url }
|
|
518
|
+
- name: OPENAI_API_KEY
|
|
519
|
+
valueFrom:
|
|
520
|
+
secretKeyRef: { name: doc2vec-secrets, key: openai-api-key }
|
|
521
|
+
volumeMounts:
|
|
522
|
+
- name: configs
|
|
523
|
+
mountPath: /etc/doc2vec
|
|
524
|
+
resources:
|
|
525
|
+
requests: { memory: 1Gi }
|
|
526
|
+
limits: { memory: 4Gi } # website sources launch headless Chromium
|
|
527
|
+
volumes:
|
|
528
|
+
- name: configs
|
|
529
|
+
configMap: { name: doc2vec-configs }
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
The controller detects ConfigMap updates without a restart. Health endpoint: `GET /api/health`.
|
|
533
|
+
|
|
534
|
+
### API
|
|
535
|
+
|
|
536
|
+
The UI is backed by a small REST API you can also use directly: `GET /api/configs`, `POST /api/configs/:id/run`, `GET /api/runs?configId=`, `GET /api/runs/:id/logs`, `POST /api/runs/:id/cancel`, `GET /api/configs/:id/stats?days=30`, plus server-sent events at `GET /api/events` and `GET /api/runs/:id/logs/stream`.
|
|
537
|
+
|
|
440
538
|
## Database Options
|
|
441
539
|
|
|
442
540
|
### SQLite (`database_config.type: 'sqlite'`)
|
|
@@ -36,7 +36,8 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
|
36
36
|
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
37
37
|
};
|
|
38
38
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
39
|
-
exports.ContentProcessor = void 0;
|
|
39
|
+
exports.ContentProcessor = exports.BrowserLaunchError = void 0;
|
|
40
|
+
exports.resolveBrowserExecutablePath = resolveBrowserExecutablePath;
|
|
40
41
|
const readability_1 = require("@mozilla/readability");
|
|
41
42
|
const axios_1 = __importDefault(require("axios"));
|
|
42
43
|
const cheerio_1 = require("cheerio");
|
|
@@ -48,6 +49,97 @@ const fs = __importStar(require("fs"));
|
|
|
48
49
|
const path = __importStar(require("path"));
|
|
49
50
|
const utils_1 = require("./utils");
|
|
50
51
|
const code_chunker_1 = require("./code-chunker");
|
|
52
|
+
/**
|
|
53
|
+
* Thrown when the browser cannot be launched at all (even after a retry).
|
|
54
|
+
* Without a browser no page in the crawl can be processed, so this aborts the
|
|
55
|
+
* whole website source — the run is then reported as failed instead of
|
|
56
|
+
* "succeeding" with an error on every URL.
|
|
57
|
+
*/
|
|
58
|
+
class BrowserLaunchError extends Error {
|
|
59
|
+
constructor(message) {
|
|
60
|
+
super(message);
|
|
61
|
+
this.name = 'BrowserLaunchError';
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
exports.BrowserLaunchError = BrowserLaunchError;
|
|
65
|
+
/**
|
|
66
|
+
* Resolve which browser binary to launch, in order of preference:
|
|
67
|
+
* 1. PUPPETEER_EXECUTABLE_PATH (explicit override)
|
|
68
|
+
* 2. Puppeteer's own downloaded Chrome for Testing (version-matched, the
|
|
69
|
+
* reliable choice — Debian's chromium package has shipped builds that
|
|
70
|
+
* crash at startup in containers, e.g. 150.0.7871 SIGTRAPs immediately)
|
|
71
|
+
* 3. A system chromium, as the fallback (also covers linux/arm64, where
|
|
72
|
+
* Chrome for Testing is not published)
|
|
73
|
+
* 4. undefined — let puppeteer decide
|
|
74
|
+
*/
|
|
75
|
+
function resolveBrowserExecutablePath() {
|
|
76
|
+
const override = process.env.PUPPETEER_EXECUTABLE_PATH;
|
|
77
|
+
if (override)
|
|
78
|
+
return override;
|
|
79
|
+
try {
|
|
80
|
+
const bundled = puppeteer_1.default.executablePath();
|
|
81
|
+
if (bundled && fs.existsSync(bundled))
|
|
82
|
+
return bundled;
|
|
83
|
+
}
|
|
84
|
+
catch {
|
|
85
|
+
// no browser downloaded for this platform
|
|
86
|
+
}
|
|
87
|
+
if (fs.existsSync('/usr/bin/chromium'))
|
|
88
|
+
return '/usr/bin/chromium';
|
|
89
|
+
if (fs.existsSync('/usr/bin/chromium-browser'))
|
|
90
|
+
return '/usr/bin/chromium-browser';
|
|
91
|
+
return undefined;
|
|
92
|
+
}
|
|
93
|
+
/**
|
|
94
|
+
* Best-effort snapshot of process + container memory, for diagnosing browser
|
|
95
|
+
* launch failures (the usual cause in Kubernetes is the pod's memory cgroup —
|
|
96
|
+
* Chromium forks, then its renderers are immediately OOM-killed).
|
|
97
|
+
*/
|
|
98
|
+
function memoryDiagnostics() {
|
|
99
|
+
const gib = (bytes) => `${(bytes / 1024 ** 3).toFixed(2)}GiB`;
|
|
100
|
+
const parts = [`node rss ${gib(process.memoryUsage().rss)}`];
|
|
101
|
+
try {
|
|
102
|
+
// cgroup v2, else v1 — absent entirely outside Linux/containers
|
|
103
|
+
let current;
|
|
104
|
+
let max;
|
|
105
|
+
if (fs.existsSync('/sys/fs/cgroup/memory.current')) {
|
|
106
|
+
current = fs.readFileSync('/sys/fs/cgroup/memory.current', 'utf8').trim();
|
|
107
|
+
max = fs.readFileSync('/sys/fs/cgroup/memory.max', 'utf8').trim();
|
|
108
|
+
}
|
|
109
|
+
else if (fs.existsSync('/sys/fs/cgroup/memory/memory.usage_in_bytes')) {
|
|
110
|
+
current = fs.readFileSync('/sys/fs/cgroup/memory/memory.usage_in_bytes', 'utf8').trim();
|
|
111
|
+
max = fs.readFileSync('/sys/fs/cgroup/memory/memory.limit_in_bytes', 'utf8').trim();
|
|
112
|
+
}
|
|
113
|
+
if (current !== undefined && max !== undefined) {
|
|
114
|
+
const unlimited = max === 'max' || Number(max) > 2 ** 60;
|
|
115
|
+
parts.push(`cgroup ${gib(Number(current))} used / ${unlimited ? 'no limit' : gib(Number(max))}`);
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
catch {
|
|
119
|
+
// diagnostics only — never fail because of them
|
|
120
|
+
}
|
|
121
|
+
try {
|
|
122
|
+
// /proc/meminfo reflects the NODE in Kubernetes — catches host-level pressure
|
|
123
|
+
const meminfo = fs.readFileSync('/proc/meminfo', 'utf8');
|
|
124
|
+
const available = meminfo.match(/^MemAvailable:\s+(\d+) kB/m);
|
|
125
|
+
if (available) {
|
|
126
|
+
parts.push(`node available ${gib(Number(available[1]) * 1024)}`);
|
|
127
|
+
}
|
|
128
|
+
}
|
|
129
|
+
catch {
|
|
130
|
+
// not Linux
|
|
131
|
+
}
|
|
132
|
+
try {
|
|
133
|
+
// Chromium writes its profile and (with --disable-dev-shm-usage) its
|
|
134
|
+
// shared memory under /tmp — a full disk breaks launches
|
|
135
|
+
const stat = fs.statfsSync('/tmp');
|
|
136
|
+
parts.push(`/tmp free ${gib(stat.bavail * stat.bsize)}`);
|
|
137
|
+
}
|
|
138
|
+
catch {
|
|
139
|
+
// statfs unavailable
|
|
140
|
+
}
|
|
141
|
+
return parts.join(', ');
|
|
142
|
+
}
|
|
51
143
|
class ContentProcessor {
|
|
52
144
|
constructor(logger) {
|
|
53
145
|
this.logger = logger;
|
|
@@ -329,16 +421,8 @@ class ContentProcessor {
|
|
|
329
421
|
let browser = null;
|
|
330
422
|
let page = null;
|
|
331
423
|
const launchBrowser = async () => {
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
if (fs.existsSync('/usr/bin/chromium')) {
|
|
335
|
-
executablePath = '/usr/bin/chromium';
|
|
336
|
-
}
|
|
337
|
-
else if (fs.existsSync('/usr/bin/chromium-browser')) {
|
|
338
|
-
executablePath = '/usr/bin/chromium-browser';
|
|
339
|
-
}
|
|
340
|
-
}
|
|
341
|
-
const b = await puppeteer_1.default.launch({
|
|
424
|
+
const executablePath = resolveBrowserExecutablePath();
|
|
425
|
+
const launchOptions = {
|
|
342
426
|
executablePath,
|
|
343
427
|
args: [
|
|
344
428
|
'--no-sandbox',
|
|
@@ -348,7 +432,25 @@ class ContentProcessor {
|
|
|
348
432
|
'--disable-extensions',
|
|
349
433
|
],
|
|
350
434
|
protocolTimeout: 60000,
|
|
351
|
-
}
|
|
435
|
+
};
|
|
436
|
+
let b;
|
|
437
|
+
try {
|
|
438
|
+
b = await puppeteer_1.default.launch(launchOptions);
|
|
439
|
+
}
|
|
440
|
+
catch (firstError) {
|
|
441
|
+
// One retry after a pause — the failure may be transient (e.g.
|
|
442
|
+
// memory pressure from a page that just closed). If the browser
|
|
443
|
+
// cannot launch at all, no page in this crawl can be processed,
|
|
444
|
+
// so escalate as fatal instead of failing every URL one by one.
|
|
445
|
+
logger.warn(`Browser launch failed (${memoryDiagnostics()}), retrying once in 5s...`, firstError);
|
|
446
|
+
await new Promise(resolve => setTimeout(resolve, 5000));
|
|
447
|
+
try {
|
|
448
|
+
b = await puppeteer_1.default.launch(launchOptions);
|
|
449
|
+
}
|
|
450
|
+
catch (secondError) {
|
|
451
|
+
throw new BrowserLaunchError(`browser failed to launch twice (${memoryDiagnostics()}): ${secondError instanceof Error ? secondError.message : String(secondError)}`);
|
|
452
|
+
}
|
|
453
|
+
}
|
|
352
454
|
const p = await b.newPage();
|
|
353
455
|
return { browser: b, page: p };
|
|
354
456
|
};
|
|
@@ -682,6 +784,13 @@ class ContentProcessor {
|
|
|
682
784
|
}
|
|
683
785
|
}
|
|
684
786
|
catch (error) {
|
|
787
|
+
// A browser that cannot launch is fatal for the whole crawl —
|
|
788
|
+
// rethrow so the source (and the run) is reported as failed
|
|
789
|
+
// rather than logging an error per URL and finishing "green".
|
|
790
|
+
if (error instanceof BrowserLaunchError) {
|
|
791
|
+
logger.error(`Aborting crawl of ${baseUrl}: ${error.message}`);
|
|
792
|
+
throw error;
|
|
793
|
+
}
|
|
685
794
|
const status = this.getHttpStatus(error);
|
|
686
795
|
// ── Handle 429 (Too Many Requests) with retry ──
|
|
687
796
|
if (status === 429) {
|
|
@@ -846,17 +955,8 @@ class ContentProcessor {
|
|
|
846
955
|
}
|
|
847
956
|
else {
|
|
848
957
|
// Standalone mode: launch a browser for this single page
|
|
849
|
-
let executablePath = process.env.PUPPETEER_EXECUTABLE_PATH;
|
|
850
|
-
if (!executablePath) {
|
|
851
|
-
if (fs.existsSync('/usr/bin/chromium')) {
|
|
852
|
-
executablePath = '/usr/bin/chromium';
|
|
853
|
-
}
|
|
854
|
-
else if (fs.existsSync('/usr/bin/chromium-browser')) {
|
|
855
|
-
executablePath = '/usr/bin/chromium-browser';
|
|
856
|
-
}
|
|
857
|
-
}
|
|
858
958
|
browser = await puppeteer_1.default.launch({
|
|
859
|
-
executablePath,
|
|
959
|
+
executablePath: resolveBrowserExecutablePath(),
|
|
860
960
|
args: [
|
|
861
961
|
'--no-sandbox',
|
|
862
962
|
'--disable-setuid-sandbox',
|
|
@@ -0,0 +1,277 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
|
|
3
|
+
if (k2 === undefined) k2 = k;
|
|
4
|
+
var desc = Object.getOwnPropertyDescriptor(m, k);
|
|
5
|
+
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
|
|
6
|
+
desc = { enumerable: true, get: function() { return m[k]; } };
|
|
7
|
+
}
|
|
8
|
+
Object.defineProperty(o, k2, desc);
|
|
9
|
+
}) : (function(o, m, k, k2) {
|
|
10
|
+
if (k2 === undefined) k2 = k;
|
|
11
|
+
o[k2] = m[k];
|
|
12
|
+
}));
|
|
13
|
+
var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
|
|
14
|
+
Object.defineProperty(o, "default", { enumerable: true, value: v });
|
|
15
|
+
}) : function(o, v) {
|
|
16
|
+
o["default"] = v;
|
|
17
|
+
});
|
|
18
|
+
var __importStar = (this && this.__importStar) || (function () {
|
|
19
|
+
var ownKeys = function(o) {
|
|
20
|
+
ownKeys = Object.getOwnPropertyNames || function (o) {
|
|
21
|
+
var ar = [];
|
|
22
|
+
for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
|
|
23
|
+
return ar;
|
|
24
|
+
};
|
|
25
|
+
return ownKeys(o);
|
|
26
|
+
};
|
|
27
|
+
return function (mod) {
|
|
28
|
+
if (mod && mod.__esModule) return mod;
|
|
29
|
+
var result = {};
|
|
30
|
+
if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
|
|
31
|
+
__setModuleDefault(result, mod);
|
|
32
|
+
return result;
|
|
33
|
+
};
|
|
34
|
+
})();
|
|
35
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
36
|
+
exports.ConfigRegistry = void 0;
|
|
37
|
+
exports.parseConfigMeta = parseConfigMeta;
|
|
38
|
+
exports.isValidCron = isValidCron;
|
|
39
|
+
exports.hashContent = hashContent;
|
|
40
|
+
const crypto = __importStar(require("crypto"));
|
|
41
|
+
const fs = __importStar(require("fs"));
|
|
42
|
+
const path = __importStar(require("path"));
|
|
43
|
+
const yaml = __importStar(require("js-yaml"));
|
|
44
|
+
const croner_1 = require("croner");
|
|
45
|
+
const types_1 = require("./types");
|
|
46
|
+
/**
|
|
47
|
+
* Parse raw config YAML WITHOUT ${ENV} substitution (secrets stay as placeholders)
|
|
48
|
+
* and without Doc2Vec.loadConfig()'s process.exit() behavior — invalid files are
|
|
49
|
+
* reported, never fatal.
|
|
50
|
+
*/
|
|
51
|
+
function parseConfigMeta(content, filePath) {
|
|
52
|
+
const fallbackName = path.basename(filePath).replace(/\.ya?ml$/i, '');
|
|
53
|
+
try {
|
|
54
|
+
const parsed = yaml.load(content);
|
|
55
|
+
if (!parsed || typeof parsed !== 'object') {
|
|
56
|
+
return { name: fallbackName, schedule: null, sourceSummary: [], parseError: 'config is empty or not a YAML mapping' };
|
|
57
|
+
}
|
|
58
|
+
if (!Array.isArray(parsed.sources) || parsed.sources.length === 0) {
|
|
59
|
+
return { name: parsed.name || fallbackName, schedule: null, sourceSummary: [], parseError: 'config has no sources' };
|
|
60
|
+
}
|
|
61
|
+
const sourceSummary = parsed.sources.map((s) => ({
|
|
62
|
+
type: String(s?.type ?? 'unknown'),
|
|
63
|
+
product_name: String(s?.product_name ?? 'unknown'),
|
|
64
|
+
...(s?.version !== undefined && { version: String(s.version) }),
|
|
65
|
+
}));
|
|
66
|
+
let schedule = null;
|
|
67
|
+
let parseError = null;
|
|
68
|
+
if (parsed.schedule !== undefined && parsed.schedule !== null) {
|
|
69
|
+
schedule = String(parsed.schedule);
|
|
70
|
+
if (!isValidCron(schedule)) {
|
|
71
|
+
parseError = `invalid cron expression: ${schedule}`;
|
|
72
|
+
schedule = null;
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
return { name: String(parsed.name || fallbackName), schedule, sourceSummary, parseError };
|
|
76
|
+
}
|
|
77
|
+
catch (err) {
|
|
78
|
+
return {
|
|
79
|
+
name: fallbackName,
|
|
80
|
+
schedule: null,
|
|
81
|
+
sourceSummary: [],
|
|
82
|
+
parseError: err instanceof Error ? err.message : String(err),
|
|
83
|
+
};
|
|
84
|
+
}
|
|
85
|
+
}
|
|
86
|
+
function isValidCron(expr) {
|
|
87
|
+
try {
|
|
88
|
+
new croner_1.Cron(expr, { paused: true }).stop();
|
|
89
|
+
return true;
|
|
90
|
+
}
|
|
91
|
+
catch {
|
|
92
|
+
return false;
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
function hashContent(content) {
|
|
96
|
+
return crypto.createHash('sha256').update(content).digest('hex');
|
|
97
|
+
}
|
|
98
|
+
/**
|
|
99
|
+
* Watches the config files/directories passed on the CLI (plus --config-dir in
|
|
100
|
+
* read-write mode), keeps the d2v_configs table in sync, and — in read-write mode —
|
|
101
|
+
* persists UI edits back to YAML files on disk. Disk is the source of truth.
|
|
102
|
+
*
|
|
103
|
+
* Change detection is content-hash polling rather than fs.watch: Kubernetes
|
|
104
|
+
* ConfigMap updates arrive as atomic symlink swaps that fs.watch misses.
|
|
105
|
+
*/
|
|
106
|
+
class ConfigRegistry {
|
|
107
|
+
constructor(opts, store, events, logger) {
|
|
108
|
+
this.opts = opts;
|
|
109
|
+
this.store = store;
|
|
110
|
+
this.events = events;
|
|
111
|
+
this.logger = logger;
|
|
112
|
+
this.byPath = new Map();
|
|
113
|
+
this.timer = null;
|
|
114
|
+
this.syncing = false;
|
|
115
|
+
}
|
|
116
|
+
async start() {
|
|
117
|
+
await this.sync();
|
|
118
|
+
this.timer = setInterval(() => {
|
|
119
|
+
this.sync().catch(err => this.logger.error('Config re-sync failed:', err));
|
|
120
|
+
}, Math.max(this.opts.reloadIntervalSec, 5) * 1000);
|
|
121
|
+
this.timer.unref();
|
|
122
|
+
}
|
|
123
|
+
stop() {
|
|
124
|
+
if (this.timer)
|
|
125
|
+
clearInterval(this.timer);
|
|
126
|
+
this.timer = null;
|
|
127
|
+
}
|
|
128
|
+
list() {
|
|
129
|
+
return [...this.byPath.values()].sort((a, b) => a.name.localeCompare(b.name));
|
|
130
|
+
}
|
|
131
|
+
get(id) {
|
|
132
|
+
return [...this.byPath.values()].find(c => c.id === id);
|
|
133
|
+
}
|
|
134
|
+
/** Expand CLI args (files and directories) into the current set of config files. */
|
|
135
|
+
discoverFiles() {
|
|
136
|
+
const files = new Set();
|
|
137
|
+
const addDir = (dir) => {
|
|
138
|
+
for (const entry of fs.readdirSync(dir)) {
|
|
139
|
+
if (/\.ya?ml$/i.test(entry) && !entry.startsWith('.')) {
|
|
140
|
+
const full = path.join(dir, entry);
|
|
141
|
+
if (fs.statSync(full).isFile())
|
|
142
|
+
files.add(full);
|
|
143
|
+
}
|
|
144
|
+
}
|
|
145
|
+
};
|
|
146
|
+
for (const arg of this.opts.configArgs) {
|
|
147
|
+
const abs = path.resolve(arg);
|
|
148
|
+
if (!fs.existsSync(abs)) {
|
|
149
|
+
this.logger.warn(`Config path does not exist: ${abs}`);
|
|
150
|
+
continue;
|
|
151
|
+
}
|
|
152
|
+
if (fs.statSync(abs).isDirectory()) {
|
|
153
|
+
addDir(abs);
|
|
154
|
+
}
|
|
155
|
+
else {
|
|
156
|
+
files.add(abs);
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
if (this.opts.readWrite && this.opts.configDir && fs.existsSync(this.opts.configDir)) {
|
|
160
|
+
addDir(path.resolve(this.opts.configDir));
|
|
161
|
+
}
|
|
162
|
+
return [...files];
|
|
163
|
+
}
|
|
164
|
+
async sync() {
|
|
165
|
+
if (this.syncing)
|
|
166
|
+
return;
|
|
167
|
+
this.syncing = true;
|
|
168
|
+
try {
|
|
169
|
+
const files = this.discoverFiles();
|
|
170
|
+
const seen = new Set();
|
|
171
|
+
let changed = false;
|
|
172
|
+
for (const file of files) {
|
|
173
|
+
seen.add(file);
|
|
174
|
+
let content;
|
|
175
|
+
try {
|
|
176
|
+
content = fs.readFileSync(file, 'utf8');
|
|
177
|
+
}
|
|
178
|
+
catch (err) {
|
|
179
|
+
this.logger.warn(`Failed to read config ${file}:`, err);
|
|
180
|
+
continue;
|
|
181
|
+
}
|
|
182
|
+
const contentHash = hashContent(content);
|
|
183
|
+
const existing = this.byPath.get(file);
|
|
184
|
+
if (existing && existing.content_hash === contentHash)
|
|
185
|
+
continue;
|
|
186
|
+
const meta = parseConfigMeta(content, file);
|
|
187
|
+
if (meta.parseError) {
|
|
188
|
+
this.logger.warn(`Config ${file} has a problem: ${meta.parseError}`);
|
|
189
|
+
}
|
|
190
|
+
const record = await this.store.upsertConfig({
|
|
191
|
+
path: file,
|
|
192
|
+
name: meta.name,
|
|
193
|
+
content,
|
|
194
|
+
contentHash,
|
|
195
|
+
schedule: meta.schedule,
|
|
196
|
+
sourceSummary: meta.sourceSummary,
|
|
197
|
+
parseError: meta.parseError,
|
|
198
|
+
});
|
|
199
|
+
this.byPath.set(file, record);
|
|
200
|
+
changed = true;
|
|
201
|
+
this.logger.info(`${existing ? 'Reloaded' : 'Loaded'} config '${record.name}' from ${file}` +
|
|
202
|
+
(record.schedule ? ` (schedule: ${record.schedule})` : ' (manual runs only)'));
|
|
203
|
+
}
|
|
204
|
+
for (const [file] of this.byPath) {
|
|
205
|
+
if (!seen.has(file)) {
|
|
206
|
+
this.logger.info(`Config file removed: ${file}`);
|
|
207
|
+
await this.store.markConfigDeleted(file);
|
|
208
|
+
this.byPath.delete(file);
|
|
209
|
+
changed = true;
|
|
210
|
+
}
|
|
211
|
+
}
|
|
212
|
+
if (changed)
|
|
213
|
+
this.events.emitConfigUpdate();
|
|
214
|
+
}
|
|
215
|
+
finally {
|
|
216
|
+
this.syncing = false;
|
|
217
|
+
}
|
|
218
|
+
}
|
|
219
|
+
// ------------------------------------------------------------- read-write mode
|
|
220
|
+
/** Validate content for create/update: must parse and declare sources; cron must be valid. */
|
|
221
|
+
validateForWrite(content, filePath) {
|
|
222
|
+
const meta = parseConfigMeta(content, filePath);
|
|
223
|
+
if (meta.parseError) {
|
|
224
|
+
throw new types_1.ValidationError(meta.parseError);
|
|
225
|
+
}
|
|
226
|
+
return meta;
|
|
227
|
+
}
|
|
228
|
+
atomicWrite(target, content) {
|
|
229
|
+
const tmp = `${target}.tmp-${process.pid}`;
|
|
230
|
+
fs.writeFileSync(tmp, content, 'utf8');
|
|
231
|
+
fs.renameSync(tmp, target);
|
|
232
|
+
}
|
|
233
|
+
async create(filename, content) {
|
|
234
|
+
if (!this.opts.configDir) {
|
|
235
|
+
throw new types_1.ValidationError('no --config-dir configured');
|
|
236
|
+
}
|
|
237
|
+
if (!/^[A-Za-z0-9][A-Za-z0-9._-]*\.ya?ml$/.test(filename)) {
|
|
238
|
+
throw new types_1.ValidationError('filename must be a plain name ending in .yaml or .yml');
|
|
239
|
+
}
|
|
240
|
+
const target = path.join(path.resolve(this.opts.configDir), filename);
|
|
241
|
+
if (fs.existsSync(target)) {
|
|
242
|
+
throw new types_1.ConflictError(`config file ${filename} already exists`);
|
|
243
|
+
}
|
|
244
|
+
this.validateForWrite(content, target);
|
|
245
|
+
this.atomicWrite(target, content);
|
|
246
|
+
await this.sync();
|
|
247
|
+
const record = this.byPath.get(target);
|
|
248
|
+
if (!record)
|
|
249
|
+
throw new Error('config was written but failed to load');
|
|
250
|
+
return record;
|
|
251
|
+
}
|
|
252
|
+
async update(id, content, baseHash) {
|
|
253
|
+
const existing = this.get(id);
|
|
254
|
+
if (!existing)
|
|
255
|
+
throw new types_1.NotFoundError(`config ${id} not found`);
|
|
256
|
+
if (existing.content_hash !== baseHash) {
|
|
257
|
+
throw new types_1.ConflictError('config changed on disk since you loaded it — reload and reapply your edits');
|
|
258
|
+
}
|
|
259
|
+
this.validateForWrite(content, existing.path);
|
|
260
|
+
this.atomicWrite(existing.path, content);
|
|
261
|
+
await this.sync();
|
|
262
|
+
const record = this.byPath.get(existing.path);
|
|
263
|
+
if (!record)
|
|
264
|
+
throw new Error('config was written but failed to load');
|
|
265
|
+
return record;
|
|
266
|
+
}
|
|
267
|
+
async delete(id) {
|
|
268
|
+
const existing = this.get(id);
|
|
269
|
+
if (!existing)
|
|
270
|
+
throw new types_1.NotFoundError(`config ${id} not found`);
|
|
271
|
+
fs.unlinkSync(existing.path);
|
|
272
|
+
await this.store.markConfigDeleted(existing.path);
|
|
273
|
+
this.byPath.delete(existing.path);
|
|
274
|
+
this.events.emitConfigUpdate();
|
|
275
|
+
}
|
|
276
|
+
}
|
|
277
|
+
exports.ConfigRegistry = ConfigRegistry;
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
exports.ControllerEvents = void 0;
|
|
4
|
+
const events_1 = require("events");
|
|
5
|
+
/**
|
|
6
|
+
* In-process event bus connecting the job runner / config registry to SSE clients.
|
|
7
|
+
*
|
|
8
|
+
* Events:
|
|
9
|
+
* - 'run:update' (run: RunRecord) — a run was created or changed status
|
|
10
|
+
* - 'run:log' ({ runId: number, lines: LogRow[] }) — new log lines for a running job
|
|
11
|
+
* - 'config:update' () — the set of configs changed on disk
|
|
12
|
+
*/
|
|
13
|
+
class ControllerEvents extends events_1.EventEmitter {
|
|
14
|
+
constructor() {
|
|
15
|
+
super();
|
|
16
|
+
// Every SSE client adds a listener per event; don't warn at the default 10
|
|
17
|
+
this.setMaxListeners(1000);
|
|
18
|
+
}
|
|
19
|
+
emitRunUpdate(run) {
|
|
20
|
+
this.emit('run:update', run);
|
|
21
|
+
}
|
|
22
|
+
emitRunLogs(runId, lines) {
|
|
23
|
+
this.emit('run:log', { runId, lines });
|
|
24
|
+
}
|
|
25
|
+
emitConfigUpdate() {
|
|
26
|
+
this.emit('config:update');
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
exports.ControllerEvents = ControllerEvents;
|