grepmax 0.26.61 → 0.26.63
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +12 -0
- package/dist/commands/mcp.js +8 -5
- package/dist/commands/status.js +14 -10
- package/dist/commands/watch.js +5 -2
- package/dist/lib/daemon/daemon.js +11 -8
- package/dist/lib/daemon/watcher-manager.js +53 -8
- package/dist/lib/index/batch-processor.js +117 -21
- package/dist/lib/index/watcher-ignore.js +104 -0
- package/dist/lib/index/watcher.js +45 -10
- package/dist/lib/output/index-state-footer.js +7 -0
- package/dist/lib/output/mcp-results.js +9 -0
- package/package.json +1 -1
- package/plugins/grepmax/.claude-plugin/plugin.json +1 -1
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
"name": "grepmax",
|
|
11
11
|
"source": "./plugins/grepmax",
|
|
12
12
|
"description": "Semantic code search for Claude Code. Automatically indexes your project and provides intelligent search capabilities.",
|
|
13
|
-
"version": "0.26.
|
|
13
|
+
"version": "0.26.63",
|
|
14
14
|
"author": {
|
|
15
15
|
"name": "Robert Owens",
|
|
16
16
|
"email": "robowens@me.com"
|
package/README.md
CHANGED
|
@@ -227,6 +227,8 @@ The CLI and MCP output include score, max similarity, pair count, representative
|
|
|
227
227
|
|
|
228
228
|
A single daemon watches your projects via native OS file events (FSEvents/inotify). Changes are detected in sub-second and incrementally reindexed. All writes to LanceDB are routed through the daemon via IPC, eliminating lock contention.
|
|
229
229
|
|
|
230
|
+
Filesystem watching belongs to gmax, in both daemon and standalone watch modes. Hetchy Developer is an optional monitor/controller that reads gmax status and invokes gmax commands. Watching, indexing and search must work with the dev app and its helper absent or stopped. Any future shared watcher extraction must be an independent component or package with an interface and lifecycle usable without Hetchy Developer; the dev app cannot become a gmax runtime dependency.
|
|
231
|
+
|
|
230
232
|
The daemon watches only projects that are in use: each Claude Code session (its MCP server and session hooks) holds a lease on its project, and `gmax add` / `gmax index` take a short one. When the last lease ends the project is unwatched; its index stays searchable, and the next session's catch-up scan picks up whatever changed in between. Set `GMAX_WATCH_ALL=1` to watch every registered project instead.
|
|
231
233
|
|
|
232
234
|
```bash
|
|
@@ -240,6 +242,10 @@ Adding or indexing a project can start the daemon automatically. MCP acquires pr
|
|
|
240
242
|
|
|
241
243
|
`gmax watch status` reports failed files, native watcher recovery or polling, dropped-event counts, and the last complete filesystem reconciliation. Polling scans run every five minutes, so recent edits can lag. Search and MCP responses retain health warnings even after the file queue drains. A complete reconciliation means the filesystem scan completed; queued embedding work can still be pending.
|
|
242
244
|
|
|
245
|
+
Live filesystem edits take priority over background catchup and cleanup. Background batches dispatch one file at a time, with at most four catchup files or 50 cleanup paths. They yield between files when a live edit arrives or the two-second dispatch window is reached; an active file or store commit finishes before yielding. Catchup and cleanup alternate, and one background file is admitted after three consecutive live batches so cleanup continues during sustained editing. Existing retry backoff and worker limits remain in effect.
|
|
246
|
+
|
|
247
|
+
`gmax status --json` exposes `projects[].health.queue`; daemon IPC and MCP `index_status` expose the same `indexState.queue`. Counts `live`, `catchup` and `cleanup` cover waiting paths; `activeFiles` covers the selected current batch, including files not yet dispatched. `oldestLiveEditAgeMs` includes both waiting and current-batch live edits until commit, and is null when none remain. Human/agent status, `gmax watch status` and MCP health text show these measures separately. Initial full indexing uses its existing progress counters rather than these watcher queues. Older daemons may omit queue diagnostics.
|
|
248
|
+
|
|
243
249
|
Temporary embedding backend outages preserve existing indexed rows and retry with backoff from five seconds up to one minute, without exhausting individual files' retry budgets. MLX failures report bounded transport/protocol details in the daemon log. `gmax status --json` includes the most recent resource snapshot when a recycle threshold has been reached; these counters help investigate memory growth, whose cause requires measurement over time.
|
|
244
250
|
|
|
245
251
|
## Running under the Claude Code sandbox
|
|
@@ -570,3 +576,9 @@ worker limits remain. The default `strict` policy keeps aggregate admission.
|
|
|
570
576
|
Restart the daemon after changing policy. An existing safety-stop marker requires
|
|
571
577
|
explicit operator review and clearing; selecting a policy never clears it.
|
|
572
578
|
Full-table compaction remains disabled in either policy.
|
|
579
|
+
|
|
580
|
+
Root `.gmaxignore` exclusions also filter native filesystem events. Simple
|
|
581
|
+
positive rules update both daemon and standalone subscriptions when the policy
|
|
582
|
+
changes. Rules with negations, escapes or advanced pattern syntax remain in the
|
|
583
|
+
indexing policy only, preserving re-included files. Source JSON outside an
|
|
584
|
+
excluded path stays indexable.
|
package/dist/commands/mcp.js
CHANGED
|
@@ -1842,7 +1842,8 @@ exports.mcp = new commander_1.Command("mcp")
|
|
|
1842
1842
|
}
|
|
1843
1843
|
function handleIndexStatus() {
|
|
1844
1844
|
return __awaiter(this, void 0, void 0, function* () {
|
|
1845
|
-
var _a
|
|
1845
|
+
var _a;
|
|
1846
|
+
var _b, _c, _d, _e, _f;
|
|
1846
1847
|
try {
|
|
1847
1848
|
const globalConfig = storeConfig();
|
|
1848
1849
|
const projects = storeProjects();
|
|
@@ -1860,7 +1861,7 @@ exports.mcp = new commander_1.Command("mcp")
|
|
|
1860
1861
|
: (0, watcher_store_1.getWatcherCoveringPath)(currentRoot());
|
|
1861
1862
|
let watcherLine = "Watcher: not running";
|
|
1862
1863
|
if (watcher) {
|
|
1863
|
-
const status = (
|
|
1864
|
+
const status = (_b = watcher.status) !== null && _b !== void 0 ? _b : "unknown";
|
|
1864
1865
|
const root = path.basename(watcher.projectRoot);
|
|
1865
1866
|
const reindex = watcher.lastReindex
|
|
1866
1867
|
? `last reindex: ${Math.round((Date.now() - watcher.lastReindex) / 60000)}m ago`
|
|
@@ -1871,12 +1872,14 @@ exports.mcp = new commander_1.Command("mcp")
|
|
|
1871
1872
|
}
|
|
1872
1873
|
}
|
|
1873
1874
|
if (daemonProject) {
|
|
1874
|
-
watcherLine = `Watcher: ${(
|
|
1875
|
+
watcherLine = `Watcher: ${(_c = daemonProject.status) !== null && _c !== void 0 ? _c : "unknown"} (${path.basename(daemonProject.root)}/)`;
|
|
1875
1876
|
const health = (0, index_state_footer_1.formatIndexStateFooter)(daemonProject.indexState, {
|
|
1876
1877
|
agent: true,
|
|
1877
1878
|
});
|
|
1878
1879
|
if (health)
|
|
1879
1880
|
watcherLine += `\n${health}`;
|
|
1881
|
+
if ((_a = daemonProject.indexState) === null || _a === void 0 ? void 0 : _a.queue)
|
|
1882
|
+
watcherLine += `\nQueue: ${(0, index_state_footer_1.formatWatchQueue)(daemonProject.indexState.queue)}`;
|
|
1880
1883
|
}
|
|
1881
1884
|
// Health overview only. The per-project listing (names, roots,
|
|
1882
1885
|
// per-project chunk counts) lives in `list_projects` — keep this tool
|
|
@@ -1919,8 +1922,8 @@ exports.mcp = new commander_1.Command("mcp")
|
|
|
1919
1922
|
: null,
|
|
1920
1923
|
},
|
|
1921
1924
|
watcher: {
|
|
1922
|
-
status: (
|
|
1923
|
-
indexState: (
|
|
1925
|
+
status: (_e = (_d = daemonProject === null || daemonProject === void 0 ? void 0 : daemonProject.status) !== null && _d !== void 0 ? _d : watcher === null || watcher === void 0 ? void 0 : watcher.status) !== null && _e !== void 0 ? _e : (secondary() || !daemonStatus.ok ? "unobserved" : "not_running"),
|
|
1926
|
+
indexState: (_f = daemonProject === null || daemonProject === void 0 ? void 0 : daemonProject.indexState) !== null && _f !== void 0 ? _f : null,
|
|
1924
1927
|
},
|
|
1925
1928
|
compaction: compaction.success ? compaction.data : null,
|
|
1926
1929
|
});
|
package/dist/commands/status.js
CHANGED
|
@@ -273,8 +273,8 @@ Examples:
|
|
|
273
273
|
gmax status --json The same, plus daemon and settings, as JSON
|
|
274
274
|
`)
|
|
275
275
|
.action((opts) => __awaiter(void 0, void 0, void 0, function* () {
|
|
276
|
-
var _a, _b, _c, _d;
|
|
277
|
-
var
|
|
276
|
+
var _a, _b, _c, _d, _e, _f, _g, _h;
|
|
277
|
+
var _j, _k, _l, _m, _o, _p, _q, _r;
|
|
278
278
|
const globalConfig = (0, index_config_1.readGlobalConfig)();
|
|
279
279
|
const projects = (0, project_registry_1.listProjects)();
|
|
280
280
|
const indexing = (0, lock_1.isLocked)(config_1.PATHS.globalRoot);
|
|
@@ -316,7 +316,7 @@ Examples:
|
|
|
316
316
|
console.log(`\n${style.bold("gmax")} · ${globalConfig.modelTier} (${globalConfig.vectorDim}d, ${globalConfig.embedMode})${indexing ? style.yellow(" · indexing...") : ""}`);
|
|
317
317
|
}
|
|
318
318
|
if (((_b = (_a = view.daemon) === null || _a === void 0 ? void 0 : _a.service) === null || _b === void 0 ? void 0 : _b.mode) === "paused") {
|
|
319
|
-
const reason = ((
|
|
319
|
+
const reason = ((_j = view.daemon.service.reason) !== null && _j !== void 0 ? _j : "host pressure").replace(/[\r\n\t]/g, " ");
|
|
320
320
|
console.log(opts.agent
|
|
321
321
|
? `daemon_service\tmode=paused\treason=${reason}`
|
|
322
322
|
: style.yellow(`Bounded reads available; indexing and embeddings paused: ${reason}`));
|
|
@@ -331,14 +331,14 @@ Examples:
|
|
|
331
331
|
if (opts.agent) {
|
|
332
332
|
console.log(`store\t${store.home}\t${store.state}\tprefixes=${store.prefixes.join(",")}${store.error ? `\terror=${store.error}` : ""}`);
|
|
333
333
|
for (const project of store.projects)
|
|
334
|
-
console.log(`store_project\t${store.home}\t${project.name}\t${(
|
|
334
|
+
console.log(`store_project\t${store.home}\t${project.name}\t${(_k = project.chunkCount) !== null && _k !== void 0 ? _k : 0}\t${(_l = project.status) !== null && _l !== void 0 ? _l : "indexed"}\tcounts=cached`);
|
|
335
335
|
}
|
|
336
336
|
else {
|
|
337
337
|
console.log(`\nStore: ${shortenPath(store.home)} — ${store.state}${store.error ? ` (${store.error})` : ""}`);
|
|
338
338
|
if (store.state === "offline")
|
|
339
339
|
console.log(" Connect the drive to list its projects.");
|
|
340
340
|
for (const project of store.projects)
|
|
341
|
-
console.log(` ${project.name} ${formatChunks(project.chunkCount)} chunks (cached) ${(
|
|
341
|
+
console.log(` ${project.name} ${formatChunks(project.chunkCount)} chunks (cached) ${(_m = project.status) !== null && _m !== void 0 ? _m : "indexed"}`);
|
|
342
342
|
}
|
|
343
343
|
}
|
|
344
344
|
if (projects.length === 0) {
|
|
@@ -355,9 +355,10 @@ Examples:
|
|
|
355
355
|
for (const project of projects) {
|
|
356
356
|
const st = projectState(project, watchers.get(project.root));
|
|
357
357
|
const isCurrent = project.root === currentRoot;
|
|
358
|
-
const count = (
|
|
358
|
+
const count = (_o = chunkCounts.get(project.root)) !== null && _o !== void 0 ? _o : project.chunkCount;
|
|
359
359
|
const identity = (0, embedding_status_1.projectEmbeddingStatus)(project, globalConfig);
|
|
360
|
-
|
|
360
|
+
const queue = (_d = (_c = view.health) === null || _c === void 0 ? void 0 : _c.get(project.root)) === null || _d === void 0 ? void 0 : _d.queue;
|
|
361
|
+
console.log(`${project.name}\t${formatChunks(count)}\t${formatAge(project.lastIndexed)}\t${st}\tembedding=${identity.state}${isCurrent ? "\tcurrent" : ""}${((_e = view.health) === null || _e === void 0 ? void 0 : _e.get(project.root)) ? `\t${(_p = (0, index_state_footer_1.formatIndexStateFooter)(view.health.get(project.root), { agent: true })) !== null && _p !== void 0 ? _p : ""}` : ""}${queue ? `\t${(0, index_state_footer_1.formatWatchQueue)(queue)}` : ""}`);
|
|
361
362
|
}
|
|
362
363
|
const legacyNotice = (0, embedding_status_1.formatLegacyEmbeddingNotice)((0, embedding_status_1.countLegacyEmbeddingProjects)(projects, globalConfig), { agent: true });
|
|
363
364
|
if (legacyNotice)
|
|
@@ -373,7 +374,7 @@ Examples:
|
|
|
373
374
|
const watcher = watchers.get(project.root);
|
|
374
375
|
// Status column
|
|
375
376
|
let statusStr;
|
|
376
|
-
const projectStatus = (
|
|
377
|
+
const projectStatus = (_q = project.status) !== null && _q !== void 0 ? _q : "indexed";
|
|
377
378
|
if (projectStatus === "pending") {
|
|
378
379
|
statusStr = style.yellow("pending");
|
|
379
380
|
}
|
|
@@ -393,7 +394,7 @@ Examples:
|
|
|
393
394
|
statusStr = style.dim("idle");
|
|
394
395
|
}
|
|
395
396
|
// Chunks column
|
|
396
|
-
const count = (
|
|
397
|
+
const count = (_r = chunkCounts.get(project.root)) !== null && _r !== void 0 ? _r : project.chunkCount;
|
|
397
398
|
const chunks = `${formatChunks(count)} chunks`;
|
|
398
399
|
// Age column
|
|
399
400
|
const age = formatAge(project.lastIndexed);
|
|
@@ -402,11 +403,14 @@ Examples:
|
|
|
402
403
|
const marker = isCurrent ? style.cyan(" ←") : "";
|
|
403
404
|
const name = project.name.padEnd(nameWidth);
|
|
404
405
|
console.log(` ${name} ${chunks.padEnd(12)} ${age.padEnd(10)} ${statusStr} embedding:${embedding}${marker}`);
|
|
405
|
-
const health = (0, index_state_footer_1.formatIndexStateFooter)((
|
|
406
|
+
const health = (0, index_state_footer_1.formatIndexStateFooter)((_f = view.health) === null || _f === void 0 ? void 0 : _f.get(project.root), {
|
|
406
407
|
agent: false,
|
|
407
408
|
});
|
|
408
409
|
if (health)
|
|
409
410
|
console.log(` ${health}`);
|
|
411
|
+
const queue = (_h = (_g = view.health) === null || _g === void 0 ? void 0 : _g.get(project.root)) === null || _h === void 0 ? void 0 : _h.queue;
|
|
412
|
+
if (queue)
|
|
413
|
+
console.log(` queue: ${(0, index_state_footer_1.formatWatchQueue)(queue)}`);
|
|
410
414
|
}
|
|
411
415
|
const legacyNotice = (0, embedding_status_1.formatLegacyEmbeddingNotice)((0, embedding_status_1.countLegacyEmbeddingProjects)(projects, globalConfig));
|
|
412
416
|
if (legacyNotice)
|
package/dist/commands/watch.js
CHANGED
|
@@ -401,7 +401,8 @@ exports.watch
|
|
|
401
401
|
.command("status")
|
|
402
402
|
.description("Show running watchers")
|
|
403
403
|
.action(() => __awaiter(void 0, void 0, void 0, function* () {
|
|
404
|
-
var _a
|
|
404
|
+
var _a;
|
|
405
|
+
var _b, _c, _d;
|
|
405
406
|
const { sendDaemonCommand } = yield Promise.resolve().then(() => __importStar(require("../lib/utils/daemon-client")));
|
|
406
407
|
const resp = yield sendDaemonCommand({ cmd: "status" });
|
|
407
408
|
if (resp.ok) {
|
|
@@ -417,8 +418,10 @@ exports.watch
|
|
|
417
418
|
const health = (0, index_state_footer_1.formatIndexStateFooter)(p.indexState, { agent: true });
|
|
418
419
|
if (health)
|
|
419
420
|
console.log(` ${health}`);
|
|
421
|
+
if ((_a = p.indexState) === null || _a === void 0 ? void 0 : _a.queue)
|
|
422
|
+
console.log(` queue: ${(0, index_state_footer_1.formatWatchQueue)(p.indexState.queue)}`);
|
|
420
423
|
if (p.indexState)
|
|
421
|
-
console.log(` watcher=${(
|
|
424
|
+
console.log(` watcher=${(_b = p.indexState.watcherMode) !== null && _b !== void 0 ? _b : "unknown"} overflows=${(_c = p.indexState.overflowCount) !== null && _c !== void 0 ? _c : 0} lastReconciled=${p.indexState.lastReconciledAt ? new Date(p.indexState.lastReconciledAt).toISOString() : "unknown"} catchupMs=${(_d = p.indexState.catchupMs) !== null && _d !== void 0 ? _d : "unknown"}`);
|
|
422
425
|
}
|
|
423
426
|
if (projects.length === 0) {
|
|
424
427
|
console.log(" (no projects)");
|
|
@@ -980,13 +980,14 @@ class Daemon {
|
|
|
980
980
|
* on every search to annotate partial-result responses (Phase 6).
|
|
981
981
|
*/
|
|
982
982
|
indexState(root) {
|
|
983
|
-
var _a
|
|
984
|
-
var
|
|
983
|
+
var _a;
|
|
984
|
+
var _b, _c, _d, _e;
|
|
985
985
|
const processor = this.processors.get(root);
|
|
986
|
-
const
|
|
987
|
-
const
|
|
986
|
+
const progress = processor === null || processor === void 0 ? void 0 : processor.progress;
|
|
987
|
+
const batchPending = (_b = progress === null || progress === void 0 ? void 0 : progress.pendingFiles) !== null && _b !== void 0 ? _b : 0;
|
|
988
|
+
const processing = (_c = progress === null || progress === void 0 ? void 0 : progress.processing) !== null && _c !== void 0 ? _c : false;
|
|
988
989
|
// status === "pending" means the initial full index hasn't completed.
|
|
989
|
-
const initialPending = ((
|
|
990
|
+
const initialPending = ((_a = (0, project_registry_1.getProject)(root)) === null || _a === void 0 ? void 0 : _a.status) === "pending";
|
|
990
991
|
// A full index (--reset / initial) bypasses the batch processor; its
|
|
991
992
|
// onProgress feeds indexProgress, giving a real remaining count.
|
|
992
993
|
const fullIdx = this.indexProgress.get(root);
|
|
@@ -998,7 +999,7 @@ class Daemon {
|
|
|
998
999
|
// index genuinely has work outstanding; a post-FSEvents-drop catchup mostly
|
|
999
1000
|
// re-verifies unchanged files. Only claim "verifying" with a real sample and
|
|
1000
1001
|
// no initial-index work pending.
|
|
1001
|
-
const { recentFiles, recentReindexed } =
|
|
1002
|
+
const { recentFiles, recentReindexed } = progress !== null && progress !== void 0 ? progress : {
|
|
1002
1003
|
recentFiles: 0,
|
|
1003
1004
|
recentReindexed: 0,
|
|
1004
1005
|
};
|
|
@@ -1008,8 +1009,10 @@ class Daemon {
|
|
|
1008
1009
|
recentReindexed === 0 &&
|
|
1009
1010
|
pendingFiles > 0;
|
|
1010
1011
|
const health = this.watcherManager.health(root);
|
|
1011
|
-
return Object.assign(Object.assign({ indexing: this.pausedReason == null &&
|
|
1012
|
-
(!!fullIdx || processing || batchPending > 0 || initialPending), pendingFiles: this.pausedReason == null ? pendingFiles : 0, verifying: this.pausedReason == null && verifying }, health),
|
|
1012
|
+
return Object.assign(Object.assign(Object.assign({ indexing: this.pausedReason == null &&
|
|
1013
|
+
(!!fullIdx || processing || batchPending > 0 || initialPending), pendingFiles: this.pausedReason == null ? pendingFiles : 0, verifying: this.pausedReason == null && verifying }, health), (this.pausedReason == null && (progress === null || progress === void 0 ? void 0 : progress.queue)
|
|
1014
|
+
? { queue: progress.queue }
|
|
1015
|
+
: {})), { failedFiles: Math.max((_d = progress === null || progress === void 0 ? void 0 : progress.failedFiles) !== null && _d !== void 0 ? _d : 0, (_e = health.failedFiles) !== null && _e !== void 0 ? _e : 0) });
|
|
1013
1016
|
}
|
|
1014
1017
|
search(payload, signal) {
|
|
1015
1018
|
return __awaiter(this, void 0, void 0, function* () {
|
|
@@ -57,6 +57,7 @@ const cache_coherence_1 = require("../index/cache-coherence");
|
|
|
57
57
|
const file_policy_1 = require("../index/file-policy");
|
|
58
58
|
const walker_1 = require("../index/walker");
|
|
59
59
|
const watcher_1 = require("../index/watcher");
|
|
60
|
+
const watcher_ignore_1 = require("../index/watcher-ignore");
|
|
60
61
|
const file_utils_1 = require("../utils/file-utils");
|
|
61
62
|
const logger_1 = require("../utils/logger");
|
|
62
63
|
const project_registry_1 = require("../utils/project-registry");
|
|
@@ -114,6 +115,7 @@ class WatcherManager {
|
|
|
114
115
|
this.degradedRoots = new Set();
|
|
115
116
|
this.terminalFailures = new Map();
|
|
116
117
|
this.watchLifecycles = new Map();
|
|
118
|
+
this.subscriptionUpdates = new Map();
|
|
117
119
|
}
|
|
118
120
|
health(root) {
|
|
119
121
|
var _a;
|
|
@@ -155,7 +157,17 @@ class WatcherManager {
|
|
|
155
157
|
let lastReindexAt;
|
|
156
158
|
processor = new batch_processor_1.ProjectBatchProcessor(Object.assign(Object.assign({ projectRoot: root, vectorDb,
|
|
157
159
|
metaCache }, (workerPool ? { workerPool } : {})), { filePolicy, onPolicyChange: () => {
|
|
158
|
-
void this.
|
|
160
|
+
void this.subscribeWatcher(root, processor)
|
|
161
|
+
.then(() => {
|
|
162
|
+
if (this.deps.processors.get(root) === processor &&
|
|
163
|
+
!this.deps.getShuttingDown())
|
|
164
|
+
return this.runCatchup(root, processor);
|
|
165
|
+
return;
|
|
166
|
+
}, (error) => {
|
|
167
|
+
this.recoverWatcher(root, processor);
|
|
168
|
+
throw error;
|
|
169
|
+
})
|
|
170
|
+
.catch((err) => {
|
|
159
171
|
console.error(`[daemon:${path.basename(root)}] Policy reconciliation failed:`, err);
|
|
160
172
|
});
|
|
161
173
|
}, onReindex: (files, ms) => __awaiter(this, void 0, void 0, function* () {
|
|
@@ -277,8 +289,27 @@ class WatcherManager {
|
|
|
277
289
|
}
|
|
278
290
|
subscribeWatcher(root, processor) {
|
|
279
291
|
return __awaiter(this, void 0, void 0, function* () {
|
|
292
|
+
var _a;
|
|
293
|
+
const previous = (_a = this.subscriptionUpdates.get(root)) !== null && _a !== void 0 ? _a : Promise.resolve();
|
|
294
|
+
const update = previous
|
|
295
|
+
.catch(() => { })
|
|
296
|
+
.then(() => this.subscribeWatcherNow(root, processor));
|
|
297
|
+
this.subscriptionUpdates.set(root, update);
|
|
298
|
+
try {
|
|
299
|
+
yield update;
|
|
300
|
+
}
|
|
301
|
+
finally {
|
|
302
|
+
if (this.subscriptionUpdates.get(root) === update)
|
|
303
|
+
this.subscriptionUpdates.delete(root);
|
|
304
|
+
}
|
|
305
|
+
});
|
|
306
|
+
}
|
|
307
|
+
subscribeWatcherNow(root, processor) {
|
|
308
|
+
return __awaiter(this, void 0, void 0, function* () {
|
|
309
|
+
var _a, _b;
|
|
280
310
|
const name = path.basename(root);
|
|
281
|
-
if (this.deps.processors.get(root) !== processor
|
|
311
|
+
if (this.deps.processors.get(root) !== processor ||
|
|
312
|
+
((_b = (_a = this.deps).getShuttingDown) === null || _b === void 0 ? void 0 : _b.call(_a)))
|
|
282
313
|
return;
|
|
283
314
|
// Unsubscribe existing watcher if any (e.g. during recovery)
|
|
284
315
|
const existingSub = this.deps.subscriptions.get(root);
|
|
@@ -286,11 +317,13 @@ class WatcherManager {
|
|
|
286
317
|
try {
|
|
287
318
|
yield existingSub.unsubscribe();
|
|
288
319
|
}
|
|
289
|
-
catch (
|
|
320
|
+
catch (_c) { }
|
|
290
321
|
this.deps.subscriptions.delete(root);
|
|
291
322
|
}
|
|
292
323
|
const sub = yield watcher.subscribe(root, (err, events) => {
|
|
293
324
|
var _a, _b;
|
|
325
|
+
if (this.deps.processors.get(root) !== processor)
|
|
326
|
+
return;
|
|
294
327
|
if (err) {
|
|
295
328
|
this.overflowCounts.set(root, ((_a = this.overflowCounts.get(root)) !== null && _a !== void 0 ? _a : 0) + 1);
|
|
296
329
|
const now = Date.now();
|
|
@@ -311,7 +344,12 @@ class WatcherManager {
|
|
|
311
344
|
processor.handleFileEvent(event.type === "delete" ? "unlink" : "change", event.path);
|
|
312
345
|
}
|
|
313
346
|
this.deps.touchActivity();
|
|
314
|
-
}, {
|
|
347
|
+
}, {
|
|
348
|
+
ignore: [
|
|
349
|
+
...watcher_1.WATCHER_IGNORE_GLOBS,
|
|
350
|
+
...(yield (0, watcher_ignore_1.readProjectWatcherIgnores)(root)),
|
|
351
|
+
],
|
|
352
|
+
});
|
|
315
353
|
if (this.deps.processors.get(root) !== processor) {
|
|
316
354
|
yield sub.unsubscribe();
|
|
317
355
|
return;
|
|
@@ -568,6 +606,7 @@ class WatcherManager {
|
|
|
568
606
|
else if (reconciliation.action === "reprocess") {
|
|
569
607
|
processor.handleFileEvent("change", absPath, {
|
|
570
608
|
forceReprocess: true,
|
|
609
|
+
workKind: "catchup",
|
|
571
610
|
});
|
|
572
611
|
queued++;
|
|
573
612
|
if (queued % 500 === 0) {
|
|
@@ -585,7 +624,9 @@ class WatcherManager {
|
|
|
585
624
|
});
|
|
586
625
|
if (snapshot.size !== stats.size ||
|
|
587
626
|
snapshot.mtimeMs !== stats.mtimeMs) {
|
|
588
|
-
processor.handleFileEvent("change", absPath
|
|
627
|
+
processor.handleFileEvent("change", absPath, {
|
|
628
|
+
workKind: "catchup",
|
|
629
|
+
});
|
|
589
630
|
continue;
|
|
590
631
|
}
|
|
591
632
|
const hash = (0, file_utils_1.computeContentHash)(snapshot.buffer, absPath);
|
|
@@ -605,7 +646,7 @@ class WatcherManager {
|
|
|
605
646
|
}
|
|
606
647
|
if (signal.aborted)
|
|
607
648
|
return false;
|
|
608
|
-
processor.handleFileEvent("change", absPath);
|
|
649
|
+
processor.handleFileEvent("change", absPath, { workKind: "catchup" });
|
|
609
650
|
queued++;
|
|
610
651
|
// Throttle: pause periodically during large catchup scans to let the
|
|
611
652
|
// batch processor drain and compaction run between bursts.
|
|
@@ -639,7 +680,10 @@ class WatcherManager {
|
|
|
639
680
|
return false;
|
|
640
681
|
if (!seenPaths.has(cachedPath) &&
|
|
641
682
|
!(0, walker_1.isPathProtectedByWalkState)(cachedPath, walkState)) {
|
|
642
|
-
processor.handleFileEvent("unlink", cachedPath, {
|
|
683
|
+
processor.handleFileEvent("unlink", cachedPath, {
|
|
684
|
+
forceDelete: true,
|
|
685
|
+
workKind: "cleanup",
|
|
686
|
+
});
|
|
643
687
|
purged++;
|
|
644
688
|
}
|
|
645
689
|
}
|
|
@@ -725,7 +769,7 @@ class WatcherManager {
|
|
|
725
769
|
}
|
|
726
770
|
unwatchProject(root) {
|
|
727
771
|
return __awaiter(this, void 0, void 0, function* () {
|
|
728
|
-
var _a;
|
|
772
|
+
var _a, _b;
|
|
729
773
|
// Stop poll-mode timers + their FSEvents recovery probe first, so a removed
|
|
730
774
|
// project can't keep scanning until full daemon shutdown. These live
|
|
731
775
|
// independently of the processor, so clear them even on the early return.
|
|
@@ -750,6 +794,7 @@ class WatcherManager {
|
|
|
750
794
|
this.pendingOps.delete(`recover:${root}`);
|
|
751
795
|
const processor = this.deps.processors.get(root);
|
|
752
796
|
this.deps.processors.delete(root);
|
|
797
|
+
yield (_b = this.subscriptionUpdates.get(root)) === null || _b === void 0 ? void 0 : _b.catch(() => { });
|
|
753
798
|
yield this.stopCatchup(root);
|
|
754
799
|
if (processor)
|
|
755
800
|
yield processor.close();
|
|
@@ -58,6 +58,10 @@ const watcher_batch_1 = require("./watcher-batch");
|
|
|
58
58
|
const DEBOUNCE_MS = 2000;
|
|
59
59
|
const MAX_RETRIES = 5;
|
|
60
60
|
const MAX_BATCH_SIZE = 50;
|
|
61
|
+
const MAX_CATCHUP_BATCH_SIZE = 4;
|
|
62
|
+
const BACKGROUND_DISPATCH_WINDOW_MS = 2000;
|
|
63
|
+
const MAX_CONSECUTIVE_LIVE_BATCHES = 3;
|
|
64
|
+
const WORK_PRIORITY = { live: 0, catchup: 1, cleanup: 2 };
|
|
61
65
|
// Pool priority cannot preempt an active processFile, so keep one worker free
|
|
62
66
|
// for latency-sensitive encodeQuery/rerank work during indexing bursts.
|
|
63
67
|
const DEFAULT_BATCH_CONCURRENCY = Math.max(1, config_1.CONFIG.WORKER_THREADS - 1);
|
|
@@ -70,6 +74,10 @@ class ProjectBatchProcessor {
|
|
|
70
74
|
constructor(opts) {
|
|
71
75
|
var _a, _b, _c;
|
|
72
76
|
this.pending = new Map();
|
|
77
|
+
this.pendingWork = new Map();
|
|
78
|
+
this.activeWork = new Map();
|
|
79
|
+
this.liveBatches = 0;
|
|
80
|
+
this.nextBackground = "catchup";
|
|
73
81
|
this.retryCount = new Map();
|
|
74
82
|
this.retryAt = new Map();
|
|
75
83
|
this.terminalFailures = new Set();
|
|
@@ -110,6 +118,7 @@ class ProjectBatchProcessor {
|
|
|
110
118
|
}
|
|
111
119
|
handleFileEvent(event, absPath, options) {
|
|
112
120
|
var _a, _b;
|
|
121
|
+
var _c;
|
|
113
122
|
if (this.closed)
|
|
114
123
|
return;
|
|
115
124
|
const normalize = this.filePolicy.normalizeEventPath;
|
|
@@ -149,10 +158,35 @@ class ProjectBatchProcessor {
|
|
|
149
158
|
this.forcedReprocess.set(normalized, ++this.forceGeneration);
|
|
150
159
|
}
|
|
151
160
|
this.retryAt.delete(normalized);
|
|
152
|
-
this.
|
|
161
|
+
this.enqueueWork(normalized, event, {
|
|
162
|
+
kind: (_c = options === null || options === void 0 ? void 0 : options.workKind) !== null && _c !== void 0 ? _c : "live",
|
|
163
|
+
queuedAt: Date.now(),
|
|
164
|
+
});
|
|
153
165
|
(_b = this.onActivity) === null || _b === void 0 ? void 0 : _b.call(this);
|
|
154
166
|
this.scheduleBatch();
|
|
155
167
|
}
|
|
168
|
+
enqueueWork(absPath, event, work) {
|
|
169
|
+
var _a;
|
|
170
|
+
const previous = (_a = this.pendingWork.get(absPath)) !== null && _a !== void 0 ? _a : this.activeWork.get(absPath);
|
|
171
|
+
this.pending.set(absPath, event);
|
|
172
|
+
this.pendingWork.set(absPath, !previous || WORK_PRIORITY[work.kind] < WORK_PRIORITY[previous.kind]
|
|
173
|
+
? work
|
|
174
|
+
: {
|
|
175
|
+
kind: previous.kind,
|
|
176
|
+
queuedAt: previous.kind === work.kind
|
|
177
|
+
? Math.min(previous.queuedAt, work.queuedAt)
|
|
178
|
+
: previous.queuedAt,
|
|
179
|
+
});
|
|
180
|
+
}
|
|
181
|
+
hasReadyLiveWork() {
|
|
182
|
+
var _a;
|
|
183
|
+
const now = Date.now();
|
|
184
|
+
for (const [file, work] of this.pendingWork) {
|
|
185
|
+
if (work.kind === "live" && ((_a = this.retryAt.get(file)) !== null && _a !== void 0 ? _a : 0) <= now)
|
|
186
|
+
return true;
|
|
187
|
+
}
|
|
188
|
+
return false;
|
|
189
|
+
}
|
|
156
190
|
/** Live (re)index progress: files queued + whether a batch is running.
|
|
157
191
|
*
|
|
158
192
|
* recentFiles/recentReindexed summarize the last few batches so callers can
|
|
@@ -162,18 +196,40 @@ class ProjectBatchProcessor {
|
|
|
162
196
|
* not pending work. Without this, a large pendingFiles alone reads as "the
|
|
163
197
|
* index is far behind" and search callers caveat correct results. */
|
|
164
198
|
get progress() {
|
|
199
|
+
var _a;
|
|
165
200
|
let recentFiles = 0;
|
|
166
201
|
let recentReindexed = 0;
|
|
167
202
|
for (const b of this.recentBatches) {
|
|
168
203
|
recentFiles += b.files;
|
|
169
204
|
recentReindexed += b.reindexed;
|
|
170
205
|
}
|
|
206
|
+
const queue = {
|
|
207
|
+
live: 0,
|
|
208
|
+
catchup: 0,
|
|
209
|
+
cleanup: 0,
|
|
210
|
+
activeFiles: this.activeWork.size,
|
|
211
|
+
oldestLiveEditAgeMs: null,
|
|
212
|
+
};
|
|
213
|
+
let oldest = Number.POSITIVE_INFINITY;
|
|
214
|
+
for (const file of this.pending.keys()) {
|
|
215
|
+
const work = this.pendingWork.get(file);
|
|
216
|
+
queue[(_a = work === null || work === void 0 ? void 0 : work.kind) !== null && _a !== void 0 ? _a : "live"]++;
|
|
217
|
+
if ((work === null || work === void 0 ? void 0 : work.kind) === "live")
|
|
218
|
+
oldest = Math.min(oldest, work.queuedAt);
|
|
219
|
+
}
|
|
220
|
+
for (const work of this.activeWork.values()) {
|
|
221
|
+
if (work.kind === "live")
|
|
222
|
+
oldest = Math.min(oldest, work.queuedAt);
|
|
223
|
+
}
|
|
224
|
+
if (Number.isFinite(oldest))
|
|
225
|
+
queue.oldestLiveEditAgeMs = Math.max(0, Date.now() - oldest);
|
|
171
226
|
return {
|
|
172
227
|
pendingFiles: this.pending.size,
|
|
173
228
|
processing: this.processing,
|
|
174
229
|
failedFiles: this.terminalFailures.size,
|
|
175
230
|
recentFiles,
|
|
176
231
|
recentReindexed,
|
|
232
|
+
queue,
|
|
177
233
|
};
|
|
178
234
|
}
|
|
179
235
|
loadReusableEmbeddings(absPath) {
|
|
@@ -253,8 +309,8 @@ class ProjectBatchProcessor {
|
|
|
253
309
|
}
|
|
254
310
|
processBatch(operationSignal) {
|
|
255
311
|
return __awaiter(this, void 0, void 0, function* () {
|
|
256
|
-
var _a, _b, _c, _d;
|
|
257
|
-
var
|
|
312
|
+
var _a, _b, _c, _d, _e;
|
|
313
|
+
var _f, _g, _h, _j;
|
|
258
314
|
if (this.closed || this.processing || this.pending.size === 0)
|
|
259
315
|
return;
|
|
260
316
|
const diskPressure = this.vectorDb.checkDiskPressure();
|
|
@@ -263,25 +319,55 @@ class ProjectBatchProcessor {
|
|
|
263
319
|
}
|
|
264
320
|
const batch = new Map();
|
|
265
321
|
const batchForceGenerations = new Map();
|
|
322
|
+
const batchWork = new Map();
|
|
266
323
|
const now = Date.now();
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
324
|
+
const ready = {
|
|
325
|
+
live: [],
|
|
326
|
+
catchup: [],
|
|
327
|
+
cleanup: [],
|
|
328
|
+
};
|
|
329
|
+
for (const absPath of this.pending.keys()) {
|
|
330
|
+
if (((_f = this.retryAt.get(absPath)) !== null && _f !== void 0 ? _f : 0) > now)
|
|
270
331
|
continue;
|
|
332
|
+
const kind = (_g = (_a = this.pendingWork.get(absPath)) === null || _a === void 0 ? void 0 : _a.kind) !== null && _g !== void 0 ? _g : "live";
|
|
333
|
+
if (ready[kind].length < MAX_BATCH_SIZE)
|
|
334
|
+
ready[kind].push(absPath);
|
|
335
|
+
}
|
|
336
|
+
const background = ready[this.nextBackground].length
|
|
337
|
+
? this.nextBackground
|
|
338
|
+
: this.nextBackground === "catchup"
|
|
339
|
+
? "cleanup"
|
|
340
|
+
: "catchup";
|
|
341
|
+
const fairBackground = ready.live.length > 0 &&
|
|
342
|
+
ready[background].length > 0 &&
|
|
343
|
+
this.liveBatches >= MAX_CONSECUTIVE_LIVE_BATCHES;
|
|
344
|
+
const kind = ready.live.length > 0 && !fairBackground ? "live" : background;
|
|
345
|
+
const limit = fairBackground
|
|
346
|
+
? 1
|
|
347
|
+
: kind === "catchup"
|
|
348
|
+
? MAX_CATCHUP_BATCH_SIZE
|
|
349
|
+
: MAX_BATCH_SIZE;
|
|
350
|
+
for (const absPath of ready[kind].slice(0, limit)) {
|
|
351
|
+
const event = this.pending.get(absPath);
|
|
271
352
|
batch.set(absPath, event);
|
|
353
|
+
batchWork.set(absPath, (_h = this.pendingWork.get(absPath)) !== null && _h !== void 0 ? _h : { kind, queuedAt: now });
|
|
272
354
|
const forceGeneration = this.forcedReprocess.get(absPath);
|
|
273
355
|
if (forceGeneration !== undefined) {
|
|
274
356
|
batchForceGenerations.set(absPath, forceGeneration);
|
|
275
357
|
}
|
|
276
|
-
taken++;
|
|
277
|
-
if (taken >= MAX_BATCH_SIZE)
|
|
278
|
-
break;
|
|
279
358
|
}
|
|
280
359
|
if (batch.size === 0) {
|
|
281
360
|
this.schedulePendingBatch(30000);
|
|
282
361
|
return;
|
|
283
362
|
}
|
|
284
363
|
this.processing = true;
|
|
364
|
+
this.activeWork = batchWork;
|
|
365
|
+
if (kind === "live")
|
|
366
|
+
this.liveBatches++;
|
|
367
|
+
else {
|
|
368
|
+
this.liveBatches = 0;
|
|
369
|
+
this.nextBackground = kind === "catchup" ? "cleanup" : "catchup";
|
|
370
|
+
}
|
|
285
371
|
const batchAc = new AbortController();
|
|
286
372
|
const abortBatch = () => batchAc.abort(operationSignal.reason);
|
|
287
373
|
if (operationSignal.aborted)
|
|
@@ -295,6 +381,7 @@ class ProjectBatchProcessor {
|
|
|
295
381
|
}, this.batchTimeoutMs);
|
|
296
382
|
for (const key of batch.keys()) {
|
|
297
383
|
this.pending.delete(key);
|
|
384
|
+
this.pendingWork.delete(key);
|
|
298
385
|
this.retryAt.delete(key);
|
|
299
386
|
}
|
|
300
387
|
const filenames = [...batch.keys()].map((p) => path.basename(p));
|
|
@@ -331,7 +418,7 @@ class ProjectBatchProcessor {
|
|
|
331
418
|
this.retryCount.set(absPath, retry.failures);
|
|
332
419
|
this.retryAt.set(absPath, Date.now() + retry.backoffMs);
|
|
333
420
|
}
|
|
334
|
-
this.
|
|
421
|
+
this.enqueueWork(absPath, event, batchWork.get(absPath));
|
|
335
422
|
};
|
|
336
423
|
let stopDispatch = false;
|
|
337
424
|
const processOne = (absPath, event) => __awaiter(this, void 0, void 0, function* () {
|
|
@@ -473,7 +560,7 @@ class ProjectBatchProcessor {
|
|
|
473
560
|
}
|
|
474
561
|
}
|
|
475
562
|
});
|
|
476
|
-
const batchConcurrency = Math.min(this.concurrency, Math.max(1, Math.ceil(batch.size / FILES_PER_BATCH_SLOT)));
|
|
563
|
+
const batchConcurrency = Math.min(kind === "live" ? this.concurrency : 1, Math.max(1, Math.ceil(batch.size / FILES_PER_BATCH_SLOT)));
|
|
477
564
|
const activeTasks = [];
|
|
478
565
|
const schedule = (task) => __awaiter(this, void 0, void 0, function* () {
|
|
479
566
|
const taskPromise = task();
|
|
@@ -491,6 +578,14 @@ class ProjectBatchProcessor {
|
|
|
491
578
|
for (const [absPath, event] of batch) {
|
|
492
579
|
if (batchAc.signal.aborted || stopDispatch)
|
|
493
580
|
break;
|
|
581
|
+
// Do not enqueue a whole background batch into the worker pool. Finish
|
|
582
|
+
// the current file, commit its results, then admit waiting live edits.
|
|
583
|
+
// A single fairness file still runs during continuous live traffic.
|
|
584
|
+
if (kind !== "live" &&
|
|
585
|
+
settled > 0 &&
|
|
586
|
+
(Date.now() - start >= BACKGROUND_DISPATCH_WINDOW_MS ||
|
|
587
|
+
(!fairBackground && this.hasReadyLiveWork())))
|
|
588
|
+
break;
|
|
494
589
|
yield schedule(() => processOne(absPath, event));
|
|
495
590
|
}
|
|
496
591
|
yield Promise.allSettled(activeTasks);
|
|
@@ -561,9 +656,9 @@ class ProjectBatchProcessor {
|
|
|
561
656
|
}
|
|
562
657
|
const duration = Date.now() - start;
|
|
563
658
|
if (reindexed > 0) {
|
|
564
|
-
(
|
|
659
|
+
(_b = this.onReindex) === null || _b === void 0 ? void 0 : _b.call(this, reindexed, duration);
|
|
565
660
|
}
|
|
566
|
-
this.recentBatches.push({ files:
|
|
661
|
+
this.recentBatches.push({ files: settled, reindexed });
|
|
567
662
|
if (this.recentBatches.length > ProjectBatchProcessor.RECENT_BATCH_WINDOW) {
|
|
568
663
|
this.recentBatches.shift();
|
|
569
664
|
}
|
|
@@ -578,7 +673,7 @@ class ProjectBatchProcessor {
|
|
|
578
673
|
this.retryCount.delete(absPath);
|
|
579
674
|
this.retryAt.delete(absPath);
|
|
580
675
|
this.terminalFailures.delete(absPath);
|
|
581
|
-
(
|
|
676
|
+
(_c = this.onPathSuccess) === null || _c === void 0 ? void 0 : _c.call(this, absPath);
|
|
582
677
|
}
|
|
583
678
|
// Trigger compaction if fragments are accumulating
|
|
584
679
|
if (reindexed > 0 && diskPressure !== "critical") {
|
|
@@ -599,7 +694,7 @@ class ProjectBatchProcessor {
|
|
|
599
694
|
if (this.terminalFailures.has(absPath))
|
|
600
695
|
continue;
|
|
601
696
|
if (!this.pending.has(absPath)) {
|
|
602
|
-
this.
|
|
697
|
+
this.enqueueWork(absPath, event, batchWork.get(absPath));
|
|
603
698
|
}
|
|
604
699
|
}
|
|
605
700
|
(0, logger_1.log)(this.wtag, "Disk pressure — requeued batch, will retry in 60s");
|
|
@@ -622,7 +717,7 @@ class ProjectBatchProcessor {
|
|
|
622
717
|
if (this.terminalFailures.has(absPath))
|
|
623
718
|
continue;
|
|
624
719
|
if (!this.pending.has(absPath))
|
|
625
|
-
this.
|
|
720
|
+
this.enqueueWork(absPath, event, batchWork.get(absPath));
|
|
626
721
|
}
|
|
627
722
|
backoffOverrideMs = 30 * 60 * 1000;
|
|
628
723
|
}
|
|
@@ -635,19 +730,19 @@ class ProjectBatchProcessor {
|
|
|
635
730
|
continue;
|
|
636
731
|
if (this.pending.has(absPath))
|
|
637
732
|
continue;
|
|
638
|
-
const retry = (0, watcher_batch_1.computePathRetry)((
|
|
733
|
+
const retry = (0, watcher_batch_1.computePathRetry)((_j = this.retryCount.get(absPath)) !== null && _j !== void 0 ? _j : 0, MAX_RETRIES, DEBOUNCE_MS);
|
|
639
734
|
if (!retry.retry) {
|
|
640
735
|
this.retryCount.delete(absPath);
|
|
641
736
|
this.retryAt.delete(absPath);
|
|
642
737
|
this.terminalFailures.add(absPath);
|
|
643
|
-
(
|
|
738
|
+
(_d = this.onTerminalFailure) === null || _d === void 0 ? void 0 : _d.call(this, absPath);
|
|
644
739
|
dropped++;
|
|
645
740
|
droppedPaths.push(absPath);
|
|
646
741
|
continue;
|
|
647
742
|
}
|
|
648
743
|
this.retryCount.set(absPath, retry.failures);
|
|
649
744
|
this.retryAt.set(absPath, Date.now() + retry.backoffMs);
|
|
650
|
-
this.
|
|
745
|
+
this.enqueueWork(absPath, event, batchWork.get(absPath));
|
|
651
746
|
}
|
|
652
747
|
if (dropped > 0) {
|
|
653
748
|
(0, logger_1.log)(this.wtag, `Dropped ${dropped} file(s) after ${MAX_RETRIES} retries: ${droppedPaths.map((p) => path.basename(p)).join(", ")}`);
|
|
@@ -666,10 +761,11 @@ class ProjectBatchProcessor {
|
|
|
666
761
|
if (!this.pending.has(absPath)) {
|
|
667
762
|
this.retryCount.delete(absPath);
|
|
668
763
|
this.retryAt.delete(absPath);
|
|
669
|
-
this.
|
|
764
|
+
this.enqueueWork(absPath, "change", batchWork.get(absPath));
|
|
670
765
|
}
|
|
671
766
|
}
|
|
672
767
|
}
|
|
768
|
+
this.activeWork = new Map();
|
|
673
769
|
if (!this.closed && this.pending.size > 0) {
|
|
674
770
|
if (backoffOverrideMs > 0) {
|
|
675
771
|
if (this.debounceTimer)
|
|
@@ -686,7 +782,7 @@ class ProjectBatchProcessor {
|
|
|
686
782
|
}
|
|
687
783
|
}
|
|
688
784
|
if (!this.closed)
|
|
689
|
-
(
|
|
785
|
+
(_e = this.onBatchSettled) === null || _e === void 0 ? void 0 : _e.call(this);
|
|
690
786
|
}
|
|
691
787
|
});
|
|
692
788
|
}
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
|
|
3
|
+
if (k2 === undefined) k2 = k;
|
|
4
|
+
var desc = Object.getOwnPropertyDescriptor(m, k);
|
|
5
|
+
if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
|
|
6
|
+
desc = { enumerable: true, get: function() { return m[k]; } };
|
|
7
|
+
}
|
|
8
|
+
Object.defineProperty(o, k2, desc);
|
|
9
|
+
}) : (function(o, m, k, k2) {
|
|
10
|
+
if (k2 === undefined) k2 = k;
|
|
11
|
+
o[k2] = m[k];
|
|
12
|
+
}));
|
|
13
|
+
var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
|
|
14
|
+
Object.defineProperty(o, "default", { enumerable: true, value: v });
|
|
15
|
+
}) : function(o, v) {
|
|
16
|
+
o["default"] = v;
|
|
17
|
+
});
|
|
18
|
+
var __importStar = (this && this.__importStar) || (function () {
|
|
19
|
+
var ownKeys = function(o) {
|
|
20
|
+
ownKeys = Object.getOwnPropertyNames || function (o) {
|
|
21
|
+
var ar = [];
|
|
22
|
+
for (var k in o) if (Object.prototype.hasOwnProperty.call(o, k)) ar[ar.length] = k;
|
|
23
|
+
return ar;
|
|
24
|
+
};
|
|
25
|
+
return ownKeys(o);
|
|
26
|
+
};
|
|
27
|
+
return function (mod) {
|
|
28
|
+
if (mod && mod.__esModule) return mod;
|
|
29
|
+
var result = {};
|
|
30
|
+
if (mod != null) for (var k = ownKeys(mod), i = 0; i < k.length; i++) if (k[i] !== "default") __createBinding(result, mod, k[i]);
|
|
31
|
+
__setModuleDefault(result, mod);
|
|
32
|
+
return result;
|
|
33
|
+
};
|
|
34
|
+
})();
|
|
35
|
+
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
|
|
36
|
+
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
|
|
37
|
+
return new (P || (P = Promise))(function (resolve, reject) {
|
|
38
|
+
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
|
|
39
|
+
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
|
|
40
|
+
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
|
|
41
|
+
step((generator = generator.apply(thisArg, _arguments || [])).next());
|
|
42
|
+
});
|
|
43
|
+
};
|
|
44
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
45
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
46
|
+
};
|
|
47
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
48
|
+
exports.nativeProjectIgnoreGlobs = nativeProjectIgnoreGlobs;
|
|
49
|
+
exports.readProjectWatcherIgnores = readProjectWatcherIgnores;
|
|
50
|
+
const fs = __importStar(require("node:fs/promises"));
|
|
51
|
+
const path = __importStar(require("node:path"));
|
|
52
|
+
const ignore_1 = __importDefault(require("ignore"));
|
|
53
|
+
/** Only the common positive gitignore/glob subset is safe to push into Parcel.
|
|
54
|
+
* Negations, escapes and unsupported syntax stay with ProjectFilePolicy, so a
|
|
55
|
+
* native ignore can never hide a file that an explicit re-inclusion needs. */
|
|
56
|
+
function nativeProjectIgnoreGlobs(content) {
|
|
57
|
+
const lines = content
|
|
58
|
+
.split("\n")
|
|
59
|
+
.map((line) => line.replace(/\r$/, "").replace(/ +$/, ""));
|
|
60
|
+
if (lines.some((line) => line.startsWith("!")))
|
|
61
|
+
return [];
|
|
62
|
+
const result = new Set();
|
|
63
|
+
for (const line of lines) {
|
|
64
|
+
if (!line || line.startsWith("#") || !/^[A-Za-z0-9_.*/-]+$/.test(line))
|
|
65
|
+
continue;
|
|
66
|
+
const pattern = line.replace(/^\//, "").replace(/\/$/, "");
|
|
67
|
+
if (!pattern ||
|
|
68
|
+
pattern.split("/").some((part) => part === "." || part === ".."))
|
|
69
|
+
continue;
|
|
70
|
+
// Keep root policy changes observable, including removal of a blanket rule.
|
|
71
|
+
const filter = (0, ignore_1.default)().add(line);
|
|
72
|
+
if (filter.ignores(".gmaxignore") || filter.ignores(".gitignore"))
|
|
73
|
+
continue;
|
|
74
|
+
if (pattern.split("/").some((part) => part.includes("**") && part !== "**"))
|
|
75
|
+
continue;
|
|
76
|
+
const glob = line.startsWith("/") || pattern.includes("/") ? pattern : `**/${pattern}`;
|
|
77
|
+
// A trailing slash ignores directories, never a same-named source file.
|
|
78
|
+
if (!line.endsWith("/"))
|
|
79
|
+
result.add(glob);
|
|
80
|
+
result.add(`${glob}/**`);
|
|
81
|
+
}
|
|
82
|
+
return [...result];
|
|
83
|
+
}
|
|
84
|
+
/** Root .gmaxignore rules are explicit project choices. Nested scopes and
|
|
85
|
+
* .gitignore remain authoritative in file policy, without lossy translation. */
|
|
86
|
+
function readProjectWatcherIgnores(root) {
|
|
87
|
+
return __awaiter(this, void 0, void 0, function* () {
|
|
88
|
+
try {
|
|
89
|
+
const file = path.join(root, ".gmaxignore");
|
|
90
|
+
const stat = yield fs.stat(file);
|
|
91
|
+
if (!stat.isFile() || stat.size > 65536)
|
|
92
|
+
return [];
|
|
93
|
+
const contents = yield fs.readFile(file, "utf8");
|
|
94
|
+
if (Buffer.byteLength(contents) > 65536)
|
|
95
|
+
return [];
|
|
96
|
+
return nativeProjectIgnoreGlobs(contents);
|
|
97
|
+
}
|
|
98
|
+
catch (_a) {
|
|
99
|
+
// File policy handles unreadable policy conservatively. Native watching
|
|
100
|
+
// must still deliver changes rather than treating a failed read as empty.
|
|
101
|
+
return [];
|
|
102
|
+
}
|
|
103
|
+
});
|
|
104
|
+
}
|
|
@@ -58,6 +58,7 @@ const cache_coherence_1 = require("./cache-coherence");
|
|
|
58
58
|
const file_policy_1 = require("./file-policy");
|
|
59
59
|
const ignore_patterns_1 = require("./ignore-patterns");
|
|
60
60
|
const walker_1 = require("./walker");
|
|
61
|
+
const watcher_ignore_1 = require("./watcher-ignore");
|
|
61
62
|
// Directory names ignored at ANY depth. @parcel/watcher treats a non-glob
|
|
62
63
|
// ignore entry as a literal path resolved against the watch root, so a bare
|
|
63
64
|
// "dist" only covered <root>/dist — nested build dirs (packages/*/dist,
|
|
@@ -121,6 +122,42 @@ function startWatcher(opts) {
|
|
|
121
122
|
let degradedRepairQueued = false;
|
|
122
123
|
const terminalFailures = new Set();
|
|
123
124
|
let processor;
|
|
125
|
+
let subscription = null;
|
|
126
|
+
let subscriptionUpdate = Promise.resolve();
|
|
127
|
+
let subscriptionPolicy = "";
|
|
128
|
+
const updateSubscription = () => {
|
|
129
|
+
const update = subscriptionUpdate
|
|
130
|
+
.catch(() => { })
|
|
131
|
+
.then(() => __awaiter(this, void 0, void 0, function* () {
|
|
132
|
+
const ignores = [
|
|
133
|
+
...exports.WATCHER_IGNORE_GLOBS,
|
|
134
|
+
...(yield (0, watcher_ignore_1.readProjectWatcherIgnores)(projectRoot)),
|
|
135
|
+
];
|
|
136
|
+
const policy = JSON.stringify(ignores);
|
|
137
|
+
if (closing || (subscription && subscriptionPolicy === policy))
|
|
138
|
+
return;
|
|
139
|
+
yield subscription === null || subscription === void 0 ? void 0 : subscription.unsubscribe();
|
|
140
|
+
subscription = null;
|
|
141
|
+
if (closing)
|
|
142
|
+
return;
|
|
143
|
+
const next = yield watcher.subscribe(projectRoot, (err, events) => {
|
|
144
|
+
if (err) {
|
|
145
|
+
console.error(`[${wtag}] Watcher error:`, err);
|
|
146
|
+
return;
|
|
147
|
+
}
|
|
148
|
+
for (const event of events)
|
|
149
|
+
processor.handleFileEvent(event.type === "delete" ? "unlink" : "change", event.path);
|
|
150
|
+
}, { ignore: ignores });
|
|
151
|
+
if (closing) {
|
|
152
|
+
yield next.unsubscribe();
|
|
153
|
+
return;
|
|
154
|
+
}
|
|
155
|
+
subscription = next;
|
|
156
|
+
subscriptionPolicy = policy;
|
|
157
|
+
}));
|
|
158
|
+
subscriptionUpdate = update;
|
|
159
|
+
return update;
|
|
160
|
+
};
|
|
124
161
|
const reportHealthyIfSettled = () => {
|
|
125
162
|
var _a;
|
|
126
163
|
if (scanHealthy &&
|
|
@@ -144,6 +181,9 @@ function startWatcher(opts) {
|
|
|
144
181
|
do {
|
|
145
182
|
reconcileRequested = false;
|
|
146
183
|
filePolicy.invalidateIgnoreCache();
|
|
184
|
+
yield updateSubscription();
|
|
185
|
+
if (closing)
|
|
186
|
+
return;
|
|
147
187
|
const rootPrefix = projectRoot.endsWith("/")
|
|
148
188
|
? projectRoot
|
|
149
189
|
: `${projectRoot}/`;
|
|
@@ -170,6 +210,7 @@ function startWatcher(opts) {
|
|
|
170
210
|
}
|
|
171
211
|
processor.handleFileEvent("change", absolute, {
|
|
172
212
|
forceReprocess: reconciliation.action === "reprocess",
|
|
213
|
+
workKind: "catchup",
|
|
173
214
|
});
|
|
174
215
|
}
|
|
175
216
|
}
|
|
@@ -187,6 +228,7 @@ function startWatcher(opts) {
|
|
|
187
228
|
!(0, walker_1.isPathProtectedByWalkState)(cachedPath, state)) {
|
|
188
229
|
processor.handleFileEvent("unlink", cachedPath, {
|
|
189
230
|
forceDelete: true,
|
|
231
|
+
workKind: "cleanup",
|
|
190
232
|
});
|
|
191
233
|
}
|
|
192
234
|
}
|
|
@@ -234,15 +276,7 @@ function startWatcher(opts) {
|
|
|
234
276
|
}
|
|
235
277
|
reportHealthyIfSettled();
|
|
236
278
|
} }));
|
|
237
|
-
|
|
238
|
-
if (err) {
|
|
239
|
-
console.error(`[${wtag}] Watcher error:`, err);
|
|
240
|
-
return;
|
|
241
|
-
}
|
|
242
|
-
for (const event of events) {
|
|
243
|
-
processor.handleFileEvent(event.type === "delete" ? "unlink" : "change", event.path);
|
|
244
|
-
}
|
|
245
|
-
}, { ignore: exports.WATCHER_IGNORE_GLOBS });
|
|
279
|
+
yield updateSubscription();
|
|
246
280
|
reconcile();
|
|
247
281
|
return {
|
|
248
282
|
get progress() {
|
|
@@ -250,7 +284,8 @@ function startWatcher(opts) {
|
|
|
250
284
|
},
|
|
251
285
|
close: () => __awaiter(this, void 0, void 0, function* () {
|
|
252
286
|
closing = true;
|
|
253
|
-
yield
|
|
287
|
+
yield subscriptionUpdate.catch(() => { });
|
|
288
|
+
yield subscription === null || subscription === void 0 ? void 0 : subscription.unsubscribe();
|
|
254
289
|
yield reconciliation;
|
|
255
290
|
yield processor.close();
|
|
256
291
|
}),
|
|
@@ -6,7 +6,14 @@
|
|
|
6
6
|
// historically did not. This formats a single machine-readable footer so an
|
|
7
7
|
// agent can decide to caveat its answer or retry once indexing settles.
|
|
8
8
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
9
|
+
exports.formatWatchQueue = formatWatchQueue;
|
|
9
10
|
exports.formatIndexStateFooter = formatIndexStateFooter;
|
|
11
|
+
function formatWatchQueue(queue) {
|
|
12
|
+
const age = queue.oldestLiveEditAgeMs === null
|
|
13
|
+
? "none"
|
|
14
|
+
: `${Math.ceil(queue.oldestLiveEditAgeMs / 1000)}s`;
|
|
15
|
+
return `live=${queue.live} catchup=${queue.catchup} cleanup=${queue.cleanup} active=${queue.activeFiles} oldestLiveEdit=${age}`;
|
|
16
|
+
}
|
|
10
17
|
/**
|
|
11
18
|
* One-line footer describing an in-progress index, or null when there's
|
|
12
19
|
* nothing to say (no state, or the index is settled). Suppressing the
|
|
@@ -111,6 +111,15 @@ exports.MCP_READ_OUTPUT_SCHEMAS = {
|
|
|
111
111
|
.object({
|
|
112
112
|
indexing: zod_1.z.boolean(),
|
|
113
113
|
pendingFiles: zod_1.z.number(),
|
|
114
|
+
queue: zod_1.z
|
|
115
|
+
.object({
|
|
116
|
+
live: zod_1.z.number().int().nonnegative(),
|
|
117
|
+
catchup: zod_1.z.number().int().nonnegative(),
|
|
118
|
+
cleanup: zod_1.z.number().int().nonnegative(),
|
|
119
|
+
activeFiles: zod_1.z.number().int().nonnegative(),
|
|
120
|
+
oldestLiveEditAgeMs: zod_1.z.number().nonnegative().nullable(),
|
|
121
|
+
})
|
|
122
|
+
.optional(),
|
|
114
123
|
verifying: zod_1.z.boolean().optional(),
|
|
115
124
|
failedFiles: zod_1.z.number().optional(),
|
|
116
125
|
degraded: zod_1.z.boolean().optional(),
|
package/package.json
CHANGED