@soulcraft/cor 3.0.5 → 3.0.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -72,6 +72,18 @@ export declare class GraphAdjacencyIndex implements GraphIndexProvider {
|
|
|
72
72
|
private unifiedCache;
|
|
73
73
|
private config;
|
|
74
74
|
private initialized;
|
|
75
|
+
/**
|
|
76
|
+
* Canonical storage holds verbs but NO durable graph state exists — the
|
|
77
|
+
* 7.x → 3.0 migration / restore-dir shape. Detected once per cold open in
|
|
78
|
+
* {@link ensureInitialized} by a bounded O(1) canonical probe; suppresses
|
|
79
|
+
* the edgeless-graph ready verdict in {@link isReady} so brainy's cold-open
|
|
80
|
+
* gate runs the one rebuild that reconstructs the adjacency (without this, a
|
|
81
|
+
* migrated brain served empty neighbors()/related() with no error — the
|
|
82
|
+
* graph twin of the vector leg's silent-empty migration bug). Cleared by a
|
|
83
|
+
* completed {@link rebuild}, whose durable output also makes later boots
|
|
84
|
+
* skip the probe.
|
|
85
|
+
*/
|
|
86
|
+
private canonicalUnindexed;
|
|
75
87
|
private isRebuilding;
|
|
76
88
|
private flushTimer?;
|
|
77
89
|
private rebuildStartTime;
|
|
@@ -129,6 +141,19 @@ export declare class GraphAdjacencyIndex implements GraphIndexProvider {
|
|
|
129
141
|
* graph on first touch.
|
|
130
142
|
*/
|
|
131
143
|
private ensureInitialized;
|
|
144
|
+
/**
|
|
145
|
+
* Detect the canonical-but-unindexed cold-open shape: storage holds verbs
|
|
146
|
+
* while NO durable graph state loaded (edges-ready latch unset, zero
|
|
147
|
+
* resident relationships) — a 7.x → 3.0 migration dir or a restore that
|
|
148
|
+
* never built the graph LSM. The edgeless-ready rule (BR-8-BOOT-INDEX) is
|
|
149
|
+
* correct only when there are genuinely no relationships anywhere; here a
|
|
150
|
+
* rebuild is NOT redundant, and reporting ready serves empty adjacency.
|
|
151
|
+
* Bounded O(1): one single-verb canonical page, only on the nothing-durable
|
|
152
|
+
* path — after the gate-triggered rebuild persists the trees, later boots
|
|
153
|
+
* latch edges-ready and never reach the probe. Truly edgeless brains (no
|
|
154
|
+
* canonical verbs) stay ready, so the boot-rebuild/502 class stays dead.
|
|
155
|
+
*/
|
|
156
|
+
private probeCanonicalCoverage;
|
|
132
157
|
/**
|
|
133
158
|
* Cold-load the four u64 LSM trees from persistence: load each tree's
|
|
134
159
|
* manifest and register every persisted SSTable (mmap tier, then binary
|
|
@@ -217,12 +242,21 @@ export declare class GraphAdjacencyIndex implements GraphIndexProvider {
|
|
|
217
242
|
isHealthy(): boolean;
|
|
218
243
|
/**
|
|
219
244
|
* Whether the graph's edges are loaded and queryable WITHOUT a rebuild —
|
|
220
|
-
* brainy 8.0's rebuild gate (#36).
|
|
221
|
-
*
|
|
222
|
-
*
|
|
223
|
-
*
|
|
224
|
-
*
|
|
225
|
-
*
|
|
245
|
+
* brainy 8.0's rebuild gate (#36). Synchronous (no await) and **never gated
|
|
246
|
+
* on the verb-id membership walk**, so the deferred O(N) membership fast-follow
|
|
247
|
+
* can never depress readiness for a graph whose edges are already mapped in.
|
|
248
|
+
*
|
|
249
|
+
* The native `is_ready()` latch fires only when EDGES load (SSTable cold-load,
|
|
250
|
+
* a non-empty manifest, or a memtable flush). That latch is never set for an
|
|
251
|
+
* **edgeless** brain (entities but no relationships — a very common shape:
|
|
252
|
+
* per-user brains, catalogs, CMS content), so a naive delegate reported
|
|
253
|
+
* `false` and made brainy spuriously rebuild the graph on every boot — which
|
|
254
|
+
* cascades into an id-mapper hydration + a full metadata rebuild (the observed
|
|
255
|
+
* multi-minute boot / 502 window). An edgeless graph is trivially ready: a
|
|
256
|
+
* rebuild would reconstruct zero edges. So once cold-load has run
|
|
257
|
+
* ({@link initialized}) we also report ready when the durable graph stores are
|
|
258
|
+
* genuinely EMPTY — distinguished from "edges exist on disk but didn't load"
|
|
259
|
+
* (which must still rebuild) via the persisted SSTable + memtable counts.
|
|
226
260
|
*/
|
|
227
261
|
isReady(): boolean;
|
|
228
262
|
/**
|
|
@@ -76,6 +76,18 @@ export class GraphAdjacencyIndex {
|
|
|
76
76
|
unifiedCache;
|
|
77
77
|
config;
|
|
78
78
|
initialized = false;
|
|
79
|
+
/**
|
|
80
|
+
* Canonical storage holds verbs but NO durable graph state exists — the
|
|
81
|
+
* 7.x → 3.0 migration / restore-dir shape. Detected once per cold open in
|
|
82
|
+
* {@link ensureInitialized} by a bounded O(1) canonical probe; suppresses
|
|
83
|
+
* the edgeless-graph ready verdict in {@link isReady} so brainy's cold-open
|
|
84
|
+
* gate runs the one rebuild that reconstructs the adjacency (without this, a
|
|
85
|
+
* migrated brain served empty neighbors()/related() with no error — the
|
|
86
|
+
* graph twin of the vector leg's silent-empty migration bug). Cleared by a
|
|
87
|
+
* completed {@link rebuild}, whose durable output also makes later boots
|
|
88
|
+
* skip the probe.
|
|
89
|
+
*/
|
|
90
|
+
canonicalUnindexed = false;
|
|
79
91
|
isRebuilding = false;
|
|
80
92
|
flushTimer;
|
|
81
93
|
rebuildStartTime = 0;
|
|
@@ -218,9 +230,41 @@ export class GraphAdjacencyIndex {
|
|
|
218
230
|
return;
|
|
219
231
|
}
|
|
220
232
|
await this.openGraphLsmEngine();
|
|
233
|
+
await this.probeCanonicalCoverage();
|
|
221
234
|
this.startAutoFlush();
|
|
222
235
|
this.initialized = true;
|
|
223
236
|
}
|
|
237
|
+
/**
|
|
238
|
+
* Detect the canonical-but-unindexed cold-open shape: storage holds verbs
|
|
239
|
+
* while NO durable graph state loaded (edges-ready latch unset, zero
|
|
240
|
+
* resident relationships) — a 7.x → 3.0 migration dir or a restore that
|
|
241
|
+
* never built the graph LSM. The edgeless-ready rule (BR-8-BOOT-INDEX) is
|
|
242
|
+
* correct only when there are genuinely no relationships anywhere; here a
|
|
243
|
+
* rebuild is NOT redundant, and reporting ready serves empty adjacency.
|
|
244
|
+
* Bounded O(1): one single-verb canonical page, only on the nothing-durable
|
|
245
|
+
* path — after the gate-triggered rebuild persists the trees, later boots
|
|
246
|
+
* latch edges-ready and never reach the probe. Truly edgeless brains (no
|
|
247
|
+
* canonical verbs) stay ready, so the boot-rebuild/502 class stays dead.
|
|
248
|
+
*/
|
|
249
|
+
async probeCanonicalCoverage() {
|
|
250
|
+
if (this.native.isReady() || this.native.size() > 0)
|
|
251
|
+
return;
|
|
252
|
+
if (typeof this.storage?.getVerbs !== 'function')
|
|
253
|
+
return;
|
|
254
|
+
try {
|
|
255
|
+
const page = await this.storage.getVerbs({ pagination: { limit: 1 } });
|
|
256
|
+
if ((page?.items ?? []).length > 0) {
|
|
257
|
+
this.canonicalUnindexed = true;
|
|
258
|
+
prodLog.warn('GraphAdjacencyIndex: canonical storage holds verbs but no durable ' +
|
|
259
|
+
'graph state exists (migration/restore dir) — reporting not-ready ' +
|
|
260
|
+
'so the cold-open gate rebuilds the adjacency once from canonical');
|
|
261
|
+
}
|
|
262
|
+
}
|
|
263
|
+
catch {
|
|
264
|
+
// Probe is advisory: an unreadable canonical page must not block open.
|
|
265
|
+
// Worst case we report ready and behave as before the probe existed.
|
|
266
|
+
}
|
|
267
|
+
}
|
|
224
268
|
/**
|
|
225
269
|
* Cold-load the four u64 LSM trees from persistence: load each tree's
|
|
226
270
|
* manifest and register every persisted SSTable (mmap tier, then binary
|
|
@@ -497,15 +541,42 @@ export class GraphAdjacencyIndex {
|
|
|
497
541
|
}
|
|
498
542
|
/**
|
|
499
543
|
* Whether the graph's edges are loaded and queryable WITHOUT a rebuild —
|
|
500
|
-
* brainy 8.0's rebuild gate (#36).
|
|
501
|
-
*
|
|
502
|
-
*
|
|
503
|
-
*
|
|
504
|
-
*
|
|
505
|
-
*
|
|
544
|
+
* brainy 8.0's rebuild gate (#36). Synchronous (no await) and **never gated
|
|
545
|
+
* on the verb-id membership walk**, so the deferred O(N) membership fast-follow
|
|
546
|
+
* can never depress readiness for a graph whose edges are already mapped in.
|
|
547
|
+
*
|
|
548
|
+
* The native `is_ready()` latch fires only when EDGES load (SSTable cold-load,
|
|
549
|
+
* a non-empty manifest, or a memtable flush). That latch is never set for an
|
|
550
|
+
* **edgeless** brain (entities but no relationships — a very common shape:
|
|
551
|
+
* per-user brains, catalogs, CMS content), so a naive delegate reported
|
|
552
|
+
* `false` and made brainy spuriously rebuild the graph on every boot — which
|
|
553
|
+
* cascades into an id-mapper hydration + a full metadata rebuild (the observed
|
|
554
|
+
* multi-minute boot / 502 window). An edgeless graph is trivially ready: a
|
|
555
|
+
* rebuild would reconstruct zero edges. So once cold-load has run
|
|
556
|
+
* ({@link initialized}) we also report ready when the durable graph stores are
|
|
557
|
+
* genuinely EMPTY — distinguished from "edges exist on disk but didn't load"
|
|
558
|
+
* (which must still rebuild) via the persisted SSTable + memtable counts.
|
|
506
559
|
*/
|
|
507
560
|
isReady() {
|
|
508
|
-
|
|
561
|
+
if (this.native.isReady())
|
|
562
|
+
return true;
|
|
563
|
+
if (!this.initialized)
|
|
564
|
+
return false;
|
|
565
|
+
// Canonical verbs exist with no durable graph state (migration/restore
|
|
566
|
+
// dir): a rebuild is NOT redundant — see {@link probeCanonicalCoverage}.
|
|
567
|
+
if (this.canonicalUnindexed)
|
|
568
|
+
return false;
|
|
569
|
+
// Cold-load has run but the edges-ready latch is unset. Ready ⇔ there is
|
|
570
|
+
// nothing durable to load: no SSTables and empty memtables on either the
|
|
571
|
+
// source or target tree. If any durable edges exist but didn't load, this
|
|
572
|
+
// is false and brainy correctly rebuilds.
|
|
573
|
+
const raw = this.native.getStats();
|
|
574
|
+
const s = typeof raw === 'string' ? JSON.parse(raw) : raw;
|
|
575
|
+
const durableEmpty = Number(s?.sourceSstableCount ?? 0) === 0 &&
|
|
576
|
+
Number(s?.targetSstableCount ?? 0) === 0 &&
|
|
577
|
+
Number(s?.sourceMemTableSize ?? 0) === 0 &&
|
|
578
|
+
Number(s?.targetMemTableSize ?? 0) === 0;
|
|
579
|
+
return durableEmpty;
|
|
509
580
|
}
|
|
510
581
|
/**
|
|
511
582
|
* @description **#18 migration lock — feature-detected by Brainy.** `true` while
|
|
@@ -644,6 +715,9 @@ export class GraphAdjacencyIndex {
|
|
|
644
715
|
}
|
|
645
716
|
const rebuildTime = Date.now() - this.rebuildStartTime;
|
|
646
717
|
const memoryUsage = this.calculateMemoryUsage();
|
|
718
|
+
// The rebuild reconstructed the adjacency from canonical — the cold-open
|
|
719
|
+
// not-ready verdict (migration/restore dir) is satisfied.
|
|
720
|
+
this.canonicalUnindexed = false;
|
|
647
721
|
prodLog.info(`GraphAdjacencyIndex: Rebuild complete in ${rebuildTime}ms`);
|
|
648
722
|
prodLog.info(` - Total verbs tracked: ${totalVerbs.toLocaleString()}`);
|
|
649
723
|
if (reconstructAdjacency) {
|
|
@@ -209,6 +209,17 @@ export declare class NativeDiskAnnWrapper implements VectorIndexProvider {
|
|
|
209
209
|
private manifestNextId;
|
|
210
210
|
/** Segment-manifest cold-load memo (see {@link ensureSegments}). */
|
|
211
211
|
private segmentsLoaded;
|
|
212
|
+
/**
|
|
213
|
+
* Canonical storage holds vectored nouns but NO durable vector index exists —
|
|
214
|
+
* the 7.x → 3.0 migration / restore-dir shape. Detected once per cold open by
|
|
215
|
+
* {@link probeCanonicalCoverage}; makes {@link isReady} report `false` so
|
|
216
|
+
* brainy's cold-open gate runs the one rebuild that indexes the canonical
|
|
217
|
+
* vectors (without this, a migrated brain served silent-empty vector search —
|
|
218
|
+
* the gate-305/306 `indexEpochMigration` failure). Cleared by a completed
|
|
219
|
+
* {@link runRebuild}, whose durable output also makes every later boot skip
|
|
220
|
+
* the probe entirely.
|
|
221
|
+
*/
|
|
222
|
+
private canonicalUnindexed;
|
|
212
223
|
private segmentsLoading;
|
|
213
224
|
/**
|
|
214
225
|
* True when a durable base index was present on disk but failed to load
|
|
@@ -289,6 +300,14 @@ export declare class NativeDiskAnnWrapper implements VectorIndexProvider {
|
|
|
289
300
|
* it loaded so an operator can see the reconstruction happen.
|
|
290
301
|
*
|
|
291
302
|
* @param dim - Index dimension; nouns whose vector width differs are skipped.
|
|
303
|
+
* Optional: when absent (a zero-config brain with no base index or L0
|
|
304
|
+
* segment to read it from), the width is adopted from the first canonical
|
|
305
|
+
* vector — every vector in a brain shares one dim (brainy enforces the
|
|
306
|
+
* embedder's). Passing `undefined` previously skipped EVERY vector
|
|
307
|
+
* (`length !== undefined` is always true), silently rebuilding an empty
|
|
308
|
+
* index on a zero-config restore.
|
|
309
|
+
* @returns The dimension actually used — the passed `dim`, or the width
|
|
310
|
+
* adopted from canonical data, or `undefined` if nothing was read.
|
|
292
311
|
*/
|
|
293
312
|
private hydrateDeltaFromStorage;
|
|
294
313
|
search(queryVector: Vector, k?: number, filter?: (id: string) => Promise<boolean>, options?: {
|
|
@@ -432,8 +451,24 @@ export declare class NativeDiskAnnWrapper implements VectorIndexProvider {
|
|
|
432
451
|
* still gets the segments lazily; only the pre-gate timing needs init().)
|
|
433
452
|
*/
|
|
434
453
|
init(): Promise<void>;
|
|
435
|
-
/** Memoized segment-manifest cold-load. */
|
|
454
|
+
/** Memoized segment-manifest cold-load (+ canonical-coverage probe). */
|
|
436
455
|
private ensureSegments;
|
|
456
|
+
/**
|
|
457
|
+
* Detect the canonical-but-unindexed cold-open shape: storage holds vectored
|
|
458
|
+
* nouns while NO durable vector artifact exists (no base index, no L0
|
|
459
|
+
* segment, empty delta) — a 7.x → 3.0 migration dir or a restore that never
|
|
460
|
+
* built the index. The readiness contract's "fresh brain is ready" rule is
|
|
461
|
+
* correct only when there is genuinely nothing to index; here a rebuild is
|
|
462
|
+
* NOT redundant, and reporting ready serves silent-empty vector search.
|
|
463
|
+
*
|
|
464
|
+
* Bounded O(32), never an O(N) walk (the 48s-per-boot mandate): one small
|
|
465
|
+
* canonical page, checked for any vectored noun. Runs only when the brain
|
|
466
|
+
* has zero durable vector state — after the gate-triggered rebuild persists
|
|
467
|
+
* its output, every later boot sees the durable base and skips the probe.
|
|
468
|
+
* Metadata-only brains (no vectors in the probe page) stay ready, so the
|
|
469
|
+
* boot-rebuild class this contract was built to kill stays dead.
|
|
470
|
+
*/
|
|
471
|
+
private probeCanonicalCoverage;
|
|
437
472
|
/**
|
|
438
473
|
* Read `__vector_index_manifest__` and open every live L0 segment. A missing
|
|
439
474
|
* manifest is the legacy single-index layout (the base at `indexPath` IS the
|
|
@@ -311,6 +311,17 @@ export class NativeDiskAnnWrapper {
|
|
|
311
311
|
manifestNextId = 1;
|
|
312
312
|
/** Segment-manifest cold-load memo (see {@link ensureSegments}). */
|
|
313
313
|
segmentsLoaded = false;
|
|
314
|
+
/**
|
|
315
|
+
* Canonical storage holds vectored nouns but NO durable vector index exists —
|
|
316
|
+
* the 7.x → 3.0 migration / restore-dir shape. Detected once per cold open by
|
|
317
|
+
* {@link probeCanonicalCoverage}; makes {@link isReady} report `false` so
|
|
318
|
+
* brainy's cold-open gate runs the one rebuild that indexes the canonical
|
|
319
|
+
* vectors (without this, a migrated brain served silent-empty vector search —
|
|
320
|
+
* the gate-305/306 `indexEpochMigration` failure). Cleared by a completed
|
|
321
|
+
* {@link runRebuild}, whose durable output also makes every later boot skip
|
|
322
|
+
* the probe entirely.
|
|
323
|
+
*/
|
|
324
|
+
canonicalUnindexed = false;
|
|
314
325
|
segmentsLoading = null;
|
|
315
326
|
/**
|
|
316
327
|
* True when a durable base index was present on disk but failed to load
|
|
@@ -398,7 +409,12 @@ export class NativeDiskAnnWrapper {
|
|
|
398
409
|
* `rebuild()` call, which folds the delta into the main index.
|
|
399
410
|
*/
|
|
400
411
|
async addItem(item) {
|
|
401
|
-
|
|
412
|
+
// item.vector.length is the authoritative width for THIS write; fall back to
|
|
413
|
+
// it when the brain is zero-config (config.dimensions undefined). Otherwise
|
|
414
|
+
// bytesPerVector is NaN and every `NaN >= cap` backpressure/fold check is
|
|
415
|
+
// false — the delta grows unbounded and never folds (MEMORY-COR-RANGE-ERROR
|
|
416
|
+
// family: the same undefined-dim that broke consolidation).
|
|
417
|
+
const bytesPerVector = (this.config.dimensions ?? item.vector.length) * 4;
|
|
402
418
|
// HARD cap — backpressure: the delta may not grow unbounded. Wait for
|
|
403
419
|
// the in-flight fold (or start one) before accepting the write. Serving
|
|
404
420
|
// is unaffected — the fold runs off the JS thread and reads keep
|
|
@@ -409,7 +425,19 @@ export class NativeDiskAnnWrapper {
|
|
|
409
425
|
// safe (the rebuild folds only its snapshot; the new segment survives
|
|
410
426
|
// its retirement filter and the serialized manifest saves keep the
|
|
411
427
|
// on-disk list consistent).
|
|
412
|
-
|
|
428
|
+
//
|
|
429
|
+
// A committed write must NEVER fail on post-commit index maintenance — the
|
|
430
|
+
// caller would retry and double-write (MEMORY-COR-RANGE-ERROR isolation
|
|
431
|
+
// ask). A backpressure-flush failure is a loud alarm, not a rejected
|
|
432
|
+
// write: log it, keep the delta for the next trigger, accept the write.
|
|
433
|
+
try {
|
|
434
|
+
await (this.flushInFlight ?? this.flushDeltaToL0());
|
|
435
|
+
}
|
|
436
|
+
catch (error) {
|
|
437
|
+
prodLog.error(`NativeDiskAnnWrapper: backpressure flush failed (write accepted; ` +
|
|
438
|
+
`delta retained for retry): ` +
|
|
439
|
+
`${error instanceof Error ? error.message : String(error)}`);
|
|
440
|
+
}
|
|
413
441
|
}
|
|
414
442
|
if (this.tombstones.has(item.id)) {
|
|
415
443
|
this.tombstones.delete(item.id);
|
|
@@ -475,6 +503,14 @@ export class NativeDiskAnnWrapper {
|
|
|
475
503
|
* it loaded so an operator can see the reconstruction happen.
|
|
476
504
|
*
|
|
477
505
|
* @param dim - Index dimension; nouns whose vector width differs are skipped.
|
|
506
|
+
* Optional: when absent (a zero-config brain with no base index or L0
|
|
507
|
+
* segment to read it from), the width is adopted from the first canonical
|
|
508
|
+
* vector — every vector in a brain shares one dim (brainy enforces the
|
|
509
|
+
* embedder's). Passing `undefined` previously skipped EVERY vector
|
|
510
|
+
* (`length !== undefined` is always true), silently rebuilding an empty
|
|
511
|
+
* index on a zero-config restore.
|
|
512
|
+
* @returns The dimension actually used — the passed `dim`, or the width
|
|
513
|
+
* adopted from canonical data, or `undefined` if nothing was read.
|
|
478
514
|
*/
|
|
479
515
|
async hydrateDeltaFromStorage(dim) {
|
|
480
516
|
// Best-effort: only when the storage adapter supports bulk noun reads
|
|
@@ -482,7 +518,8 @@ export class NativeDiskAnnWrapper {
|
|
|
482
518
|
// adapters used in direct-construction tests do not). When absent, the
|
|
483
519
|
// caller falls back to the in-memory rebuild.
|
|
484
520
|
if (typeof this.storage?.getNouns !== 'function')
|
|
485
|
-
return;
|
|
521
|
+
return dim;
|
|
522
|
+
let resolved = dim;
|
|
486
523
|
let cursor = undefined;
|
|
487
524
|
let hasMore = true;
|
|
488
525
|
let loaded = 0;
|
|
@@ -498,7 +535,13 @@ export class NativeDiskAnnWrapper {
|
|
|
498
535
|
if (this.tombstones.has(noun.id))
|
|
499
536
|
continue;
|
|
500
537
|
const vector = noun.vector;
|
|
501
|
-
if (!vector
|
|
538
|
+
if (!vector)
|
|
539
|
+
continue;
|
|
540
|
+
// Adopt the width from the first canonical vector when the caller
|
|
541
|
+
// couldn't supply it; thereafter enforce it (one dim per brain).
|
|
542
|
+
if (resolved === undefined)
|
|
543
|
+
resolved = vector.length;
|
|
544
|
+
if (vector.length !== resolved)
|
|
502
545
|
continue;
|
|
503
546
|
this.delta.set(noun.id, vector);
|
|
504
547
|
loaded++;
|
|
@@ -510,6 +553,7 @@ export class NativeDiskAnnWrapper {
|
|
|
510
553
|
prodLog?.info?.(`NativeDiskAnnWrapper: cold rebuild hydrated ${loaded} vectors from ` +
|
|
511
554
|
`canonical storage (#44 migration/restore path)`);
|
|
512
555
|
}
|
|
556
|
+
return resolved;
|
|
513
557
|
}
|
|
514
558
|
async search(queryVector, k = 10, filter, options) {
|
|
515
559
|
// Predicate pushdown: a SELECTIVE metadata∩graph universe is evaluated
|
|
@@ -743,6 +787,10 @@ export class NativeDiskAnnWrapper {
|
|
|
743
787
|
isReady() {
|
|
744
788
|
if (this.durableBaseLoadFailed)
|
|
745
789
|
return false;
|
|
790
|
+
// Canonical vectors exist with no durable index (migration/restore dir):
|
|
791
|
+
// a rebuild is NOT redundant — see {@link probeCanonicalCoverage}.
|
|
792
|
+
if (this.canonicalUnindexed)
|
|
793
|
+
return false;
|
|
746
794
|
// init() (or any prior async op) has attached the segment set. Before that,
|
|
747
795
|
// report not-ready so the gate never skips over an unloaded index; brainy's
|
|
748
796
|
// contract calls isReady() only after init(), where this is true.
|
|
@@ -978,15 +1026,36 @@ export class NativeDiskAnnWrapper {
|
|
|
978
1026
|
// (non-empty `liveOldSlots`) both skip it, so the hot paths pay
|
|
979
1027
|
// nothing. `hydrateDeltaFromStorage` only adds ids absent from the
|
|
980
1028
|
// delta, so any straggler already fed by restore is preserved.
|
|
1029
|
+
// Resolve the vector dimension robustly — the SAME fallback flushDeltaToL0
|
|
1030
|
+
// uses. brainy's zero-config vector factory does not always pass
|
|
1031
|
+
// config.dimensions; consolidation must still resolve the width from an
|
|
1032
|
+
// authoritative source, or it throws "≠ index dim undefined" and corrupts
|
|
1033
|
+
// the fold buffers with a NaN stride on every zero-config consolidation
|
|
1034
|
+
// (MEMORY-COR-RANGE-ERROR — the flush path was fixed for this, rebuild was
|
|
1035
|
+
// not). Priority: declared config → attached base index header → any live
|
|
1036
|
+
// L0 segment header → (below) cold-hydrated canonical data → the delta's
|
|
1037
|
+
// own vectors.
|
|
1038
|
+
let dim = this.config.dimensions ??
|
|
1039
|
+
(this.native ? this.native.header().dim : undefined) ??
|
|
1040
|
+
builtL0.find((s) => s.nodeCount > 0)?.native.header().dim;
|
|
981
1041
|
if (liveOldSlots.length === 0 && typeof this.storage?.getNouns === 'function') {
|
|
982
1042
|
const probe = await this.storage.getNouns({ pagination: { limit: 1 } });
|
|
983
1043
|
if (probe.totalCount === undefined || probe.totalCount > this.delta.size) {
|
|
984
|
-
|
|
1044
|
+
// Cold/restore rebuild: hydrate also adopts the dim from canonical data
|
|
1045
|
+
// when we still don't have it, and returns the width it used.
|
|
1046
|
+
dim = await this.hydrateDeltaFromStorage(dim);
|
|
985
1047
|
}
|
|
986
1048
|
}
|
|
987
1049
|
// Delta snapshot AFTER the cold-hydrate (which populates this.delta).
|
|
988
1050
|
const builtDelta = new Map(this.delta);
|
|
989
|
-
|
|
1051
|
+
// Final fallback: the delta's own vectors (an addItem-only zero-config brain
|
|
1052
|
+
// with no base index yet), mirroring flushDeltaToL0.
|
|
1053
|
+
dim ??= builtDelta.values().next().value?.length;
|
|
1054
|
+
if (dim === undefined) {
|
|
1055
|
+
// Nothing to fold and no source to learn the width from → empty rebuild.
|
|
1056
|
+
prodLog?.warn?.('NativeDiskAnnWrapper.rebuild: nothing to build (no vectors, no resolvable dimension)');
|
|
1057
|
+
return;
|
|
1058
|
+
}
|
|
990
1059
|
const deltaCount = builtDelta.size;
|
|
991
1060
|
let deltaBuf = null;
|
|
992
1061
|
const deltaInts = [];
|
|
@@ -1027,6 +1096,9 @@ export class NativeDiskAnnWrapper {
|
|
|
1027
1096
|
}
|
|
1028
1097
|
if (liveOldSlots.length + deltaCount + l0Entries.length === 0) {
|
|
1029
1098
|
prodLog?.warn?.('NativeDiskAnnWrapper.rebuild: nothing to build');
|
|
1099
|
+
// The rebuild ran over canonical and found nothing to index — its verdict
|
|
1100
|
+
// supersedes the cold-open probe's; never leave the gate looping.
|
|
1101
|
+
this.canonicalUnindexed = false;
|
|
1030
1102
|
return;
|
|
1031
1103
|
}
|
|
1032
1104
|
const totalCount = liveOldSlots.length + deltaCount + l0Entries.length;
|
|
@@ -1123,6 +1195,9 @@ export class NativeDiskAnnWrapper {
|
|
|
1123
1195
|
}
|
|
1124
1196
|
}
|
|
1125
1197
|
}
|
|
1198
|
+
// The rebuild's durable output now covers canonical — the cold-open
|
|
1199
|
+
// not-ready verdict (migration/restore dir) is satisfied.
|
|
1200
|
+
this.canonicalUnindexed = false;
|
|
1126
1201
|
}
|
|
1127
1202
|
/**
|
|
1128
1203
|
* Flush the delta buffer to disk. For DiskANN the delta is in-memory
|
|
@@ -1191,18 +1266,58 @@ export class NativeDiskAnnWrapper {
|
|
|
1191
1266
|
async init() {
|
|
1192
1267
|
await this.ensureSegments();
|
|
1193
1268
|
}
|
|
1194
|
-
/** Memoized segment-manifest cold-load. */
|
|
1269
|
+
/** Memoized segment-manifest cold-load (+ canonical-coverage probe). */
|
|
1195
1270
|
async ensureSegments() {
|
|
1196
1271
|
if (this.segmentsLoaded)
|
|
1197
1272
|
return;
|
|
1198
1273
|
if (this.segmentsLoading)
|
|
1199
1274
|
return this.segmentsLoading;
|
|
1200
|
-
this.segmentsLoading = this.loadSegmentManifest()
|
|
1275
|
+
this.segmentsLoading = this.loadSegmentManifest()
|
|
1276
|
+
.then(() => this.probeCanonicalCoverage())
|
|
1277
|
+
.finally(() => {
|
|
1201
1278
|
this.segmentsLoaded = true;
|
|
1202
1279
|
this.segmentsLoading = null;
|
|
1203
1280
|
});
|
|
1204
1281
|
return this.segmentsLoading;
|
|
1205
1282
|
}
|
|
1283
|
+
/**
|
|
1284
|
+
* Detect the canonical-but-unindexed cold-open shape: storage holds vectored
|
|
1285
|
+
* nouns while NO durable vector artifact exists (no base index, no L0
|
|
1286
|
+
* segment, empty delta) — a 7.x → 3.0 migration dir or a restore that never
|
|
1287
|
+
* built the index. The readiness contract's "fresh brain is ready" rule is
|
|
1288
|
+
* correct only when there is genuinely nothing to index; here a rebuild is
|
|
1289
|
+
* NOT redundant, and reporting ready serves silent-empty vector search.
|
|
1290
|
+
*
|
|
1291
|
+
* Bounded O(32), never an O(N) walk (the 48s-per-boot mandate): one small
|
|
1292
|
+
* canonical page, checked for any vectored noun. Runs only when the brain
|
|
1293
|
+
* has zero durable vector state — after the gate-triggered rebuild persists
|
|
1294
|
+
* its output, every later boot sees the durable base and skips the probe.
|
|
1295
|
+
* Metadata-only brains (no vectors in the probe page) stay ready, so the
|
|
1296
|
+
* boot-rebuild class this contract was built to kill stays dead.
|
|
1297
|
+
*/
|
|
1298
|
+
async probeCanonicalCoverage() {
|
|
1299
|
+
if (this.native || this.l0Segments.length > 0 || this.delta.size > 0)
|
|
1300
|
+
return;
|
|
1301
|
+
if (this.durableBaseLoadFailed)
|
|
1302
|
+
return; // already not-ready, loudly
|
|
1303
|
+
const storage = this.storage;
|
|
1304
|
+
if (!storage || typeof storage.getNouns !== 'function')
|
|
1305
|
+
return;
|
|
1306
|
+
try {
|
|
1307
|
+
const page = await storage.getNouns({ pagination: { limit: 32 } });
|
|
1308
|
+
const vectored = (page?.items ?? []).some((n) => Array.isArray(n.vector) && n.vector.length > 0);
|
|
1309
|
+
if (vectored) {
|
|
1310
|
+
this.canonicalUnindexed = true;
|
|
1311
|
+
prodLog?.warn?.('NativeDiskAnnWrapper: canonical storage holds vectors but no durable ' +
|
|
1312
|
+
'vector index exists (migration/restore dir) — reporting not-ready so ' +
|
|
1313
|
+
'the cold-open gate rebuilds once from canonical');
|
|
1314
|
+
}
|
|
1315
|
+
}
|
|
1316
|
+
catch {
|
|
1317
|
+
// Probe is advisory: an unreadable canonical page must not block open.
|
|
1318
|
+
// Worst case we report ready and behave as before the probe existed.
|
|
1319
|
+
}
|
|
1320
|
+
}
|
|
1206
1321
|
/**
|
|
1207
1322
|
* Read `__vector_index_manifest__` and open every live L0 segment. A missing
|
|
1208
1323
|
* manifest is the legacy single-index layout (the base at `indexPath` IS the
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@soulcraft/cor",
|
|
3
|
-
"version": "3.0.
|
|
4
|
-
"description": "Native Rust acceleration for Brainy
|
|
3
|
+
"version": "3.0.7",
|
|
4
|
+
"description": "Native Rust acceleration for Brainy — SIMD distance, vector quantization, zero-copy mmap, native embeddings. Free tier for storage, Pro license for compute acceleration.",
|
|
5
5
|
"main": "dist/index.js",
|
|
6
6
|
"types": "dist/index.d.ts",
|
|
7
7
|
"type": "module",
|