@soulcraft/cor 3.0.3 → 3.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/hnsw/NativeDiskAnnWrapper.d.ts +38 -4
- package/dist/hnsw/NativeDiskAnnWrapper.js +98 -8
- package/dist/utils/NativeColumnStore.d.ts +2 -1
- package/dist/utils/NativeColumnStore.js +1 -1
- package/dist/utils/NativeMetadataIndex.d.ts +21 -0
- package/dist/utils/NativeMetadataIndex.js +79 -3
- package/docs/migration-3.0.md +1 -1
- package/docs/scaling.md +5 -3
- package/package.json +2 -2
|
@@ -210,6 +210,14 @@ export declare class NativeDiskAnnWrapper implements VectorIndexProvider {
|
|
|
210
210
|
/** Segment-manifest cold-load memo (see {@link ensureSegments}). */
|
|
211
211
|
private segmentsLoaded;
|
|
212
212
|
private segmentsLoading;
|
|
213
|
+
/**
|
|
214
|
+
* True when a durable base index was present on disk but failed to load
|
|
215
|
+
* (corrupt file, or a missing slotmap sidecar). Distinct from a fresh brain
|
|
216
|
+
* (no file at all): drives {@link isReady} to report NOT-ready so brainy's
|
|
217
|
+
* cold-open gate recovers via `rebuild()` instead of skipping an unusable
|
|
218
|
+
* index. Set in {@link tryOpenExisting}; cleared by a successful rebuild.
|
|
219
|
+
*/
|
|
220
|
+
private durableBaseLoadFailed;
|
|
213
221
|
/** Serializes manifest writes: the storage adapter has no per-key write
|
|
214
222
|
* ordering, so two in-flight saves (a flush racing a consolidation) could
|
|
215
223
|
* physically land out of order and persist a STALE segment list — losing a
|
|
@@ -325,6 +333,24 @@ export declare class NativeDiskAnnWrapper implements VectorIndexProvider {
|
|
|
325
333
|
* @returns `true` when the brain must not be read from or written to.
|
|
326
334
|
*/
|
|
327
335
|
isMigrating(): boolean;
|
|
336
|
+
/**
|
|
337
|
+
* @description **Readiness contract (brainy ≥ 8.0.12).** Honest durability
|
|
338
|
+
* signal for brainy's per-leg cold-open rebuild gate: `true` ⇔ the persisted
|
|
339
|
+
* index is loaded (or there is none to load) and a rebuild from canonical
|
|
340
|
+
* would be redundant. Brainy defers to this INSTEAD of the `size()===0`
|
|
341
|
+
* heuristic — a disk-native index legitimately reports zero resident nodes
|
|
342
|
+
* while fully durable, which is exactly why `size()` is the wrong gate.
|
|
343
|
+
*
|
|
344
|
+
* Returns `false` only when a durable base index was present but FAILED to
|
|
345
|
+
* load ({@link durableBaseLoadFailed}) — never claim ready over a failed load,
|
|
346
|
+
* or the gate would serve silent-empty results instead of recovering via
|
|
347
|
+
* `rebuild()`. A fresh brain (no durable index) is ready: loading nothing
|
|
348
|
+
* succeeds and a rebuild is a no-op. Partially-skipped L0 segments do NOT make
|
|
349
|
+
* it not-ready — their entries remain in canonical and heal via consolidation
|
|
350
|
+
* (see {@link loadSegmentManifest}).
|
|
351
|
+
* @returns `true` when a cold-open rebuild would be redundant.
|
|
352
|
+
*/
|
|
353
|
+
isReady(): boolean;
|
|
328
354
|
/**
|
|
329
355
|
* @description **#18 migration lock — structured progress** Brainy relays into
|
|
330
356
|
* `getIndexStatus().migration`. `null` unless a migration is in flight.
|
|
@@ -392,10 +418,18 @@ export declare class NativeDiskAnnWrapper implements VectorIndexProvider {
|
|
|
392
418
|
/** Membership across the base + every L0 segment (int → any slot). */
|
|
393
419
|
private hasIntAnywhere;
|
|
394
420
|
/**
|
|
395
|
-
* @description
|
|
396
|
-
*
|
|
397
|
-
*
|
|
398
|
-
*
|
|
421
|
+
* @description **Readiness contract (brainy ≥ 8.0.12).** Provider init — brainy
|
|
422
|
+
* feature-detects and awaits it once during brain open, after the metadata
|
|
423
|
+
* provider's `init()` (id-mapper hydrated) and BEFORE its cold-open rebuild
|
|
424
|
+
* gate. The base index is mmap-attached in the constructor ({@link
|
|
425
|
+
* tryOpenExisting}); this eager cold-load attaches the L0 **segment** set too,
|
|
426
|
+
* so `size()`/{@link isReady} reflect the FULL persisted index at the gate
|
|
427
|
+
* rather than the base alone. Without it the async segment cold-load defers
|
|
428
|
+
* past the gate, `size()` reads low/zero, and brainy rebuilds from canonical on
|
|
429
|
+
* every open (the 48s-per-boot class). An mmap attach — O(pages touched), never
|
|
430
|
+
* an O(N) rebuild — so it stays cheap at billion scale. (The wrapper's own
|
|
431
|
+
* async paths also call {@link ensureSegments}, so a caller that skips init()
|
|
432
|
+
* still gets the segments lazily; only the pre-gate timing needs init().)
|
|
399
433
|
*/
|
|
400
434
|
init(): Promise<void>;
|
|
401
435
|
/** Memoized segment-manifest cold-load. */
|
|
@@ -312,6 +312,14 @@ export class NativeDiskAnnWrapper {
|
|
|
312
312
|
/** Segment-manifest cold-load memo (see {@link ensureSegments}). */
|
|
313
313
|
segmentsLoaded = false;
|
|
314
314
|
segmentsLoading = null;
|
|
315
|
+
/**
|
|
316
|
+
* True when a durable base index was present on disk but failed to load
|
|
317
|
+
* (corrupt file, or a missing slotmap sidecar). Distinct from a fresh brain
|
|
318
|
+
* (no file at all): drives {@link isReady} to report NOT-ready so brainy's
|
|
319
|
+
* cold-open gate recovers via `rebuild()` instead of skipping an unusable
|
|
320
|
+
* index. Set in {@link tryOpenExisting}; cleared by a successful rebuild.
|
|
321
|
+
*/
|
|
322
|
+
durableBaseLoadFailed = false;
|
|
315
323
|
/** Serializes manifest writes: the storage adapter has no per-key write
|
|
316
324
|
* ordering, so two in-flight saves (a flush racing a consolidation) could
|
|
317
325
|
* physically land out of order and persist a STALE segment list — losing a
|
|
@@ -470,7 +478,7 @@ export class NativeDiskAnnWrapper {
|
|
|
470
478
|
*/
|
|
471
479
|
async hydrateDeltaFromStorage(dim) {
|
|
472
480
|
// Best-effort: only when the storage adapter supports bulk noun reads
|
|
473
|
-
// (production FileSystemStorage
|
|
481
|
+
// (production brainy FileSystemStorage does; some partial
|
|
474
482
|
// adapters used in direct-construction tests do not). When absent, the
|
|
475
483
|
// caller falls back to the in-memory rebuild.
|
|
476
484
|
if (typeof this.storage?.getNouns !== 'function')
|
|
@@ -715,6 +723,31 @@ export class NativeDiskAnnWrapper {
|
|
|
715
723
|
isMigrating() {
|
|
716
724
|
return this.migrationCoordinatorGetter?.()?.isMigrating() ?? false;
|
|
717
725
|
}
|
|
726
|
+
/**
|
|
727
|
+
* @description **Readiness contract (brainy ≥ 8.0.12).** Honest durability
|
|
728
|
+
* signal for brainy's per-leg cold-open rebuild gate: `true` ⇔ the persisted
|
|
729
|
+
* index is loaded (or there is none to load) and a rebuild from canonical
|
|
730
|
+
* would be redundant. Brainy defers to this INSTEAD of the `size()===0`
|
|
731
|
+
* heuristic — a disk-native index legitimately reports zero resident nodes
|
|
732
|
+
* while fully durable, which is exactly why `size()` is the wrong gate.
|
|
733
|
+
*
|
|
734
|
+
* Returns `false` only when a durable base index was present but FAILED to
|
|
735
|
+
* load ({@link durableBaseLoadFailed}) — never claim ready over a failed load,
|
|
736
|
+
* or the gate would serve silent-empty results instead of recovering via
|
|
737
|
+
* `rebuild()`. A fresh brain (no durable index) is ready: loading nothing
|
|
738
|
+
* succeeds and a rebuild is a no-op. Partially-skipped L0 segments do NOT make
|
|
739
|
+
* it not-ready — their entries remain in canonical and heal via consolidation
|
|
740
|
+
* (see {@link loadSegmentManifest}).
|
|
741
|
+
* @returns `true` when a cold-open rebuild would be redundant.
|
|
742
|
+
*/
|
|
743
|
+
isReady() {
|
|
744
|
+
if (this.durableBaseLoadFailed)
|
|
745
|
+
return false;
|
|
746
|
+
// init() (or any prior async op) has attached the segment set. Before that,
|
|
747
|
+
// report not-ready so the gate never skips over an unloaded index; brainy's
|
|
748
|
+
// contract calls isReady() only after init(), where this is true.
|
|
749
|
+
return this.segmentsLoaded;
|
|
750
|
+
}
|
|
718
751
|
/**
|
|
719
752
|
* @description **#18 migration lock — structured progress** Brainy relays into
|
|
720
753
|
* `getIndexStatus().migration`. `null` unless a migration is in flight.
|
|
@@ -1056,6 +1089,9 @@ export class NativeDiskAnnWrapper {
|
|
|
1056
1089
|
// the #72 Phase A+B win. slot ↔ uuid resolution flows through the
|
|
1057
1090
|
// mmap slotmaps + the canonical mapper on every query.
|
|
1058
1091
|
this.native = newNative;
|
|
1092
|
+
// A completed rebuild regenerates the base index + its slotmap sidecar, so
|
|
1093
|
+
// any prior failed-durable-load state is now resolved — the index is ready.
|
|
1094
|
+
this.durableBaseLoadFailed = false;
|
|
1059
1095
|
// Reconcile the LIVE delta/tombstones against the fold snapshot: only
|
|
1060
1096
|
// entries the fold actually baked in are cleared. Adds/updates/removes
|
|
1061
1097
|
// that landed during the async build stay behind for the next fold (an
|
|
@@ -1094,7 +1130,32 @@ export class NativeDiskAnnWrapper {
|
|
|
1094
1130
|
* size for parity with HNSW's flush contract.
|
|
1095
1131
|
*/
|
|
1096
1132
|
async flush() {
|
|
1097
|
-
|
|
1133
|
+
// **Durable close (readiness contract).** Persist the in-memory delta to a
|
|
1134
|
+
// durable L0 segment. brainy calls flush() on shutdown/close (never per
|
|
1135
|
+
// write — verified against its flushOnShutdown + close paths), so a brain
|
|
1136
|
+
// that added fewer than the soft auto-fold threshold ({@link
|
|
1137
|
+
// DELTA_SOFT_FOLD_BYTES}) still lands ALL its writes durably on a clean
|
|
1138
|
+
// close. WITHOUT this, that sub-threshold delta lived only in canonical
|
|
1139
|
+
// storage and the vector index rebuilt it from canonical on the NEXT open —
|
|
1140
|
+
// a cold-open rebuild on every restart (the 48s-per-boot class; a small
|
|
1141
|
+
// brain never folds, so it paid it forever). After this flush the durable
|
|
1142
|
+
// base + segments reflect every persisted entity, so {@link isReady}
|
|
1143
|
+
// reports true at brainy's cold-open gate and the rebuild is skipped.
|
|
1144
|
+
//
|
|
1145
|
+
// Drain any in-flight fold first (it may already be persisting this delta),
|
|
1146
|
+
// then persist whatever remains. flushDeltaToL0 no-ops on an empty delta and
|
|
1147
|
+
// coalesces with a concurrent flush; a transient base/L0 overlap is
|
|
1148
|
+
// deduplicated at search time by the per-hit authority check. Bounded cost:
|
|
1149
|
+
// repeated small closes make small L0s that consolidation merges at
|
|
1150
|
+
// L0_COMPACT_THRESHOLD.
|
|
1151
|
+
const pending = this.delta.size;
|
|
1152
|
+
if (this.flushInFlight)
|
|
1153
|
+
await this.flushInFlight;
|
|
1154
|
+
if (this.rebuildInFlight)
|
|
1155
|
+
await this.rebuildInFlight;
|
|
1156
|
+
if (this.delta.size > 0)
|
|
1157
|
+
await this.flushDeltaToL0();
|
|
1158
|
+
return pending;
|
|
1098
1159
|
}
|
|
1099
1160
|
getPersistMode() {
|
|
1100
1161
|
return this.persistMode;
|
|
@@ -1114,10 +1175,18 @@ export class NativeDiskAnnWrapper {
|
|
|
1114
1175
|
return false;
|
|
1115
1176
|
}
|
|
1116
1177
|
/**
|
|
1117
|
-
* @description
|
|
1118
|
-
*
|
|
1119
|
-
*
|
|
1120
|
-
*
|
|
1178
|
+
* @description **Readiness contract (brainy ≥ 8.0.12).** Provider init — brainy
|
|
1179
|
+
* feature-detects and awaits it once during brain open, after the metadata
|
|
1180
|
+
* provider's `init()` (id-mapper hydrated) and BEFORE its cold-open rebuild
|
|
1181
|
+
* gate. The base index is mmap-attached in the constructor ({@link
|
|
1182
|
+
* tryOpenExisting}); this eager cold-load attaches the L0 **segment** set too,
|
|
1183
|
+
* so `size()`/{@link isReady} reflect the FULL persisted index at the gate
|
|
1184
|
+
* rather than the base alone. Without it the async segment cold-load defers
|
|
1185
|
+
* past the gate, `size()` reads low/zero, and brainy rebuilds from canonical on
|
|
1186
|
+
* every open (the 48s-per-boot class). An mmap attach — O(pages touched), never
|
|
1187
|
+
* an O(N) rebuild — so it stays cheap at billion scale. (The wrapper's own
|
|
1188
|
+
* async paths also call {@link ensureSegments}, so a caller that skips init()
|
|
1189
|
+
* still gets the segments lazily; only the pre-gate timing needs init().)
|
|
1121
1190
|
*/
|
|
1122
1191
|
async init() {
|
|
1123
1192
|
await this.ensureSegments();
|
|
@@ -1228,7 +1297,17 @@ export class NativeDiskAnnWrapper {
|
|
|
1228
1297
|
const builtDelta = new Map(this.delta);
|
|
1229
1298
|
if (builtDelta.size === 0)
|
|
1230
1299
|
return;
|
|
1231
|
-
|
|
1300
|
+
// Resolve the vector dimension robustly. `this.config.dimensions` is the
|
|
1301
|
+
// declared dim, but brainy's zero-config vector factory does not always pass
|
|
1302
|
+
// it (the wrapper marks it "required", yet a provider constructed without it
|
|
1303
|
+
// must still flush on close). Fall back to the attached base index's header,
|
|
1304
|
+
// then to the delta's own vectors — every vector in a brain shares one dim
|
|
1305
|
+
// (brainy enforces the embedding model's). Without this fallback, flush()
|
|
1306
|
+
// on a zero-config brain threw `≠ index dim undefined` on every close.
|
|
1307
|
+
const firstVec = builtDelta.values().next().value;
|
|
1308
|
+
const dim = this.config.dimensions ??
|
|
1309
|
+
(this.native ? this.native.header().dim : undefined) ??
|
|
1310
|
+
firstVec.length;
|
|
1232
1311
|
const mapper = this.mapper();
|
|
1233
1312
|
const deltaBuf = new Float32Array(builtDelta.size * dim);
|
|
1234
1313
|
const slotIdsArr = new BigUint64Array(builtDelta.size);
|
|
@@ -1320,15 +1399,26 @@ export class NativeDiskAnnWrapper {
|
|
|
1320
1399
|
// .slots.json parse). No mapper is needed here; slot↔uuid
|
|
1321
1400
|
// resolution happens lazily at query/rebuild time.
|
|
1322
1401
|
if (!this.native.slotmapReady()) {
|
|
1402
|
+
// The base index file is present but its slotmap sidecar is missing —
|
|
1403
|
+
// the index exists on disk but can't map hits to entity ints. It must
|
|
1404
|
+
// be regenerated by rebuild(); until then this is a FAILED durable load,
|
|
1405
|
+
// not a fresh/empty brain. isReady() reports it so brainy's gate
|
|
1406
|
+
// recovers via rebuild instead of skipping over an unusable index.
|
|
1323
1407
|
this.native = null;
|
|
1408
|
+
this.durableBaseLoadFailed = true;
|
|
1324
1409
|
}
|
|
1325
1410
|
else {
|
|
1326
1411
|
this.removeLegacySlotsSidecar();
|
|
1327
1412
|
}
|
|
1328
1413
|
}
|
|
1329
1414
|
catch {
|
|
1330
|
-
//
|
|
1415
|
+
// openExisting threw. Distinguish a fresh brain (no file — normal, index
|
|
1416
|
+
// stays empty until the first rebuild) from a corrupt/unreadable durable
|
|
1417
|
+
// index that IS on disk (a failed load that needs rebuild recovery).
|
|
1331
1418
|
this.native = null;
|
|
1419
|
+
const path = this.config.indexPath;
|
|
1420
|
+
if (path !== undefined && existsSync(path))
|
|
1421
|
+
this.durableBaseLoadFailed = true;
|
|
1332
1422
|
}
|
|
1333
1423
|
}
|
|
1334
1424
|
/**
|
|
@@ -35,7 +35,8 @@ import type { ColumnStoreProvider, EntityIdMapperLike } from './columnStoreTypes
|
|
|
35
35
|
import { ValueType } from './columnStoreTypes.js';
|
|
36
36
|
import { RoaringBitmap32 } from '../native/NativeRoaringBitmap32.js';
|
|
37
37
|
/**
|
|
38
|
-
* @description The subset of
|
|
38
|
+
* @description The subset of brainy's `FileSystemStorage` (whose index files cor
|
|
39
|
+
* memory-maps) that the column
|
|
39
40
|
* store needs for billion-scale raw segment persistence. cor builds against
|
|
40
41
|
* published `@soulcraft/brainy`, whose `StorageAdapter` type does not yet
|
|
41
42
|
* declare these binary-blob methods, so we narrow the adapter to this local
|
|
@@ -130,7 +130,7 @@ export class NativeColumnStore {
|
|
|
130
130
|
if (!isBinaryBlobStorage(storage)) {
|
|
131
131
|
throw new Error('NativeColumnStore requires a storage adapter with binary-blob support ' +
|
|
132
132
|
'(saveBinaryBlob/loadBinaryBlob/deleteBinaryBlob/getBinaryBlobPath). ' +
|
|
133
|
-
'Use
|
|
133
|
+
'Use brainy FileSystemStorage or another adapter implementing BinaryBlobStorage.');
|
|
134
134
|
}
|
|
135
135
|
this.storage = storage;
|
|
136
136
|
// Discover fields by listing manifest files under the column-index prefix.
|
|
@@ -97,6 +97,12 @@ export declare class MetadataIndexManager implements MetadataIndexProvider {
|
|
|
97
97
|
private config;
|
|
98
98
|
private native;
|
|
99
99
|
private isRebuilding;
|
|
100
|
+
/**
|
|
101
|
+
* True once {@link init} has finished loading the durable derived state
|
|
102
|
+
* (registry + LSM cold-load + column store). Gates {@link isReady} so brainy's
|
|
103
|
+
* cold-open rebuild gate never skips over an index that has not loaded yet.
|
|
104
|
+
*/
|
|
105
|
+
private readinessInitialized;
|
|
100
106
|
/**
|
|
101
107
|
* **#18** — set true by {@link guardIndexEpoch} outcome 4 (large stale brain,
|
|
102
108
|
* auto-migrate enabled) and read once at the END of {@link init}, which kicks the
|
|
@@ -497,6 +503,21 @@ export declare class MetadataIndexManager implements MetadataIndexProvider {
|
|
|
497
503
|
*/
|
|
498
504
|
getIdMapper(): EntityIdMapperLike;
|
|
499
505
|
init(): Promise<void>;
|
|
506
|
+
/**
|
|
507
|
+
* @description **Readiness contract (brainy ≥ 8.0.12).** Honest durability
|
|
508
|
+
* signal for brainy's per-leg cold-open rebuild gate: `true` ⇔ this metadata
|
|
509
|
+
* index is loaded and a rebuild from canonical would be redundant. Brainy
|
|
510
|
+
* defers to this INSTEAD of the `getStats().totalEntries === 0` heuristic.
|
|
511
|
+
*
|
|
512
|
+
* Returns `false` when: `init()` has not completed; a migration is in flight
|
|
513
|
+
* (the gate skips a migrating leg, and the migration itself rebuilds); or the
|
|
514
|
+
* field registry lists fields but the LSM engine recovered NO durable state —
|
|
515
|
+
* the derived index is genuinely missing and a rebuild must run (never let the
|
|
516
|
+
* gate skip a legitimate recovery). Otherwise (durable LSM loaded, or a
|
|
517
|
+
* genuinely empty brain) a rebuild is redundant and this returns `true`.
|
|
518
|
+
* @returns `true` when a cold-open rebuild would be redundant.
|
|
519
|
+
*/
|
|
520
|
+
isReady(): boolean;
|
|
500
521
|
/**
|
|
501
522
|
* **#72 Phase C.** Build the shared mmap `BinaryIdMapper` from the
|
|
502
523
|
* brain's `_id_mapper/*` storage paths and inject it into the Rust
|
|
@@ -171,6 +171,12 @@ export class MetadataIndexManager {
|
|
|
171
171
|
config;
|
|
172
172
|
native;
|
|
173
173
|
isRebuilding = false;
|
|
174
|
+
/**
|
|
175
|
+
* True once {@link init} has finished loading the durable derived state
|
|
176
|
+
* (registry + LSM cold-load + column store). Gates {@link isReady} so brainy's
|
|
177
|
+
* cold-open rebuild gate never skips over an index that has not loaded yet.
|
|
178
|
+
*/
|
|
179
|
+
readinessInitialized = false;
|
|
174
180
|
/**
|
|
175
181
|
* **#18** — set true by {@link guardIndexEpoch} outcome 4 (large stale brain,
|
|
176
182
|
* auto-migrate enabled) and read once at the END of {@link init}, which kicks the
|
|
@@ -405,13 +411,31 @@ export class MetadataIndexManager {
|
|
|
405
411
|
// Resolve the metadata directory for the persistent engine, mirroring
|
|
406
412
|
// wireIdMapper's filesystem guard. `getBinaryBlobPath` returns null on
|
|
407
413
|
// non-local adapters → fall back to the in-RAM engine.
|
|
408
|
-
const
|
|
409
|
-
|
|
410
|
-
|
|
414
|
+
const getBlobPath = blobStorage.getBinaryBlobPath;
|
|
415
|
+
const hasBlobPath = typeof getBlobPath === 'function';
|
|
416
|
+
const metaDir = hasBlobPath ? getBlobPath.call(blobStorage, '_metadata') : null;
|
|
411
417
|
if (metaDir === null) {
|
|
412
418
|
// Fallback: non-persistent in-RAM engine (the legacy `new` path).
|
|
413
419
|
// U64 IdSpace matches the cor 3.0 production default + the metadata
|
|
414
420
|
// index core constructed above.
|
|
421
|
+
//
|
|
422
|
+
// Two very different situations land here — do NOT let the durable one
|
|
423
|
+
// pass silently (the invisible-degrade the whole GA bar forbids):
|
|
424
|
+
// - method ABSENT → a genuinely non-local adapter (cloud/in-memory);
|
|
425
|
+
// an in-RAM metadata engine is correct. Quiet.
|
|
426
|
+
// - method PRESENT but returned null for '_metadata' → a filesystem-
|
|
427
|
+
// class adapter that should have yielded a path but didn't. The
|
|
428
|
+
// metadata index will run WITHOUT on-disk durability — every restart
|
|
429
|
+
// rebuilds from canonical (the 48s-per-boot class). Announce it LOUD
|
|
430
|
+
// so an operator sees it instead of silently paying the cost forever.
|
|
431
|
+
if (hasBlobPath) {
|
|
432
|
+
prodLog.error('[NativeMetadataIndex] getBinaryBlobPath("_metadata") returned null on ' +
|
|
433
|
+
'an adapter that implements it — the metadata LSM is running IN-RAM, ' +
|
|
434
|
+
'NOT persisted. Every restart will rebuild the index from canonical ' +
|
|
435
|
+
'(slow cold opens). This is a durability misconfiguration, not normal ' +
|
|
436
|
+
'operation — check the storage adapter/path. Reachable durable state is ' +
|
|
437
|
+
'the "restart = warm" contract.');
|
|
438
|
+
}
|
|
415
439
|
this.lsmEngine = new bindings.NativeLsmEngine({ idSpace: 'u64' });
|
|
416
440
|
this.lsmManifest = null;
|
|
417
441
|
this.lsmMetaDir = null;
|
|
@@ -1040,6 +1064,30 @@ export class MetadataIndexManager {
|
|
|
1040
1064
|
'NO rebuild.');
|
|
1041
1065
|
return;
|
|
1042
1066
|
}
|
|
1067
|
+
// **Tripwire — a rebuild here on a PREVIOUSLY-HEALTHY brain is a bug,
|
|
1068
|
+
// not routine.** A rebuild is legitimate exactly twice: a fresh brain
|
|
1069
|
+
// (no marker) and a 7.x→8.0 upgrade (stale marker) — both have no
|
|
1070
|
+
// durable derived state yet. But if the on-disk brain-format marker is
|
|
1071
|
+
// present AND current, an 8.x/3.x already fully built and STAMPED this
|
|
1072
|
+
// brain's indexes; their absence now means the field registry + LSM
|
|
1073
|
+
// both failed to persist/reload (the "restart = milliseconds" contract
|
|
1074
|
+
// broken — every boot pays an O(N) rebuild). That must never be
|
|
1075
|
+
// silent: surface it at ERROR with the entity count so it's caught the
|
|
1076
|
+
// first time, not after a customer measures a 48s boot.
|
|
1077
|
+
try {
|
|
1078
|
+
const marker = await readOnDiskBrainFormat(this.storage);
|
|
1079
|
+
if (!isEpochStale(marker)) {
|
|
1080
|
+
prodLog.error(`[NativeMetadataIndex] DURABILITY TRIPWIRE: rebuilding ${probe.totalCount ?? 'unknown'} ` +
|
|
1081
|
+
'entities on a brain whose format marker is present + current — the ' +
|
|
1082
|
+
'derived indexes (field registry AND LSM) should already be on disk. ' +
|
|
1083
|
+
'This is an unexpected full rebuild on a previously-healthy brain ' +
|
|
1084
|
+
'(the every-boot-rebuild class); please report it with the brain layout. ' +
|
|
1085
|
+
'Data is safe (canonical entities intact); only startup time is affected.');
|
|
1086
|
+
}
|
|
1087
|
+
}
|
|
1088
|
+
catch {
|
|
1089
|
+
// marker read failed — fall through to the normal rebuild path
|
|
1090
|
+
}
|
|
1043
1091
|
console.warn(`[NativeMetadataIndex] Field registry missing but ${probe.totalCount ?? 'unknown'} entities exist — rebuilding index`);
|
|
1044
1092
|
await this.rebuild();
|
|
1045
1093
|
return;
|
|
@@ -1066,11 +1114,39 @@ export class MetadataIndexManager {
|
|
|
1066
1114
|
// `.catch` only prevents an unhandled rejection on the start() promise when a
|
|
1067
1115
|
// failed migration surfaces (it is logged loudly and keeps the brain locked). No coordinator
|
|
1068
1116
|
// wired (raw-napi test / non-migrating tier) → no-op.
|
|
1117
|
+
// Readiness contract (brainy ≥ 8.0.12): everything durable has now been
|
|
1118
|
+
// loaded (registry + LSM cold-load + column store) or determined absent.
|
|
1119
|
+
// Set BEFORE the migration kick — during a migration isReady() reports
|
|
1120
|
+
// not-ready via isMigrating(), and brainy's gate skips a migrating leg anyway.
|
|
1121
|
+
this.readinessInitialized = true;
|
|
1069
1122
|
if (this.epochMigrationPending) {
|
|
1070
1123
|
this.epochMigrationPending = false;
|
|
1071
1124
|
this.migrationCoordinatorGetter?.()?.start().catch(() => { });
|
|
1072
1125
|
}
|
|
1073
1126
|
}
|
|
1127
|
+
/**
|
|
1128
|
+
* @description **Readiness contract (brainy ≥ 8.0.12).** Honest durability
|
|
1129
|
+
* signal for brainy's per-leg cold-open rebuild gate: `true` ⇔ this metadata
|
|
1130
|
+
* index is loaded and a rebuild from canonical would be redundant. Brainy
|
|
1131
|
+
* defers to this INSTEAD of the `getStats().totalEntries === 0` heuristic.
|
|
1132
|
+
*
|
|
1133
|
+
* Returns `false` when: `init()` has not completed; a migration is in flight
|
|
1134
|
+
* (the gate skips a migrating leg, and the migration itself rebuilds); or the
|
|
1135
|
+
* field registry lists fields but the LSM engine recovered NO durable state —
|
|
1136
|
+
* the derived index is genuinely missing and a rebuild must run (never let the
|
|
1137
|
+
* gate skip a legitimate recovery). Otherwise (durable LSM loaded, or a
|
|
1138
|
+
* genuinely empty brain) a rebuild is redundant and this returns `true`.
|
|
1139
|
+
* @returns `true` when a cold-open rebuild would be redundant.
|
|
1140
|
+
*/
|
|
1141
|
+
isReady() {
|
|
1142
|
+
if (!this.readinessInitialized)
|
|
1143
|
+
return false;
|
|
1144
|
+
if (this.isMigrating())
|
|
1145
|
+
return false;
|
|
1146
|
+
if (this.knownFields.size > 0 && !this.lsmHasDurableState())
|
|
1147
|
+
return false;
|
|
1148
|
+
return true;
|
|
1149
|
+
}
|
|
1074
1150
|
// ==========================================================================
|
|
1075
1151
|
// Storage I/O helpers
|
|
1076
1152
|
// ==========================================================================
|
package/docs/migration-3.0.md
CHANGED
|
@@ -355,7 +355,7 @@ Memory is the canonical first 3.0 consumer (production on Brainy 7.31.6 + Cortex
|
|
|
355
355
|
|
|
356
356
|
**Memory-specific gotchas:**
|
|
357
357
|
|
|
358
|
-
-
|
|
358
|
+
- **Storage** is Brainy's standard `filesystem` adapter (there is no separate "mmap" storage class to configure — cor memory-maps its own index files under whatever filesystem path you give Brainy). cor 3.0's DiskANN auto-engagement WILL fire here, which is what you want.
|
|
359
359
|
- **Per-user brain cold start** was 8.3 s on Brainy 7.31.3 + Cortex 2.7.0 (per the BRAINY-COR-MMAP-VECTOR-NOT-WIRED thread). On 8.0 + 3.0 with DiskANN + the cor shadow-page LSM, cold start should drop substantially — Adaptive DiskANN auto-selects `in-memory` mode at the per-user scale (each user is currently ≤ 1 K memories), giving sub-millisecond reads with no warm-up. Re-measure after upgrade and update the production-impact section of that thread.
|
|
360
360
|
- **`db.asOf(g)` / `db.with(...)`** time-travel reads are now available — if Memory wants to surface "what did this user remember last week", that's a feature you can build on top of brainy 8.0's historical query surface without any cor-side work.
|
|
361
361
|
- **No external Memory user has 1 M+ memories yet**, so the migration window is operationally cheap; do it during the morning low-traffic window per current ops practice.
|
package/docs/scaling.md
CHANGED
|
@@ -104,9 +104,11 @@ are evicted before small active ones).
|
|
|
104
104
|
|
|
105
105
|
### Memory-mapped storage
|
|
106
106
|
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
107
|
+
You use Brainy's standard `filesystem` storage — there is no separate
|
|
108
|
+
storage class to choose. Cor's native engines memory-map their own index
|
|
109
|
+
files (vectors, graph adjacency, metadata LSM) via `memmap2`, letting the
|
|
110
|
+
Linux kernel manage which pages stay in RAM. Zero-copy binary I/O comes
|
|
111
|
+
from the native layer, not from a special adapter.
|
|
110
112
|
|
|
111
113
|
What mmap provides:
|
|
112
114
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@soulcraft/cor",
|
|
3
|
-
"version": "3.0.
|
|
3
|
+
"version": "3.0.5",
|
|
4
4
|
"description": "Native Rust acceleration for Brainy \u2014 SIMD distance, vector quantization, zero-copy mmap, native embeddings. Free tier for storage, Pro license for compute acceleration.",
|
|
5
5
|
"main": "dist/index.js",
|
|
6
6
|
"types": "dist/index.d.ts",
|
|
@@ -82,7 +82,7 @@
|
|
|
82
82
|
},
|
|
83
83
|
"devDependencies": {
|
|
84
84
|
"@napi-rs/cli": "^3.0.0",
|
|
85
|
-
"@soulcraft/brainy": "8.0.
|
|
85
|
+
"@soulcraft/brainy": "8.0.12",
|
|
86
86
|
"@types/node": "^22.0.0",
|
|
87
87
|
"pg": "^8.21.0",
|
|
88
88
|
"tsx": "^4.21.0",
|