mnemocyte 0.1.5 → 0.2.0

package/README.md

@@ -16,7 +16,7 @@ logs, and duplicate-consolidation tools on top of your existing infrastructure.
 - Hybrid recall with vector similarity, lexical matching, recency, confidence,
   access count, and importance.
 - Prompt-ready context output in Markdown, plain text, or XML.
-- Provider timeouts, retries, and `AbortSignal` cancellation.
+- Provider retries, operation timeouts, and caller `AbortSignal` cancellation.
 - Pruning, duplicate detection, audit logs, and experimental consolidation.
 
 Mnemocyte is ESM-only. Use `import`, not CommonJS `require`.
@@ -103,8 +103,8 @@ supported for consumers that prefer provider-specific imports.
 
 ## Postgres
 
-For persistent storage, apply the bundled migration to a Postgres database with
-pgvector enabled, then pass `databaseUrl`.
+For persistent storage, enable pgvector in your Postgres database, apply the
+bundled migrations, then pass `databaseUrl`.
 
 ```ts
 const client = createMnemocyte({
@@ -117,13 +117,46 @@ The package includes:
 
 ```txt
 migrations/0000_initial.sql
+migrations/0001_add_mnemocyte_meta.sql
+migrations/0000_initial.sql.template
+migrations/render-initial.mjs
 ```
 
-The current Postgres schema uses `embedding vector(1536)`, so the Postgres
-backend currently requires a 1536-dimensional embedder.
+For the default 1536-dimensional schema on a fresh install, apply these in
+order:
 
-If dimensions do not match, `createMnemocyte` throws a `MnemocyteError` with
-code `"CONFIG"` before opening the connection pool.
+```txt
+migrations/0000_initial.sql
+migrations/0001_add_mnemocyte_meta.sql
+```
+
+For an existing 0.1.x Postgres install, apply
+`migrations/0001_add_mnemocyte_meta.sql` to record the current 1536-dimensional
+installation metadata.
+
+For a different embedding dimension on a fresh installation, render an explicit
+initial migration from the template and apply that rendered file instead of the
+default `0000_initial.sql`. The rendered initial migration includes the
+matching `mnemocyte_meta` row.
+
+```bash
+node node_modules/mnemocyte/migrations/render-initial.mjs --dimensions 768 --out migrations/0000_initial.768.sql
+```
+
+Inside this repository, the equivalent development shortcut is:
+
+```bash
+pnpm migration:render -- --dimensions 768 --out migrations/0000_initial.768.sql
+```
+
+The Postgres backend supports one embedding dimension per installation. On the
+first Postgres operation, Mnemocyte reads `mnemocyte_meta` and validates it
+against `embedder.dimensions` before calling the embedder. A mismatch throws a
+`MnemocyteError` with code `"CONFIG"`; missing v0.2.0 metadata throws code
+`"MIGRATION"`.
+
+Use the embedding dimension documented for your chosen model, render or apply a
+matching schema, and keep one dimension per Mnemocyte installation.
 
 The migration creates the bundled HNSW pgvector index
 `mnemocyte_memories_embedding_hnsw_idx` for cosine search. HNSW is approximate:
@@ -141,7 +174,8 @@ add full-text or tag-specific GIN indexes yet; add and benchmark expression
 indexes such as `to_tsvector('english', content)` or tag-oriented indexes only
 when your workload needs them.
 
-Configurable embedding dimensions are planned for `0.2.0`.
+Existing 0.1.x deployments remain on 1536 unless you plan and run your own
+Postgres migration to a different pgvector dimension.
 
 ## API
 
@@ -266,6 +300,7 @@ VALIDATION
 CONFIG
 DB
 NOT_FOUND
+MIGRATION
 ```
 
 ## Retrieval Tuning
@@ -293,13 +328,41 @@ const client = createMnemocyte({
 
 Planned larger milestones:
 
-- configurable embedding dimensions with `mnemocyte_meta`
 - `MemoryStore` abstraction
 - `drizzleStore(db)` for caller-owned Drizzle clients
 - `@mnemocyte/mcp`
 
 See [ROADMAP.md](./ROADMAP.md).
 
+## Pre-v1 Notes
+
+Current behavior:
+
+- `createMnemocyte()` selects either the in-memory backend or the
+  Postgres/pgvector backend.
+- Postgres schema setup is explicit; the client does not create tables or
+  indexes for you.
+- `findDuplicates`, audit-log workflows, and `experimental.consolidate` are
+  available before v1 but still subject to API refinement.
+
+Known limitations before v1:
+
+- The two current backends share behavior by convention rather than through a
+  formal `MemoryStore` adapter boundary.
+- Configured provider timeouts fail the Mnemocyte operation with `"TIMEOUT"`;
+  actively aborting the underlying provider request on timeout remains a
+  pre-v1 hardening follow-up.
+- Postgres dimension metadata is installation-wide. Mixed embedding dimensions
+  in one database are intentionally out of scope for now.
+
+Planned v1 direction:
+
+- Extract a `MemoryStore` boundary so storage adapters own persistence while
+  shared core code owns validation, embedding, scoring, observability,
+  resilience, and context building.
+- Keep root `mnemocyte` imports provider-SDK-free.
+- Keep migrations, dimensions, and production index choices explicit.
+
 ## Development
 
 ```bash

package/dist/embedders/openai.d.mts

@@ -1,4 +1,4 @@
-import { c as Embedder } from "../types-QK_43oKJ.mjs";
+import { c as Embedder } from "../types-DXEGcc47.mjs";
 
 //#region src/embedders/openai.d.ts
 interface OpenAIEmbedderOptions {

package/dist/index.d.mts

@@ -1,4 +1,4 @@
-import { A as RetrievalConfig, C as MnemocyteOperation, D as PruneResult, E as PruneInput, M as RetrievalScoreWeights, N as RetrievalScores, O as RecallInput, P as TokenCounter, S as MnemocyteObservationPhase, T as ProviderResilienceConfig, _ as MemoryWithScore, a as ConsolidateResult, b as MnemocyteConfig, c as Embedder, d as FindDuplicatesInput, f as GlobalStats, g as MemoryType, h as Memory, i as ConsolidateInput, j as RetrievalExplanation, k as RememberInput, l as EntityStats, m as ListAuditLogInput, n as AuditEvent, o as ContextFormat, p as ImportanceLevel, r as BuildContextInput, s as DuplicatePair, t as AuditConfig, u as ExperimentalMnemocyteClient, v as MnemocyteBackend, w as ObservabilityConfig, x as MnemocyteObservation, y as MnemocyteClient } from "./types-QK_43oKJ.mjs";
+import { A as RetrievalConfig, C as MnemocyteOperation, D as PruneResult, E as PruneInput, M as RetrievalScoreWeights, N as RetrievalScores, O as RecallInput, P as TokenCounter, S as MnemocyteObservationPhase, T as ProviderResilienceConfig, _ as MemoryWithScore, a as ConsolidateResult, b as MnemocyteConfig, c as Embedder, d as FindDuplicatesInput, f as GlobalStats, g as MemoryType, h as Memory, i as ConsolidateInput, j as RetrievalExplanation, k as RememberInput, l as EntityStats, m as ListAuditLogInput, n as AuditEvent, o as ContextFormat, p as ImportanceLevel, r as BuildContextInput, s as DuplicatePair, t as AuditConfig, u as ExperimentalMnemocyteClient, v as MnemocyteBackend, w as ObservabilityConfig, x as MnemocyteObservation, y as MnemocyteClient } from "./types-DXEGcc47.mjs";
 
 //#region src/client.d.ts
 /**
@@ -16,15 +16,13 @@ import { A as RetrievalConfig, C as MnemocyteOperation, D as PruneResult, E as P
  * @param config - Client configuration. {@link MnemocyteConfig.embedder} is required.
  * @returns A {@link MnemocyteClient} instance.
  * @throws {MnemocyteError} With code `"CONFIG"` if `embedder` is missing or
- * invalid (including when `databaseUrl` is set and `embedder.dimensions`
- * is not 1536 — the dimensionality pinned by the bundled migration), or
- * `"VALIDATION"` if `databaseUrl` is provided but empty.
+ * invalid, or `"VALIDATION"` if `databaseUrl` is provided but empty.
  *
  * @example Postgres-backed client
  * ```ts
  * const client = createMnemocyte({
  *   databaseUrl: process.env.DATABASE_URL,
- *   embedder, // embedder.dimensions MUST be 1536 for the Postgres backend
+ *   embedder, // embedder.dimensions must match mnemocyte_meta
  * });
  * ```
  */

package/dist/index.d.mts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.mts","names":[],"sources":["../src/client.ts","../src/errors.ts"],"mappings":";;;;;AAqCA;;;;;;;;AAAyE;;;;ACzBzE;;;;AAA8B;AA6B9B;;;;;;;;iBDJgB,eAAA,CAAgB,MAAA,EAAQ,eAAA,GAAkB,eAAe;;;;;;AAAzE;;;;;;;;AAAyE;KCzB7D,kBAAA;;;AAAZ;;;;AAA8B;AA6B9B;;;;;;;;;;;;cAAa,cAAA,SAAuB,KAAA;EAWvB;EAAA,SATH,IAAA,EAAM,kBAAA;EASc;EAAA,SAPX,KAAA;EAOoD;AAAA;AAiBvE;;;cAjBa,OAAA,UAAiB,IAAA,EAAM,kBAAA,EAAoB,KAAA;AAAA;;;;AAiBiB;;;;;;iBAAzD,gBAAA,CAAiB,KAAA,YAAiB,KAAA,IAAS,cAAc"}
+{"version":3,"file":"index.d.mts","names":[],"sources":["../src/client.ts","../src/errors.ts"],"mappings":";;;;;AA+BA;;;;;;;;AAAyE;;;;ACnBzE;;;;AAA8B;AA6B9B;;;;;;iBDVgB,eAAA,CAAgB,MAAA,EAAQ,eAAA,GAAkB,eAAe;;;;;;AAAzE;;;;;;;;AAAyE;KCnB7D,kBAAA;;;AAAZ;;;;AAA8B;AA6B9B;;;;;;;;;;;;cAAa,cAAA,SAAuB,KAAA;EAWvB;EAAA,SATH,IAAA,EAAM,kBAAA;EASc;EAAA,SAPX,KAAA;EAOoD;AAAA;AAiBvE;;;cAjBa,OAAA,UAAiB,IAAA,EAAM,kBAAA,EAAoB,KAAA;AAAA;;;;AAiBiB;;;;;;iBAAzD,gBAAA,CAAiB,KAAA,YAAiB,KAAA,IAAS,cAAc"}

package/dist/index.mjs

@@ -18,11 +18,12 @@ var __exportAll = (all, no_symbols) => {
 //#region src/db/schema.ts
 var schema_exports = /* @__PURE__ */ __exportAll({
 	eventsTable: () => eventsTable,
-	memoriesTable: () => memoriesTable
+	memoriesTable: () => memoriesTable,
+	metaTable: () => metaTable
 });
 const vector = customType({
 	dataType(config) {
-		return `vector(${typeof config === "number" ? config : 1536})`;
+		return `vector(${config?.dimensions ?? 1536})`;
 	},
 	toDriver(value) {
 		return `[${value.map(formatVectorComponent$1).join(",")}]`;
@@ -67,6 +68,10 @@ const eventsTable = pgTable("mnemocyte_events", {
 	metadata: jsonb("metadata").notNull().default({}),
 	timestamp: timestamp("timestamp", { withTimezone: true }).notNull().defaultNow()
 }, (table) => [index("mnemocyte_events_entity_time_idx").on(table.entityId, table.timestamp)]);
+const metaTable = pgTable("mnemocyte_meta", {
+	key: text("key").primaryKey(),
+	embeddingDimensions: integer("embedding_dimensions").notNull()
+});
 //#endregion
 //#region src/db/index.ts
 function parseSslFromUrl(url) {
@@ -1393,6 +1398,9 @@ async function lexicalSearch(db, input) {
 		LIMIT ${input.limit}
 	`);
 }
+async function getInstallationMeta(db) {
+	return (await db.select().from(metaTable).where(eq(metaTable.key, "installation")).limit(1))[0] ?? null;
+}
 //#endregion
 //#region src/retrieval/index.ts
 async function hybridRecall(input) {
@@ -1458,6 +1466,9 @@ const IMPORTANCE_LEVELS = [
 function importanceCeilingLevels(max) {
 	return IMPORTANCE_LEVELS.filter((level) => IMPORTANCE_RANK[level] <= IMPORTANCE_RANK[max]);
 }
+function hasPostgresErrorCode(error, code) {
+	return typeof error === "object" && error !== null && "code" in error && error.code === code;
+}
 function toPruneFilter(input) {
 	const filter = {};
 	if (input.entityId !== void 0) filter.entityId = input.entityId;
@@ -1472,9 +1483,31 @@ function toPruneFilter(input) {
 }
 function createPostgresClient(config, handle) {
 	let closed = false;
+	let schemaValidated = false;
+	let schemaValidationPromise;
 	function assertOpen() {
 		if (closed) throw new MnemocyteError("Mnemocyte client is closed.", "DB");
 	}
+	async function ensureSchemaMatchesEmbedder() {
+		if (schemaValidated) return;
+		if (!schemaValidationPromise) schemaValidationPromise = validateSchemaMatchesEmbedder().catch((error) => {
+			schemaValidationPromise = void 0;
+			throw error;
+		});
+		await schemaValidationPromise;
+	}
+	async function validateSchemaMatchesEmbedder() {
+		let meta;
+		try {
+			meta = await getInstallationMeta(handle.db);
+		} catch (error) {
+			if (hasPostgresErrorCode(error, "42P01")) throw new MnemocyteError("mnemocyte_meta is missing. Apply the v0.2.0 migration or render a dimension-specific initial migration before using the Postgres backend.", "MIGRATION", error);
+			throw error;
+		}
+		if (!meta) throw new MnemocyteError("mnemocyte_meta is missing the \"installation\" row. Apply the v0.2.0 migration or render a dimension-specific initial migration before using the Postgres backend.", "MIGRATION");
+		if (meta.embeddingDimensions !== config.embedder.dimensions) throw new MnemocyteError(`embedder.dimensions (${config.embedder.dimensions}) must match mnemocyte_meta.embedding_dimensions (${meta.embeddingDimensions}). Render and apply a migration for the selected embedding dimension, or configure an embedder that matches this installation.`, "CONFIG");
+		schemaValidated = true;
+	}
 	async function recordAudit(entityId, description, metadata) {
 		if (config.audit?.enabled !== true) return;
 		try {
@@ -1491,6 +1524,7 @@ function createPostgresClient(config, handle) {
 		return observe(config, "postgres", "remember", { entityId: input.entityId }, async () => {
 			assertOpen();
 			validateRememberInput(input);
+			await ensureSchemaMatchesEmbedder();
 			const embedding = await embedOne(config.embedder, input.content, {
 				...input.signal === void 0 ? {} : { signal: input.signal },
 				...config.provider === void 0 ? {} : { resilience: config.provider }
@@ -1535,6 +1569,7 @@ function createPostgresClient(config, handle) {
 				assertOpen();
 				if (inputs.length === 0) return [];
 				for (const input of inputs) validateRememberInput(input);
+				await ensureSchemaMatchesEmbedder();
 				const embeddings = await embedMany(config.embedder, inputs.map((i) => i.content), {
 					...inputs[0]?.signal === void 0 ? {} : { signal: inputs[0].signal },
 					...config.provider === void 0 ? {} : { resilience: config.provider }
@@ -1578,6 +1613,7 @@ function createPostgresClient(config, handle) {
 				const minScore = input.minScore ?? config.defaults?.minScore ?? 0;
 				assertLimit(limit);
 				assertMinScore(minScore);
+				await ensureSchemaMatchesEmbedder();
 				return hybridRecall({
 					db: handle.db,
 					embedder: config.embedder,
@@ -1593,6 +1629,7 @@ function createPostgresClient(config, handle) {
 		async buildContext(input) {
 			return observe(config, "postgres", "buildContext", { entityId: input.entityId }, async () => {
 				assertOpen();
+				await ensureSchemaMatchesEmbedder();
 				return buildContext({
 					input,
 					recall: (contextInput) => {
@@ -1613,6 +1650,7 @@ function createPostgresClient(config, handle) {
 				assertOpen();
 				assertNonEmptyString(input.entityId, "entityId");
 				assertNonEmptyString(input.memoryId, "memoryId");
+				await ensureSchemaMatchesEmbedder();
 				if (!await deleteMemory(handle.db, input.entityId, input.memoryId)) throw new MnemocyteError("Memory was not found.", "NOT_FOUND");
 				await recordAudit(input.entityId, "memory.deleted", { memoryId: input.memoryId });
 			});
@@ -1621,6 +1659,7 @@ function createPostgresClient(config, handle) {
 			return observe(config, "postgres", "forgetAll", { entityId: input.entityId }, async () => {
 				assertOpen();
 				assertNonEmptyString(input.entityId, "entityId");
+				await ensureSchemaMatchesEmbedder();
 				const deletedCount = await deleteMemoriesForEntity(handle.db, input.entityId);
 				await recordAudit(input.entityId, "entity.cleared", { count: deletedCount });
 				return deletedCount;
@@ -1630,6 +1669,7 @@ function createPostgresClient(config, handle) {
 			return observe(config, "postgres", "prune", input.entityId === void 0 ? {} : { entityId: input.entityId }, async () => {
 				assertOpen();
 				validatePruneInput(input);
+				await ensureSchemaMatchesEmbedder();
 				const filter = toPruneFilter(input);
 				if (input.dryRun === true) return {
 					matchedCount: await countPruneMatches(handle.db, filter),
@@ -1649,6 +1689,7 @@ function createPostgresClient(config, handle) {
 			return observe(config, "postgres", "findDuplicates", { entityId: input.entityId }, async () => {
 				assertOpen();
 				validateFindDuplicatesInput(input);
+				await ensureSchemaMatchesEmbedder();
 				const threshold = input.threshold ?? .95;
 				const limit = input.limit ?? 100;
 				return (await findDuplicatePairs(handle.db, {
@@ -1670,6 +1711,7 @@ function createPostgresClient(config, handle) {
 			return observe(config, "postgres", "listAuditLog", { entityId: input.entityId }, async () => {
 				assertOpen();
 				validateListAuditLogInput(input);
+				await ensureSchemaMatchesEmbedder();
 				return (await listEvents(handle.db, {
 					entityId: input.entityId,
 					limit: input.limit ?? 50,
@@ -1681,6 +1723,7 @@ function createPostgresClient(config, handle) {
 		async stats(input) {
 			return observe(config, "postgres", "stats", input?.entityId ? { entityId: input.entityId } : {}, async () => {
 				assertOpen();
+				await ensureSchemaMatchesEmbedder();
 				const now = /* @__PURE__ */ new Date();
 				if (input?.entityId) {
 					const counts = await getEntityMemoryStatsCounts(handle.db, input.entityId, now);
@@ -1709,6 +1752,7 @@ function createPostgresClient(config, handle) {
 			}, async () => {
 				assertOpen();
 				validateConsolidateInput(input);
+				await ensureSchemaMatchesEmbedder();
 				const survivor = await getMemoryById(handle.db, input.entityId, input.survivorId);
 				if (!survivor) throw new MnemocyteError("Survivor memory was not found.", "NOT_FOUND");
 				if (survivor.supersededBy !== null) throw new MnemocyteError("Survivor memory is itself superseded.", "VALIDATION");
@@ -1766,8 +1810,6 @@ function createPostgresClient(config, handle) {
 }
 //#endregion
 //#region src/client.ts
-/** Embedding dimensionality pinned by `migrations/0000_initial.sql`. */
-const POSTGRES_EMBEDDING_DIMENSIONS = 1536;
 /**
 * Create a Mnemocyte client.
 *
@@ -1783,15 +1825,13 @@ const POSTGRES_EMBEDDING_DIMENSIONS = 1536;
 * @param config - Client configuration. {@link MnemocyteConfig.embedder} is required.
 * @returns A {@link MnemocyteClient} instance.
 * @throws {MnemocyteError} With code `"CONFIG"` if `embedder` is missing or
-* invalid (including when `databaseUrl` is set and `embedder.dimensions`
-* is not 1536 — the dimensionality pinned by the bundled migration), or
-* `"VALIDATION"` if `databaseUrl` is provided but empty.
+* invalid, or `"VALIDATION"` if `databaseUrl` is provided but empty.
 *
 * @example Postgres-backed client
 * ```ts
 * const client = createMnemocyte({
 *   databaseUrl: process.env.DATABASE_URL,
-*   embedder, // embedder.dimensions MUST be 1536 for the Postgres backend
+*   embedder, // embedder.dimensions must match mnemocyte_meta
 * });
 * ```
 */
@@ -1799,7 +1839,6 @@ function createMnemocyte(config) {
 	assertEmbedder(config.embedder);
 	if (config.databaseUrl) {
 		assertNonEmptyString(config.databaseUrl, "databaseUrl");
-		if (config.embedder.dimensions !== POSTGRES_EMBEDDING_DIMENSIONS) throw new MnemocyteError(`embedder.dimensions must be ${POSTGRES_EMBEDDING_DIMENSIONS} for the Postgres backend (got ${config.embedder.dimensions}). The bundled migration pins the vector column to vector(${POSTGRES_EMBEDDING_DIMENSIONS}).`, "CONFIG");
 		return createPostgresClient(config, createDatabase(config.databaseUrl));
 	}
 	return createInMemoryClient(config);