mnemocyte 0.0.8 → 0.1.1

package/README.md

@@ -1,19 +1,25 @@
 # Mnemocyte
 
-Persistent memory for TypeScript AI apps.
+Persistent memory for TypeScript AI applications.
 
-> Warning
->
-> Mnemocyte is in active early development. APIs may change significantly before v1.0.
+Mnemocyte stores small, durable memories for users, agents, sessions, or
+other entities. It provides hybrid retrieval, context formatting, lifecycle
+operations, audit logging, and duplicate-consolidation tools without requiring
+a separate vector database.
 
-## Current status
+> Mnemocyte is early software. APIs may change before v1.0.
 
-Mnemocyte currently provides an MVP API with two backends:
+## Features
 
-- **In-memory backend** when `databaseUrl` is omitted.
-- **Postgres + pgvector backend** when `databaseUrl` is provided.
+- In-memory backend for tests, demos, and short-lived processes.
+- Postgres + pgvector backend for persistent storage.
+- Hybrid recall using vector similarity, lexical matching, recency,
+  confidence, access count, and importance.
+- Prompt-ready context output in Markdown, plain text, or XML.
+- Provider timeouts, retries, and `AbortSignal` cancellation.
+- Pruning, duplicate detection, audit logs, and experimental consolidation.
 
-The package is ESM-only. Use `import` rather than CommonJS `require`.
+Mnemocyte is ESM-only. Use `import`, not CommonJS `require`.
 
 ## Install
 
@@ -21,134 +27,218 @@ The package is ESM-only. Use `import` rather than CommonJS `require`.
 pnpm add mnemocyte
 ```
 
-## Basic usage
+## Quick Start
 
 ```ts
 import { createMnemocyte } from "mnemocyte";
 
 const client = createMnemocyte({
-embedder: {
-model: "demo",
-dimensions: 3,
-async embed(texts) {
-return texts.map((text) => [text.length, 1, 0]);
-},
-},
+  embedder: {
+    model: "demo",
+    dimensions: 3,
+    async embed(texts) {
+      return texts.map((text) => [text.length, 1, 0]);
+    },
+  },
 });
 
 await client.remember({
-entityId: "user_123",
-content: "Prefers short, direct answers.",
-type: "preference",
+  entityId: "user_123",
+  content: "Prefers short, direct answers.",
+  type: "preference",
 });
 
 const memories = await client.recall({
-entityId: "user_123",
-query: "How should I respond?",
-limit: 5,
-explain: true,
+  entityId: "user_123",
+  query: "How should I respond?",
+  limit: 5,
+  explain: true,
 });
 
 await client.close();
 ```
 
-## Postgres usage
+When `databaseUrl` is omitted, Mnemocyte uses the in-memory backend.
 
-Run the included migration against a Postgres database with pgvector enabled, then pass `databaseUrl`:
+## Postgres
+
+Apply the bundled migration to a Postgres database with pgvector enabled, then
+pass `databaseUrl`:
 
 ```ts
 const client = createMnemocyte({
-databaseUrl: process.env.DATABASE_URL,
-embedder,
+  databaseUrl: process.env.DATABASE_URL,
+  embedder, // embedder.dimensions must be 1536 for the Postgres backend.
 });
 ```
 
 The published package includes `migrations/0000_initial.sql`.
 
-## Provider resilience
+The current Postgres schema uses `embedding vector(1536)`. If
+`embedder.dimensions` is anything else, `createMnemocyte` throws a
+`MnemocyteError` with code `"CONFIG"` before opening the connection pool. The
+in-memory backend does not enforce this constraint.
+
+## Provider Resilience
 
-Mnemocyte applies an optional timeout and retry policy to every outbound
-embedder call, and forwards `AbortSignal` cancellation to in-flight
-attempts. Defaults disable both retries and timeouts so existing setups
-are unaffected.
+Mnemocyte can apply timeouts and retries to embedder calls, and forwards
+`AbortSignal` cancellation to in-flight attempts. Retries and timeouts are
+disabled by default.
 
 ```ts
 const client = createMnemocyte({
-embedder,
-provider: {
-timeoutMs: 5_000,
-retries: 2,
-baseDelayMs: 200,
-maxDelayMs: 2_000,
-},
+  embedder,
+  provider: {
+    timeoutMs: 5_000,
+    retries: 2,
+    baseDelayMs: 200,
+    maxDelayMs: 2_000,
+  },
 });
 
 const controller = new AbortController();
 setTimeout(() => controller.abort(), 1_000);
 
 await client.remember({
-entityId: "user_123",
-content: "Prefers short answers.",
-signal: controller.signal,
+  entityId: "user_123",
+  content: "Prefers short answers.",
+  signal: controller.signal,
 });
 ```
 
-Failures surface as `MnemocyteError` with stable `code`s:
+Provider failures surface as `MnemocyteError` values with stable `code`s:
 
-- `"TIMEOUT"` — a single provider attempt exceeded `timeoutMs`.
-- `"ABORTED"` — the operation was cancelled via `signal` (never retried).
-- `"EMBEDDING"` — the embedder failed after all retries are exhausted.
+- `"TIMEOUT"`: a provider attempt exceeded `timeoutMs`.
+- `"ABORTED"`: the operation was cancelled via `signal` and is not retried.
+- `"EMBEDDING"`: the embedder failed after retries were exhausted.
 
-## Pruning memories
+## Pruning
 
-`prune` bulk-deletes memories matching a filter. At least one selector
-is required; `prune({})` is rejected to avoid accidental full deletion.
+`prune` deletes memories matching a filter. At least one selector is required;
+`prune({})` is rejected to avoid accidental full deletion.
 
 ```ts
-// Drop expired memories for one entity.
 await client.prune({ entityId: "user_123", expired: true });
 
-// Evict memories not accessed in the last 30 days, but never drop
-// "high" or "critical" memories.
 await client.prune({
-notAccessedSince: new Date(Date.now() - 30 * 24 * 3600 * 1000),
-maxImportance: "normal",
+  notAccessedSince: new Date(Date.now() - 30 * 24 * 3600 * 1000),
+  maxImportance: "normal",
 });
 
-// Count what would be deleted without touching the store.
 const preview = await client.prune({
-entityId: "user_123",
-superseded: true,
-dryRun: true,
+  entityId: "user_123",
+  superseded: true,
+  dryRun: true,
 });
 console.log(preview.matchedCount);
 ```
 
 Available selectors: `entityId`, `expired`, `superseded`, `createdBefore`,
-`notAccessedSince`, `types`, `tags`, `maxImportance`. They AND together.
+`notAccessedSince`, `types`, `tags`, `maxImportance`. Selectors are combined
+with AND semantics.
+
+## Duplicate Detection
+
+`findDuplicates` is an experimental, read-only scan that surfaces
+near-identical memories for one entity using pairwise cosine similarity.
+
+```ts
+const pairs = await client.findDuplicates({
+  entityId: "user_123",
+  threshold: 0.95,
+  limit: 50,
+});
+
+for (const { a, b, similarity } of pairs) {
+  console.log(similarity.toFixed(3), a.content, "<>", b.content);
+}
+```
+
+Optional selectors: `types`, `tags`, `includeSuperseded`, `includeExpired`.
+`types` and `tags` must match on both memories in each pair. Superseded and
+expired memories are excluded by default.
+
+## Audit Log
+
+When `audit.enabled` is `true`, Mnemocyte records an entry for state-changing
+operations. Read entries with `client.listAuditLog()`.
+
+```ts
+const client = createMnemocyte({
+  embedder,
+  audit: { enabled: true },
+});
+
+await client.remember({ entityId: "user_123", content: "Likes tea." });
+await client.prune({ entityId: "user_123", expired: true });
+
+const log = await client.listAuditLog({
+  entityId: "user_123",
+  limit: 100,
+});
+
+for (const event of log) {
+  console.log(event.timestamp, event.description, event.metadata);
+}
+```
+
+Recorded `description` slugs:
+
+- `"memory.created"`: single insert.
+- `"memory.deleted"`: single `forget`.
+- `"entity.cleared"`: `forgetAll`.
+- `"memory.pruned"`: non-dry-run `prune` with an `entityId` selector.
+- `"memory.superseded"`: experimental consolidation.
+
+Audit entries are sticky: `forgetAll` does not erase prior log entries. Disable
+audit by leaving `audit.enabled` unset.
+
+## Consolidation
+
+`client.experimental.consolidate()` marks likely-duplicate memories as
+superseded by a survivor. Superseded memories are excluded from `recall` by
+default. Tags are unioned onto the survivor unless `mergeTags: false`.
+
+```ts
+const pairs = await client.findDuplicates({
+  entityId: "user_123",
+  threshold: 0.97,
+});
+
+for (const { a, b } of pairs) {
+  await client.experimental.consolidate({
+    entityId: "user_123",
+    survivorId: a.id,
+    supersededIds: [b.id],
+  });
+}
+```
+
+The `experimental` namespace is intentionally unstable. Members may change or
+move before v1.0.
 
-## Retrieval tuning
+## Retrieval Tuning
 
 ```ts
 const client = createMnemocyte({
-embedder,
-retrieval: {
-weights: {
-vector: 0.55,
-lexical: 0.2,
-recency: 0.1,
-confidence: 0.05,
-access: 0.05,
-importance: 0.05,
-},
-recencyHalfLifeDays: 90,
-accessSaturation: 10,
-candidateMultiplier: 3,
-},
+  embedder,
+  retrieval: {
+    weights: {
+      vector: 0.55,
+      lexical: 0.2,
+      recency: 0.1,
+      confidence: 0.05,
+      access: 0.05,
+      importance: 0.05,
+    },
+    recencyHalfLifeDays: 90,
+    accessSaturation: 10,
+    candidateMultiplier: 3,
+  },
 });
 ```
 
-## Development checks
+## Development
 
 ```bash
 pnpm checktypes
@@ -162,4 +252,5 @@ pnpm run pack:check
 
 ## Architecture
 
-See `ARCHITECTURE.md` for the canonical architecture, roadmap, and production plan.
+See [ARCHITECTURE.md](./ARCHITECTURE.md) for implementation details and
+[ROADMAP.md](./ROADMAP.md) for planned work.

package/dist/index.d.mts

@@ -40,7 +40,7 @@ type MnemocyteBackend = "in-memory" | "postgres";
  * The set of public {@link MnemocyteClient} methods that emit
  * {@link MnemocyteObservation} events through {@link ObservabilityConfig.onEvent}.
  */
-type MnemocyteOperation = "remember" | "rememberMany" | "recall" | "buildContext" | "forget" | "forgetAll" | "prune" | "stats" | "close";
+type MnemocyteOperation = "remember" | "rememberMany" | "recall" | "buildContext" | "forget" | "forgetAll" | "prune" | "findDuplicates" | "listAuditLog" | "consolidate" | "stats" | "close";
 /**
  * Lifecycle phase of a {@link MnemocyteObservation}.
  *
@@ -149,6 +149,30 @@ interface MnemocyteConfig {
    * {@link ProviderResilienceConfig}.
    */
   provider?: ProviderResilienceConfig;
+  /**
+   * Audit-log configuration. When enabled, Mnemocyte records an
+   * {@link AuditEvent} for every state-changing operation
+   * (`remember`, `forget`, `forgetAll`, `prune`). Read with
+   * {@link MnemocyteClient.listAuditLog}.
+   *
+   * @experimental Part of Phase 6 (consolidation tooling).
+   */
+  audit?: AuditConfig;
+}
+/**
+ * Audit-log configuration. See {@link MnemocyteConfig.audit}.
+ *
+ * @experimental Part of Phase 6 (consolidation tooling).
+ */
+interface AuditConfig {
+  /**
+   * Master switch for audit logging. When `false` (default), Mnemocyte
+   * neither writes nor relies on the audit log; existing log entries
+   * remain readable via {@link MnemocyteClient.listAuditLog}.
+   *
+   * @defaultValue `false`
+   */
+  enabled?: boolean;
 }
 /**
  * Pluggable embedding provider. Wraps an external embedding API
@@ -184,7 +208,7 @@ interface Embedder {
  * new memory and link the previous one via {@link Memory.supersededBy}.
  */
 interface Memory {
-  /** Unique identifier (UUID for Postgres backend). */
+  /** Unique identifier (`mem_*` prefixed string). */
   id: string;
   /** Owning entity — typically a user, agent, or session ID. */
   entityId: string;
@@ -208,6 +232,13 @@ interface Memory {
   embeddingDimensions: number;
   /** ID of the memory that supersedes this one, or `null` if active. */
   supersededBy: string | null;
+  /**
+   * Wall-clock time when this memory was superseded, or `null` if it
+   * has not been superseded. Set together with
+   * {@link Memory.supersededBy} by
+   * {@link ExperimentalMnemocyteClient.consolidate}.
+   */
+  supersededAt: Date | null;
   /** Optional expiry; expired memories are filtered out by default. */
   expiresAt: Date | null;
   /** Last time this memory was returned by {@link MnemocyteClient.recall}. */
@@ -476,6 +507,142 @@ interface PruneResult {
   /** `true` when {@link PruneInput.dryRun} was set. */
   dryRun: boolean;
 }
+/**
+ * Input for {@link MnemocyteClient.findDuplicates}.
+ *
+ * Performs a pairwise cosine-similarity scan of stored memories for a
+ * single entity and returns pairs whose similarity meets
+ * {@link FindDuplicatesInput.threshold}. The scan is read-only — no
+ * memories are modified or deleted.
+ *
+ * @experimental Part of Phase 6 (consolidation tooling). API may change.
+ */
+interface FindDuplicatesInput {
+  /** Entity whose memories should be scanned for duplicates. */
+  entityId: string;
+  /**
+   * Cosine-similarity cutoff in `[0, 1]`. Pairs whose similarity is at
+   * or above this value are returned. @defaultValue `0.95`
+   */
+  threshold?: number;
+  /**
+   * Maximum number of duplicate pairs to return, ordered by descending
+   * similarity. @defaultValue `100`
+   */
+  limit?: number;
+  /** Restrict the scan to these memory types. */
+  types?: readonly MemoryType[];
+  /** Require both memories in the pair to include all of these tags. */
+  tags?: readonly string[];
+  /** Include superseded memories in the scan. @defaultValue `false` */
+  includeSuperseded?: boolean;
+  /** Include expired memories in the scan. @defaultValue `false` */
+  includeExpired?: boolean;
+  /** Optional cancellation signal. */
+  signal?: AbortSignal;
+}
+/**
+ * A pair of memories that are likely duplicates.
+ *
+ * The pair is unordered (i.e. `{a, b}` is the same logical result as
+ * `{b, a}`); Mnemocyte returns each pair only once.
+ *
+ * @experimental Part of Phase 6 (consolidation tooling). Shape may change.
+ */
+interface DuplicatePair {
+  /** One of the two memories in the duplicate pair. */
+  a: Memory;
+  /** The other memory in the duplicate pair. */
+  b: Memory;
+  /** Cosine similarity between the two embeddings, in `[0, 1]`. */
+  similarity: number;
+}
+/**
+ * A single audit-log entry recording a state-changing operation.
+ *
+ * @experimental Part of Phase 6 (consolidation tooling).
+ */
+interface AuditEvent {
+  /** Unique identifier (`evt_*` prefixed string). */
+  id: string;
+  /** Entity whose state was changed. */
+  entityId: string;
+  /**
+   * Stable, dotted slug describing the change. Examples:
+   * `"memory.created"`, `"memory.deleted"`, `"memory.pruned"`,
+   * `"entity.cleared"`.
+   */
+  description: string;
+  /** Free-form structured details about the change. */
+  metadata: Record<string, unknown>;
+  /** When the change was recorded. */
+  timestamp: Date;
+}
+/**
+ * Input for {@link MnemocyteClient.listAuditLog}.
+ *
+ * @experimental Part of Phase 6 (consolidation tooling).
+ */
+interface ListAuditLogInput {
+  /** Entity whose audit log should be returned. */
+  entityId: string;
+  /**
+   * Maximum number of entries to return, ordered newest first.
+   * @defaultValue `50`
+   */
+  limit?: number;
+  /** Only return entries strictly before this timestamp. */
+  before?: Date;
+  /** Only return entries strictly after this timestamp. */
+  after?: Date;
+  /** Optional cancellation signal. */
+  signal?: AbortSignal;
+}
+/**
+ * Input for {@link ExperimentalMnemocyteClient.consolidate}.
+ *
+ * Marks each memory in {@link ConsolidateInput.supersededIds} as
+ * superseded by {@link ConsolidateInput.survivorId}. Optionally
+ * unions the losers' tags into the survivor.
+ *
+ * @experimental Part of Phase 6 (consolidation tooling).
+ */
+interface ConsolidateInput {
+  /** Entity that owns every memory referenced. */
+  entityId: string;
+  /** Memory that should win the merge. Must not itself be superseded. */
+  survivorId: string;
+  /**
+   * Memories to mark as superseded by the survivor. Already-superseded
+   * entries in this list are skipped silently for idempotency. Must
+   * not contain {@link ConsolidateInput.survivorId}.
+   */
+  supersededIds: readonly string[];
+  /**
+   * When `true`, the survivor's `tags` becomes the union of its own
+   * tags and the tags of every memory that was actually superseded.
+   * @defaultValue `true`
+   */
+  mergeTags?: boolean;
+  /** Optional cancellation signal. */
+  signal?: AbortSignal;
+}
+/**
+ * Result returned by {@link ExperimentalMnemocyteClient.consolidate}.
+ *
+ * @experimental Part of Phase 6 (consolidation tooling).
+ */
+interface ConsolidateResult {
+  /** Echoes {@link ConsolidateInput.survivorId}. */
+  survivorId: string;
+  /**
+   * Number of memories actually marked as superseded. Excludes entries
+   * that were already superseded (idempotent skip).
+   */
+  supersededCount: number;
+  /** The IDs that were newly superseded during this call. */
+  supersededIds: readonly string[];
+}
 /**
  * Stats for a single entity, returned by
  * {@link MnemocyteClient.stats} when an `entityId` is provided.
@@ -572,6 +739,55 @@ interface MnemocyteClient {
    * ```
    */
   prune(input: PruneInput): Promise<PruneResult>;
+  /**
+   * Scan an entity's memories for likely duplicates by pairwise cosine
+   * similarity and return matching pairs, ordered by descending
+   * similarity. Read-only — no memories are modified.
+   *
+   * @experimental Part of Phase 6 (consolidation tooling). API may change.
+   *
+   * @example
+   * ```ts
+   * const pairs = await client.findDuplicates({
+   *   entityId: "user_123",
+   *   threshold: 0.95,
+   *   limit: 50,
+   * });
+   * for (const { a, b, similarity } of pairs) {
+   *   console.log(similarity, a.content, "<>", b.content);
+   * }
+   * ```
+   */
+  findDuplicates(input: FindDuplicatesInput): Promise<DuplicatePair[]>;
+  /**
+   * Return audit-log entries for `entityId`, newest first.
+   *
+   * Entries are recorded automatically when
+   * {@link MnemocyteConfig.audit}.`enabled` is `true`. Historic entries
+   * remain readable even after toggling `enabled` back to `false`.
+   *
+   * @experimental Part of Phase 6 (consolidation tooling). API may change.
+   *
+   * @example
+   * ```ts
+   * const log = await client.listAuditLog({
+   *   entityId: "user_123",
+   *   limit: 100,
+   * });
+   * for (const event of log) {
+   *   console.log(event.timestamp, event.description, event.metadata);
+   * }
+   * ```
+   */
+  listAuditLog(input: ListAuditLogInput): Promise<AuditEvent[]>;
+  /**
+   * Unstable, opt-in extension surface. Members may change or be
+   * removed without a semver bump until they graduate to the main
+   * client interface.
+   *
+   * @experimental Part of Phase 6 (consolidation tooling).
+   */
+  experimental: ExperimentalMnemocyteClient;
   /**
    * Return statistics. With an `entityId`, returns {@link EntityStats};
    * without one, returns {@link GlobalStats}.
@@ -585,6 +801,45 @@ interface MnemocyteClient {
    */
   close(): Promise<void>;
 }
+/**
+ * Extension surface for unstable Phase 6 operations. Reached via
+ * {@link MnemocyteClient.experimental}.
+ *
+ * @experimental Members may change or be removed without a semver bump.
+ */
+interface ExperimentalMnemocyteClient {
+  /**
+   * Consolidate likely-duplicate memories into a single survivor by
+   * marking the others as superseded. Optionally unions their tags
+   * onto the survivor. Idempotent for memories that are already
+   * superseded.
+   *
+   * Emits one `"memory.superseded"` audit event per newly superseded
+   * memory when {@link MnemocyteConfig.audit}.`enabled` is `true`.
+   *
+   * @experimental Part of Phase 6 (consolidation tooling). API may change.
+   *
+   * @throws {MnemocyteError} `"VALIDATION"` for empty `supersededIds`,
+   * a `survivorId` that also appears in `supersededIds`, or a survivor
+   * that is itself already superseded.
+   * @throws {MnemocyteError} `"NOT_FOUND"` if the survivor or any
+   * non-already-superseded memory in `supersededIds` does not belong
+   * to `entityId`.
+   *
+   * @example
+   * ```ts
+   * const pairs = await client.findDuplicates({ entityId: "user_123" });
+   * for (const { a, b } of pairs) {
+   *   await client.experimental.consolidate({
+   *     entityId: "user_123",
+   *     survivorId: a.id,
+   *     supersededIds: [b.id],
+   *   });
+   * }
+   * ```
+   */
+  consolidate(input: ConsolidateInput): Promise<ConsolidateResult>;
+}
 //#endregion
 //#region src/client.d.ts
 /**
@@ -601,14 +856,16 @@ interface MnemocyteClient {
  *
  * @param config - Client configuration. {@link MnemocyteConfig.embedder} is required.
  * @returns A {@link MnemocyteClient} instance.
- * @throws {MnemocyteError} With code `"CONFIG"` if `embedder` is missing or invalid,
- * or `"VALIDATION"` if `databaseUrl` is provided but empty.
+ * @throws {MnemocyteError} With code `"CONFIG"` if `embedder` is missing or
+ * invalid (including when `databaseUrl` is set and `embedder.dimensions`
+ * is not 1536 — the dimensionality pinned by the bundled migration), or
+ * `"VALIDATION"` if `databaseUrl` is provided but empty.
  *
  * @example Postgres-backed client
  * ```ts
  * const client = createMnemocyte({
  *   databaseUrl: process.env.DATABASE_URL,
- *   embedder,
+ *   embedder, // embedder.dimensions MUST be 1536 for the Postgres backend
  * });
  * ```
  */
@@ -670,5 +927,5 @@ declare class MnemocyteError extends Error {
  */
 declare function isMnemocyteError(error: unknown): error is MnemocyteError;
 //#endregion
-export { type BuildContextInput, type ContextFormat, type Embedder, type EntityStats, type GlobalStats, type ImportanceLevel, type Memory, type MemoryType, type MemoryWithScore, type MnemocyteBackend, type MnemocyteClient, type MnemocyteConfig, MnemocyteError, type MnemocyteErrorCode, type MnemocyteObservation, type MnemocyteObservationPhase, type MnemocyteOperation, type ObservabilityConfig, type ProviderResilienceConfig, type PruneInput, type PruneResult, type RecallInput, type RememberInput, type RetrievalConfig, type RetrievalExplanation, type RetrievalScoreWeights, type RetrievalScores, type TokenCounter, createMnemocyte, isMnemocyteError };
+export { type AuditConfig, type AuditEvent, type BuildContextInput, type ConsolidateInput, type ConsolidateResult, type ContextFormat, type DuplicatePair, type Embedder, type EntityStats, type ExperimentalMnemocyteClient, type FindDuplicatesInput, type GlobalStats, type ImportanceLevel, type ListAuditLogInput, type Memory, type MemoryType, type MemoryWithScore, type MnemocyteBackend, type MnemocyteClient, type MnemocyteConfig, MnemocyteError, type MnemocyteErrorCode, type MnemocyteObservation, type MnemocyteObservationPhase, type MnemocyteOperation, type ObservabilityConfig, type ProviderResilienceConfig, type PruneInput, type PruneResult, type RecallInput, type RememberInput, type RetrievalConfig, type RetrievalExplanation, type RetrievalScoreWeights, type RetrievalScores, type TokenCounter, createMnemocyte, isMnemocyteError };
 //# sourceMappingURL=index.d.mts.map