npm - @danielsimonjr/memoryjs - Versions diffs - 1.15.0 → 2.1.0 - Mend

@danielsimonjr/memoryjs 1.15.0 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (10) hide show

package/README.md CHANGED Viewed

@@ -1,14 +1,14 @@
 # MemoryJS
-[![Version](https://img.shields.io/badge/version-1.14.0-blue.svg)](https://github.com/danielsimonjr/memoryjs)
+[![Version](https://img.shields.io/badge/version-2.1.0-blue.svg)](https://github.com/danielsimonjr/memoryjs)
 [![NPM](https://img.shields.io/npm/v/@danielsimonjr/memoryjs.svg)](https://www.npmjs.com/package/@danielsimonjr/memoryjs)
 [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
 [![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue.svg)](https://www.typescriptlang.org/)
-[![Tests](https://img.shields.io/badge/tests-6157%20passing-brightgreen.svg)](https://github.com/danielsimonjr/memoryjs)
+[![Tests](https://img.shields.io/badge/tests-7127%20passing-brightgreen.svg)](https://github.com/danielsimonjr/memoryjs)
-A **TypeScript knowledge graph library** for managing entities, relations, and observations with **advanced search**, **hierarchical organization**, **bitemporal versioning**, **causal reasoning**, **role-based access control**, **multi-agent collaboration**, and **multiple storage backends**.
+A **TypeScript knowledge graph library** for managing entities, relations, and observations with **advanced search**, **hierarchical organization**, **bitemporal versioning**, **causal reasoning**, **role-based access control**, **multi-agent collaboration**, **memory-mapped I/O**, **segment-sharded JSONL**, **tiered indexing**, and **multiple storage backends**.
-> Core library powering [@danielsimonjr/memory-mcp](https://www.npmjs.com/package/@danielsimonjr/memory-mcp). **183 TypeScript files**, **62,889 lines of code**, **6157 passing tests**, dual storage backends (JSONL/SQLite + pluggable `IMemoryBackend`), comprehensive search (BM25, TF-IDF, fuzzy with N-gram pre-filter, semantic, hybrid, temporal, LLM-planned, active iterative retrieval), and a complete **Agent Memory System** for AI agents — role profiles, entropy filtering, recursive consolidation, collaborative synthesis with conflict resolution, failure distillation, cognitive load analysis, visibility hierarchies, RBAC, optimistic concurrency, audit attribution, procedural memory, causal reasoning, and a world-model orchestrator.
+> Core library powering [@danielsimonjr/memory-mcp](https://www.npmjs.com/package/@danielsimonjr/memory-mcp). **235 TypeScript files**, **79,378 lines of code**, **7127+ passing tests**, dual storage backends (JSONL/SQLite + pluggable `IMemoryBackend`), comprehensive search (BM25 with incremental indexing, TF-IDF, fuzzy with N-gram pre-filter, semantic, hybrid, temporal, LLM-planned, active iterative retrieval, minimal SPARQL subset), and a complete **Agent Memory System** for AI agents — role profiles, entropy filtering, recursive consolidation, collaborative synthesis with conflict resolution, failure distillation, cognitive load analysis, visibility hierarchies, RBAC, optimistic concurrency, audit attribution, procedural memory, causal reasoning, world-model orchestrator, and the Phase 2 catalog-aligned memory-type slots: **prospective** (intentions-to-act), **failure** (structured pre-task lookup), **plan** (hierarchical goal trees), **reflection** (derived pattern + trajectory summary) + a discriminated `TrustLevel` provenance mixin.
 ## Table of Contents
@@ -58,103 +58,121 @@ A **TypeScript knowledge graph library** for managing entities, relations, and o
 | Module | Files | Key Components |
 |--------|-------|----------------|
-| `agent/` | 61 | AgentMemoryManager, SessionManager, DecayEngine, WorkingMemoryManager, ArtifactManager, DistillationPipeline, RoleProfiles, EntropyFilter, ConsolidationScheduler, MemoryFormatter, CollaborativeSynthesis, FailureDistillation, CognitiveLoadAnalyzer, VisibilityResolver, **MemoryEngine**, **MemoryValidator**, **TrajectoryCompressor**, **ExperienceExtractor**, **CausalReasoner**, **ProcedureManager**, **WorldModelManager**, **ActiveRetrievalController**, **CollaborationAuditEnforcer**, **RbacMiddleware**, **InMemoryBackend** / **SQLiteBackend** |
-| `core/` | 14 | ManagerContext, EntityManager (with OCC), RelationManager (with temporal validity), ObservationManager (with bitemporal axis), HierarchyManager, GraphStorage, SQLiteStorage, GraphTraversal, TransactionManager, RefIndex |
-| `search/` | 37 | SearchManager, RankedSearch (TF-IDF), BM25Search, BooleanSearch, FuzzySearch, SemanticSearch, HybridSearchManager, NGramIndex, TemporalQueryParser, TemporalSearch, LLMQueryPlanner, LLMSearchExecutor, EmbeddingService, VectorStore |
-| `features/` | 17 | IOManager (with RDF/Turtle/JSON-LD export), ArchiveManager, CompressionManager, StreamingExporter, FreshnessManager, AuditLog, GovernanceManager, ContradictionDetector, SemanticForget, AutoLinker |
-| `utils/` | 26 | BatchProcessor, CompressedCache, WorkerPoolManager, MemoryMonitor, schemas (Zod) |
-| `types/` | 7 | Entity, Relation, AgentEntity, SessionEntity, ArtifactEntity, Procedure |
-| `security/` | 2 | **PiiRedactor** + bundled patterns (email/SSN/CC/phone/IP) |
-| `cli/` | 16 | `memory` / `memoryjs` binary commands (entity, relation, search, observation, tag, hierarchy, graph, io, maintenance) |
+| `agent/` | 65 | AgentMemoryManager, SessionManager, DecayEngine, WorkingMemoryManager, ArtifactManager, DistillationPipeline, RoleProfiles, EntropyFilter, ConsolidationScheduler, MemoryFormatter, CollaborativeSynthesis, FailureDistillation, CognitiveLoadAnalyzer, VisibilityResolver, **MemoryEngine**, **MemoryValidator**, **TrajectoryCompressor**, **ExperienceExtractor**, **CausalReasoner**, **ProcedureManager**, **WorldModelManager**, **ActiveRetrievalController**, **CollaborationAuditEnforcer**, **RbacMiddleware**, **InMemoryBackend** / **SQLiteBackend**, **ProspectiveMemoryManager**, **FailureManager**, **PlanManager**, **ReflectionManager** (Phase 2 memory-type slots), **ReflectionStage** + **ProspectivePromotionStage** (pipeline stages) |
+| `core/` | 25 | ManagerContext, EntityManager (with OCC), RelationManager (with temporal validity), ObservationManager (with bitemporal axis), HierarchyManager, GraphStorage, SQLiteStorage, GraphTraversal, TransactionManager, RefIndex, **FileSegmentStorage**, **WriteAheadLog** + **EntityProxy**, **JsonlColumnStore**, **TieredIndex** (`LRUHotTier`/`DiskWarmTier`/`BrotliColdTier`), **IMmapBackend** / **BufferMmapBackend** / **FsReadMmapBackend** |
+| `search/` | 55 | SearchManager, RankedSearch (TF-IDF), BM25Search (incremental), BooleanSearch, FuzzySearch, SemanticSearch, HybridSearchManager, NGramIndex, TemporalQueryParser, TemporalSearch, LLMQueryPlanner, LLMSearchExecutor, EmbeddingService, VectorStore, **SparqlExecutor** (minimal subset), **PartialIndexAdvisor** |
+| `features/` | 20 | IOManager (with RDF/Turtle/JSON-LD export), **BackupManager**, ArchiveManager, CompressionManager, StreamingExporter, FreshnessManager, AuditLog, GovernanceManager, ContradictionDetector, SemanticForget, AutoLinker, **CRDT**, **AnomalyDetector** |
+| `utils/` | 34 | BatchProcessor, CompressedCache, WorkerPoolManager, MemoryMonitor, schemas (Zod), **ICompressionAdapter** / **BrotliCompressionAdapter** / **ZlibCompressionAdapter** / **IdentityCompressionAdapter** / **CompressedMap**, structured `logger`, scheduler/explainPlan/indexHealth diagnostics |
+| `types/` | 7 | Entity, Relation, AgentEntity, SessionEntity, ArtifactEntity, Procedure, **ProspectiveEntity** / **FailureEntity** / **PlanEntity** / **ReflectionEntity** (Phase 2 per-type entities), **TrustLevel** mixin on `MemorySource` |
+| `security/` | 5 | **PiiRedactor** + bundled patterns (email/SSN/CC/phone/IP), **ABAC + RLS + API keys** |
+| `cli/` | 16 | `memory` / `memoryjs` binary commands (entity, relation, search, observation, tag, hierarchy, graph, io, maintenance) + pipe support |
+| `adapters/` | 4 | `IDatabaseAdapter` / `IVectorDBAdapter` interfaces, `LangChainMemoryAdapter`, `RestRouter` |
 | `workers/` | 2 | Levenshtein distance calculations |
-**Total:** 183 TypeScript files | 62,889 lines of code | 6157 passing tests
-### Recent additions (Unreleased — built on top of v1.14.0)
-| Feature | Entry Point |
-|---------|-------------|
-| **η.4.4 Bitemporal Versioning** | `EntityManager.invalidateEntity()` / `entityAsOf()` / `entityTimeline()`; `ObservationManager.invalidateObservation()` / `observationsAsOf()`; `Entity.validFrom` / `validUntil` / `observationMeta[]` |
-| **η.5.4 Linked Data Export** | `ioManager.exportGraph(g, 'turtle' \| 'rdf-xml' \| 'json-ld')` — W3C RDF 1.1; reification fallback for non-NCName predicates |
-| **η.5.5.a Multi-Agent Conflict View** | `SynthesisResult.conflicts[]` + `CollaborativeSynthesis.resolveConflicts(result, policy)` (most_recent / highest_confidence / highest_score / trusted_agent) |
-| **η.5.5.b Visibility Expansion** | `AgentEntity.visibleFrom` / `visibleUntil` / `allowedRoles[]`; `VisibilityResolver` adds time-window gate + role predicate |
-| **η.5.5.c Optimistic Concurrency** | `EntityManager.updateEntity(name, updates, { expectedVersion })` → throws `VersionConflictError` on mismatch |
-| **η.5.5.d Attribution Enforcer** | `CollaborationAuditEnforcer` — strict-mode requires `agentId` on every mutation; appends to `AuditLog` |
-| **η.6.1 RBAC** | `ctx.rbacMiddleware.checkPermission(agentId, action, resourceType, resourceName?)`; `ctx.roleAssignmentStore` with optional JSONL persistence |
-| **η.6.3 PII Redactor** | `new PiiRedactor().redactGraph(graph)` — pluggable regex bank with `redactWithStats()` for compliance audit trails |
-| **3B.4 Procedural Memory** | `ctx.procedureManager.addProcedure({ steps })` / `matchProcedure(context)` / `refineProcedure(id, feedback)` (EWMA success-rate) |
-| **3B.5 Active Retrieval** | `ctx.activeRetrieval.adaptiveRetrieve({ query })` — iterative query rewriting with token-overlap expansion |
-| **3B.6 Causal Reasoning** | `ctx.causalReasoner.findCauses()` / `findEffects()` / `counterfactual()` / `detectCycles()` |
-| **3B.7 World Model** | `ctx.worldModelManager.getCurrentState()` / `validateFact()` / `predictOutcome()` / `detectStateChange()` |
-### v1.13.0 — Memory Intelligence Services (Phase δ)
-| Feature | Entry Point |
-|---------|-------------|
-| Memory Validator | `ctx.memoryValidator.validateConsistency()` / `detectContradictions()` / `repairWithResolver()` / `validateTemporalOrder()` / `calculateReliability()` |
-| Trajectory Compressor | `ctx.trajectoryCompressor.distill()` / `abstractAtLevel()` / `findRedundancies()` / `mergeRedundant()` |
-| Experience Extractor | `ctx.experienceExtractor.extractFromContrastivePairs()` / `clusterTrajectories()` / `synthesizeExperience()` |
-### v1.12.0 — Pluggable Memory Backends (Phase β)
-| Feature | Entry Point |
-|---------|-------------|
-| `IMemoryBackend` interface | `MEMORY_BACKEND=in-memory \| sqlite` selects via `ctx.memoryBackend` |
-| `InMemoryBackend` | Ephemeral, dedup on `(sessionId, content)` |
-| `SQLiteBackend` | Wraps `MemoryEngine` + `DecayEngine.calculatePrdEffectiveImportance()` |
-| Audit-tooling guard | `npm run audit:plans` PostToolUse hook catches plan-doc rot |
-### v1.11.0 — Turn-Aware Memory Engine
-| Feature | Entry Point |
-|---------|-------------|
-| `MemoryEngine` | `ctx.memoryEngine.addTurn()` / `getSessionTurns()` — four-tier dedup (exact / prefix / Jaccard / optional semantic) |
-| `ImportanceScorer` | length × keyword × recent-turn-overlap → integer [0, 10] |
-| O(1) exact-equality dedup | `Entity.contentHash` (SHA-256) + SQLite `idx_entities_content_hash` |
-### v1.9.0 — Temporal Relations + Conversation Ingest
-| Feature | Entry Point |
-|---------|-------------|
-| Temporal Relations | `RelationManager.invalidateRelation()` / `queryAsOf(date)` / `timeline()` |
-| 4-Layer Wake-Up Stack | `ContextWindowManager.wakeUp({ compress })` — ~600-token context bootstrap |
-| Conversation Ingestion | `IOManager.ingest({ messages, ...options })` — format-agnostic pipeline |
-| Per-Agent Diary | `AgentMemoryManager.writeDiary()` / `readDiary()` |
-| Local Embeddings (default) | `MEMORY_EMBEDDING_PROVIDER=local` — zero-config semantic search |
-### v1.8.0 — Memory Versioning + Project Scoping
-| Feature | Entry Point |
-|---------|-------------|
-| Contradiction-driven supersession | `Entity.version` / `parentEntityName` / `rootEntityName` / `supersededBy` |
+**Total:** 235 TypeScript files | 79,378 lines of code | 7127+ passing tests | 11 modules | 1 runtime + 3 type-only circular dependencies (see `docs/architecture/DEPENDENCY_GRAPH.md`)
+### Agent Memory
+| Capability | Entry point |
+|-----------|-------------|
+| Sessions, working memory, episodic memory | `ctx.agentMemory()` — `startSession` / `addWorkingMemory` / `retrieveForContext` |
+| Turn-aware conversation memory with 4-tier dedup (exact / prefix / Jaccard / semantic) | `ctx.memoryEngine.addTurn()` / `getSessionTurns()` |
+| Time-based decay, salience scoring, freshness | `DecayEngine`, `SalienceEngine`, `ctx.freshnessManager` |
+| Role profiles (`researcher` / `planner` / `executor` / `reviewer` / `coordinator`) | `MEMORY_AGENT_ROLE` / `RoleProfileManager.apply(role)` |
+| Memory consolidation, summarization, pattern detection | `ConsolidationPipeline`, `ConsolidationScheduler` |
+| Collaborative synthesis across agents with conflict resolution | `CollaborativeSynthesis.synthesize()` / `resolveConflicts()` |
+| Failure-driven distillation, cognitive load analysis | `FailureDistillation`, `CognitiveLoadAnalyzer` |
+| Procedural memory (executable procedures with feedback refinement) | `ctx.procedureManager.addProcedure()` / `matchProcedure()` / `refineProcedure()` |
+| Active retrieval (iterative query rewriting) | `ctx.activeRetrieval.adaptiveRetrieve()` |
+| Causal reasoning (causes / effects / counterfactuals / cycle detection) | `ctx.causalReasoner.findCauses()` / `findEffects()` / `counterfactual()` |
+| World-state orchestrator | `ctx.worldModelManager.getCurrentState()` / `predictOutcome()` |
+| Per-agent persistent journal | `AgentMemoryManager.writeDiary()` / `readDiary()` |
+| Prospective memory (intentions-to-act with discriminated lifecycle) | `ctx.prospectiveMemory.schedule()` / `fire()` / `cancel()` |
+| Failure memory (pre-task `applicability_hint` lookup) | `ctx.failureManager.record()` / `lookupForTask()` / `markResolved()` |
+| Plan memory (hierarchical goal trees with invariant validation) | `ctx.plan.createPlan()` / `pushSubGoal()` / `transitionNode()` / `getCurrentPath()` |
+| Reflection memory (additive derived insights with content-hash dedup) | `ctx.reflectionManager.create()` / `getRelevantForSession()` / `archive()` |
+| Trust hierarchy (`ground-truth` / `verified` / `inferred` / `unverified`) | `MemorySource.trustLevel?:` + `'trust_level'` `ConflictStrategy` + `inferTrustLevel()` backfill |
+### Search & Retrieval
+| Capability | Entry point |
+|-----------|-------------|
+| Auto-selecting search with method explanation | `ctx.searchManager.autoSearch(query)` |
+| TF-IDF + BM25 ranked search (incremental indexing) | `ctx.rankedSearch`, `BM25Search` |
+| Boolean (AND / OR / NOT) with AST parser | `ctx.searchManager.booleanSearch()` |
+| Fuzzy matching (Levenshtein, N-gram pre-filtered) | `ctx.searchManager.fuzzySearch()` |
+| Semantic search with pluggable embedding provider | `ctx.semanticSearch` (set `MEMORY_EMBEDDING_PROVIDER`) |
+| Hybrid (semantic + lexical + symbolic) | `ctx.hybridSearch` |
+| Temporal range queries with natural-language parsing | `ctx.searchManager.searchByTime("last hour")` |
+| LLM-planned natural-language queries | `ctx.queryNaturalLanguage(query, llmProvider?)` |
+| Minimal SPARQL subset (BGP / FILTER / OPTIONAL / UNION) | `ctx.sparqlExecutor.query()` |
+| Query diagnostics (`explainPlan`, index health) | `ctx.diagnostics` |
+### Knowledge Graph
+| Capability | Entry point |
+|-----------|-------------|
+| Entity / relation / observation CRUD | `ctx.entityManager`, `ctx.relationManager`, `ctx.observationManager` |
+| Optimistic concurrency control on updates | `entityManager.updateEntity(name, updates, { expectedVersion })` |
+| Bitemporal versioning (entities + observations) | `invalidateEntity()` / `entityAsOf()` / `entityTimeline()` |
+| Temporal relations (validity windows) | `relationManager.invalidateRelation()` / `queryAsOf()` / `timeline()` |
+| Memory versioning with contradiction-driven supersession | `Entity.version` / `parentEntityName` / `rootEntityName` / `supersededBy` |
 | Project scoping | `Entity.projectId` + `MEMORY_DEFAULT_PROJECT_ID` |
-| Two-tier deletion | `ctx.semanticForget` — exact match → 0.85 semantic fallback with audit logging |
-### v1.7.0 — Multi-Agent Memory
-| Feature | Entry Point |
-|---------|-------------|
-| Role-Aware Customization | `RoleProfileManager.apply(role)` — salience weights + budget splits |
-| Entropy-Aware Filtering | `EntropyFilter` — Shannon entropy gate in `ConsolidationPipeline` |
-| Recursive Memory Consolidation | `ConsolidationScheduler` — background dedup + merge to fixed point |
-| Salience Budget Allocation | `MemoryFormatter.formatWithSalienceBudget()` |
-| Collaborative Memory Synthesis | `CollaborativeSynthesis.synthesize(entity, hopDepth)` |
-| Failure-Driven Distillation | `FailureDistillation.distill(failureEntity)` |
-| Cognitive Load Metrics | `CognitiveLoadAnalyzer.analyze(memories)` → `CognitiveLoadReport` |
-| Shared Visibility Hierarchies | `VisibilityResolver.canAccess()` — 5-level model |
-### v1.6.0 — Governance + Temporal
-| Feature | Entry Point |
-|---------|-------------|
-| Stable Index Dereferencing | `ctx.refIndex` — `register` / `resolve` / `deregister` |
-| Artifact-Level Granularity | `ctx.agentMemory().artifactManager.createArtifact()` |
-| Temporal Range Queries | `ctx.searchManager.searchByTime()` / `ctx.temporalSearch` |
-| Memory Distillation Policy | `IDistillationPolicy` — wired into `ContextWindowManager` |
-| Freshness Auditing | `ctx.freshnessManager` — TTL, confidence, staleness report |
-| N-gram Pre-filtering | Automatic in `FuzzySearch` via `NGramIndex` |
-| LLM Query Planner | `ctx.queryNaturalLanguage(query, llmProvider?)` |
-| Dynamic Memory Governance | `ctx.governanceManager` — `withTransaction` / `GovernancePolicy` |
+| Two-tier deletion (exact match → 0.85 semantic fallback) | `ctx.semanticForget` |
+| Hierarchical nesting + traversal (ancestors / descendants / subtrees) | `ctx.hierarchyManager` |
+| Graph algorithms — shortest path, all paths, BFS / DFS | `ctx.graphTraversal.findShortestPath()` |
+| Centrality — degree, betweenness, PageRank, HITS hub/authority | `ctx.graphTraversal.pageRank()` / `hits()` |
+| Community detection (Louvain), clique enumeration | `ctx.graphTraversal.louvainCommunities()` / `findMaximalCliques()` |
+| Stable named references for entity name changes | `ctx.refIndex.register()` / `resolve()` |
+| Tag aliases, importance scores (0–10) | `ctx.tagManager`, `Entity.importance` |
+| Multi-format import / export — JSON, CSV, GraphML, GEXF, DOT, Markdown, Mermaid | `ctx.ioManager.exportGraph(format)` |
+| W3C Linked Data export — Turtle, RDF/XML, JSON-LD | `ctx.ioManager.exportGraph('turtle' \| 'rdf-xml' \| 'json-ld')` |
+| Conversation ingestion (format-agnostic) | `ctx.ioManager.ingest(input, options)` |
+### Storage & Performance
+| Capability | Entry point |
+|-----------|-------------|
+| JSONL or SQLite backend (FTS5, BM25, WAL mode) | `MEMORY_STORAGE_TYPE=jsonl\|sqlite` |
+| Pluggable Memory Engine backend | `MEMORY_BACKEND=sqlite\|in-memory` |
+| Memory-mapped file loading for stores > 100 MB | `MEMORY_USE_MMAP=true`, `MEMORY_MMAP_THRESHOLD_BYTES` |
+| Segment-sharded JSONL (FNV-routed N-way shards) | `MEMORY_STORAGE_SEGMENT_COUNT=1..1024` |
+| Columnar observation storage | `JsonlColumnStore`, `ObservationColumn` |
+| Tiered index — LRU hot / disk warm / Brotli cold | `LRUHotTier` → `DiskWarmTier` → `BrotliColdTier` via `TieredIndex` |
+| In-memory entity-cache compression (Zlib / Brotli / Identity) | `ctx.compressedEntityCache`, `CompressedMap` |
+| Write-ahead log + entity proxy for durable mutations | `WriteAheadLog`, `EntityProxy` |
+| Backup lifecycle (create / list / restore / delete / cleanOld) with symlink-attack guards | `ctx.ioManager` (delegates to `BackupManager`) |
+| Streaming export with Brotli compression | `ctx.streamingExporter` |
+| Entity archival to compressed storage | `ctx.archiveManager` |
+| Duplicate detection + entity merging | `ctx.compressionManager` |
+| LSH-based anomaly detection | `ctx.anomalyDetector` |
+### Governance, Security, Multi-Agent
+| Capability | Entry point |
+|-----------|-------------|
+| Policy enforcement + transactional rollback | `ctx.governanceManager.withTransaction()` / `GovernancePolicy` |
+| Immutable JSONL audit trail | `ctx.auditLog` |
+| Strict-mode attribution enforcer | `CollaborationAuditEnforcer` (requires `agentId` on every mutation) |
+| RBAC — role / permission / matrix / middleware | `ctx.rbacMiddleware.checkPermission()` / `ctx.roleAssignmentStore` |
+| ABAC + row-level security + API-key scoping | `src/security/abac.ts`, `rls.ts`, `apiKeys.ts` |
+| PII redactor (email / SSN / CC / phone / IP) with per-pattern stats | `new PiiRedactor().redactGraph()` / `redactWithStats()` |
+| Visibility hierarchies — 5-level (`private` / `team` / `org` / `shared` / `public`) | `VisibilityResolver.canAccess()` |
+| Time-window + role-gated visibility | `AgentEntity.visibleFrom` / `visibleUntil` / `allowedRoles[]` |
+| CRDT primitives for multi-writer scenarios | `src/features/CRDT.ts` |
+| Path-traversal protection (defense-in-depth) | `validateFilePath(path, baseDir?, confineToBase=true)` |
+## What's New
+**Unreleased (Phase 2 memory-types expansion, 2026-05)** — Four catalog-aligned `MemoryType` slots and one provenance mixin land on top of v1.15:
+- **Sprint 4 — Failure Memory**: `FailureManager` + `MemoryType: 'failure'` + `ctx.failureManager`. Structured `FailureRecord` with `applicability_hint` retrieval key; pre-task `lookupForTask(taskContext)` scoring; discriminated `MarkResolvedResult`.
+- **Sprint 5 — Plan / Goal Stack**: `PlanManager` + `MemoryType: 'plan'` + `ctx.plan`. Recursive `GoalNode` tree with discriminated `PlanLifecycle` / `GoalNodeLifecycle`, branded `PlanId` / `GoalNodeId`, `validatePlanInvariants` after every mutation, cycle-protected DFS.
+- **Sprint 6 — Trust Hierarchy formalization (partial)**: `TrustLevel` discriminated mixin on `MemorySource` (`'ground-truth' | 'verified' | 'inferred' | 'unverified'`) with `inferTrustLevel` backfill and `'trust_level'` `ConflictStrategy`. `CollaborativeSynthesis.resolveConflicts` ordering integration deferred.
+- **Sprint 8 — Reflection Log scheduled pass**: `ReflectionManager` + `MemoryType: 'reflection'` + `ReflectionStage` pipeline stage + `ctx.reflectionManager` (publicly aliased as `ReflectionMemoryManager`). Additive (no supersession); content-hash dedup at create; session-end scheduling via `runOnSessionEnd(sessionId)` helper.
+**v1.15.0** — Twelve-phase performance & scale track adding mmap-backed I/O, segment-sharded JSONL, columnar observation storage, tiered indexing, pluggable in-memory compression, a minimal SPARQL subset, write-ahead log, extracted `BackupManager`, CRDT primitives, ABAC + RLS + API keys, HITS / clique / Louvain graph algorithms, and a hardened security baseline (`crypto.randomBytes` for IDs, ReDoS-resistant regex escapes, bounded `TaskQueue`).
+See [CHANGELOG.md](CHANGELOG.md) for the full per-version history. The roadmap and remaining items live in [`docs/roadmap/ROADMAP.md`](docs/roadmap/ROADMAP.md) and [`docs/roadmap/MEMORY_TYPES_EXPANSION_PHASE_2.md`](docs/roadmap/MEMORY_TYPES_EXPANSION_PHASE_2.md).
 ## Installation
@@ -259,7 +277,7 @@ await enforcer.createEntities([{ name: 'X', entityType: 't', observations: ['fac
   'agent-alice'); // throws AttributionRequiredError if agentId is missing
 ```
-### 6. Bitemporal versioning (η.4.4)
+### 6. Bitemporal versioning
 ```typescript
 // Mark an entity as no longer valid at a specific time
@@ -277,7 +295,7 @@ const obsAtTime = await ctx.observationManager.observationsAsOf(
 );
 ```
-### 7. Causal reasoning + world model (3B.6 / 3B.7)
+### 7. Causal reasoning and world model
 ```typescript
 // Forward inference — "what does X cause?"
@@ -294,7 +312,7 @@ const after = await ctx.worldModelManager.getCurrentState();
 const change = ctx.worldModelManager.detectStateChange(before, after);
 ```
-### 8. RBAC + PII redaction (η.6.1 / η.6.3)
+### 8. RBAC and PII redaction
 ```typescript
 // Grant a role
@@ -327,11 +345,11 @@ interface Entity {
   createdAt?: string;        // ISO 8601 timestamp
   lastModified?: string;     // ISO 8601 timestamp
-  // v1.6.0 — Freshness
+  // Freshness
   ttl?: number;              // Seconds until stale
   confidence?: number;       // [0, 1] belief strength
-  // v1.8.0 — Project scoping + supersession
+  // Project scoping + supersession
   projectId?: string;        // Multi-project isolation
   version?: number;          // Supersession chain version (also drives η.5.5.c OCC)
   parentEntityName?: string;
@@ -339,7 +357,7 @@ interface Entity {
   isLatest?: boolean;
   supersededBy?: string;
-  // v1.11.0 — Memory Engine dedup
+  // Memory Engine dedup
   contentHash?: string;      // SHA-256 for O(1) Tier-1 dedup
   // η.4.4 — Bitemporal validity (orthogonal to supersession)
@@ -408,13 +426,13 @@ ctx.memoryBackend        // Pluggable IMemoryBackend (in-memory / sqlite)
 ctx.contextWindowManager // 4-layer wake-up stack + token budgeting
 ctx.agentMemory()        // Full Agent Memory System facade
-// Memory intelligence (Phase δ)
+// Memory intelligence
 ctx.memoryValidator         // Validate consistency / contradictions / temporal order
 ctx.trajectoryCompressor    // Distill / abstract / merge redundant trajectories
 ctx.experienceExtractor     // Cluster trajectories → reusable experience patterns
 ctx.patternDetector         // Trigger / sequence / outcome pattern mining
-// Memory theory (Phase 3B)
+// Memory theory
 ctx.procedureManager     // 3B.4 — executable procedure memory + EWMA refinement
 ctx.causalReasoner       // 3B.6 — findCauses / findEffects / counterfactual
 ctx.worldModelManager    // 3B.7 — snapshot orchestrator + state-change diff
@@ -650,13 +668,27 @@ await agent.endSession(session.name);
 ### Memory Types
 ```typescript
-type MemoryType = 'working' | 'episodic' | 'semantic' | 'procedural';
+type MemoryType =
+  | 'working'      // Short-term, session-scoped memories that may be promoted
+  | 'episodic'     // Timeline-based event memories with temporal ordering
+  | 'semantic'     // Long-term factual knowledge
+  | 'procedural'   // Learned behaviors and patterns (3B.4)
+  | 'prospective'  // (Phase 1) Intentions-to-act at a future time / event / condition
+  | 'failure'      // (Phase 2 Sprint 4) Pre-task failure lookup with applicability_hint
+  | 'plan'         // (Phase 2 Sprint 5) Hierarchical goal trees with sub-tasks + acceptance criteria
+  | 'reflection';  // (Phase 2 Sprint 8) Additive derived insights with content-hash dedup
 ```
 - **Working Memory**: Short-term, session-scoped memories that may be promoted
 - **Episodic Memory**: Timeline-based event memories with temporal ordering
 - **Semantic Memory**: Long-term factual knowledge
-- **Procedural Memory**: Learned behaviors and patterns
+- **Procedural Memory**: Learned behaviors and patterns (`ctx.procedureManager`)
+- **Prospective Memory**: Forward-looking intentions with discriminated `ProspectiveLifecycle` (`pending` / `fired` / `cancelled` / `expired`); `ctx.prospectiveMemory`. Catalog Type 4
+- **Failure Memory**: Structured `FailureRecord` with `applicability_hint` as the retrieval key; `markResolved` returns discriminated `MarkResolvedResult`; `ctx.failureManager`. Catalog Type 9
+- **Plan Memory**: Recursive `GoalNode` tree with discriminated `PlanLifecycle` / `GoalNodeLifecycle`, branded `PlanId` / `GoalNodeId`, `validatePlanInvariants` after every mutation; `ctx.plan`. Catalog Type 6
+- **Reflection Memory**: Additive (no supersession of evidence entities) with `ReflectionScope` discriminator (`session` / `project` / `global`); content-hash dedup at `create`; `ctx.reflectionManager`. Catalog Type 10. Produced by `ReflectionStage` pipeline stage
+**Trust-hierarchy mixin** (Phase 2 Sprint 6, Catalog Type 12): every `MemorySource` may carry an optional categorical `trustLevel?: TrustLevel` (`'ground-truth' | 'verified' | 'inferred' | 'unverified'`) — backfilled from `method` + `reliability` via `inferTrustLevel(source)`. Powers the `'trust_level'` `ConflictStrategy` with recency tiebreak.
 ### Decay System
@@ -713,7 +745,7 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `batchUpdate(updates[])` | Atomic multi-entity update |
 | `addTags(name, tags)` / `removeTags(name, tags)` | Tag management |
 | `setImportance(name, score)` | Set importance (0-10) |
-| `getVersionChain(name)` / `getLatestVersion(name)` | v1.8.0 supersession chains |
+| `getVersionChain(name)` / `getLatestVersion(name)` | supersession chains |
 | `invalidateEntity(name, ended?)` | η.4.4 — set `validUntil` |
 | `entityAsOf(name, asOf)` | η.4.4 — time-travel query |
 | `entityTimeline(name)` | η.4.4 — versions sorted by `validFrom` |
@@ -725,9 +757,9 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `createRelations(relations)` | Create multiple relations |
 | `getRelations(entityName)` | Get incoming/outgoing relations |
 | `deleteRelations(relations)` | Delete specific relations |
-| `invalidateRelation(from, type, to, ended?)` | v1.9.0 — set `validUntil` on a relation |
-| `queryAsOf(entity, asOf, { direction? })` | v1.9.0 — relations valid at time T |
-| `timeline(entity, { direction? })` | v1.9.0 — chronological history |
+| `invalidateRelation(from, type, to, ended?)` | set `validUntil` on a relation |
+| `queryAsOf(entity, asOf, { direction? })` | relations valid at time T |
+| `timeline(entity, { direction? })` | chronological history |
 ### ObservationManager
@@ -756,9 +788,9 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `exportGraph(graph, format)` | Export to `json` / `csv` / `graphml` / `gexf` / `dot` / `markdown` / `mermaid` / `turtle` / `rdf-xml` / `json-ld` |
 | `exportGraphWithCompression(graph, format, options?)` | Brotli-compressed export |
 | `importGraph(format, data, options)` | Import with merge strategies (`replace` / `skip` / `merge` / `fail`) |
-| `ingest({ messages }, options?)` | v1.9.0 — conversation ingestion pipeline |
-| `splitSessions(content, options?)` | v1.9.0 — split multi-session transcripts |
-| `visualizeGraph(options?)` | v1.9.1 — interactive HTML visualization |
+| `ingest({ messages }, options?)` | conversation ingestion pipeline |
+| `splitSessions(content, options?)` | split multi-session transcripts |
+| `visualizeGraph(options?)` | interactive HTML visualization |
 | `createBackup(options)` / `restoreBackup(path)` | Backup management |
 ### GraphTraversal
@@ -771,7 +803,7 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `getConnectedComponents()` | Find isolated subgraphs |
 | `bfs(start, options)` / `dfs(start, options)` | Traversal |
-### CausalReasoner (3B.6)
+### CausalReasoner
 | Method | Description |
 |--------|-------------|
@@ -780,7 +812,7 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `counterfactual({ seed, removeFrom, removeTo, predict })` | Chains surviving edge removal (pure; no graph mutation) |
 | `detectCycles(seed, maxDepth?)` | Depth-bounded DFS over causal subgraph |
-### ProcedureManager (3B.4)
+### ProcedureManager
 | Method | Description |
 |--------|-------------|
@@ -790,7 +822,7 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `matchProcedure(context, candidates, threshold?)` | Token-overlap match |
 | `refineProcedure(id, { succeeded, notes? })` | EWMA success-rate update |
-### WorldModelManager (3B.7)
+### WorldModelManager
 | Method | Description |
 |--------|-------------|
@@ -799,21 +831,21 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `predictOutcome(action, candidates)` | Delegates to `CausalReasoner.findEffects` |
 | `detectStateChange(before, after)` | Pure snapshot diff |
-### ActiveRetrievalController (3B.5)
+### ActiveRetrievalController
 | Method | Description |
 |--------|-------------|
 | `shouldRetrieve(context)` | Cost heuristic; rejects empty / over-budget |
 | `adaptiveRetrieve(context)` | Iterative rewrite + retrieve until coverage threshold |
-### CollaborativeSynthesis (η.5.5.a)
+### CollaborativeSynthesis
 | Method | Description |
 |--------|-------------|
 | `synthesize(seedEntity, context?)` | BFS + salience scoring; surfaces multi-agent `conflicts[]` |
 | `resolveConflicts(result, policy)` | Pick winners per `most_recent` / `highest_confidence` / `highest_score` / `trusted_agent` |
-### MemoryValidator (Phase δ)
+### MemoryValidator
 | Method | Description |
 |--------|-------------|
@@ -823,7 +855,7 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `validateTemporalOrder(observations)` | Sync `[T=ISO]` ordering check |
 | `calculateReliability(entity)` | Confidence × confirmation × age penalty |
-### RbacMiddleware (η.6.1)
+### RbacMiddleware
 | Method | Description |
 |--------|-------------|
@@ -832,7 +864,7 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `roleAssignmentStore.revoke(agentId, role, resourceType?)` | Remove a grant |
 | `roleAssignmentStore.listActive(agentId, now?)` | Active grants at a point in time |
-### PiiRedactor (η.6.3)
+### PiiRedactor
 | Method | Description |
 |--------|-------------|
@@ -840,6 +872,46 @@ const results = await agent.searchCrossAgent('agent_2', 'query');
 | `redactWithStats(text)` | Returns `{ text, stats: { totalRedactedBytes, countsByPattern } }` |
 | `redactGraph(graph)` | Apply to every observation in a graph-shaped object |
+### ProspectiveMemoryManager (`ctx.prospectiveMemory`)
+| Method | Description |
+|--------|-------------|
+| `schedule(intention, options?)` | Persist a `ProspectiveEntity`; trigger kind `time-based` / `time-window` / `event` / `conditional` |
+| `fire(id, result?)` | Transition `pending → fired`; returns discriminated `MarkResolvedResult` |
+| `cancel(id, reason?)` | Transition `pending → cancelled`; returns `CancelResult` |
+| `expireDueIntentions()` | Bulk-expire past-due pending intentions; returns count |
+| `getPending(filter?)` / `getFired(filter?)` | Query by `sessionId` / `agentId` |
+### FailureManager (`ctx.failureManager`)
+| Method | Description |
+|--------|-------------|
+| `record(input, options?)` | Persist a structured `FailureRecord`; validates five required non-empty fields |
+| `lookupForTask(taskContext, options?)` | Pre-task substring-match scoring (`applicability_hint` 3× / `context` 2× / `attempted` 1×) |
+| `markResolved(id, reason?)` | Returns discriminated `MarkResolvedResult` (`resolved` / `already-resolved` / `not-found` / `vanished-mid-update`) |
+| `getAll(options?)` | Filter by `status` and/or `sourceSessionId` |
+### PlanManager (`ctx.plan`)
+| Method | Description |
+|--------|-------------|
+| `createPlan(rootDescription, options?)` | Create a single-node plan tree; mints branded `PlanId` |
+| `pushSubGoal(planId, parentNodeId, description, options?)` | Append a child `GoalNode`; throws on `persistPlan` failure |
+| `transitionNode(planId, nodeId, transition)` | Unified `GoalNodeLifecycle` state-machine entry point |
+| `markPlanComplete(planId, note?)` / `abandonPlan(planId, reason?)` | Plan-level lifecycle; returns `MarkResolvedResult` |
+| `findPlan` / `findNode` / `getCurrentPath` | `Readonly<>` reads (clone-free) |
+| `getActivePlan(sessionId)` / `listPlans(options?)` | Session-scoped + filtered queries |
+### ReflectionManager (`ctx.reflectionManager`)
+| Method | Description |
+|--------|-------------|
+| `create(input, options?)` | Persist a `ReflectionRecord`; content-hash dedup on `sha256(scope\|sorted(evidence))` |
+| `list(options?)` | Filter by `scope` / `sourceSessionId` / `minConfidence` / `includeArchived` / `limit` |
+| `getRelevantForSession(sessionId, options?)` | Reflections matching `sourceSessionId` OR overlapping evidence; confidence-sorted |
+| `archive(id)` | Soft-delete; returns discriminated `ArchiveReflectionResult` |
+| `ReflectionStage.runOnSessionEnd(sessionId)` | Pipeline stage helper — runs the reflection pass scoped to one session |
 ## Configuration
 ### Environment Variables
@@ -863,10 +935,15 @@ The full env-var reference lives in [CLAUDE.md](CLAUDE.md). Most-used:
 | `MEMORY_AUDIT_ATTRIBUTION_REQUIRED` | `CollaborationAuditEnforcer` strict mode | `false` |
 | `MEMORY_RBAC_ENABLED` | Wire `RbacMiddleware` into `GovernancePolicy` | `false` |
 | `MEMORY_DEFAULT_VISIBILITY` | Default `AgentEntity.visibility` | `private` |
+| `MEMORY_USE_MMAP` | Use mmap (`FsReadMmapBackend`) for `GraphStorage.loadFromDisk` | `false` (strict `'true'` literal match) |
+| `MEMORY_MMAP_THRESHOLD_BYTES` | Minimum file size to trigger mmap path; `0` means always-on | `104857600` (100 MB) |
+| `MEMORY_STORAGE_SEGMENT_COUNT` | Number of FNV-routed JSONL shards in `FileSegmentStorage` (1–1024) | `1` (off) |
+| `MEMORY_SQLITE_READ_POOL_SIZE` | Read-connection pool size for `SQLiteStorage` | `4` |
+| `MEMORY_SQLITE_AUTO_INDEX` | Enable `PartialIndexAdvisor`-driven automatic partial-index DDL | `false` |
 | `SKIP_BENCHMARKS` | Skip perf benchmark tests | `false` |
 | `LOG_LEVEL` | `debug` / `info` / `warn` / `error` | (none) |
-See [CLAUDE.md](CLAUDE.md#environment-variables) for the complete list (~50 variables across decay/salience/context-window/freshness/RBAC/PRD scoring/MemoryEngine knobs).
+See [CLAUDE.md](CLAUDE.md#environment-variables) for the complete list (~60 variables across decay/salience/context-window/freshness/RBAC/PRD scoring/MemoryEngine/mmap/segment/SQLite-pool/PartialIndex knobs).
 ## Development
@@ -962,9 +1039,9 @@ memoryjs/
 │   │   ├── MemoryBackend.ts            # IMemoryBackend interface (v1.12)
 │   │   ├── InMemoryBackend.ts          # Ephemeral adapter
 │   │   ├── SQLiteBackend.ts            # SQLite-backed adapter
-│   │   ├── MemoryValidator.ts          # Phase δ — consistency checks
-│   │   ├── TrajectoryCompressor.ts     # Phase δ — distill / merge
-│   │   ├── ExperienceExtractor.ts      # Phase δ — pattern abstraction
+│   │   ├── MemoryValidator.ts          # Memory consistency / contradiction checks
+│   │   ├── TrajectoryCompressor.ts     # Distill / merge trajectory data
+│   │   ├── ExperienceExtractor.ts      # Pattern abstraction from contrastive pairs
 │   │   ├── PatternDetector.ts          # Sequence + outcome mining
 │   │   ├── causal/                     # 3B.6 — CausalReasoner
 │   │   ├── procedural/                 # 3B.4 — ProcedureManager + Sequencer
@@ -1051,7 +1128,7 @@ Comprehensive architecture documentation in `docs/architecture/`:
 ADRs and roadmap:
-- [ARCHITECTURE_DECISIONS.md](docs/development/ARCHITECTURE_DECISIONS.md) - including ADR-011 (Phase δ wrap-and-extend pattern)
+- [ARCHITECTURE_DECISIONS.md](docs/development/ARCHITECTURE_DECISIONS.md) - including ADR-011 (wrap-and-extend pattern for memory intelligence services)
 - [ROADMAP.md](docs/roadmap/ROADMAP.md) - Feature roadmap with implementation details
 - [docs/superpowers/plans/](docs/superpowers/plans/) - Phase α–η implementation plans