ai-memory-layer 2.0.0

package/docs/DEPLOYMENT.md

@@ -0,0 +1,84 @@
+# Deployment Guide
+
+`memory-layer` can run embedded in-process or as a standalone HTTP/MCP service.
+
+## Embedded Package
+
+Use the package directly when your application already runs in Node.js:
+
+```ts
+import { createMemory } from 'ai-memory-layer';
+
+const memory = createMemory({
+  adapter: 'sqlite',
+  path: './data/memory.db',
+  preset: 'ai_ide',
+  scope: 'default',
+});
+```
+
+This is the lowest-friction option for AI IDEs, copilots, and single-service agents.
+
+The zero-config quick path is pure-JS and ephemeral. For durable local storage, install the optional `better-sqlite3` package and pass `adapter: 'sqlite'` with a file path.
+
+## Standalone HTTP Service
+
+Run the built-in server when multiple processes need shared memory:
+
+```bash
+npx memory-layer serve \
+  --transport http \
+  --db ./data/memory.db \
+  --preset autonomous_agent \
+  --port 3100
+```
+
+Recommended environment variables:
+
+```bash
+MEMORY_DB_PATH=./data/memory.db
+MEMORY_TRANSPORT=http
+MEMORY_PORT=3100
+MEMORY_API_KEY=replace-me
+MEMORY_ADMIN_API_KEY=replace-me-admin
+```
+
+## Docker
+
+Build and run the provided image:
+
+```bash
+docker build -t memory-layer .
+docker run --rm \
+  -p 3100:3100 \
+  -v "$(pwd)/data:/data" \
+  -e MEMORY_API_KEY=local-dev-key \
+  -e MEMORY_ADMIN_API_KEY=local-dev-admin \
+  memory-layer
+```
+
+The container persists SQLite data under `/data/memory.db`.
+
+## MCP Transport
+
+Use the MCP server when integrating with AI tools that speak the Model Context Protocol:
+
+```bash
+npx memory-layer serve --transport mcp --db ./data/memory.db --preset ai_ide
+```
+
+The server reads and writes on stdio, so it fits directly into MCP-compatible runtimes.
+
+## Production Notes
+
+- Use a file-backed SQLite database or Postgres for durable deployments.
+- Install `better-sqlite3` only when you want the durable SQLite path.
+- Install `pg` only when you want the hosted Postgres path.
+- Treat SQLite as the lowest-friction embedded path. Its semantic retrieval is an in-process scan over local embeddings, which is appropriate for local and moderate-sized workloads but not the strongest scaling path.
+- Treat SQLite HTTP/MCP deployments as a single-process service contract. It is the right fit when one runtime owns writes and other components talk to that one service.
+- Use Postgres when multiple processes, workers, or hosted instances need to write shared memory concurrently. That is the operationally safe multi-writer path.
+- For the strongest hosted retrieval path, use Postgres with the `pgvector` extension enabled and keep the `knowledge_embeddings` HNSW index from `src/adapters/postgres/schema.sql`.
+- Put `MEMORY_API_KEY` behind an API gateway or private network if the service is shared.
+- Reserve `MEMORY_ADMIN_API_KEY` for compaction and maintenance automation.
+- Keep `bodyLimitBytes` low unless you intentionally ingest large prompts or transcripts.
+- Enable SSE consumers on `/v1/events` when you want real-time observability hooks.

package/docs/INTEGRATIONS.md

@@ -0,0 +1,64 @@
+# Integration Guide
+
+## Core Choices
+
+Pick the narrowest integration surface that fits your system:
+
+- Package API: best for Node apps and AI IDE extensions
+- HTTP API: best for polyglot services and hosted memory
+- MCP server: best for tools that already use MCP
+
+## AI IDE Pattern
+
+Use `createMemoryRuntime()` to inject prompt-ready context before each model call and persist the exchange afterward:
+
+```ts
+const prepared = await runtime.beforeModelCall(userInput);
+const result = await model(prepared.prompt);
+await runtime.afterModelCall({
+  userInput,
+  assistantOutput: result,
+});
+```
+
+## Hosted Service Pattern
+
+Run one HTTP service and route each request into its own scope:
+
+- `tenant_id`: customer or org
+- `system_id`: product surface
+- `workspace_id`: shared project memory
+- `scope_id`: thread, task, run, or conversation
+
+Operational contract:
+
+- Use a single SQLite-backed service when one process should own writes.
+- Use Postgres-backed hosting when multiple workers or agents need concurrent shared-memory writes.
+- Use `collaboration_id` when memory must be intentionally shared across distinct systems without collapsing all workspace memory together.
+
+## Autonomous Agent Pattern
+
+Use the `autonomous_agent` preset, aggressive compaction, work-item tracking, and periodic maintenance:
+
+```ts
+await manager.trackWorkItem('Finish migration rollout', 'objective', 'in_progress');
+await runtime.afterModelCall({ userInput, assistantOutput });
+await manager.runMaintenance();
+```
+
+## Framework Adapters
+
+The repo now ships tested integrations for:
+
+- Claude-adjacent agent wrapping via `wrapClaudeAgentModel()`
+- OpenAI/Claude tool-call surfaces via `createOpenAIMemoryTools()` and `createClaudeMemoryTools()`
+- Vercel AI SDK middleware-style wrapping via `wrapVercelAIModel()`
+- LangChain chat-history style bridging via `createLangChainMemoryBridge()`
+
+Runnable examples:
+
+- `examples/autonomous-agent.ts`: Claude-style lifecycle wrapping without requiring a provider SDK just to understand the flow
+- `examples/tool-calling-agent.ts`: OpenAI-compatible tool surface
+- `examples/langchain.ts`: LangChain memory variable bridge
+- `examples/multi-agent-postgres.ts`: real Postgres-backed shared memory using `MEMORY_DATABASE_URL`
+- `clients/python/`: hosted Python client helpers for service-oriented deployments

package/docs/MEMORY_QUALITY_BASELINE.md

@@ -0,0 +1,55 @@
+# Memory Quality Baseline
+
+This document records the current release-quality baseline used by the enforced delta report in `evals/memory-quality/baseline.json`.
+
+It is the known-good anchor that future releases must not regress from.
+
+## Baseline Run
+
+Command:
+
+```bash
+npm run eval:memory-quality
+```
+
+Result:
+
+- overall score: `100`
+- passed: `true`
+
+## Metrics
+
+| Metric | Baseline | Threshold | Status |
+|---|---:|---:|---|
+| `constraintRetentionRate` | `1.00` | `0.92` | pass |
+| `preferenceRetentionRate` | `1.00` | `0.90` | pass |
+| `identityRetentionRate` | `1.00` | `0.95` | pass |
+| `procedureRetentionRate` | `1.00` | `0.88` | pass |
+| `updateCorrectnessRate` | `1.00` | `0.88` | pass |
+| `strategyOutcomeRecallRate` | `1.00` | `0.85` | pass |
+| `falseMemoryRate` | `0.00` | `0.05` | pass |
+| `contradictionResolutionAccuracy` | `1.00` | `0.85` | pass |
+| `trustedMemoryPrecision` | `1.00` | `0.90` | pass |
+| `trustedMemoryRecall` | `1.00` | `0.88` | pass |
+| `memoryIsolationAccuracy` | `1.00` | `0.95` | pass |
+| `provisionalLeakRate` | `0.00` | `0.08` | pass |
+| `postCompactionFidelityScore` | `1.00` | `0.88` | pass |
+| `postMaintenanceFidelityScore` | `1.00` | `0.86` | pass |
+
+## What This Baseline Means
+
+This baseline represents the current release claim:
+
+- evidence-grounded promotion is working
+- contradiction handling is explicit and safe
+- trust-aware retrieval prefers durable memory correctly
+- long-horizon compaction and maintenance preserve critical memory
+- isolation and cross-scope behavior remain safe by default
+- fresh-install no-provider replay still preserves the right local memory contract
+- hosted shared-memory replay still surfaces the right cross-scope knowledge
+
+Because the baseline is a known-good release anchor, the delta gate has a stricter meaning:
+
+- green delta output means the current build has not regressed from the proven release baseline
+- baseline refreshes should only happen after a full hard-gate pass
+- any future baseline change should be treated as a deliberate quality reset, not a convenience update

package/docs/MEMORY_QUALITY_RELEASE_GATE.md

@@ -0,0 +1,63 @@
+# Memory Quality Release Gate
+
+`memory-layer` treats memory quality as a release blocker, not a best-effort benchmark.
+
+## Required Commands
+
+Run these before shipping:
+
+```bash
+npm run eval:retrieval:enforce
+npm run eval:memory-quality:enforce
+npm run eval:memory-quality:delta:enforce
+npm run python:check
+npm run eval:platform-quality:enforce
+```
+
+The enforced memory-quality run must pass every threshold. The enforced delta report blocks regressions versus the recorded baseline in `evals/memory-quality/baseline.json`. Refresh that baseline only after a full hard-gate pass so it remains a known-good release anchor. The Python and platform checks prove that hosted HTTP, Node CLI inspection, Python client surfaces, the fresh-install no-provider replay, and the hosted shared-memory replay all still work against the same product contract.
+
+## Final Thresholds
+
+| Metric | Threshold |
+|---|---:|
+| `constraintRetentionRate` | `>= 0.92` |
+| `preferenceRetentionRate` | `>= 0.90` |
+| `identityRetentionRate` | `>= 0.95` |
+| `procedureRetentionRate` | `>= 0.88` |
+| `updateCorrectnessRate` | `>= 0.88` |
+| `strategyOutcomeRecallRate` | `>= 0.85` |
+| `falseMemoryRate` | `<= 0.05` |
+| `contradictionResolutionAccuracy` | `>= 0.85` |
+| `trustedMemoryPrecision` | `>= 0.90` |
+| `trustedMemoryRecall` | `>= 0.88` |
+| `memoryIsolationAccuracy` | `>= 0.95` |
+| `provisionalLeakRate` | `<= 0.08` |
+| `postCompactionFidelityScore` | `>= 0.88` |
+| `postMaintenanceFidelityScore` | `>= 0.86` |
+
+## How To Read The Score
+
+- `overallScore = 100` means every tracked metric met or exceeded its threshold.
+- `passed = true` means there are no threshold failures.
+- The score is evidence-backed only when the scenario list and per-metric evaluations also pass.
+
+## Quality Modes
+
+The quick factory reports mode behavior separately from the aggregate score:
+
+- `fast_adoption`: easiest start, weakest trust posture.
+- `balanced_memory`: recommended default.
+- `high_fidelity_memory`: strictest safety and lifecycle posture.
+
+Mode reporting is descriptive. The release gate is still the main memory-quality suite.
+
+## Platform Proof
+
+`memory-layer` does not treat core-engine quality as sufficient proof on its own. The final gate also requires:
+
+- `npm run python:check`
+  Verifies the Python package can be installed in a clean virtualenv, built, linted, and tested.
+- `npm run eval:platform-quality:enforce`
+  Starts the hosted server, seeds real memory, verifies hosted inspection routes, verifies the Node inspection CLI, verifies the Python CLI against the same live service, and replays a shared-memory hosted trace across multiple scopes.
+
+The release claim is only defensible when both the engine-quality gate and the platform-quality gate are green.

package/docs/MEMORY_QUALITY_RUBRIC.md

@@ -0,0 +1,249 @@
+# Memory Quality Rubric
+
+This rubric defines what "memory quality" means for `memory-layer`.
+
+The goal is not to measure how many features the system has. The goal is to measure whether an AI-heavy product can safely keep context, learn over time, and recall the right memory later.
+
+## Scoring Philosophy
+
+The overall score is a weighted summary of memory-quality metrics. A higher score means:
+
+- important memory survives over time
+- durable knowledge is correct more often
+- updates and contradictions are handled safely
+- retrieval surfaces trustworthy memory
+- workflow memory behavior is safe across scopes and lineages
+
+The score does **not** give extra credit for:
+
+- packaging quality
+- transport surfaces
+- documentation quality
+- number of tests
+- size or complexity of the implementation
+
+Those matter for adoption, but not for memory quality.
+
+## Metrics
+
+All metrics are normalized to `0.0` through `1.0`.
+
+### `constraintRetentionRate`
+
+How often durable constraints that should survive are still available when needed later.
+
+Formula:
+
+`correctly_recalled_constraints / total_expected_constraints`
+
+Target:
+
+`>= 0.92`
+
+### `preferenceRetentionRate`
+
+How often user or system preferences survive long enough to be recalled correctly.
+
+Formula:
+
+`correctly_recalled_preferences / total_expected_preferences`
+
+Target:
+
+`>= 0.90`
+
+### `identityRetentionRate`
+
+How often durable identity information survives and is recalled correctly.
+
+Formula:
+
+`correctly_recalled_identity_facts / total_expected_identity_facts`
+
+Target:
+
+`>= 0.95`
+
+### `procedureRetentionRate`
+
+How often durable procedural knowledge remains available and correctly recalled.
+
+Formula:
+
+`correctly_recalled_procedures / total_expected_procedures`
+
+Target:
+
+`>= 0.88`
+
+### `updateCorrectnessRate`
+
+How often the system prefers the latest correct fact when a value is revised or reversed.
+
+Formula:
+
+`correct_updates_handled / total_update_scenarios`
+
+Target:
+
+`>= 0.88`
+
+### `strategyOutcomeRecallRate`
+
+How often the system remembers that a strategy succeeded or failed, and recalls that outcome appropriately.
+
+Formula:
+
+`correct_strategy_outcome_recalls / total_strategy_outcome_checks`
+
+Target:
+
+`>= 0.85`
+
+### `falseMemoryRate`
+
+How often the system promotes or recalls unsupported memory as if it were durable truth.
+
+Formula:
+
+`false_durable_memories_detected / total_false_memory_checks`
+
+Target:
+
+`<= 0.05`
+
+### `contradictionResolutionAccuracy`
+
+How often contradictions are handled safely instead of silently preserving outdated memory.
+
+Formula:
+
+`correctly_resolved_contradictions / total_contradiction_checks`
+
+Target:
+
+`>= 0.85`
+
+### `trustedMemoryPrecision`
+
+How often memory that is surfaced as durable/trusted is actually correct and appropriate.
+
+Formula:
+
+`correct_trusted_recalls / total_trusted_recalls`
+
+Target:
+
+`>= 0.90`
+
+### `trustedMemoryRecall`
+
+How often the system successfully surfaces trusted memory when it should.
+
+Formula:
+
+`trusted_memories_recalled_when_needed / total_trusted_memory_needs`
+
+Target:
+
+`>= 0.88`
+
+### `memoryIsolationAccuracy`
+
+How often the system preserves the right boundary behavior between local, lineage, workspace, and cross-scope memory.
+
+Formula:
+
+`correct_isolation_or_inheritance_behaviors / total_isolation_checks`
+
+Target:
+
+`>= 0.95`
+
+### `provisionalLeakRate`
+
+How often weak, provisional, or otherwise unsafe memory leaks into default recall behavior.
+
+Formula:
+
+`unsafe_provisional_surface_events / total_provisional_safety_checks`
+
+Target:
+
+`<= 0.08`
+
+### `postCompactionFidelityScore`
+
+How much important information survives compaction without corruption.
+
+Formula:
+
+`important_facts_preserved_after_compaction / total_important_facts_checked_after_compaction`
+
+Target:
+
+`>= 0.88`
+
+### `postMaintenanceFidelityScore`
+
+How much important memory remains correct and available after lifecycle maintenance.
+
+Formula:
+
+`important_memories_preserved_after_maintenance / total_important_memories_checked_after_maintenance`
+
+Target:
+
+`>= 0.86`
+
+## Overall Score
+
+Each metric contributes equally to the overall score for now. The weighted metric score is the average of all per-metric normalized scores, multiplied by `100`.
+
+For metrics where higher is better:
+
+`normalized = min(actual / target, 1)`
+
+For metrics where lower is better:
+
+`normalized = 1` when `actual <= target`, otherwise `target / actual`
+
+Overall score:
+
+`overallScore = average(normalized_metrics) * 100`
+
+## Pass / Fail
+
+The suite passes only if **all** threshold metrics pass.
+
+This is intentionally strict. A memory system is only as safe as its weakest major behavior.
+
+## Score Interpretation
+
+- `95-100`: elite memory behavior with strong evidence
+- `90-94`: very strong, but still has a few meaningful edge weaknesses
+- `80-89`: useful and serious, but not yet trustworthy enough for the hardest autonomous use cases
+- `70-79`: capable memory platform, but still too error-prone or lossy in core behaviors
+- `< 70`: significant memory-quality gaps remain
+
+## Failure Interpretation
+
+If a metric fails:
+
+- `constraintRetentionRate` or `identityRetentionRate`: the system is forgetting durable high-value memory
+- `updateCorrectnessRate` or `contradictionResolutionAccuracy`: the system is not safely handling change over time
+- `falseMemoryRate`: the system is learning things it should not
+- `trustedMemoryPrecision`: the system is surfacing weak or unsafe memory as durable truth
+- `memoryIsolationAccuracy`: the system is leaking or misapplying memory across boundaries
+- `postCompactionFidelityScore`: the compaction path is destroying important information
+- `postMaintenanceFidelityScore`: lifecycle automation is too destructive
+
+## Required Eval Behavior
+
+The eval harness must:
+
+- run deterministically
+- produce structured JSON output
+- expose per-scenario results
+- support `--enforce`
+- be able to fail on real regressions

package/docs/OPERATIONS.md

@@ -0,0 +1,49 @@
+# Operations Guide
+
+## Health Endpoints
+
+- `GET /healthz`: process liveness
+- `GET /readyz`: process readiness and active scope count
+- `GET /v1/health`: per-scope memory counters
+- `GET /v1/events`: server-sent events for memory activity
+
+## Compaction and Maintenance
+
+Use the admin surface for lifecycle operations:
+
+```bash
+curl -X POST http://localhost:3100/v1/compact \
+  -H "Authorization: Bearer $MEMORY_API_KEY" \
+  -H "x-admin-key: $MEMORY_ADMIN_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"scope":{"tenant_id":"acme","system_id":"ai-ide","scope_id":"task-42"}}'
+```
+
+```bash
+curl -X POST http://localhost:3100/v1/maintenance \
+  -H "Authorization: Bearer $MEMORY_API_KEY" \
+  -H "x-admin-key: $MEMORY_ADMIN_API_KEY"
+```
+
+## Scope Routing
+
+Requests can resolve scope three ways:
+
+1. `scope` object in the JSON body
+2. Query parameters: `tenant_id`, `system_id`, `workspace_id`, `scope_id`
+3. Headers: `x-memory-tenant`, `x-memory-system`, `x-memory-workspace`, `x-memory-scope`
+
+Use body scope when the request already has a JSON payload. Use headers for shared gateways or framework middleware.
+
+## Recommended Alerts
+
+- Sustained growth in `activeTurnCount`
+- Low or zero `knowledgeCount` for a workload that should learn
+- Repeated `compacted: false` results during forced compaction
+- Large `expiredWorkingMemory` or `retiredKnowledge` spikes during maintenance
+
+## Data Hygiene
+
+- Redact sensitive text at ingest using `redactText`.
+- Prefer tenant and workspace boundaries that match your product’s isolation model.
+- Export memory before major schema or provider changes.

package/docs/SECURITY.md

@@ -0,0 +1,25 @@
+# Security Guide
+
+## Authentication
+
+- `apiKey` protects the general HTTP surface with bearer auth.
+- `adminApiKey` separately protects force-compaction and maintenance endpoints.
+- MCP transport is intended for local or already-trusted stdio integrations.
+
+## Recommended Defaults
+
+- Bind the HTTP server to `127.0.0.1` unless you intentionally expose it.
+- Keep `bodyLimitBytes` close to your expected prompt sizes.
+- Run the service behind TLS termination when exposed beyond localhost.
+
+## Sensitive Data Handling
+
+- Use `redactText` to scrub secrets before they enter turns, working memory, or knowledge memory.
+- Avoid sharing tenant-level scopes across customers.
+- Audit `relevantKnowledge` and `searchCrossScope()` use before enabling broad cross-scope retrieval.
+
+## Release Hygiene
+
+- Only publish built assets under `dist/`.
+- Keep provider SDKs as optional peers so non-provider installs stay lean.
+- Use `npm run release:check` before publishing.