@mnexium/core 0.1.0

package/.env.example

@@ -0,0 +1,37 @@
+POSTGRES_HOST=localhost
+POSTGRES_PORT=5432
+POSTGRES_DB=mnexium_core
+POSTGRES_USER=postgres
+POSTGRES_PASSWORD=change_me
+
+# Optional
+CORE_DEFAULT_PROJECT_ID=default-project
+PORT=8080
+CORE_DEBUG=false
+
+# AI routing mode:
+# - auto (default): cerebras -> openai -> simple fallback
+# - cerebras: force Cerebras, fallback to simple if key missing
+# - openai: force OpenAI ChatGPT model, fallback to simple if key missing
+# - simple: no LLM calls, heuristic mode only
+CORE_AI_MODE=auto
+
+# Controls search-time retrieval expansion/reranking.
+# true  = LLM classify/expand/rerank path (when LLM is available)
+# false = simple search mode only
+USE_RETRIEVAL_EXPAND=true
+
+# Model API keys
+OPENAI_API_KEY=
+
+# Optional: enables Cerebras-powered recall routing/reranking.
+CEREBRAS_API=
+
+# Shared retrieval model for whichever LLM provider is selected.
+# - Cerebras example: gpt-oss-120b
+# - OpenAI example: gpt-4o-mini or gpt-4.1-mini
+RETRIEVAL_MODEL=gpt-oss-120b
+
+# Embeddings model for semantic memory search and claim vectorization.
+# Keep this aligned with schema vector dimension (schema is VECTOR(1536)).
+OPENAI_EMBED_MODEL=text-embedding-3-small

package/README.md

@@ -0,0 +1,37 @@
+# CORE
+
+CORE is Mnexium's memory engine service: a Postgres-backed HTTP API for storing memories, extracting claims, resolving truth state, and retrieving relevant context for downstream applications.
+
+It is designed as an integration-first core service that can run standalone and plug into existing auth, tenancy, and platform controls.
+
+## What CORE does
+
+- Stores subject-scoped memories and supports lifecycle operations (create, update, soft-delete, restore).
+- Extracts structured claims from natural language and persists claim assertions.
+- Maintains slot-based truth state (`slot_state`) to track active winners and retractions.
+- Supports retrieval with vector + lexical fallback and optional LLM-powered query expansion/reranking.
+- Streams memory lifecycle events over SSE for real-time consumers.
+
+## Why it is powerful
+
+- Better grounding for responses: LLMs can retrieve durable, user-specific memory instead of relying only on short chat context.
+- Lower hallucination risk on known facts: retrieval and claim state give the model a concrete memory substrate to reference.
+- Personalization that persists: preferences, history, and prior decisions survive across sessions and channels.
+- Works beyond context-window limits: important memory is stored and recalled on demand instead of repeatedly reprompted.
+- Faster LLM product development: app teams get a ready memory/truth backend rather than building custom memory pipelines from scratch.
+
+## Intended use
+
+CORE is intended to be the memory and truth substrate behind apps, agents, and workflows that need:
+
+- long-lived user memory,
+- auditable claim history,
+- query-time recall,
+- and deterministic APIs backed by Postgres.
+
+## Documentation map
+
+- Setup and initialization: [docs/SETUP.md](docs/SETUP.md)
+- Runtime behavior and decision logic: [docs/BEHAVIOR.md](docs/BEHAVIOR.md)
+- HTTP endpoints and contracts: [docs/API.md](docs/API.md)
+- Production hardening checklist: [docs/OPERATIONS.md](docs/OPERATIONS.md)

package/docs/API.md

@@ -0,0 +1,299 @@
+# 📘 API Reference
+
+All routes except `GET /health` require project context.
+
+Project context resolution:
+
+1. `x-project-id` header
+2. fallback project id configured on server startup
+
+If neither is available, the request fails with `400`:
+
+```json
+{
+  "error": "project_id_required",
+  "message": "Provide x-project-id header or configure defaultProjectId"
+}
+```
+
+## 🩺 Health
+
+### GET `/health`
+
+Returns service liveness:
+
+```json
+{
+  "ok": true,
+  "service": "mnexium-core",
+  "timestamp": "..."
+}
+```
+
+## 📡 Memory Events (SSE)
+
+### GET `/api/v1/events/memories`
+
+Query params:
+
+- `subject_id` (optional)
+
+Event types:
+
+- `connected`
+- `heartbeat`
+- `memory.created`
+- `memory.superseded`
+- `memory.updated`
+- `memory.deleted`
+
+## 🧠 Memories
+
+### GET `/api/v1/memories`
+
+Query params:
+
+- `subject_id` (required)
+- `limit` (default `50`, max `200`)
+- `offset` (default `0`)
+- `include_deleted` (`true|false`)
+- `include_superseded` (`true|false`)
+
+Returns:
+
+- `{ data: Memory[], count: number }`
+
+### POST `/api/v1/memories`
+
+Body:
+
+- `subject_id` (required)
+- `text` (required, max length `10000`)
+- `kind`, `visibility`, `importance`, `confidence`, `is_temporal`, `tags`, `metadata`, `source_type` (optional)
+- `id` (optional)
+- `extract_claims` (optional, default `true`)
+- `no_supersede` (optional, default `false`)
+
+Returns:
+
+- `201` created:
+  - `{ id, subject_id, text, kind, created: true, superseded_count, superseded_ids }`
+- `200` duplicate skip:
+  - `{ id: null, subject_id, text, kind, created: false, skipped: true, reason: "duplicate" }`
+
+### GET `/api/v1/memories/search`
+
+Query params:
+
+- `subject_id` (required)
+- `q` (required)
+- `limit` (default `25`, max `200`)
+- `min_score` (default `30`)
+- `distance` (alias of `min_score`)
+- `context` (repeatable; optional conversation context items)
+
+Returns:
+
+- when recall service is configured (default in `src/dev.ts`):
+  - `{ data, query, count, engine, mode, used_queries, predicates }`
+- internal fallback path:
+  - `{ data, query, count, engine }`
+
+### POST `/api/v1/memories/extract`
+
+Body:
+
+- `subject_id` (required)
+- `text` (required)
+- `force` (`boolean`, optional)
+- `learn` (`boolean`, optional)
+- `conversation_context` (`string[]`, optional)
+
+Query params:
+
+- `learn=true|false` (optional)
+- `force=true|false` (optional)
+
+`learn`/`force` are enabled when either body or query sets them to `true`.
+
+Returns (extraction only):
+
+- `{ ok: true, learned: false, mode, extracted_count, memories }`
+
+Returns (learn/write path):
+
+- `{ ok: true, learned: true, mode, extracted_count, learned_memory_count, learned_claim_count, memories }`
+
+### GET `/api/v1/memories/superseded`
+
+Query params:
+
+- `subject_id` (required)
+- `limit` (default `50`, max `200`)
+- `offset` (default `0`)
+
+Returns:
+
+- `{ data: Memory[], count }`
+
+### GET `/api/v1/memories/recalls`
+
+Query params:
+
+- `chat_id` OR `memory_id` (one required)
+- `stats=true|false` (used only with `memory_id` path)
+- `limit` (default `100`, max `1000`)
+
+Response modes:
+
+- by chat (`chat_id` provided): `{ data, count, chat_id }`
+- by memory (`memory_id` + no stats): `{ data, count, memory_id }`
+- memory stats (`memory_id` + `stats=true`): `{ memory_id, stats }`
+
+Note:
+
+- if both `chat_id` and `memory_id` are provided, `chat_id` path is used.
+
+### GET `/api/v1/memories/:id`
+
+Returns:
+
+- `{ data: Memory }`
+- `404` with `memory_not_found` or `memory_deleted`
+
+### PATCH `/api/v1/memories/:id`
+
+Body (any subset):
+
+- `text`, `kind`, `visibility`, `importance`, `confidence`, `is_temporal`, `tags`, `metadata`
+
+Returns:
+
+- `{ id, updated: true }`
+- `404` with `memory_not_found` or `memory_deleted`
+
+### DELETE `/api/v1/memories/:id`
+
+Soft delete.
+
+Returns:
+
+- `{ ok: true, deleted: boolean }`
+
+### GET `/api/v1/memories/:id/claims`
+
+Returns assertion-centric claims linked to memory:
+
+- `{ data: [{ id, predicate, type, value, confidence, status, first_seen_at, last_seen_at }], count }`
+
+Errors:
+
+- `404` `memory_not_found`
+- `404` `memory_deleted`
+
+### POST `/api/v1/memories/:id/restore`
+
+Returns:
+
+- `{ ok: true, restored: true, id, subject_id, text }`
+- `{ ok: true, restored: false, message: "Memory is already active" }`
+- `400` `memory_deleted`
+- `404` `memory_not_found`
+
+## 🧩 Claims
+
+### POST `/api/v1/claims`
+
+Body:
+
+- required: `subject_id`, `predicate`, `object_value`
+- optional: `claim_id`, `claim_type`, `slot`, `confidence`, `importance`, `tags`, `source_memory_id`, `source_observation_id`, `subject_entity`, `valid_from`, `valid_until`
+
+Returns:
+
+- `{ claim_id, subject_id, predicate, object_value, slot, claim_type, confidence, observation_id, linking_triggered }`
+
+### POST `/api/v1/claims/:id/retract`
+
+Body:
+
+- `reason` (optional, default `manual_retraction`)
+
+Returns:
+
+- `{ success, claim_id, slot, previous_claim_id, restored_previous, reason }`
+
+### GET `/api/v1/claims/:id`
+
+Returns:
+
+- `{ claim, assertions, edges, supersession_chain }`
+
+Notes:
+
+- `claim` excludes embedding field
+- `supersession_chain` is filtered from `edges` where `edge_type = supersedes`
+
+### GET `/api/v1/claims/subject/:subjectId/truth`
+
+Query params:
+
+- `include_source=true|false` (default `true`)
+
+Returns:
+
+- `{ subject_id, project_id, slot_count, slots }`
+
+### GET `/api/v1/claims/subject/:subjectId/slot/:slot`
+
+Returns:
+
+- `{ subject_id, project_id, slot, active_claim_id, predicate, object_value, claim_type, confidence, updated_at, tags, source }`
+- `404` `slot_not_found`
+
+### GET `/api/v1/claims/subject/:subjectId/slots`
+
+Query params:
+
+- `limit` (default `100`, max `500`)
+
+Returns grouped slot states:
+
+- `{ subject_id, total, active_count, slots: { active, superseded, other } }`
+
+### GET `/api/v1/claims/subject/:subjectId/graph`
+
+Query params:
+
+- `limit` (default `50`, max `200`)
+
+Returns:
+
+- `{ subject_id, claims_count, edges_count, edges_by_type, claims, edges }`
+
+### GET `/api/v1/claims/subject/:subjectId/history`
+
+Query params:
+
+- `slot` (optional)
+- `limit` (default `100`, max `500`)
+
+Returns:
+
+- `{ subject_id, project_id, slot_filter, by_slot, edges, total_claims }`
+
+## ⚠️ Error Conventions
+
+Common error payloads:
+
+- validation: `{ error: "subject_id_required" }`, `{ error: "q_required" }`, `{ error: "invalid_json_body" }`
+- not found: `{ error: "memory_not_found" }`, `{ error: "claim_not_found" }`, `{ error: "slot_not_found" }`, `{ error: "not_found" }`
+- server error: `{ error: "server_error", message: "..." }`
+
+Status codes:
+
+- `200` success
+- `201` created
+- `400` validation/input
+- `404` not found
+- `500` unexpected server error

package/docs/BEHAVIOR.md

@@ -0,0 +1,224 @@
+# 🧠 Runtime Behavior
+
+This document explains how CORE decides retrieval/extraction behavior at runtime and how memory + claim state changes flow through the system.
+
+## 🏗️ Request Lifecycle
+
+CORE server path:
+
+1. Receive HTTP request
+2. Resolve `project_id`
+3. Dispatch route handler
+4. Execute storage contract (`CoreStore`)
+5. Return JSON (or SSE stream)
+
+Primary implementation files:
+
+- `src/server/createCoreServer.ts`
+- `src/adapters/postgres/PostgresCoreStore.ts`
+- `src/ai/recallService.ts`
+- `src/ai/memoryExtractionService.ts`
+
+## 🤖 AI Provider Resolution
+
+Configured by `CORE_AI_MODE`:
+
+- `auto` (default)
+- `cerebras`
+- `openai`
+- `simple`
+
+Resolution behavior (`src/dev.ts`):
+
+- `auto`
+  - use Cerebras when `CEREBRAS_API` exists
+  - else use OpenAI when `OPENAI_API_KEY` exists
+  - else use `simple`
+- `cerebras`
+  - requires `CEREBRAS_API`
+  - if missing, warns and falls back to `simple`
+  - does not auto-switch to OpenAI
+- `openai`
+  - requires `OPENAI_API_KEY`
+  - if missing, warns and falls back to `simple`
+  - does not auto-switch to Cerebras
+- `simple`
+  - no LLM client
+
+`RETRIEVAL_MODEL` is passed to the selected LLM client.
+
+## 🔎 Retrieval Behavior (`GET /api/v1/memories/search`)
+
+### Retrieval toggle
+
+`USE_RETRIEVAL_EXPAND=true|false`:
+
+- `true`
+  - with LLM client: classify + expand + rerank path
+  - without LLM client: simple path
+- `false`
+  - always simple search path
+  - extraction endpoint can still use LLM if an LLM client exists
+
+### Query mode semantics (`broad|direct|indirect`)
+
+Mode is produced by an LLM classifier in `src/ai/recallService.ts`.
+
+- `broad`
+  - intent: profile/summary requests
+  - behavior: list active memories and sort by importance + recency
+  - output pattern: wider profile set (`max(limit, 20)`)
+- `direct`
+  - intent: specific fact lookup
+  - behavior: search with hints, then boost claim-backed memories when predicates match truth slots
+  - output pattern: high-precision, usually narrow (often top 5)
+- `indirect`
+  - intent: advice/planning where personal context helps
+  - behavior: query expansion + larger candidate pool + rerank when needed
+  - output pattern: context-rich supporting memories
+
+Classification fallback behavior:
+
+- classify failure/invalid mode -> defaults to `indirect`
+- empty query -> no memories returned
+
+### LLM expanded pipeline
+
+1. Classify query into `broad|direct|indirect` and extract hints/predicates.
+2. Build query set:
+   - `broad`: profile listing path (no multi-query expansion)
+   - `direct`: original + search hints
+   - `indirect`: original + search hints + expanded queries
+3. For each query, search with embedding (if available) plus lexical fallback.
+4. Merge and dedupe by memory ID.
+5. Apply mode-specific ranking:
+   - `direct`: claim/truth boosts, rerank only when needed
+   - `indirect`: rerank via LLM when candidate set exceeds limit
+
+Other runtime constraints:
+
+- conversation context is capped to last 5 items
+- classify timeout is 2s
+- rerank timeout is 3s
+
+Response fields include:
+
+- `engine` (`provider:model` or `simple`)
+- `mode` (`broad|direct|indirect|simple`)
+- `used_queries`
+- `predicates`
+
+### Simple retrieval pipeline
+
+1. Run single query search.
+2. Use embedding if available, otherwise lexical-only path.
+3. Rank via adapter scoring.
+
+No LLM classify/expand/rerank is applied.
+
+## ✂️ Extraction Behavior (`POST /api/v1/memories/extract`)
+
+This endpoint is always available.
+
+- With LLM client:
+  - uses extraction prompt
+  - normalizes output
+  - falls back to simple extraction on parse/failure
+- Without LLM client:
+  - uses simple heuristic extraction directly
+
+Learn modes:
+
+- `learn=false` (default): extraction-only response
+- `learn=true`: writes memories/claims and emits `memory.created` per learned memory
+
+## 🗂️ Memory Lifecycle Semantics
+
+### Create memory (`POST /api/v1/memories`)
+
+- writes one `memories` row
+- optional embedding generation
+- duplicate guard when embedding similarity >= 85
+- conflict supersede when embedding similarity is in `[60, 85)`
+- emits `memory.created`
+- emits `memory.superseded` for superseded conflicting memories
+- optional async claim extraction (`extract_claims=true` by default)
+
+Notes:
+
+- async extraction is skipped when `no_supersede=true`
+- extracted claims link back via `source_memory_id`
+
+### Update memory (`PATCH /api/v1/memories/:id`)
+
+- patch updates supported fields
+- emits `memory.updated`
+
+### Delete memory (`DELETE /api/v1/memories/:id`)
+
+- soft delete (`is_deleted=true`)
+- emits `memory.deleted` when delete actually occurs
+
+### Restore memory (`POST /api/v1/memories/:id/restore`)
+
+- sets `status='active'`, clears `superseded_by`
+- emits `memory.updated`
+- deleted memories cannot be restored
+
+### Superseded listing (`GET /api/v1/memories/superseded`)
+
+- returns non-deleted memories where `status='superseded'`
+
+## 🧩 Claim Semantics
+
+### Create claim (`POST /api/v1/claims`)
+
+- inserts into `claims`
+- inserts assertion row in `claim_assertions`
+- upserts `slot_state` active winner for claim slot
+
+### Retract claim (`POST /api/v1/claims/:id/retract`)
+
+- sets claim status to `retracted`
+- restores previous active claim in same slot when available
+- updates `slot_state`
+- writes `retracts` edge when prior winner is restored
+
+### Truth/slot reads
+
+- truth + slot endpoints resolve from `slot_state` joined with active `claims`
+- graph/history endpoints return claims + edges
+
+## 📡 SSE Event Behavior
+
+Endpoint:
+
+- `GET /api/v1/events/memories`
+
+Event types:
+
+- `connected`
+- `heartbeat` (every 30s)
+- `memory.created`
+- `memory.superseded`
+- `memory.updated`
+- `memory.deleted`
+
+Topology:
+
+- in-process event bus (`src/server/memoryEventBus.ts`)
+- no cross-instance fanout by default
+- for horizontal scale, replace bus with external pub/sub (Redis/NATS/Kafka)
+
+## ⚠️ Error and Degradation Model
+
+Status pattern:
+
+- `400` validation/input issues
+- `404` missing resource or route
+- `500` unexpected server error
+
+Graceful degradation:
+
+- no embedding key -> non-vector retrieval path still works
+- no LLM client or LLM failure -> simple retrieval/extraction fallback

package/docs/OPERATIONS.md

@@ -0,0 +1,116 @@
+# 🛠️ Operations Playbook
+
+This guide focuses on production hardening for CORE deployments.
+
+## 🧱 Deployment Baseline
+
+Recommended baseline:
+
+- 1+ stateless CORE instances
+- managed Postgres with `pgvector`
+- reverse proxy/load balancer with tuned upstream timeouts
+- centralized logs + metrics
+
+For multi-instance event fanout, add external pub/sub.
+
+## 🔐 Platform Integrations to Add
+
+CORE intentionally leaves platform controls to the host system. Typical production additions:
+
+1. Auth (API keys/JWT) and route-level scopes
+2. Tenant boundary checks
+3. Idempotency keys for writes
+4. External event transport for SSE fanout
+5. Alerting + SLO dashboards
+
+## 🔁 Idempotency Guidance
+
+Write routes can be retried by clients/proxies. Add:
+
+- `Idempotency-Key` for `POST/PATCH/DELETE`
+- persisted request fingerprint and response body
+- replay of original success response for duplicate key
+
+## 📡 SSE at Scale
+
+Current behavior:
+
+- memory events are emitted from an in-process bus
+- subscribers connected to instance A will not receive events produced on instance B
+
+Production recommendation:
+
+- publish lifecycle events to Redis/NATS/Kafka
+- fan out consistently across all API instances
+
+## 🗄️ Database Operations
+
+### Migrations
+
+- keep `sql/postgres/schema.sql` as bootstrap baseline
+- run versioned forward migrations in CI/CD
+- avoid startup-time auto-migration in production paths
+
+### Backup and recovery
+
+- daily full backup + PITR
+- regular restore drills
+- documented recovery RTO/RPO targets
+
+### Vector index hygiene
+
+- run `ANALYZE` after significant ingest batches
+- monitor query plans and latency
+- tune ivfflat list settings as corpus grows
+
+## 📈 Observability Model
+
+Track:
+
+- request rate and latency (`p50`, `p95`, `p99`)
+- status code distribution
+- DB query latency/error rate
+- LLM provider latency/error rate
+- SSE subscriber counts
+
+Alert on:
+
+- sustained 5xx error rate
+- DB saturation / connection pool pressure
+- high tail latency regressions
+
+## 🤖 Runtime Adaptation (Accuracy Notes)
+
+Provider selection in `src/dev.ts`:
+
+- `CORE_AI_MODE=auto`: Cerebras -> OpenAI -> simple
+- `CORE_AI_MODE=cerebras` without key: simple fallback (no OpenAI auto-switch)
+- `CORE_AI_MODE=openai` without key: simple fallback (no Cerebras auto-switch)
+
+Retrieval behavior:
+
+- `USE_RETRIEVAL_EXPAND=true`: LLM classify/expand/rerank if LLM exists
+- `USE_RETRIEVAL_EXPAND=false`: simple retrieval path
+
+Embedding behavior:
+
+- missing `OPENAI_API_KEY` causes embedder to return empty vectors
+- write/read APIs continue; retrieval uses lexical/non-vector scoring path
+
+## 🧾 Data Semantics in Production
+
+- memory deletion is soft (`is_deleted=true`)
+- supersession uses `status='superseded'` and `superseded_by`
+- truth state is read from `slot_state` joined with active claims
+- claim retraction may restore previous slot winner
+
+## ✅ Hardening Checklist
+
+- [ ] Auth + scope middleware in front of CORE routes
+- [ ] Explicit tenant/project isolation strategy
+- [ ] Idempotency-key storage and replay implemented
+- [ ] Externalized SSE/event transport for multi-instance deployments
+- [ ] Migration workflow in CI/CD
+- [ ] Backup + restore drill cadence defined
+- [ ] SLOs + alerts configured
+- [ ] Load/perf test against expected traffic profile