mumpix 1.0.19 → 1.1.3

package/CHANGELOG.md

@@ -1,16 +1,71 @@
 # Changelog
 
-## 1.0.19 - 2026-04-13
+## 1.1.3 - 2026-07-09
+
+Supersedes the unpublished `v1.1.2` release candidate.
+
+### Semantic recall
+- Fixed freshly written semantic sidecar entries so `recallMany()` exposes stable WAL-backed `id` metadata before and after reopen.
+- Hardened the semantic sidecar insert path to normalize entries at the hot-write boundary, keeping live memory, JSONL journal replay, and compacted snapshots in the same schema.
+- Corrected the semantic SIGKILL reconciliation test so it treats durable WAL replay as surviving ledger data instead of comparing only against pre-replay main-file lines.
+- Replaced the public recall-quality demo's mock embedder with a real OpenAI-compatible embeddings adapter, defaulting to Ollama `nomic-embed-text`.
+- Kept the real embedding smoke framed as a demo with counts, not a calibrated quality benchmark.
+- Removed transport URL from the demo embedder id so moving the same model to another endpoint does not force an unnecessary semantic sidecar rebuild.
+- Raised the provisional semantic fallback floor from `0.35` to `0.5` after the real `nomic-embed-text` smoke showed `0.35` allowed its nonsense control to fire; this floor remains uncalibrated pending a larger labeled suite.
+- Updated the public recall quality demo cleanup to remove both semantic sidecar files: `<db>.vectors.json` and `<db>.vectors.jsonl`.
+
+## 1.1.1 - 2026-07-09
+
+### Semantic recall seam
+- Added `Mumpix.open(path, { recall: "semantic" })` as a zero-dependency semantic fallback seam.
+- Added optional embedder contract support: `{ id, dim, embed(texts) }`.
+- Added a semantic vector sidecar keyed by WAL record `seq`, embedder id, and vector dimension. Hot writes append to `<db>.vectors.jsonl`; lifecycle compaction writes the `<db>.vectors.json` snapshot.
+- Added semantic sidecar crash reconciliation on open. Missing vectors are re-embedded from WAL records, and torn JSONL journal tails are discarded/rebuilt rather than treated as fatal.
+- `recall: "semantic"` now fails loudly when neither `{ embedder }` nor optional package `mumpix-embed` is available, instead of silently becoming a no-op.
+- Preserved 1.1.0 no-match semantics: lexical recall still runs first, semantic fallback is consulted only after lexical no-hit, and results below `semanticFloor` return `null` / `[]`.
+- Added `npm run bench:recall` with a deterministic mock embedder to demonstrate the pipeline without shipping model weights. Replaced by a real-embedder smoke in `1.1.2`.
+- Added `npm run bench:semantic` to guard against semantic sidecar write amplification regressions.
+
+## 1.1.0 - 2026-07-08
+
+### Correctness
+- Added `minScore` to `recall()` and `recallMany()` with a default local TF-IDF threshold of `0.15`.
+- Breaking change: `recall()` may now return `null`, and `recallMany()` may now return `[]`, when the best candidate is below threshold. This is correct behavior for agent memory and prevents false context injection.
+- Fixed the zero-overlap TF-IDF ranking path so recency alone no longer makes unrelated memories look relevant.
+- Added deterministic tie-breaking after score comparison to avoid unstable first-document fallthrough behavior.
 
-### Internal & Sync
-- Synchronized latest temporal engine and operator scripts from core module into the package tree.
-- Configured embedded GGUF agent context window ceilings (8K) to prevent VRAM memory allocation conflicts with local LLMs (e.g., `mumpix-mlc-llm`).
+### Reliability
+- Fixed stale crash locks by checking the lock payload PID on open. If the lock owner process is dead, Mumpix removes the lock and retries immediately.
+- Kept `lockStaleMs` mtime fallback for containers, NFS, and PID namespace cases where PID liveness is not authoritative.
+- Documented `lockStaleMs` and `lock: false`.
+
+### Verification
+- Added `npm run test:phase0` covering unrelated recall no-match behavior, explicit `minScore:false` opt-out, SIGKILL stale-lock recovery, and dead-PID lock payload recovery.
+
+## 1.0.30 - 2026-06-14
+
+### Production package
+- Republished the production-only package contents under a fresh immutable npm version after `1.0.29` became unavailable for normal registry installs.
+- Removed benchmark harnesses and dated benchmark artifacts from the npm package allowlist so production installs only include the runtime library, CLI, examples, scripts, README, changelog, and license.
+- Kept the pure-code vector retrieval optimization in the production runtime: packed `Float32Array` vector storage, lazy index construction, and metadata equality candidate narrowing before exact cosine scoring.
+- Preserved the public vector sidecar API while improving domain-scoped retrieval performance for embedders using MumpixDB as a general-purpose AI memory store.
 
-## 1.0.18 - 2026-03-30
-- General performance and sync updates
+## 1.0.28 - 2026-06-14
 
-## 1.0.17 - 2026-03-24
-- Core temporal sync patches
+### Performance and retrieval
+- Promoted the pure-code MumpixDB recall update into the bundled runtime package.
+- Added persistent inverted-index recall candidates with hash re-verification, preserving the existing trust boundary while avoiding full scans.
+- Added WAL writer policy support for coalesced eventual-mode flushes while keeping strict and verified durability paths on per-record fsync semantics.
+- Added `rememberAll()` batch persistence and index maintenance so large promotion bursts do not require per-record main-file rewrites.
+- Exported `WalWriter` and `InvertedIndex` from the package entrypoint for runtime diagnostics and tests.
+
+### Benchmark record
+- Added a scale benchmark covering write throughput, randomized reads, reopen reads, and mixed read/write pressure.
+- At 100k records, indexed retrieval measured about 24ms p95 on reopen versus about 631ms p95 for full scan.
+- Under a 5k concurrent-promotion burst, indexed retrieval stayed under 50ms p95 while full scan was about 1.29s p95.
+- Reopen p95 tracked hot p95, confirming the index is persisted and reused after load instead of acting as an in-process cache.
+- Added a pgvector RLP retrieval comparison harness that measures domain-scoped vector retrieval against an independent exact flat cosine oracle.
+- Documented MumpixDB vector retrieval as exact flat cosine today, pgvector exact scan as the default peer, pgvector HNSW as an explicit opt-in approximate mode, and FAISS-flat as the optional million-scale oracle.
 
 ## 1.0.11 - 2026-03-03
 
@@ -28,7 +83,24 @@
 - Clarified that `strict`/`verified` are tier-gated capabilities (SDK and CLI).
 - Added `verify:claims` usage to the examples section.
 
-## Unreleased
+## 1.0.22 - 2026-06-12
+
+### Truth and verification
+- Tightened README claims to match implementation exactly.
+- Clarified local recall as exact substring, TF-IDF recall, and optional custom embedding recall.
+- Clarified mandatory runtime usage sync fields and local ML-DSA event verification payloads.
+- Fixed `npm run verify:claims` so it uses a writable local Mumpix config directory during tests.
+
+## 1.0.21 - 2026-06-12
+
+### Publish hardening
+- Prepared the npm package above the published `1.0.20` line.
+- Kept install side effects out of `npm install`; account linking now happens through explicit CLI auth commands.
+- Kept runtime usage sync mandatory for billing reconciliation and license enforcement, with local queuing for failed sends.
+- Added ML-DSA-44 device identities, signed usage events, and hash-chain continuity for air-gapped reconciliation.
+- Added `mumpix auth device-id` for device-bound license issuance.
+- Tightened package files so generated `.mumpix` artifacts are not included in npm tarballs.
+- Updated smoke examples to await `Mumpix.open(...)` and run in free `eventual` mode by default.
 
 ### Licensing (offline/air-gapped)
 - Added monthly tier capability mapping for paid durability modes (`eventual`, `strict`) while active.
@@ -41,8 +113,7 @@
 - Added local account auth state support in `src/core/auth.js`.
 - Added CLI auth commands: `mumpix auth login|status|logout`.
 - Added automatic local-license loading in `MumpixDB.open()` when `licenseKey` is not passed explicitly.
-- Added install-time auth script (`postinstall`) that prompts for account sign-in in interactive terminals.
-- Added browser-open device flow on install; skip path remains `eventual` only.
+- Added explicit device login flow through `mumpix auth login`; free installs remain in `eventual` mode until linked.
 
 ## 1.0.9 - 2026-03-02
 

package/README.md

@@ -32,9 +32,9 @@ MumpixDB gives you:
 
 - local-first storage in a `.mumpix` file
 - append-oriented durable state
-- strict and verified execution modes
+- license-gated strict and verified execution modes
 - replayable memory and audit surfaces
-- semantic recall and exact recall
+- exact substring recall, local TF-IDF recall, and optional custom embedding recall
 - a structured API for long-running assistants, workflows, and applications
 
 ## Quickstart
@@ -77,7 +77,14 @@ Options:
 | Option | Type | Default | Description |
 |---|---|---|---|
 | `consistency` | `string` | `'eventual'` | `'eventual'`, `'strict'`, or `'verified'` |
-| `embedFn` | `async function` | — | Custom embedding function for semantic recall |
+| `embedFn` | `async function` | — | Custom embedding function for embedding-based recall |
+| `recall` | `string` | — | Set to `'semantic'` to enable optional semantic fallback using a local embedder |
+| `embedder` | `object` | — | Semantic recall embedder implementing `{ id, dim, embed(texts) }` |
+| `semanticFloor` | `number` | `0.5` | Minimum semantic cosine score before fallback recall returns a match; provisional and not yet calibrated on a labeled suite |
+| `vectorPath` | `string` | `<db>.vectors.json` | Optional semantic vector sidecar path |
+| `telemetry` | `boolean` | `false` | Opt in to signed license heartbeat telemetry for verified licenses |
+| `lockStaleMs` | `number` | `300000` | Milliseconds before mtime-based stale lock recovery; dead lock owner PIDs are recovered immediately |
+| `lock` | `boolean` | `true` | Set `false` only when an external single-writer lock already protects the file |
 
 ### Core methods
 
@@ -95,6 +102,93 @@ await db.stats();
 await db.close();
 ```
 
+`recall()` returns `null` when no stored memory clears the relevance threshold.
+`recallMany()` returns `[]` for the same no-match case. This is intentional:
+agent memory should prefer "I don't know" over injecting a low-confidence or
+unrelated record into context.
+
+Recall options:
+
+```js
+await db.recall('what style does the user prefer?', { minScore: 0.15 });
+await db.recallMany('deployment preference', 5, { minScore: 0.15 });
+```
+
+`minScore` defaults to `0.15` for the local TF-IDF path. Set `minScore: false`
+only when you explicitly want the old permissive behavior.
+
+### Semantic recall fallback
+
+`mumpix` keeps the core package zero-dependency. To enable semantic fallback,
+open a database with `recall: 'semantic'` and provide an embedder, or install an
+optional package that exports the same contract as `mumpix-embed`.
+
+If neither an explicit embedder nor `mumpix-embed` is available, open fails
+loudly. Mumpix does not silently fall back to lexical recall when
+`recall: 'semantic'` is requested.
+
+```js
+const db = await Mumpix.open('./agent.mumpix', {
+  recall: 'semantic',
+  embedder: {
+    id: 'my-local-embedder-v1',
+    dim: 384,
+    async embed(texts) {
+      // Return one L2-normalized vector per input text.
+      return texts.map(() => new Float32Array(384));
+    },
+  },
+  semanticFloor: 0.5,
+});
+```
+
+Embedder contract:
+
+```js
+{
+  id: string,                 // changes whenever the embedding space changes
+  dim: number,                // vector dimensionality
+  embed(texts: string[]): Promise<Float32Array[] | number[][]>
+}
+```
+
+Semantic vectors are a derived index of the WAL records. They are keyed by
+record `seq`, `embedder.id`, and vector dimension. Hot writes append to a
+`<db>.vectors.jsonl` journal, while `flush()`, `close()`, clear, prune, and
+embedding-space changes compact into the `<db>.vectors.json` snapshot. If the
+embedder id or vector dimension changes, the sidecar is rebuilt instead of
+mixing incompatible embedding spaces.
+
+On open, semantic mode reconciles the sidecar against the ledger. Records past
+the sidecar's max `seq` are re-embedded, stale vectors are pruned, and a torn
+JSONL tail from a crash is discarded and rebuilt from the WAL-backed records.
+
+Hybrid behavior is deliberately conservative:
+
+- lexical recall runs first and wins when it clears the normal threshold
+- semantic vectors are consulted only when lexical recall returns no hit
+- results below `semanticFloor` still return `null` / `[]`
+
+Run the public recall smoke/demo:
+
+```bash
+ollama pull nomic-embed-text
+npm run bench:recall
+```
+
+Run the semantic write-amplification bench:
+
+```bash
+npm run bench:semantic
+```
+
+The bundled recall example is a smoke/demo, not a calibrated quality benchmark.
+It reports counts on five paraphrases plus one nonsense control, using a real
+OpenAI-compatible embeddings endpoint. By default it expects Ollama at
+`http://127.0.0.1:11434/v1` with `nomic-embed-text` and provisional
+`semanticFloor: 0.5`. Override with `MUMPIX_EMBED_BASE_URL`,
+`MUMPIX_EMBED_MODEL`, `MUMPIX_EMBED_API_KEY`, and `MUMPIX_SEMANTIC_FLOOR`.
+
 ### Temporal API
 
 `mumpix` also exposes a deterministic temporal layer built on top of WAL-backed records.
@@ -158,6 +252,72 @@ Temporal records support scoped metadata on write:
 - `source`
 - `ts`
 
+### Behavioral Reasoning Primitives
+
+The package includes the BRP, RLP, and computational-collapse utilities from the MumpixDB papers as built-in modules. They are pure local APIs and do not add another npm package.
+
+```js
+const { brp, rlp, collapse } = require('mumpix');
+
+const gene = brp.normalizeBRP({
+  domain: 'browser',
+  trigger_signature: 'login page asks for workspace slug',
+  executable_payload: {
+    intent: 'DO',
+    sequence: [
+      { verb: 'input-text', args: { selector: '#workspace', value: 'acme' } },
+      { verb: 'click-element', args: { selector: 'button[type=submit]' } },
+    ],
+  },
+});
+
+const boundary = rlp.validateActionBoundary(gene);
+const economics = collapse.analyzeCollapse({
+  stages: [
+    { name: 'INDEX', token_cost: 200 },
+    { name: 'SCAN', token_cost: 2400 },
+    { name: 'DEBATE', token_cost: 7200 },
+    { name: 'EXECUTE', token_cost: 2050 },
+  ],
+  retrievalCost: 150,
+  promotionCost: 35550,
+  hits: 3,
+});
+
+console.log(gene.payload_type); // DO
+console.log(boundary.ok); // true
+console.log(economics.savingsPerHit); // 11700
+```
+
+Available BRP APIs:
+
+- `brp.normalizeBRP(candidate)`
+- `brp.validateBRP(candidate)`
+- `brp.validatePayload(payload)`
+- `brp.parseExecutablePayload(payload)`
+- `brp.updateSplitConfidence(brp, event)`
+- `brp.transitionStatus(brp, status, reason)`
+- `brp.structuralPrefilter(brps, query)`
+- `brp.twoStageMatch(brps, query, { k })`
+- `brp.VerbRegistry`
+
+Available RLP APIs:
+
+- `rlp.REQUIRED_MECHANISMS`
+- `rlp.FAILURE_MODES`
+- `rlp.ARCHITECTURAL_LAWS`
+- `rlp.gap(numberOrId)`
+- `rlp.validateActionBoundary(record)`
+- `rlp.mechanismChecklist(system)`
+
+Available computational-collapse APIs:
+
+- `collapse.graphCost(stages)`
+- `collapse.collapseSavings(stages, opts)`
+- `collapse.breakEvenReuseThreshold(promotionCost, cycleCost)`
+- `collapse.beta(promotionCost, cycleCost)`
+- `collapse.analyzeCollapse(input)`
+
 ### Consistency modes
 
 | Mode | Behavior | Use case |
@@ -166,6 +326,43 @@ Temporal records support scoped metadata on write:
 | `strict` | stronger durability path | production agents and application state |
 | `verified` | durability + audit surfaces | regulated, auditable, or compliance-heavy workloads |
 
+### Upgrades, licenses, and billing
+
+Free installs run in `eventual` mode. Users upgrade by linking an account or installing a signed license:
+
+```bash
+mumpix auth login
+mumpix auth login --token=<account-token>
+mumpix auth login --license=<signed-license-key>
+mumpix auth status
+mumpix auth device-id
+```
+
+The account backend exchanges an authenticated user session or account token for a signed license key. The package stores that license locally in the user's Mumpix config directory and validates it before allowing paid modes.
+
+Paid monthly licenses should be issued as short leases. The recommended policy is:
+
+- `monthly`, `developer`, `standard`, or `teams` unlock `strict`
+- `compliance`, `enterprise`, `government`, `verified`, or `pro` unlock `verified`
+- monthly leases expire and refresh through the account backend
+- canceled or unpaid subscriptions stop receiving refreshed leases, so paid modes downgrade or block after expiry
+
+License telemetry is opt-in and is used only as a signed license-verification heartbeat. By default, opening and closing a database performs no telemetry network call. To allow license heartbeat telemetry for a verified signed license, pass `telemetry: true` to `Mumpix.open()` or set `MUMPIX_TELEMETRY=1`. Free or unlicensed use stays local even when telemetry is enabled. When a verified license is present and telemetry is enabled, Mumpix sends account tier, machine/platform metadata, database stats, timestamp, the license fingerprint, and the license key transport header to the Mumpix usage endpoint on database close. Database filesystem paths are not included in telemetry events. If the endpoint is temporarily unavailable, license heartbeat events are queued locally and retried on later opted-in closes. Billing state should remain server-side in the account/subscription system; license telemetry is a reconciliation signal, not the source of truth for customer entitlement.
+
+For air-gapped installs, Mumpix creates a local ML-DSA-44 device identity. Usage events are signed with that device key and linked in a hash chain before being queued or uploaded. The event payload includes the device public key, signature, event hash, and previous hash so a reconciliation service can verify device signature, event hash, and chain continuity when the system reconnects. License issuers can bind a lease to a device by including the device public-key fingerprint from `mumpix auth device-id`.
+
+This is tamper-evident hardening for editable JavaScript packages: it makes gaps, edits, and device changes visible during reconciliation without adding cryptographic work to normal read/write paths.
+
+### Release hardening
+
+Published builds can be staged with build-time obfuscation:
+
+```bash
+npm run release:pack:obfuscated
+```
+
+This creates `.release/obfuscated` and packs from that staged tree. Runtime files under `src/` and `bin/` are minified and identifier-mangled with Terser, while examples and tests remain readable. The transform avoids control-flow flattening, eval wrappers, string-array indirection, and runtime loaders, so the package still runs as normal Node.js code with no runtime obfuscation dependency.
+
 ### Verified mode methods
 
 ```js
@@ -178,13 +375,13 @@ await db.exportAudit();
 
 ## Developer SDK
 
-This package also includes a developer client for Mumpix-backed services.
+This package also includes a small HTTP client for Mumpix-compatible services that expose the endpoints listed below.
 
 ```js
 const { MumpixDevClient } = require('mumpix');
 
 const client = new MumpixDevClient({
-  baseUrl: 'https://mumpixdb.com/benchmark'
+  baseUrl: 'http://127.0.0.1:3012'
 });
 
 const health = await client.health();
@@ -212,10 +409,83 @@ Available client methods:
 
 ## Integrations
 
-This package includes adapters for:
+This package includes two integration layers:
+
+- official peer-backed subclasses for LangChain and LlamaIndex
+- zero-dependency compatibility adapters with LangChain/LlamaIndex-shaped methods
+
+For official subclassing, install the optional peer package you need:
+
+```bash
+npm install mumpix @langchain/core
+npm install mumpix llamaindex
+```
+
+Official LangChain exports:
+
+```js
+const {
+  MumpixLangChainVectorStore,
+  MumpixLangChainRetriever,
+  MumpixLangChainMemory,
+} = require('mumpix/src/integrations/langchain-official');
+```
+
+These classes subclass LangChain's `VectorStore`, `BaseRetriever`, and `BaseMemory` from `@langchain/core`. The vector store persists embeddings in a `.vectors.json` sidecar next to the `.mumpix` file and supports `addDocuments`, `addVectors`, `addTexts`, `delete`, `similaritySearch`, `similaritySearchWithScore`, `similaritySearchVectorWithScore`, `maxMarginalRelevanceSearch`, and `save`.
+
+Official LlamaIndex exports:
+
+```js
+const {
+  MumpixLlamaIndexVectorStore,
+  MumpixLlamaIndexRetriever,
+} = require('mumpix/src/integrations/llamaindex-official');
+```
+
+These classes subclass LlamaIndex's `BaseVectorStore` and `BaseRetriever`. The vector store persists embeddings in the same `.vectors.json` sidecar and supports `client`, `get`, `add`, `delete`, `query`, and `persist`.
+
+Zero-dependency LangChain-shaped exports:
+
+These do not import `@langchain/core` and do not subclass LangChain base classes. They expose common document, retriever, and memory-shaped methods while using MumpixDB's local recall internally.
+
+```js
+const {
+  MumpixVectorStore,
+  MumpixChatMemory,
+  MumpixRetriever,
+} = require('mumpix/src/integrations/langchain');
+```
+
+Available methods:
+
+- `MumpixVectorStore.similaritySearch(query, k)`
+- `MumpixVectorStore.similaritySearchWithScore(query, k)`
+- `MumpixVectorStore.addDocuments(docs)`
+- `MumpixVectorStore.addTexts(texts, metadatas)`
+- `MumpixChatMemory.loadMemoryVariables(values)`
+- `MumpixChatMemory.saveContext(inputs, outputs)`
+- `MumpixChatMemory.clear()`
+- `MumpixRetriever.getRelevantDocuments(query)`
+
+Zero-dependency LlamaIndex-shaped exports:
+
+These do not import `llamaindex` and do not subclass LlamaIndex base classes.
+
+```js
+const {
+  MumpixIndex,
+  MumpixIndexRetriever,
+  MumpixReader,
+} = require('mumpix/src/integrations/llamaindex');
+```
+
+Available methods:
 
-- LangChain
-- LlamaIndex
+- `MumpixIndex.asRetriever({ topK })`
+- `MumpixIndex.insert(node)`
+- `MumpixIndex.insertMany(nodes)`
+- `MumpixIndexRetriever.retrieve(queryBundle)`
+- `MumpixReader.loadData()`
 
 Examples: