memorio 5.1.3 → 5.2.0

package/markdown/MEM_FORMAT.md

@@ -0,0 +1,313 @@
+# Memorio `.mem` Format
+
+This document describes the implemented MEM package and `memorio.knowledge` logical format. A `.mem`
+is data. It is not a program, module, prompt, or trusted instruction source.
+
+> **ZIP is packaging. JSON is inspection. MAP is meaning. MEM is experience.**
+
+Memorio must not require trust in Memorio to inspect a Memorio file. Ordinary ZIP and JSON tools can
+open and inspect the package; opening or extracting it does not execute its contents.
+
+## Package
+
+Current Memorio writes `.mem` as a standard ZIP archive with four ordinary JSON parts:
+
+```text
+project.mem
+├── manifest.json
+├── map.json
+├── memory/data.json
+└── security/integrity.json
+```
+
+No empty future directories are created. Protected unknown files under `extensions/` are accepted and
+preserved without interpretation. All other unexpected or unprotected entries are rejected.
+
+The archive can be listed or extracted with normal ZIP tools. `manifest.json` identifies
+`memorio.package` version 1 and locates the other standard parts. `map.json` and `memory/data.json`
+contain the same logical v2 map and data described below. ZIP is packaging; MEM semantics do not
+depend on ZIP entry metadata.
+
+```json
+{
+  "format": "memorio.package",
+  "version": 1,
+  "memory": { "format": "memorio.knowledge", "version": 2 },
+  "parts": {
+    "map": "map.json",
+    "data": "memory/data.json",
+    "integrity": "security/integrity.json"
+  }
+}
+```
+
+For a Memorio diagnostic view, run:
+
+```bash
+npx memorio inspect project.mem
+```
+
+### Integrity
+
+`security/integrity.json` uses SHA-256 and records a `sha256:<hex>` digest for `manifest.json`,
+`map.json`, `memory/data.json`, and every preserved extension part. It does not hash itself, avoiding
+a recursive self-hash. Every non-integrity archive entry must appear exactly once in the protected
+files table, and every recorded digest must match.
+
+`contentId` is the SHA-256 digest of the canonical, path-sorted protected-files digest table. It is a
+deterministic content/cache identity, not the hash of the ZIP bytes. It does not identify an author,
+prove trust, certify content, or constitute a digital signature.
+
+```json
+{
+  "algorithm": "sha256",
+  "files": {
+    "manifest.json": "sha256:...",
+    "map.json": "sha256:...",
+    "memory/data.json": "sha256:..."
+  },
+  "contentId": "sha256:..."
+}
+```
+
+Administrators can extract the package, inspect its JSON, and verify each listed digest with standard
+SHA-256 tooling. The exact command differs by operating system; hash the raw bytes of each extracted
+part and compare the lowercase hexadecimal digest with its integrity entry.
+
+Package generation sorts entry paths, recursively sorts JSON object keys, retains array order, uses
+two-space JSON indentation with a final newline, fixes ZIP timestamps to 1980-01-02 UTC, and uses
+DEFLATE level 6. Equivalent supported logical content therefore produces deterministic package bytes.
+
+### Defensive Limits
+
+- Maximum package size: 10 MiB.
+- Maximum uncompressed total: 25 MiB.
+- Maximum individual entry: 8 MiB.
+- Maximum entries: 128.
+- Maximum declared expansion ratio per entry: 200:1.
+
+Readers reject unsafe absolute/traversal paths, backslash paths, duplicate entries, directory entries,
+unsupported compression, malformed ZIP/UTF-8/JSON, missing or unprotected parts, unsupported package
+or integrity versions, and checksum mismatches. Archives are read in memory without filesystem
+extraction. Nested archive data is never recursively opened.
+
+Integrity proves only that protected bytes match the package's digest table. Because an attacker can
+replace content and recompute unsigned hashes, integrity is not authenticity, signature, certification,
+authorization, safety, or trust. Implemented optional signatures are separate security parts and do
+not change this distinction.
+
+### Optional Ed25519 Signatures
+
+Unsigned packages remain valid. An explicitly signed package adds one JSON document per signing key:
+
+```text
+security/signatures/<64-character-public-key-fingerprint>.json
+```
+
+The protected manifest also declares `"authenticity": { "signed": true }`. This makes simple
+signature-file removal structurally invalid. Converting a signed package back to an unsigned package
+requires rebuilding the protected manifest and therefore produces a different `contentId`.
+
+```json
+{
+  "format": "memorio.signature",
+  "version": 1,
+  "algorithm": "Ed25519",
+  "contentId": "sha256:...",
+  "keyId": "sha256:...",
+  "publicKey": { "format": "raw", "encoding": "base64url", "value": "..." },
+  "signature": { "encoding": "base64url", "value": "..." }
+}
+```
+
+The signed bytes are the following domain-separated UTF-8 payload:
+
+```text
+MEMORIO-PACKAGE-SIGNATURE-V1\n<contentId>\n
+```
+
+Signing the domain-separated `contentId` binds the signature to the canonical protected-parts digest
+table without binding it to incidental ZIP bytes. Signature documents are outside both the protected
+digest table and `contentId`, preventing signature recursion. Protected extensions are already part of
+that table and are therefore authenticated even when Memorio does not understand them.
+
+Ed25519 was selected for compact keys and signatures, deterministic signatures, standard raw public
+keys, and Web Crypto support in the supported modern Node/browser path. The signing API accepts
+external Ed25519 `CryptoKey` objects. Public verification material is stored as a 32-byte raw key.
+`keyId` is the SHA-256 fingerprint of those raw public-key bytes. Private keys remain external and are
+never serialized into `.mem`.
+
+Verification validates package structure and limits, verifies SHA-256 integrity and reconstructs
+`contentId`, and only then validates and verifies each signature. Inspection reports authenticity
+separately as `unsigned`, `valid`, `invalid`, or `unsupported`, with `trust: not_evaluated`. A valid
+signature proves control of the corresponding private key over that content identity. It does not
+establish a person, organization, certification, authorization, safety, claim correctness, or trust.
+
+Ordinary memory updates produce a new `contentId` and omit all prior signatures. Memorio never
+automatically re-signs. Explicit signing is available through:
+
+```ts
+const signed = await signMemPackage(unsignedPackage, privateCryptoKey, publicCryptoKey)
+```
+
+The same protected content and Ed25519 key produce deterministic signature metadata and package bytes.
+Multiple signature entry paths are structurally supported, but signer policy, trust, revocation, and
+certification are not implemented.
+
+## Logical v2 Envelope
+
+Before packaging, the logical memory has this version 2 envelope:
+
+```json
+{
+  "format": "memorio.knowledge",
+  "version": 2,
+  "map": {},
+  "data": []
+}
+```
+
+`format` identifies the file family. `version` selects its codec. `map` declares how positional
+records are interpreted. `data` contains the source's knowledge version and claims.
+
+Version 1 used a `knowledge` object shaped like the runtime `KnowledgeState`. Current Memorio detects
+content rather than relying on the `.mem` extension: it reads v1 JSON, v2 JSON, and package v1, while
+new repository writes use package v1 containing logical v2. Reading historical JSON does not rewrite
+it; a later explicit memory update writes the current package representation. Unsupported or corrupt
+input is rejected with a diagnostic status.
+
+## MAP
+
+A v2 map declares semantic identity separately from encoding position:
+
+```json
+{
+  "state": ["knowledge-version", "claims"],
+  "claim": ["id", "knowledge", "applicability", "evidence", "epistemic-type", "provenance", "acquired-at", "revision"],
+  "evidence": ["id", "source", "observed-at", "supports"],
+  "provenance": ["source", "detail"],
+  "revision": ["kind", "claim-ids", "reason"],
+  "epistemicType": ["observed", "inferred", "human-stated", "verified"],
+  "revisionKind": ["refines", "supersedes", "retracts"]
+}
+```
+
+For example, a claim value at position `0` means `id` only when `map.claim[0]` is `id`. Readers look
+up positions through the source-local map. Field and vocabulary order is not permanent.
+
+The map is an interpretation contract and a compact vocabulary declaration. It is not executable,
+an ontology, a synonym table, or authority to reinterpret historical meanings.
+
+## DATA
+
+`data` is a positional state record interpreted through `map.state`. Its `claims` value contains
+positional claim records interpreted through `map.claim`. Evidence, provenance, and revision records
+use their corresponding maps. Optional trailing `null` positions may be omitted.
+
+Vocabulary values are zero-based positions in the source-local `epistemicType` and `revisionKind`
+arrays. For example, with the map above, epistemic value `3` means `verified`.
+
+The current `memory/data.json` is therefore inspectable but deliberately compact: positional arrays
+and numeric enum indexes are not maximally human-readable in isolation. Meaningful manual inspection
+requires cross-referencing `map.json`; `memory/data.json` alone is not self-explanatory.
+
+```json
+{
+  "format": "memorio.knowledge",
+  "version": 2,
+  "map": {
+    "state": ["knowledge-version", "claims"],
+    "claim": ["id", "knowledge", "applicability", "evidence", "epistemic-type", "provenance", "acquired-at", "revision"],
+    "evidence": ["id", "source", "observed-at", "supports"],
+    "provenance": ["source", "detail"],
+    "revision": ["kind", "claim-ids", "reason"],
+    "epistemicType": ["observed", "inferred", "human-stated", "verified"],
+    "revisionKind": ["refines", "supersedes", "retracts"]
+  },
+  "data": [1, [["layout.rule", "Use layout tokens.", {"area": "layout"}, [["review-1"]], 3]]]
+}
+```
+
+## Implemented Semantics
+
+- State: knowledge version and ordered append-only claims.
+- Claim: identity, JSON knowledge, applicability, evidence, epistemic type, provenance, acquisition
+  time, and an optional revision relationship.
+- Evidence: identity, source, observation time, and bounded `supports` text.
+- Epistemic types: `observed`, `inferred`, `human-stated`, and `verified`.
+- Revision kinds: `refines`, `supersedes`, and `retracts`.
+
+Claim identity and source-file provenance are distinct. Source provenance is established by the file
+that contributes a claim; claim provenance is claim metadata.
+
+## Known, Unknown, and Invalid
+
+Known semantics are decoded into `KnowledgeState`, validated, and available to selection.
+
+Unknown but valid mapped fields and vocabulary values remain opaque JSON data. They are retained with
+their source-local map and positions, excluded from `KnowledgeState`, and not used for selection or
+reasoning. When the same source is rewritten, the current writer overlays known fields and writes the
+unknown data back unchanged. New claims leave unknown positions empty.
+
+Invalid data is rejected. This includes malformed envelopes, missing required known meanings,
+duplicate map identities, invalid positional records, non-JSON values, invalid known claims, and
+unsupported format versions. Preservation does not turn invalid data into an unknown extension.
+
+## Compatibility and Evolution
+
+Each `.mem` source carries its own version and interpretation map and is decoded independently.
+Multiple sources need not have identical field order, vocabulary order, creation time, or producer.
+Historical files therefore retain the information needed to interpret their original representation.
+
+The source-local map is not declared to be the final or only interpretation layer. A future system
+may build a separate dynamic map or index across many memories. Local maps remain valuable in that
+architecture: individual sources stay independently interpretable and can contribute the declarations
+needed to reconstruct a damaged or missing aggregate map. No global-map filename, schema, authority,
+or runtime behavior is defined by v2.
+
+A new representation does not necessarily invalidate or supersede an old representation. Different
+labels may remain useful for the same underlying concept in different contexts. The format does not
+perform synonym resolution, ontology translation, or automatic supersedence.
+
+The opaque-preservation rule permits an older runtime or future independent editor to retain valid
+extensions it does not understand. Actual composition, relationship editing, and plugin lifecycle
+behavior are not part of this format version.
+
+## Composition and Reuse Readiness
+
+A memory may eventually be used as a reusable, data-only memory plugin or as a component of a larger
+memory. Version 2 does not define mounting, discovery, certification, permissions, dependencies, or
+plugin execution.
+
+Future composition is not assumed to copy complete input files into an output file. A derived memory
+may instead record dependencies or lineage to independently retained sources, allowing a changed input
+to trigger reevaluation rather than automatic copying or replacement. The identity, integrity, trust,
+conflict, and lineage model required for this behavior remains deliberately unspecified.
+
+The format likewise does not declare differently named concepts equivalent. An AI or another future
+tool may propose relationships using the memories and current context, but the codec performs no
+semantic discovery. New vocabulary does not automatically supersede old vocabulary, and a complete
+memory may later be treated as one component of a larger memory.
+
+These constraints keep the representation independent from any future editor technology. An editor,
+CLI, library, AI, or server tool should operate on the same data and interpretation contracts.
+
+## Security
+
+All `.mem` content is untrusted data. A reader must not evaluate values, construct functions from
+them, execute expressions or commands, import modules named by them, or treat stored knowledge as
+trusted instructions. Unknown preservation carries JSON data forward; it grants no execution or
+authority.
+
+## Verified production inspection
+
+Before the 5.2.0 release checkpoint, a representative package generated through the production
+acquisition path was inspected with ordinary ZIP/JSON tooling and `inspectMem()`. It was 1,712 bytes
+with SHA-256 `568cd4a309f753bb8179d756849a2bcb403ced2eb62fcf8df8ee6448b9c83335` and contained the four
+standard unsigned entries shown above. Inspection reported package v1, memory v2, valid integrity,
+no unknown semantics, and `unsigned` / `not_evaluated` authenticity. A field-by-field semantic
+round-trip preserved both synthetic representative claims, including evidence, provenance,
+applicability, epistemic type, timestamps, and a `refines` relationship.
+
+That synthetic artifact is verification evidence, not normative production data or a fixture that
+applications must reproduce byte-for-byte.

package/markdown/SQLITE.md

@@ -153,8 +153,8 @@ await sqlite.db.journal.checkpoint('app'); // manual checkpoint
 
 | Method | Parameters | Returns | Description |
 |--------|------------|---------|-------------|
-| `sqlite.config(opts)` | `opts: { loader?, locateFile?, wasmUrl?, persistence?, namespace? }` | `sqlite` | Configure the engine. `loader` fully replaces the initializer (default = CDN `<script>`); `locateFile` customizes wasm resolution; `wasmUrl` sets the wasm base directory; `persistence` toggles automatic db snapshots; `namespace` partitions persisted snapshots. Chainable. |
-| `sqlite.db.create(name, opts)` | `name: string, opts?: { data?, persistence? }` | `Database` | Opens/creates a named in-memory db. With `persistence: true` (or global config), a snapshot is restored if present and writes are auto-saved. |
+| `sqlite.config(opts)` | `opts: { loader?, locateFile?, wasmUrl?, persistence?, namespace? }` | `sqlite` | Configure the engine. `loader` fully replaces the initializer (default = CDN `<script>`); `locateFile` customizes wasm resolution; `wasmUrl` sets the wasm base directory; `persistence` toggles journal + checkpoint persistence; `namespace` partitions persisted data. Chainable. |
+| `sqlite.db.create(name, opts)` | `name: string, opts?: { data?, persistence? }` | `Database` | Opens/creates a named in-memory db. With `persistence: true` (or global config), the latest checkpoint is restored and pending journal entries are replayed for full crash recovery. |
 | `sqlite.db.persist(name)` | `name: string` | `Promise<boolean>` | Force an immediate checkpoint (full snapshot + journal clear) of a persisted database to `store`. Alias of `flush`. |
 | `sqlite.db.flush(name)` | `name: string` | `Promise<boolean>` | Force an immediate checkpoint (alias of `persist`). |
 | `sqlite.db.download(name, filename?)` | `name: string, filename?: string` | `Promise<boolean>` | Dev-only: trigger a browser download of the db as a `.sqlite` file. |

package/markdown/STATE.md

@@ -100,15 +100,39 @@ console.debug(protect); // Array of protected keys
 
 ### Lock
 
+`state` exposes two lock scopes with distinct method names.
+
+**Global lock** - freeze the entire `state` tree at once:
+
+```javascript
+state.lock();
+state.user = { name: 'Sara' };   // Error: state is locked
+state.user.name = 'Luigi';       // Error: state is locked
+state.lock();                    // (safe to call again)
+state.unlock();                  // resume all writes
+```
+
+**Per-key lock** - freeze a single first-level node:
+
 ```javascript
-// Lock an object or array
 state.myArray = [1, 2, 3];
 state.myArray.lock();
 
-// Now any modification will fail
-state.myArray.push(4); // Error: state 'myArray' is locked
+// While locked, reassigning the key, mutating the node, array methods,
+// and deleting its properties all fail:
+state.myArray.push(4);   // Error: state 'myArray' is locked
+state.myArray[0] = 9;    // Error: state 'myArray' is locked
+delete state.myArray[0]; // Error: state 'myArray' is locked
+state.myArray = [];      // Error: state 'myArray' is locked
+
+state.myArray.unlock();  // Resume writes to myArray only
 ```
 
+Per-key locking is **first-level only**: `state.a.lock()` does not transitively lock
+`state.a.b`. Lock each descendant explicitly (`state.a.b.lock()`), or use the global
+`state.lock()` for whole-subtree immutability. While a key is per-key locked, every other
+top-level key remains writable.
+
 ---
 
 ## How It Works

package/markdown/SYNC.md

@@ -48,18 +48,23 @@ Instead:
 ```ts
 memorio.memory.remember('user.language', 'Italian')
 // and a single configuration point:
-memorio.memory.configure({ sync: { provider: myCloudProvider, namespace: '…' } })
+memorio.memory.configure({ provider: myCloudProvider, namespace: '…' })
 ```
 
-## 2. Scopes (isolation, not a security boundary)
+## 2. Namespace isolation (not a security boundary)
 
-| Scope | Lifetime | Syncs by default |
+The journal is partitioned by a `namespace` string you pass to `configure()`.
+All journal entries, pending operations, and replay calls for one namespace are
+fully isolated from another - there is no API to enumerate or open a different
+namespace's journal. You can use any naming convention, for example:
+
+| Namespace pattern | Lifetime | Syncs by default |
 |---|---|---|
-| `'device'` | this browser/device only | no (sticky) |
-| `'user'` | follows the user across devices | yes (requires provider + namespace) |
-| `'shared'` | shared across users / tenant | yes (requires provider + namespace) |
+| `device:<id>` | this browser/device only | no (sticky) |
+| `user:<id>:device:<id>` | follows the user across devices | yes (requires provider) |
+| `tenant:<id>:user:<id>` | shared across users / tenant | yes (requires provider) |
 
-> As with `memorio.createContext`, **scoping is a naming convention, not a
+> As with `memorio.createContext`, **namespace scoping is a naming convention, not a
 > security boundary.** Enforce real isolation server-side.
 
 ## 3. SQLite as the local durable store
@@ -71,7 +76,7 @@ the durable journal/value store when you opt in:
    snapshot the database to `store` (localStorage) and restore it on reopen.
  - Writes are journaled incrementally (append-only, HLC-timestamped) and
    checkpointed periodically (every 50 ops or 250 ms of idle). On reopen, the
-   last checkpoint is loaded and the journal is replayed — no writes are lost.
+   last checkpoint is loaded and the journal is replayed - no writes are lost.
  - `sqlite.db.persist(name)` / `sqlite.db.flush(name)` forces an immediate
    checkpoint; `sqlite.db.close(name)` checkpoints + closes;
    `sqlite.db.download(name, file?)` triggers a browser `.sqlite` download
@@ -87,7 +92,7 @@ because pending operations must survive a refresh for offline-first to work.
 
 | Method | Returns | Notes |
 |---|---|---|
-| `memory.journal.append(entry, operation)` | `Promise<MemoryEntry>` | records `remember\|update\|forget\|expire\|confirm\|supersede` with `sync:'pending'` |
+| `memory.journal.append(entry, operation)` | `Promise<MemoryEntry>` | records `remember\|update\|forget\|expire\|confirm\|supersede\|delete\|patch` with `sync:'pending'` |
 | `memory.journal.pending()` | `Promise<MemoryEntry[]>` | rows where `sync != 'synced'`, for the current namespace |
 | `memory.journal.markSynced(ids)` | `Promise<number>` | advances rows to `synced` (namespace-scoped) |
 | `memory.journal.get(id)` | `Promise<MemoryEntry \| null>` | single entry, namespace-scoped |
@@ -113,7 +118,8 @@ The cloud must not simply say "last write wins." Memorio tags every entry with:
 - `source` / `scope`
 
 Remote conflicts are surfaced as `sync:'conflict'` rows via
-`journal.pending()`; the provider's `resolve(op)` hint decides locally. Example:
+`journal.pending()`; the provider's `resolveConflict(local, remote)` hint decides
+which entry wins. Example:
 
 ```
 Laptop:  language=Italian, confidence=0.92
@@ -132,7 +138,7 @@ memorio.memory.configure({
   provider: {
     push(ops) { return fetch('/api/sync', { method: 'POST', body: JSON.stringify(ops), headers: authHeaders }) }
     pull(since) { return fetch(`/api/sync?since=${since}`).then(r => r.json()) }
-    resolve(op) { return op.confidence >= 0.8 ? 'local' : 'remote' }
+    resolveConflict(local, remote) { return (local.confidence ?? 0) >= 0.8 ? 'local' : 'remote' }
   },
   auto: true                           // auto-replay on focus/online (default true)
 })
@@ -142,12 +148,10 @@ memorio.memory.configure({
 interface SyncProvider {
   push(ops: MemoryEntry[]): Promise<{ synced: string[]; conflicts?: string[]; error?: string }>
   pull?(since?: number): Promise<MemoryEntry[]>
-  resolve?(op: MemoryEntry): Promise<'local' | 'remote' | 'merge'>
+  resolveConflict?(local: MemoryEntry, remote: MemoryEntry): Promise<'local' | 'remote' | 'merge'>
 }
 ```
 
-`memorio.memory.ready` resolves once the local journal substrate is chosen.
-
 ## 7. Security (NIST / OWASP / NSA posture)
 
 - **Memorio never handles credentials.** No passwords, tokens, or API keys are
@@ -189,7 +193,7 @@ layered onto the existing journal **without** adopting a full CRDT framework
 (Yjs, Automerge, etc.).
 
 > **Scope note.** These strategies target `memorio.memory` first - it already
-> carries the metadata a journal needs (`confidence`, `source`, `tag`, `scope`,
+> carries the metadata a journal needs (`confidence`, `source`, `tags`, `scope`,
 > `createdAt`, `lastConfirmedAt`). If synchronization is ever extended to
 > `state` or `store`, those layers must gain HLC timestamps and path-level
 > fields explicitly - they cannot inherit them from `memory`.