memorio 4.9.35 → 5.1.0

package/markdown/SYNC.md

@@ -0,0 +1,318 @@
+> **Status:** Published
+> **Date:** 2026-09-12
+> **Scope**: API Reference
+> **Standard**: Memorio API Specification v5
+>
+---
+# Synchronization & Cloud (optional)
+
+`memorio.memory` is **local-first**. Data is created and served from the device;
+the cloud is only ever a **transport/persistence provider**, never the source of
+truth. Enabling sync does not replace local storage - it *mirrors* it.
+
+```
+                    memorio
+                       │
+              ┌────────┴────────┐
+              │   Memory Engine │
+              └────────┬────────┘
+          ┌────────────┼────────────┐
+          ▼            ▼            ▼
+        local      SQLite         cloud
+        memory     durable        sync
+```
+
+## 1. The rule: the data is born local
+
+```ts
+memorio.memory.remember('user.language', 'Italian', { scope: 'local' })
+//        ↓ local first
+//   store / sessionStorage / IndexedDB / sql.js
+//        ↓ sync / push (when online)
+//   cloud provider
+```
+
+The cloud therefore does not **replace** memory: it **replicates** it. This
+gives you: offline-first, lowest latency, data available immediately,
+synchronization when online, multi-device, multi-user, centralized persistence.
+
+We deliberately do **not** provide:
+
+```ts
+// ❌ two mental models
+memory.cloud.save(...)
+```
+
+Instead:
+
+```ts
+memorio.memory.remember('user.language', 'Italian')
+// and a single configuration point:
+memorio.memory.configure({ sync: { provider: myCloudProvider, namespace: '…' } })
+```
+
+## 2. Scopes (isolation, not a security boundary)
+
+| Scope | Lifetime | Syncs by default |
+|---|---|---|
+| `'device'` | this browser/device only | no (sticky) |
+| `'user'` | follows the user across devices | yes (requires provider + namespace) |
+| `'shared'` | shared across users / tenant | yes (requires provider + namespace) |
+
+> As with `memorio.createContext`, **scoping is a naming convention, not a
+> security boundary.** Enforce real isolation server-side.
+
+## 3. SQLite as the local durable store
+
+SQLite (`sql.js`) is **in-memory by default** (volatie per page load). It becomes
+the durable journal/value store when you opt in:
+
+- `sqlite.config({ persistence: true })` / `sqlite.db.create('app', { persistence: true })`
+  snapshot the database to `store` (localStorage) and restore it on reopen.
+- Writes are snapshotted via sql.js `updateHook` (debounced).
+- `sqlite.db.persist(name)` forces an immediate save; `sqlite.db.close(name)`
+  flushes + closes; `sqlite.db.download(name, file?)` triggers a browser
+  `.sqlite` download (dev convenience).
+
+See `docs/markdown/SQLITE.md` for the full SQLite reference.
+
+## 4. The local operation journal
+
+The **sync journal** is the durable op log that drives cloud reconciliation.
+It is persisted on `store` (localStorage) - **not** on an in-memory sql.js db,
+because pending operations must survive a refresh for offline-first to work.
+
+| Method | Returns | Notes |
+|---|---|---|
+| `memory.journal.append(entry, operation)` | `Promise<MemoryEntry>` | records `remember\|update\|forget\|expire\|confirm\|supersede` with `sync:'pending'` |
+| `memory.journal.pending()` | `Promise<MemoryEntry[]>` | rows where `sync != 'synced'`, for the current namespace |
+| `memory.journal.markSynced(ids)` | `Promise<number>` | advances rows to `synced` (namespace-scoped) |
+| `memory.journal.get(id)` | `Promise<MemoryEntry \| null>` | single entry, namespace-scoped |
+| `memory.journal.clear()` | `Promise<void>` | wipes the current namespace's journal |
+| `memory.journal.replay()` | `Promise<SyncAck>` | pushes `pending()` to the provider, marks synced, optional `pull` |
+| `memory.journal.status()` | `Promise<'store'>` | the substrate in use |
+
+We sync **operations of memory**, never a raw database dump:
+
+```
+user A        device A
+remember X ─────► local ─────► sync ─────► cloud
+forget Z ──────► local ─────► sync ─────► cloud
+```
+
+## 5. Conflict resolution
+
+The cloud must not simply say "last write wins." Memorio tags every entry with:
+
+- `confidence` (0–1, user/system trust in the value)
+- `lastConfirmedAt` / `updatedAt` (epoch ms)
+- `version` (monotonic per-key counter)
+- `source` / `scope`
+
+Remote conflicts are surfaced as `sync:'conflict'` rows via
+`journal.pending()`; the provider's `resolve(op)` hint decides locally. Example:
+
+```
+Laptop:  language=Italian, confidence=0.92
+Phone:   language=English, confidence=0.61
+→ higher-confidence entry wins locally; the provider decides for shared scope.
+```
+
+## 6. Configuring a backend
+
+Sync is **opt-in**. You supply an application-owned `provider` that knows how to
+talk to your backend (REST, WebSocket, Supabase, a custom agent server, …).
+
+```ts
+memorio.memory.configure({
+  namespace: 'user:123:device:abc',   // tenant/user/device - partitions the journal
+  provider: {
+    push(ops) { return fetch('/api/sync', { method: 'POST', body: JSON.stringify(ops), headers: authHeaders }) }
+    pull(since) { return fetch(`/api/sync?since=${since}`).then(r => r.json()) }
+    resolve(op) { return op.confidence >= 0.8 ? 'local' : 'remote' }
+  },
+  auto: true                           // auto-replay on focus/online (default true)
+})
+```
+
+```ts
+interface SyncProvider {
+  push(ops: MemoryEntry[]): Promise<{ synced: string[]; conflicts?: string[]; error?: string }>
+  pull?(since?: number): Promise<MemoryEntry[]>
+  resolve?(op: MemoryEntry): Promise<'local' | 'remote' | 'merge'>
+}
+```
+
+`memorio.memory.ready` resolves once the local journal substrate is chosen.
+
+## 7. Security (NIST / OWASP / NSA posture)
+
+- **Memorio never handles credentials.** No passwords, tokens, or API keys are
+  read from or stored by memorio. Authentication/authorization live in your
+  `provider`/backend (OWASP A01: Broken Access Control).
+- **Namespace isolation.** The journal is keyed by `namespace:id` at the storage
+  layer; there is **no API** to enumerate or open another namespace's journal. A
+  client holding a forged/fake namespace simply sees its own (empty) journal.
+- **No dynamic code.** Journal entries are strictly JSON-round-tripped,
+  size-capped (10 MB/entry), and never `eval`'d. The sql.js loader never
+  `import()`s a bare specifier that could be hijacked at build time.
+- **Trust boundary:** memorio owns the local durable copy + operation log; the
+  provider/backend owns remote-side auth and conflict resolution. Memorio
+  surfaces `conflict`/`error` rows; it does not fabricate a winner.
+- **Data-at-rest (NSA/CISA).** memorio's `store`/`idb`/`sqlite` snapshots are
+  **not encrypted**. If you persist user data server-side or ship it through your
+  backend, encrypt it server-side with keys you manage - memorio treats the local
+  store as untrusted-from-the-browser and does not attest its own integrity.
+
+## 8. Where data lives
+
+| Substrate | API | Volatile? | Persistent? |
+|---|---|---|---|
+| in-memory `Proxy` | `state` | yes (per tab) | no |
+| `localStorage` / Map | `store` | no | yes (browser) |
+| `sessionStorage` / Map | `session` | no | per-tab (browser) |
+| IndexedDB | `idb`, `memory` durable | no | yes |
+| sql.js (WASM heap) | `sqlite` | **yes** | only with `persistence: true` (snapshot → `store`) |
+| sync journal | `memory.journal` | no | yes (`store`) |
+
+---
+
+## 9. Evolving the journal toward multi-device consistency
+
+Moving from sequential, single-device sync to concurrent offline edits across
+multiple devices is a classic local-first challenge. Below are six
+architectural strategies, in increasing order of sophistication, that can be
+layered onto the existing journal **without** adopting a full CRDT framework
+(Yjs, Automerge, etc.).
+
+> **Scope note.** These strategies target `memorio.memory` first - it already
+> carries the metadata a journal needs (`confidence`, `source`, `tag`, `scope`,
+> `createdAt`, `lastConfirmedAt`). If synchronization is ever extended to
+> `state` or `store`, those layers must gain HLC timestamps and path-level
+> fields explicitly - they cannot inherit them from `memory`.
+
+### 9.1 Field-level / path-level journaling
+
+Recording an entire entity in the journal makes orthogonal edits un-mergeable:
+
+```json
+{ "op": "set", "path": "user.role", "value": "admin", "timestamp": 1710000000 }
+```
+
+Device A writes `user.name`, device B writes `user.role` - both are patch
+operations on the **same entity**. An entity-level journal would produce one
+opaque `UPDATE user = {…}` and one write would clobber the other. Path-level
+patches merge automatically because the paths are disjoint.
+
+> **Not a panacea.** Path-level journaling merges edits to *different* fields.
+> Two devices writing the **same** path concurrently still need explicit conflict
+> resolution (Section 5). HLC tells you *when* the events happened; it does not
+> tell you *which value wins* when events are truly concurrent on the same path.
+
+### 9.2 Causal ordering with Hybrid Logical Clocks (HLC)
+
+Wall-clock timestamps alone fail under clock drift. Associate every journal
+entry with an HLC that combines a physical component, a logical counter, and a
+node identifier:
+
+```
+hlc:1710000005:2:deviceB
+```
+
+This gives constant-size causal ordering (vs. vector clocks, which grow with
+the number of writers - problematic for a bounded journal). The sync engine can
+then apply causally-dependent operations in order and invoke the conflict
+resolver only for genuinely concurrent writes on the same path.
+
+### 9.3 Ordering collections without full OT - fractional indexing
+
+Arrays and ordered lists are the hardest non-CRDT case. Numeric indices shift
+when a peer inserts or deletes nearby. Two lightweight options:
+
+1. **Keyed collections** - treat list items as a `Map<id, value>` rather than a
+   positional array. No index renumbering needed.
+2. **Fractional indexing** - assign each element a sortable key between its
+   neighbours (e.g. `1.0`, `2.0` → insert at `1.5`). On repeated re-inserts
+   between the same pair, keys grow in length and should be rebalanced
+   periodically. Use a mature library (`fractional-indexing` on npm) rather than
+   reimplementing the arithmetic.
+
+### 9.4 Explicit deletions (tombstones)
+
+A bare "remove" entry can be resurrected as a "zombie" when a concurrent
+update is replayed after it. Instead, record every `forget` / `delete` as a
+first-class journal event with its own HLC:
+
+```json
+{ "op": "delete", "path": "user.role", "timestamp": "hlc:1710000005:0:deviceA" }
+```
+
+The tombstone participates in the same causal comparison as `set` events: if
+the delete's HLC succeeds the update's, the field is gone; if it precedes, the
+update is re-applied. Keep tombstones around until all known peers have
+acknowledged them, then garbage-collect during a maintenance sweep.
+
+### 9.5 Hybrid sync - server-assisted consensus
+
+Since memorio's cloud role is transport-and-acknowledgement (not source of
+truth), the backend can resolve conflicts on the server and return the
+canonical sequence:
+
+| Pattern | Description |
+|---|---|
+| **Optimistic local apply** | Apply the local journal entry immediately and emit reactive events. |
+| **Server ack** | Backend validates causal order against the central state and returns the official sequence. |
+| **Client journal rebase** | Confirmed entries are purged from the local journal; unconfirmed local entries are replayed on top of the acknowledged state. |
+
+### 9.6 Validation - convergence simulation
+
+Causal correctness is theoretical until you test it across replay orderings:
+
+- Generate random concurrent `set` / `delete` operations across N simulated
+  devices (shared paths and disjoint paths).
+- Replay the operation log in every plausible ordering on each simulated
+  device.
+- Assert **state convergence**: every device arrives at the same final state
+  regardless of delivery order.
+- Include hand-crafted pathological cases (`update` vs concurrent `delete` on
+  the same path, `insert` vs concurrent `delete` on the same array index,
+  interleaved reorders).
+
+### 9.7 Migration
+
+Path-level journal entries with HLC and tombstones are a format change from
+entity-level entries. Mitigate with:
+
+- An explicit **version header** on every journal entry.
+- A clear migration policy: either a one-time compaction pass that folds
+  legacy entity-level entries into the current state and starts a fresh
+  journal, or a dual-format reader that can replay both formats during the
+  transition window.
+
+### 9.8 Recommended architecture diagram
+
+```text
+[ Local Mutation ]
+        │
+        ▼
+[ Field-Level Patch (set/delete) + HLC Timestamp ]
+        │
+        ├───► Local State (immediate reactive update)
+        │
+        └───► Local Journal (incl. tombstones for deletes)
+                    │
+             (Online Sync)
+                    │
+                    ▼
+        [ Backend Conflict Resolver ]
+        (concurrent path → resolveConflict;
+         otherwise apply in HLC order)
+                    │
+                    ▼
+        [ State Ack / Rebased Journal ]
+```
+
+Applying granular path-level patches, HLC for causality, fractional indexing
+for ordered collections, explicit tombstones for deletions, and a convergence
+test suite lets memorio handle high-frequency concurrent offline edits across
+devices without the overhead of a full CRDT stack.

package/markdown/TYPED.md

@@ -0,0 +1,164 @@
+> **Status:** Published
+> **Date:** 2026-09-12
+> **Scope**: API Reference
+> **Standard**: Memorio API Specification v5
+>
+---
+# Typed Stores - Memorio
+
+> ✅ **Universal**: Works in Browser, Node.js, Deno, and Edge Workers
+
+`memorio.typed<T>()` returns the global `state` proxy cast to a TypeScript type `T`, giving you **compile-time** type safety on every access and mutation.
+
+It's a **zero-runtime-cost** wrapper: the returned object is the *exact same* Proxy as `globalThis.state`, just with TypeScript types applied via a generic.
+
+---
+
+## Quick Start
+
+```typescript
+import { memorio, state, typed, useObserver } from 'memorio'
+
+interface AppState {
+  user: { name: string; age: number; email: string }
+  theme: 'light' | 'dark'
+  items: string[]
+}
+
+const app = memorio.typed<AppState>()
+
+// Type-checked at compile time:
+app.user = { name: 'Sara', age: 30, email: 'sara@test.com' }
+app.theme = 'dark'
+
+// TypeScript errors:
+// app.user = { name: 42 }            // age missing, name wrong type
+// app.theme = 'purple'               // not a valid literal
+```
+
+---
+
+## Why use typed stores?
+
+| Without typed | With `memorio.typed<T>()` |
+|---|---|
+| `state.user = { name: 42 }` - runs silently, bug at runtime | `app.user = { name: 42 }` - TypeScript error at compile time |
+| No autocomplete on `state.user.email` | Full IntelliSense: properties, types, method suggestions |
+| Rename `user` to `profile` - no compiler warning anywhere | Every `app.user` access flagged as an error |
+| AI-generated code lacks guardrails | AI gets autocomplete and type feedback inline |
+
+---
+
+## Combine with Schema Validation
+
+Typed stores catch type errors at compile time; schema validation catches invalid values at runtime. Together they form a **defense-in-depth** strategy:
+
+```typescript
+import { memorio, state } from 'memorio'
+
+interface ProfileState {
+  profile: { bio: string; avatar?: string }
+}
+
+const app = memorio.typed<ProfileState>()
+
+memorio.registerSchema('profile', {
+  type: 'object',
+  required: ['bio'],
+  properties: {
+    bio: { type: 'string', min: 1 },
+    avatar: { type: 'string' }
+  }
+})
+
+app.profile = { bio: 'Developer', avatar: 'pic.png' } // ✅ type + schema pass
+app.profile = { avatar: 'pic.png' }                  // ❌ TypeScript: bio missing
+                                                     // ❌ Runtime: bio required
+```
+
+See [Schema Validation](SCHEMA.md) for runtime validation details.
+
+---
+
+## Named import variant
+
+`typed` is also available as a named export if you prefer explicit dependencies:
+
+```typescript
+import { typed } from 'memorio'
+
+const app = typed<AppState>()
+```
+
+The `memorio` namespace object is the same across both usage styles - named imports and the `memorio/global` entrypoint share the same runtime instances.
+
+---
+
+## Full API
+
+| Method | Parameters | Returns | Description |
+|--------|-----------|---------|-------------|
+| `memorio.typed<T>()` | Generic type `T` | `T` | Returns the global `state` proxy cast to `T` |
+
+The returned object shares the same identity as `globalThis.state`:
+
+```typescript
+const app = memorio.typed<AppState>()
+console.debug(app === state) // true - same Proxy instance
+```
+
+---
+
+## React + typed stores
+
+Pair with the `useObserver` hook for type-safe, reactive React components:
+
+```tsx
+import { memorio, state, useObserver } from 'memorio'
+import { useReducer } from 'react'
+
+interface AppState {
+  user: { name: string; age: number }
+  theme: 'light' | 'dark'
+}
+
+const app = memorio.typed<AppState>()
+
+function UserProfile() {
+  const [, forceUpdate] = useReducer(x => x + 1, 0)
+
+  useObserver(forceUpdate, [state.user.name])
+
+  return (
+    <div>
+      <h1>{app.user.name}</h1>
+      <span>Theme: {app.theme}</span>
+    </div>
+  )
+}
+```
+
+---
+
+## Best Practices
+
+1. **Define your AppState at the root** of your app and import it everywhere:
+
+   ```typescript
+   // types/app-state.ts
+   export interface AppState {
+     user: { name: string; email: string }
+     theme: 'light' | 'dark'
+   }
+   ```
+
+   ```typescript
+   // anywhere in your app
+    import { memorio } from 'memorio'
+    import type { AppState } from '../types/app-state'
+    const app = memorio.typed<AppState>()
+   ```
+
+2. **Layer schema validation on top** for runtime safety, especially for data coming from APIs or user input.
+
+3. **Use alongside `memorio.help()`** (via `import 'memorio/global'`) to list available globals during development.