@x-otto/persistence 0.0.1-alpha.0

package/README.md

@@ -0,0 +1,134 @@
+# @x-otto/persistence
+
+Unified snapshot persistence + append-only log abstraction. Backends: memory, file (atomic write), HTTP REST, SQLite.
+
+## Install
+
+```bash
+pnpm add @x-otto/persistence
+```
+
+## Quick Start
+
+```ts
+import { createPersistence, createAppendLog } from '@x-otto/persistence'
+
+// File-backed snapshot persistence (atomic .tmp → rename)
+const store = createPersistence({ type: 'file', path: '/tmp/data' })
+await store.save('my-key', { hello: 'world' })
+const snap = await store.load('my-key')
+// snap → Snapshot<{ hello: string }>
+
+// Append-only log (memory)
+const log = createAppendLog({ type: 'memory' })
+const seq = log.append('stream-1', { event: 'start' })
+log.append('stream-1', { event: 'end' })
+const entries = log.read('stream-1', 0, 10)
+```
+
+## Backends
+
+| Backend         | Kind         | Save            | Load     | List | Delete | Notes                           |
+|-----------------|-------------|-----------------|----------|------|--------|----------------------------------|
+| Memory          | `memory`    | `structuredClone` | `structuredClone` | ✓ | ✓ | Test / ephemeral                 |
+| File            | `file`      | `.tmp` → `rename` | ✓       | ✓   | ✓      | Atomic per-file, mtime sort      |
+| HTTP            | `http`      | `PUT /:id`      | `GET /:id` | ✓ | ✓      | Bearer token + ETag, fetch injected |
+| SQLite          | `sqlite`    | `INSERT OR REPLACE` | ✓    | ✓   | ✓      | WAL mode, change_log per mutation |
+
+All backends support `listPaginated(page, pageSize)` returning `PaginatedResult<T>`.
+
+## AppendLog
+
+Orthogonal append-only primitive alongside snapshots:
+
+| Backend | Kind    | Write                          | Read                     | Notes                    |
+|---------|---------|--------------------------------|--------------------------|--------------------------|
+| Memory  | `memory` | Array push + listener broadcast | Full / range (0-based seq) | Real-time tail via events |
+| File    | `file`  | `.jsonl` per stream, write queue | Range read, crash-tail repair | Line-level truncate      |
+| SQLite  | `sqlite` | `append_entries` table, WAL    | Range read + `COUNT(*)`  | Shared connection via `sqlite-db-cache` |
+
+## Snapshot<T> Envelope
+
+```ts
+interface Snapshot<T> {
+  id: string
+  version: number
+  data: T
+  metadata: {
+    createdAt: number
+    updatedAt: number
+    tags?: string[]
+    [key: string]: unknown
+  }
+}
+```
+
+Use `saveDocument(persistence, id, data, opts)` as the single source of truth for envelope construction — consumers don't hand-roll `{version, id, data, metadata}`.
+
+## StorageHost
+
+Server-side semantic layer for the `RemotePersistence` wire protocol (consumed by `@x-otto/service`):
+
+- Namespace whitelist + lazy per-namespace Persistence instance
+- Snapshot envelope wrapping/unwrapping
+- `getChangesSince` — incremental sync data source
+- `streamEvents` — SSE change events (SQLite-backed only, duck-type detection)
+
+```ts
+import { createFileStorageHost } from '@x-otto/persistence'
+
+const host = createFileStorageHost({ baseDir: './data', namespaces: ['sessions', 'knowledge'] })
+```
+
+## Backend Registry
+
+`StorageBackendRegistry` — global singleton for registering custom backends (Redis, S3, etc.):
+
+```ts
+import { storageBackendRegistry } from '@x-otto/persistence'
+
+storageBackendRegistry.registerPersistence('redis', (opts) => new RedisPersistence(opts))
+const store = createPersistence({ type: 'redis', url: 'redis://...' })
+```
+
+Built-in backends auto-register at module load time.
+
+## Key Files
+
+```
+src/
+  types.ts                  # Snapshot, Persistence<T>, Codec, pagination, StorageOptions
+  errors.ts                 # PersistenceError, RemotePersistenceError
+  save-document.ts          # saveDocument() — envelope constructor
+  memory-persistence.ts     # MemoryPersistenceBase<S> + MemoryPersistence<T>
+  disk-persistence.ts       # DiskPersistence<S> — atomic write base
+  file-persistence.ts       # FilePersistence<T> — Snapshot disk backend
+  remote-persistence.ts     # RemotePersistence<S> — HTTP client base
+  http-persistence.ts       # HttpPersistence<T> — Snapshot HTTP backend
+  sqlite-persistence.ts     # SqlitePersistence — snapshots + change_log tables
+  sqlite-db-cache.ts        # Shared SQLite connection cache (ref-counted)
+  change-log.ts             # ChangeLogEntry type
+  append-log.ts             # AppendLog<E> + memory/file backends + createAppendLog
+  sqlite-append-log.ts      # SQLite append-log backend
+  storage-factory.ts        # createPersistence + backend pre-registration
+  storage-host.ts           # StorageHost — server-side semantic layer
+  atomic-write.ts           # Atomic file write utility
+  panel-state.ts            # Extension panel state persistence
+  index.ts                  # Barrel exports
+```
+
+## Dependencies
+
+- Internal: `@x-otto/env`
+- External: `better-sqlite3` (SQLite backends)
+- Consumers: `@x-otto/session`, `@x-otto/memory`, `@x-otto/runtime`, `@x-otto/coding`, `@x-otto/service`, `@x-otto/persistence-client`
+
+## Testing
+
+```bash
+pnpm --filter @x-otto/persistence typecheck
+pnpm --filter @x-otto/persistence build
+pnpm vitest run packages/persistence/tests/
+```
+
+18 test files cover all backends CRUD + pagination, AppendLog (truncate / crash-tail repair / concurrency serialization), backend registry, factory, `saveDocument`, `StorageHost`, and change tracking.