ltcai 4.0.1 → 4.2.0

package/README.md

@@ -203,34 +203,32 @@ npm run dev
 
 ## Latest Release
 
-### v4.0.1 — Digital Brain Platform Maintenance
-
-- **Brain store architecture** — `knowledge_graph.py` is now a compatibility
-  shim over focused `latticeai/brain/` modules.
-- **v2 write-mastered graph** — `nodes_v2` / `edges_v2` are authoritative;
-  legacy tables stay as a compatibility projection, with pre-flip backup and
-  DB-format guard.
-- **Durable memory and context** — conversations, decisions, experiences, and
-  context traces live in the brain database family.
-- **Real act/runtime foundation** — workflows execute governed tools, pause for
-  approval, run asynchronously with SSE progress/cancellation, and LLM-backed
-  agent runs fail closed instead of fabricating output.
-- **Durable workspace governance** — stable user UUIDs, enforced policy,
-  invitations, and SQLite-backed Workspace OS state preserve identity and
-  workspace history without destructive migration.
-- **Local sovereignty** — signed exports, device identity, scoped graph export,
-  and Brain Network peer exchange are implemented at the API layer.
-- **Post-v4 parity closure** — durable async runs, stable identity/workspace
-  state, and the complete `/app` SPA parity/legacy-retirement work are included
-  in this maintenance release.
-
-See [RELEASE_NOTES_v4.0.1.md](RELEASE_NOTES_v4.0.1.md),
+### v4.2.0 — Brain Core & Storage Rebuild
+
+- **Independent Brain Core package** — `lattice_brain` exposes the Knowledge
+  Graph, durable conversations, memory/context, signed exchange, archives, and
+  storage tools for FastAPI, CLI, tests, and future tools.
+- **Pluggable storage layer** — `StorageEngine`, `SQLiteEngine`, and
+  `PostgresEngine` give the backend an explicit storage boundary. SQLite stays
+  the default.
+- **Honest vector search** — sqlite-vec is detected when available; otherwise
+  the real local hash-embedding cosine path remains active and reported.
+- **Postgres scale mode** — pgvector setup and SQLite-to-Postgres migration are
+  opt-in. Live Docker validation covers migration integrity, idempotence, and
+  pgvector distance search. Explicit Postgres selection fails loudly when
+  DSN/dependencies are missing; it never hides failure with a SQLite fallback.
+- **Consent-gated Docker setup** — Docker Compose setup is generated locally
+  and only starts Docker after explicit user consent.
+- **Encrypted `.latticebrain` archives** — local encrypted backup/restore over
+  the brain DB and blobs, with bad passphrases failing closed.
+
+See [RELEASE_NOTES_v4.2.0.md](RELEASE_NOTES_v4.2.0.md),
 [docs/kg-schema.md](docs/kg-schema.md),
 [FEATURE_STATUS.md](FEATURE_STATUS.md).
 
 ## How it works — every source converges into the graph
 
-As of v4.0.1, data sources flow through the brain ingestion pipeline into
+As of v4.2.0, data sources flow through the brain ingestion pipeline into
 the Knowledge Graph — no source bypasses it, none becomes an isolated silo:
 
 ```text
@@ -246,7 +244,8 @@ source (file · folder · PDF · web URL · browser tab · text)
 - **The graph is the asset.** Memory, search, and agents are views over it; models
   read it. Swap a model and your knowledge is unchanged.
 - **Portable, no cloud.** Export/import the graph as JSON, or take a full local
-  binary backup (DB + blobs) and restore it.
+  binary backup (DB + blobs), encrypted `.latticebrain` archive, or explicit
+  SQLite-to-Postgres migration plan and restore it.
 - **Local-first protects the graph.** It lives in local SQLite on your machine.
 
 For the deeper design, see [ARCHITECTURE.md](ARCHITECTURE.md) and
@@ -264,6 +263,12 @@ For the deeper design, see [ARCHITECTURE.md](ARCHITECTURE.md) and
 
 - [ARCHITECTURE.md](ARCHITECTURE.md) — workspace, graph, pipeline, and model overview
 - [docs/architecture.md](docs/architecture.md) — full architecture reference
+- [docs/V4_2_BRAIN_CORE_ARCHITECTURE.md](docs/V4_2_BRAIN_CORE_ARCHITECTURE.md) — v4.2.0 Brain Core package and storage architecture
+- [docs/V4_2_STORAGE_MIGRATION_REPORT.md](docs/V4_2_STORAGE_MIGRATION_REPORT.md) — v4.2.0 storage migration and archive report
+- [docs/V4_2_VALIDATION_REPORT.md](docs/V4_2_VALIDATION_REPORT.md) — v4.2.0 validation report
+- [docs/V4_1_FRONTEND_ARCHITECTURE_REVIEW.md](docs/V4_1_FRONTEND_ARCHITECTURE_REVIEW.md) — v4.1.0 frontend and desktop architecture review
+- [docs/V4_1_FRONTEND_MIGRATION_REPORT.md](docs/V4_1_FRONTEND_MIGRATION_REPORT.md) — v4.1.0 capability migration report
+- [docs/V4_1_VALIDATION_REPORT.md](docs/V4_1_VALIDATION_REPORT.md) — v4.1.0 validation report
 - [docs/V3_BACKEND_ARCHITECTURE.md](docs/V3_BACKEND_ARCHITECTURE.md) — backend storage, search, and retrieval
 
 ### Knowledge and retrieval
@@ -282,6 +287,8 @@ For the deeper design, see [ARCHITECTURE.md](ARCHITECTURE.md) and
 ### Releases
 
 - [RELEASE_NOTES.md](RELEASE_NOTES.md) — current release notes
+- [RELEASE_NOTES_v4.2.0.md](RELEASE_NOTES_v4.2.0.md)
+- [RELEASE_NOTES_v4.1.0.md](RELEASE_NOTES_v4.1.0.md)
 - [RELEASE_NOTES_v4.0.1.md](RELEASE_NOTES_v4.0.1.md)
 - [RELEASE_NOTES_v4.0.0.md](RELEASE_NOTES_v4.0.0.md)
 - [RELEASE_NOTES_v3.6.0.md](RELEASE_NOTES_v3.6.0.md)
@@ -294,6 +301,8 @@ For the deeper design, see [ARCHITECTURE.md](ARCHITECTURE.md) and
 
 | Version | Theme |
 | --- | --- |
+| **4.2.0** | Brain Core & Storage Rebuild — independent `lattice_brain` package, pluggable storage layer, sqlite-vec/pgvector capability reporting, explicit Postgres migration, consent-gated Docker setup, encrypted `.latticebrain` archives |
+| **4.1.0** | Frontend & Desktop Rebuild RC — React/Vite/OpenAPI desktop SPA, Tauri 2.0 primary shell, graph-first navigation, and legacy static frontend removal |
 | **4.0.1** | Digital Brain Platform maintenance — closes post-tag v4 gaps with durable async runs, stable identity/workspace state, full `/app` parity, and legacy UI retirement |
 | **4.0.0** | Digital Brain Platform — decomposed brain store, v2 write-mastered Knowledge Graph, durable memory/context, real workflow/agent foundations, signed brain exchange |
 | 3.6.0 | Knowledge Graph First — unified ingestion pipeline, formalized entity/relationship model, browser/web ingestion, local export/import/backup, provenance, KG as the primary surface |

package/desktop/electron/main.cjs

@@ -0,0 +1,44 @@
+const { app, BrowserWindow } = require("electron");
+const { spawn } = require("node:child_process");
+const path = require("node:path");
+
+const origin = process.env.LATTICEAI_DESKTOP_BACKEND_ORIGIN || "http://127.0.0.1:8765";
+let backend = null;
+
+function startBackend() {
+  if (process.env.LATTICEAI_DESKTOP_NO_BACKEND) return;
+  const command = process.env.LATTICEAI_DESKTOP_BACKEND_CMD || "python3 ltcai_cli.py --host 127.0.0.1 --port 8765";
+  const [bin, ...args] = command.split(/\s+/).filter(Boolean);
+  if (!bin) return;
+  backend = spawn(bin, args, {
+    cwd: process.env.LATTICEAI_DESKTOP_BACKEND_CWD || path.resolve(__dirname, "../.."),
+    env: { ...process.env, LATTICEAI_HOST: "127.0.0.1", LATTICEAI_PORT: "8765", LATTICEAI_TUNNEL: "false" },
+    stdio: "ignore",
+  });
+}
+
+function createWindow() {
+  const win = new BrowserWindow({
+    width: 1440,
+    height: 920,
+    minWidth: 1024,
+    minHeight: 720,
+    title: "Lattice AI",
+    webPreferences: {
+      contextIsolation: true,
+      nodeIntegration: false,
+      sandbox: true,
+    },
+  });
+  win.loadURL(`${origin}/app`);
+}
+
+app.whenReady().then(() => {
+  startBackend();
+  createWindow();
+});
+
+app.on("window-all-closed", () => {
+  if (backend) backend.kill();
+  if (process.platform !== "darwin") app.quit();
+});

package/docs/CHANGELOG.md

@@ -1,5 +1,89 @@
 # Changelog
 
+## [4.2.0] - 2026-06-12
+
+> Brain Core & Storage Rebuild release. The backend Digital Brain boundary is
+> available through the independent `lattice_brain` package, while the v4.1.0
+> frontend, FastAPI contracts, and SQLite user data remain compatible.
+
+### Added
+
+- `lattice_brain` import package with `BrainCore`, Knowledge Graph,
+  conversation, memory/context, identity/network, archive, and storage facades.
+- Pluggable storage layer: `StorageEngine`, `SQLiteEngine`, `PostgresEngine`,
+  `DockerPostgresWizard`, and `SQLiteToPostgresMigrator`.
+- sqlite-vec capability detection with honest `bruteforce-cosine` local vector
+  search fallback.
+- Opt-in Postgres/pgvector scale setup and non-destructive SQLite-to-Postgres
+  migration planning/copy tooling.
+- Live Docker-backed pgvector migration validation, including rowid-less FTS5
+  shadow tables, row-count integrity, idempotent reruns, and fail-closed
+  Postgres behavior.
+- Encrypted `.latticebrain` archive create/restore support for the SQLite brain
+  database and blob directory.
+- FastAPI routes for storage status, consent-gated Docker setup,
+  SQLite-to-Postgres migration, and encrypted archive create/restore.
+
+### Changed
+
+- FastAPI constructs the graph/conversation runtime through
+  `lattice_brain.BrainCore`; root and `latticeai.brain.*` imports remain
+  compatibility shims.
+- OpenAPI client regenerated with 313 paths.
+- System settings exposes API-backed storage status, Docker setup consent, and
+  migration planning controls.
+- Synchronized package/runtime versions to `4.2.0`, including Tauri config and
+  `lattice_brain.__version__`.
+
+### Expected Artifacts
+
+- `dist/ltcai-4.2.0-py3-none-any.whl`
+- `dist/ltcai-4.2.0.tar.gz`
+- `dist/ltcai-4.2.0.vsix`
+- `ltcai-4.2.0.tgz`
+
+## [4.1.0] - 2026-06-12
+
+> Frontend & Desktop Rebuild release candidate. The existing FastAPI backend,
+> Brain Core, storage architecture, and agent/workflow runtime remain the
+> source of truth; the frontend and desktop shell are replaced by a React/Vite
+> desktop architecture.
+
+### Added
+
+- React + TypeScript + Vite SPA under `frontend/`, using TanStack Query,
+  Zustand, React Flow, Cytoscape.js, Tailwind CSS, local shadcn-style
+  primitives, and a generated OpenAPI TypeScript client.
+- Tauri 2.0 desktop shell under `src-tauri/` that launches the local backend
+  and exposes the backend origin to the SPA; Electron fallback shell retained
+  under `desktop/electron/`.
+- Primary graph-first navigation: Brain, Ask, Capture, Act, Library, System.
+- OpenAPI export/generation script and frontend lint guard for generated-client
+  usage, no-CDN static assets, and stale frontend references.
+
+### Changed
+
+- `/app` now serves the built React/Vite bundle from `static/app`.
+- Legacy static v3 frontend assets and v3 build/lint scripts are removed after
+  capability parity was migrated into the new React surfaces.
+- Release/build scripts now build Vite app assets and preserve Python, npm, and
+  VSIX packaging flows at version `4.1.0`.
+- npm Python-backed scripts use `scripts/run_python.mjs` to prefer
+  `LTCAI_PYTHON` or the repo virtualenv before falling back to system Python.
+
+### Validation Scope
+
+- Python compile check, ruff, unit tests, live integration tests, frontend lint,
+  TypeScript build, Playwright visual tests, desktop shell checks, no-CDN
+  verification, release artifact validation, wheel smoke, and npm pack dry-run.
+
+### Expected Artifacts
+
+- `dist/ltcai-4.1.0-py3-none-any.whl`
+- `dist/ltcai-4.1.0.tar.gz`
+- `dist/ltcai-4.1.0.vsix`
+- `ltcai-4.1.0.tgz`
+
 ## [4.0.1] - 2026-06-12
 
 > Digital Brain Platform maintenance release for commits on `main` after tag

package/docs/V4_1_FRONTEND_ARCHITECTURE_REVIEW.md

@@ -0,0 +1,65 @@
+# v4.1.0 Frontend Architecture Review
+
+## Scope
+
+This review covers the v4.1.0 replacement of the `/app` frontend and desktop
+shell. It does not redesign Brain Core, storage, backend API contracts, or the
+agent/workflow runtime.
+
+## Baseline
+
+v4.0.1 served a static frontend assembled under `static/v3` with bespoke build
+and lint scripts. Capability parity existed, but the implementation duplicated
+view systems, carried legacy static route assumptions, and could not cleanly
+share typed API contracts with the FastAPI backend.
+
+## Target Architecture
+
+- Desktop shell: Tauri 2.0 primary, Electron fallback only.
+- Frontend: React, TypeScript, Vite, TanStack Query, Zustand, React Flow,
+  Cytoscape.js, Tailwind CSS, local shadcn-style primitives.
+- API: generated OpenAPI TypeScript client from the existing FastAPI app.
+- Runtime: single client-side SPA, no SSR, no CDN.
+- Backend: existing FastAPI app remains the source of truth.
+
+## Implemented Architecture
+
+- `frontend/` owns the React/Vite source.
+- `frontend/src/api/openapi.ts` is generated from `frontend/openapi.json`.
+- `frontend/src/api/client.ts` wraps generated-client JSON calls and keeps
+  streaming chat plus multipart upload as explicit fetch special cases.
+- `frontend/src/routes.ts` defines Brain, Ask, Capture, Act, Library, System and
+  maps legacy hash routes into those groups.
+- `static/app` is the shipped build output and is served by `/app`.
+- `src-tauri/` contains the Tauri 2.0 shell and backend-origin bridge.
+- `desktop/electron/` contains the fallback shell.
+- `static/sw.js` precaches the React app manifest/assets for offline-capable
+  local startup.
+
+## Boundary Decisions
+
+- The backend contract was preserved; only the frontend consumer changed.
+- Brain Core and storage modules were not modified for UI convenience.
+- No CDN fallback was introduced; all visual/runtime dependencies are bundled.
+- No demo-only controls were added. Unavailable capabilities render explicit
+  unavailable/error states based on API responses.
+- npm runtime installs are kept lean: React/Vite/Tauri/Electron toolchains are
+  development dependencies because the distributable app ships built assets.
+
+## Capability Mapping
+
+| v4.0.1 capability | v4.1.0 surface |
+| --- | --- |
+| Knowledge Graph, hybrid search, memory, provenance, portability | Brain |
+| Chat, conversations, context trace, attachments | Ask |
+| Uploads, connected folders, local runtime, URL capture, index pipeline | Capture |
+| Agents, runs, approvals, workflows, triggers, hooks, tools | Act |
+| Models, embeddings, skills, MCP, templates/plugins | Library |
+| Account, auth, workspaces, snapshots, activity, network, settings, admin/security | System |
+
+## Residual Risk
+
+- The first Vite bundle is larger than 500 kB after minification because graph,
+  workflow, and app shell libraries are included in the initial desktop SPA
+  bundle. This is a performance optimization target, not a capability blocker,
+  and the build remains fully local/offline.

package/docs/V4_1_FRONTEND_MIGRATION_REPORT.md

@@ -0,0 +1,70 @@
+# v4.1.0 Frontend Migration Report
+
+## Objective
+
+Replace the v4.0.1 static frontend implementation with the Digital Brain
+desktop architecture while preserving capabilities, user data, backend API
+contracts, and local-first/offline operation.
+
+## Migration Summary
+
+- Added React + TypeScript + Vite source under `frontend/`.
+- Added generated OpenAPI schema/client files under `frontend/openapi.json` and
+  `frontend/src/api/openapi.ts`.
+- Added React capability pages for Brain, Ask, Capture, Act, Library, System.
+- Added Tauri 2.0 desktop shell under `src-tauri/`.
+- Added Electron fallback shell under `desktop/electron/`.
+- Replaced `/app` static serving with `static/app/index.html`.
+- Replaced v3 asset build/lint scripts with Vite build and frontend lint guards.
+- Removed `static/v3` and retired `scripts/build_v3_assets.mjs` /
+  `scripts/lint_v3.mjs`.
+- Updated package metadata so Python wheel/sdist, npm tgz, and VSIX builds ship
+  the new `static/app` assets.
+
+## Compatibility
+
+- Existing FastAPI routes are preserved.
+- The generated OpenAPI client is produced from the live app schema.
+- Legacy app hash routes are mapped into the new primary navigation groups.
+- Service worker caching now targets the Vite app manifest instead of v3 assets.
+- User data formats, Brain database files, Workspace OS state, snapshots,
+  conversations, memories, and graph data are not migrated or destructively
+  changed by the frontend rebuild.
+
+## Capability Parity
+
+| Capability | Migration result |
+| --- | --- |
+| Brain graph exploration | Cytoscape.js graph view in Brain |
+| Hybrid search | Brain search tab backed by `/api/search/hybrid` |
+| Memory recall | Brain memory tab backed by memory APIs |
+| Provenance and portability | Brain provenance/portability actions |
+| Chat and context trace | Ask surface with streaming chat and context panel |
+| Conversation management | Ask history backed by conversation APIs |
+| Document upload | Capture upload using multipart backend endpoint |
+| Connected folders/local runtime | Capture desktop/local runtime APIs |
+| Index pipeline and URL capture | Capture pipeline/browser ingestion APIs |
+| Agents and run records | Act agents/runs/approvals backed by runtime APIs |
+| Workflow graph | Act React Flow visualization and workflow APIs |
+| Triggers, hooks, tools | Act surfaces backed by trigger/hook/tool APIs |
+| Models and embeddings | Library model/embedding APIs |
+| Skills, MCP, templates/plugins | Library registry APIs |
+| Account/auth/profile/password | System account APIs |
+| Workspaces/invitations | System workspace APIs |
+| Snapshots/time-machine | System snapshot APIs |
+| Activity/presence/network | System activity and network APIs |
+| Admin/security/settings | System admin and settings APIs |
+
+## Removed Frontend Debt
+
+- Duplicate static v3 view modules.
+- Legacy frontend build pipeline.
+- Legacy v3 frontend lint script.
+- Static v3 asset tree in release packages.
+- Direct current-release references to v4.0.1 in latest docs.
+
+## Data Preservation
+
+No user data migration is required for v4.1.0. All persisted state remains owned
+by the existing backend stores. The frontend rebuild only changes the client and
+desktop shell used to access those stores.

package/docs/V4_1_VALIDATION_REPORT.md

@@ -0,0 +1,47 @@
+# v4.1.0 Validation Report
+
+## Summary
+
+v4.1.0 RC validation passed for the React/Vite frontend, Tauri desktop shell,
+FastAPI backend compatibility, release artifacts, and installed-wheel smoke.
+
+## Commands Run
+
+| Area | Command | Result |
+| --- | --- | --- |
+| OpenAPI generation | `npm run frontend:openapi` | Pass; generated 308 paths |
+| Static app build | `npm run build:assets` | Pass; wrote `static/app/asset-manifest.json` |
+| Frontend lint / no CDN / API compatibility | `npm run lint` | Pass; scanned 28 frontend/static files; OpenAPI exposes 308 paths |
+| TypeScript build | `npm run typecheck` | Pass; frontend TS + VS Code extension build |
+| Python compile | `npm run check:python` | Pass; compiled 206 modules |
+| Ruff | `node scripts/run_python.mjs -m ruff check .` | Pass |
+| Unit tests | `npm run test:unit -- --tb=short` | Pass; 585 passed, 2 warnings |
+| Live integration tests | `LTCAI_TEST_BASE_URL=http://127.0.0.1:8899 npm run test:integration -- --tb=short` | Pass; 9 passed |
+| Playwright visual/offline tests | `npx playwright test tests/visual/v3.spec.js` | Pass; 12 passed |
+| Tauri cargo check | `npm run desktop:tauri:check` | Pass |
+| Tauri desktop build | `LATTICEAI_DESKTOP_NO_BACKEND=1 npm run desktop:tauri:build` | Pass; built `.app` and DMG |
+| Electron fallback syntax/version | `node --check desktop/electron/main.cjs && npx electron --version` | Pass; Electron v42.4.0 |
+| Release artifacts | `npm run release:artifacts` plus resumed `npm pack && npm run package:vsix` | Pass |
+| Artifact validation | `npm run release:validate` | Pass; exact v4.1.0 wheel/sdist/VSIX/tgz found |
+| Wheel smoke | `node scripts/run_python.mjs scripts/wheel_smoke.py --wheel dist/ltcai-4.1.0-py3-none-any.whl` | Pass; imports 19 modules and `/health` reports 4.1.0 |
+| npm pack dry-run | `npm pack --dry-run` | Pass; `ltcai-4.1.0.tgz`, 259 files, 2.9 MB |
+
+## Generated Artifacts
+
+- `dist/ltcai-4.1.0-py3-none-any.whl`
+- `dist/ltcai-4.1.0.tar.gz`
+- `dist/ltcai-4.1.0.vsix`
+- `ltcai-4.1.0.tgz`
+- `src-tauri/target/release/bundle/macos/Lattice AI.app`
+- `src-tauri/target/release/bundle/dmg/Lattice AI_4.1.0_aarch64.dmg`
+
+## Notes
+
+- Vite reports one large initial bundle warning after minification. The app
+  remains fully local/offline and passes visual/offline startup validation.
+- `cargo check` reports a future-incompatibility warning in transitive
+  `block v0.1.6`; the current Tauri 2.0 build passes on Rust 1.96.
+- Release artifact validation warns that historical artifacts remain in
+  `dist/`; this is expected and reinforces the rule to upload only exact
+  v4.1.0 filenames, never `dist/*`.
+- No external registry publish was performed.

package/docs/V4_2_BRAIN_CORE_ARCHITECTURE.md

@@ -0,0 +1,97 @@
+# Lattice AI v4.2.0 — Brain Core Architecture
+
+Status: released validation complete
+
+v4.2.0 extracts the Digital Brain backend boundary into the importable
+`lattice_brain` package while preserving the existing FastAPI contracts and the
+v4.1.0 user data layout. FastAPI, CLI, tests, and future tools can now import
+Brain Core directly instead of reaching through root compatibility modules.
+
+## Package Boundary
+
+- `lattice_brain` is the independent Brain Core package namespace.
+- `latticeai.brain` remains as a compatibility namespace for existing callers.
+- Root modules such as `knowledge_graph.py` and `kg_schema.py` remain
+  compatibility shims.
+- FastAPI now constructs the graph and durable conversation store through
+  `lattice_brain.BrainCore`.
+
+## Brain Core Surfaces
+
+The package exposes the implemented v4 brain modules:
+
+- Knowledge system: `KnowledgeGraphStore`
+- Memory system: `BrainMemory`
+- Context assembler: `ContextAssembler`
+- Durable conversations: `ConversationStore`
+- Device identity and signed exchange helpers
+- Brain Network compatibility exports
+- Encrypted `.latticebrain` archives
+- Storage abstraction and migration tools
+
+The frontend still talks only to FastAPI localhost APIs. No frontend code calls
+Python directly.
+
+## Storage Layer
+
+`lattice_brain.storage` introduces:
+
+- `StorageEngine` ABC
+- `SQLiteEngine` default engine
+- `PostgresEngine` opt-in engine with pgvector extension setup
+- `DockerPostgresWizard` explicit-consent local Docker setup
+- `SQLiteToPostgresMigrator` idempotent migration planner/runner, including
+  rowid-less FTS5 shadow table support through declared primary keys
+
+SQLite remains the default. Postgres is never required. If
+`LATTICEAI_STORAGE_ENGINE=postgres` is selected without a DSN or optional
+dependency support, startup fails honestly instead of silently falling back to
+SQLite.
+
+## Vector Search
+
+SQLite vector search remains real and local:
+
+- Existing vector rows stay in `vector_embeddings`.
+- The active fallback is deterministic local hash embeddings with brute-force
+  cosine scoring.
+- `sqlite-vec` is detected and loaded when available.
+- Capability reports distinguish `sqlite-vec` from `bruteforce-cosine`; the
+  fallback is reported honestly and is still a real search path.
+
+Postgres scale mode initializes a pgvector-backed `brain_vectors` table when
+the `vector` extension is available.
+
+## Archive Model
+
+v4.2.0 adds encrypted `.latticebrain` archives:
+
+- AES-256-GCM payload encryption
+- PBKDF2-HMAC-SHA256 key derivation
+- Encrypted SQLite database and blob payload
+- Restore replaces the target DB/WAL/SHM safely and restores blobs
+- Wrong passphrase or tampered data fails closed
+
+The existing JSON export/import and ZIP backup/restore paths remain compatible.
+
+## FastAPI APIs
+
+New localhost APIs:
+
+- `GET /api/brain/storage`
+- `POST /api/brain/storage/postgres/docker`
+- `POST /api/brain/storage/migrate-postgres`
+- `POST /api/knowledge-graph/archive`
+- `POST /api/knowledge-graph/archive/restore`
+
+All mutating operations require admin authorization. Docker starts only when the
+request explicitly carries consent.
+
+## Compatibility
+
+- Existing v4.1.0 SQLite data remains in `knowledge_graph.sqlite`.
+- Existing `knowledge_graph.py`, `kg_schema.py`, and `latticeai.brain.*`
+  imports continue to work.
+- Existing FastAPI routes remain available.
+- Existing release artifacts still build.
+- No data-loss migration is performed automatically.

package/docs/V4_2_STORAGE_MIGRATION_REPORT.md

@@ -0,0 +1,91 @@
+# Lattice AI v4.2.0 — Storage Migration Report
+
+Status: released validation complete
+
+v4.2.0 adds a pluggable storage layer without changing the default local-first
+runtime. Existing users continue on SQLite. Postgres is an explicit opt-in
+scale target, not a required dependency and not an automatic fallback.
+
+## Existing Data
+
+- Existing v4.1.0 data remains in `~/.ltcai/knowledge_graph.sqlite`.
+- Blob payloads remain in `~/.ltcai/knowledge_graph_blobs/`.
+- Durable conversations continue to share the same SQLite DB family.
+- No startup migration rewrites user data for v4.2.0.
+
+## SQLite Engine
+
+`SQLiteEngine` owns SQLite connection setup:
+
+- WAL mode
+- foreign keys enabled
+- local path creation
+- backup and restore helpers
+- sqlite-vec loading attempt when installed
+- honest capability report when sqlite-vec is unavailable
+
+The current graph store continues to use SQLite-specific SQL and is therefore
+wired only to `SQLiteEngine` for the active FastAPI runtime.
+
+## Postgres Engine
+
+`PostgresEngine` is opt-in and fail-closed:
+
+- Requires `LATTICEAI_POSTGRES_DSN` when explicitly selected.
+- Requires optional `psycopg` support.
+- Creates `lattice_brain` schema by default.
+- Runs `CREATE EXTENSION IF NOT EXISTS vector`.
+- Creates a pgvector-backed `brain_vectors` table.
+
+If Postgres is selected but unavailable, Lattice AI reports the error. It does
+not hide the failure by falling back to SQLite.
+
+## Docker Setup
+
+`DockerPostgresWizard` writes a local Docker Compose file for
+`pgvector/pgvector:pg16`. It never runs Docker unless the caller explicitly
+passes consent.
+
+API behavior:
+
+- `consent=false`: writes the compose file and returns `consent_required`.
+- `dry_run=true`: returns the exact Docker command without starting anything.
+- `consent=true` and `dry_run=false`: runs `docker compose up -d postgres`.
+
+## SQLite to Postgres Migration
+
+`SQLiteToPostgresMigrator` plans and copies all user tables from a SQLite brain
+database into Postgres:
+
+- Introspects non-internal SQLite tables.
+- Preserves every row.
+- Uses table `id` as the idempotence key when present.
+- Uses declared primary keys, including composite keys on rowid-less FTS5 shadow
+  tables.
+- Uses preserved `__source_rowid` when no declared key exists and SQLite rowid
+  is available.
+- Upserts rows on repeated runs.
+- Leaves the source SQLite database untouched.
+
+The API defaults to dry-run migration planning. Actual copy requires an
+explicit DSN and `dry_run=false`.
+
+Live v4.2.0 validation used explicit Docker consent to start
+`pgvector/pgvector:pg16`, copy a seeded v4 SQLite brain database into
+Postgres, verify source and destination row counts, rerun the migration
+idempotently, query pgvector distance ordering, and tear down the test Compose
+stack with volumes.
+
+## Encrypted Archives
+
+`.latticebrain` archive support was added for local encrypted backup/restore:
+
+- Database and blobs are zipped locally.
+- Payload is encrypted with AES-256-GCM.
+- Keys derive from the user passphrase via PBKDF2-HMAC-SHA256.
+- Restore rejects bad passphrases or tampered payloads.
+
+## Compatibility Result
+
+No v4.1.0 capability is removed. SQLite remains the default and does not depend
+on Docker, Postgres, pgvector, or network access.