ltcai 3.5.0 → 4.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (181) hide show
  1. package/README.md +73 -35
  2. package/docs/CARRYOVER_AUDIT_v3.6.0.md +61 -0
  3. package/docs/CHANGELOG.md +32 -0
  4. package/docs/HANDOVER_v3.6.0.md +46 -0
  5. package/docs/RUNTIME_HOOK_COVERAGE_v3.6.0.md +49 -0
  6. package/docs/V4_BRAIN_ARCHITECTURE.md +322 -0
  7. package/docs/V4_DIGITAL_BRAIN_RECOVERY.md +509 -0
  8. package/docs/V4_IMPLEMENTATION_PLAN.md +470 -0
  9. package/docs/architecture.md +13 -12
  10. package/docs/kg-schema.md +102 -53
  11. package/docs/privacy.md +18 -2
  12. package/docs/security-model.md +17 -0
  13. package/kg_schema.py +139 -10
  14. package/knowledge_graph.py +874 -26
  15. package/knowledge_graph_api.py +11 -127
  16. package/latticeai/__init__.py +1 -1
  17. package/latticeai/api/admin.py +1 -1
  18. package/latticeai/api/agents.py +7 -1
  19. package/latticeai/api/auth.py +27 -4
  20. package/latticeai/api/browser.py +217 -0
  21. package/latticeai/api/chat.py +112 -76
  22. package/latticeai/api/health.py +1 -1
  23. package/latticeai/api/hooks.py +1 -1
  24. package/latticeai/api/knowledge_graph.py +146 -0
  25. package/latticeai/api/local_files.py +1 -1
  26. package/latticeai/api/mcp.py +23 -11
  27. package/latticeai/api/memory.py +1 -1
  28. package/latticeai/api/models.py +1 -1
  29. package/latticeai/api/network.py +81 -0
  30. package/latticeai/api/portability.py +93 -0
  31. package/latticeai/api/realtime.py +1 -1
  32. package/latticeai/api/search.py +26 -2
  33. package/latticeai/api/security_dashboard.py +2 -3
  34. package/latticeai/api/setup.py +2 -2
  35. package/latticeai/api/static_routes.py +2 -4
  36. package/latticeai/api/tools.py +3 -0
  37. package/latticeai/api/workflow_designer.py +46 -0
  38. package/latticeai/api/workspace.py +71 -49
  39. package/latticeai/app_factory.py +1710 -0
  40. package/latticeai/brain/__init__.py +18 -0
  41. package/latticeai/brain/context.py +213 -0
  42. package/latticeai/brain/conversations.py +236 -0
  43. package/latticeai/brain/identity.py +175 -0
  44. package/latticeai/brain/memory.py +102 -0
  45. package/latticeai/brain/network.py +205 -0
  46. package/latticeai/core/agent.py +31 -7
  47. package/latticeai/core/audit.py +0 -7
  48. package/latticeai/core/config.py +1 -1
  49. package/latticeai/core/context_builder.py +1 -2
  50. package/latticeai/core/enterprise.py +1 -1
  51. package/latticeai/core/graph_curator.py +2 -2
  52. package/latticeai/core/marketplace.py +1 -1
  53. package/latticeai/core/mcp_registry.py +791 -0
  54. package/latticeai/core/model_compat.py +1 -1
  55. package/latticeai/core/model_resolution.py +0 -1
  56. package/latticeai/core/multi_agent.py +238 -4
  57. package/latticeai/core/security.py +1 -1
  58. package/latticeai/core/sessions.py +37 -7
  59. package/latticeai/core/workflow_engine.py +114 -2
  60. package/latticeai/core/workspace_os.py +58 -10
  61. package/latticeai/models/__init__.py +7 -0
  62. package/latticeai/models/router.py +779 -0
  63. package/latticeai/server_app.py +29 -1504
  64. package/latticeai/services/agent_runtime.py +1 -0
  65. package/latticeai/services/app_context.py +75 -14
  66. package/latticeai/services/ingestion.py +318 -0
  67. package/latticeai/services/kg_portability.py +207 -0
  68. package/latticeai/services/memory_service.py +39 -11
  69. package/latticeai/services/model_runtime.py +2 -5
  70. package/latticeai/services/platform_runtime.py +100 -23
  71. package/latticeai/services/search_service.py +17 -8
  72. package/latticeai/services/tool_dispatch.py +12 -2
  73. package/latticeai/services/triggers.py +241 -0
  74. package/latticeai/services/upload_service.py +37 -12
  75. package/latticeai/services/workspace_service.py +31 -0
  76. package/llm_router.py +29 -772
  77. package/ltcai_cli.py +1 -2
  78. package/mcp_registry.py +25 -788
  79. package/p_reinforce.py +124 -14
  80. package/package.json +11 -8
  81. package/scripts/build_vsix.mjs +72 -0
  82. package/scripts/bump_version.py +99 -0
  83. package/scripts/generate_diagrams.py +0 -1
  84. package/scripts/lint_v3.mjs +82 -18
  85. package/scripts/validate_release_artifacts.py +0 -1
  86. package/scripts/wheel_smoke.py +142 -0
  87. package/server.py +11 -7
  88. package/setup_wizard.py +1142 -0
  89. package/static/account.html +2 -4
  90. package/static/admin.html +3 -5
  91. package/static/chat.html +3 -6
  92. package/static/graph.html +2 -4
  93. package/static/sw.js +81 -52
  94. package/static/v3/asset-manifest.json +20 -19
  95. package/static/v3/css/{lattice.base.e4cdd05d.css → lattice.base.49deefb5.css} +1 -1
  96. package/static/v3/css/lattice.base.css +1 -1
  97. package/static/v3/css/{lattice.components.9b49d614.css → lattice.components.cde18231.css} +1 -1
  98. package/static/v3/css/lattice.components.css +1 -1
  99. package/static/v3/css/{lattice.shell.8fcc9d33.css → lattice.shell.29d36d85.css} +1 -1
  100. package/static/v3/css/lattice.shell.css +1 -1
  101. package/static/v3/css/{lattice.tokens.e7018963.css → lattice.tokens.304cbc40.css} +3 -0
  102. package/static/v3/css/lattice.tokens.css +3 -0
  103. package/static/v3/css/{lattice.views.22f69117.css → lattice.views.0a18b6c5.css} +2 -2
  104. package/static/v3/css/lattice.views.css +2 -2
  105. package/static/v3/index.html +3 -4
  106. package/static/v3/js/{app.d086489d.js → app.356e6452.js} +1 -1
  107. package/static/v3/js/core/{api.12b568ad.js → api.7a308b89.js} +39 -1
  108. package/static/v3/js/core/api.js +38 -0
  109. package/static/v3/js/core/{routes.d214b399.js → routes.7222343d.js} +22 -22
  110. package/static/v3/js/core/routes.js +22 -22
  111. package/static/v3/js/core/{shell.d05266f5.js → shell.a1657f20.js} +4 -4
  112. package/static/v3/js/core/shell.js +1 -1
  113. package/static/v3/js/core/{store.34ebd5e6.js → store.204a08b2.js} +1 -1
  114. package/static/v3/js/core/store.js +1 -1
  115. package/static/v3/js/views/graph-canvas.17c15d65.js +509 -0
  116. package/static/v3/js/views/graph-canvas.js +509 -0
  117. package/static/v3/js/views/{hybrid-search.b22b97e0.js → hybrid-search.2fb63ed9.js} +1 -2
  118. package/static/v3/js/views/hybrid-search.js +1 -2
  119. package/static/v3/js/views/knowledge-graph.5e40cbeb.js +509 -0
  120. package/static/v3/js/views/knowledge-graph.js +326 -54
  121. package/static/vendor/chart.umd.min.js +20 -0
  122. package/static/vendor/fonts/inter-latin-300-normal.woff2 +0 -0
  123. package/static/vendor/fonts/inter-latin-400-normal.woff2 +0 -0
  124. package/static/vendor/fonts/inter-latin-500-normal.woff2 +0 -0
  125. package/static/vendor/fonts/inter-latin-600-normal.woff2 +0 -0
  126. package/static/vendor/fonts/inter-latin-700-normal.woff2 +0 -0
  127. package/static/vendor/fonts/inter-latin-800-normal.woff2 +0 -0
  128. package/static/vendor/fonts/inter.css +44 -0
  129. package/static/vendor/icons/tabler-icons.min.css +4 -0
  130. package/static/vendor/icons/tabler-icons.woff2 +0 -0
  131. package/static/vendor/marked.min.js +69 -0
  132. package/static/workspace.html +2 -2
  133. package/telegram_bot.py +1 -2
  134. package/tools/commands.py +4 -2
  135. package/tools/computer.py +1 -1
  136. package/tools/documents.py +1 -3
  137. package/tools/filesystem.py +0 -4
  138. package/tools/knowledge.py +1 -3
  139. package/tools/network.py +1 -3
  140. package/codex_telegram_bot.py +0 -195
  141. package/docs/assets/v3.4.0/agent-run.png +0 -0
  142. package/docs/assets/v3.4.0/agents.png +0 -0
  143. package/docs/assets/v3.4.0/before/chat-before.png +0 -0
  144. package/docs/assets/v3.4.0/before/files-before.png +0 -0
  145. package/docs/assets/v3.4.0/chat.png +0 -0
  146. package/docs/assets/v3.4.0/connect-folder.png +0 -0
  147. package/docs/assets/v3.4.0/files.png +0 -0
  148. package/docs/assets/v3.4.0/home.png +0 -0
  149. package/docs/assets/v3.4.0/hooks-dispatch.png +0 -0
  150. package/docs/assets/v3.4.0/knowledge-graph.png +0 -0
  151. package/docs/assets/v3.4.0/local-agent.png +0 -0
  152. package/docs/assets/v3.4.0/memory.png +0 -0
  153. package/docs/assets/v3.4.0/settings.png +0 -0
  154. package/docs/assets/v3.4.0/vision-input.png +0 -0
  155. package/docs/assets/v3.4.0/workflows.png +0 -0
  156. package/docs/assets/v3.4.1/e2e_runtime_log.txt +0 -42
  157. package/docs/assets/v3.4.1/hooks-dispatch.png +0 -0
  158. package/docs/assets/v3.4.1/local-agent.png +0 -0
  159. package/docs/images/admin-dashboard.png +0 -0
  160. package/docs/images/architecture.png +0 -0
  161. package/docs/images/enterprise.png +0 -0
  162. package/docs/images/graph.png +0 -0
  163. package/docs/images/hero.gif +0 -0
  164. package/docs/images/knowledge-graph.png +0 -0
  165. package/docs/images/lattice-ai-demo.gif +0 -0
  166. package/docs/images/lattice-ai-hero.png +0 -0
  167. package/docs/images/logo.svg +0 -33
  168. package/docs/images/mobile-responsive.png +0 -0
  169. package/docs/images/model-recommendation.png +0 -0
  170. package/docs/images/onboarding.png +0 -0
  171. package/docs/images/organization.png +0 -0
  172. package/docs/images/pipeline.png +0 -0
  173. package/docs/images/screenshot-admin.png +0 -0
  174. package/docs/images/screenshot-chat.png +0 -0
  175. package/docs/images/screenshot-graph.png +0 -0
  176. package/docs/images/skills.png +0 -0
  177. package/docs/images/workspace-dark.png +0 -0
  178. package/docs/images/workspace-light.png +0 -0
  179. package/docs/images/workspace.png +0 -0
  180. package/requirements.txt +0 -16
  181. package/static/v3/js/views/knowledge-graph.a14ea7e7.js +0 -237
@@ -0,0 +1,470 @@
1
+ # Lattice AI v4.0.0 — Implementation Plan
2
+
3
+ Companion to `docs/V4_BRAIN_ARCHITECTURE.md`. Audit evidence in
4
+ `docs/v4-audit/*.json`. Program state in `docs/V4_DIGITAL_BRAIN_RECOVERY.md`.
5
+
6
+ ## Ground rules (every track)
7
+
8
+ - Branch: `feat/v4-digital-brain` only. Conventional commits. Commit only
9
+ verified work; one commit (or a few) per track.
10
+ - Gate: `.venv/bin/python -m pytest tests/unit -q` green (baseline 455 →
11
+ grows; zero regressions), plus the new tests each track ships. The 9
12
+ pre-existing integration failures (need a live server) are not the gate but
13
+ must not grow.
14
+ - Python 3.11-compatible syntax (no nested same-quote f-strings). `.venv` is
15
+ 3.14 — fine for running, but write for 3.11.
16
+ - **No placeholder code, no demo data, no fabricated numbers, no
17
+ silently-skipped work.** A capability that cannot complete must surface an
18
+ explicit state (`simulation`, `awaiting_approval`, `unavailable`), never a
19
+ fake success.
20
+ - Data safety: every schema migration is preceded by an automatic backup
21
+ (existing `kg_portability` machinery) and is idempotent + re-entrant.
22
+ User data directories (`~/.ltcai`, `~/.ltcai-brain`) are never deleted —
23
+ absorptions import, then deactivate the old writer, leaving source files
24
+ in place.
25
+ - Update `docs/V4_DIGITAL_BRAIN_RECOVERY.md` after every track (status,
26
+ files touched, validation result) and before any foreseeable session limit.
27
+
28
+ ## Track sequence and ownership
29
+
30
+ Tracks are sequenced by dependency and run **strictly serially** — several
31
+ files have multiple owners across tracks (`api/chat.py`: T1/T2/T5;
32
+ `server_app.py`: T2/T4/T6; `api/workspace.py`: T1/T9; `brain/store.py`:
33
+ T3/T6/T8), so sequence, not disjointness, is the conflict protection.
34
+ "Owns" below means: *during that track*, only that track touches the file,
35
+ and only within the named scope. T1/T2 unblock everything; T3→T6 are the
36
+ brain spine; T7/T8 build on the spine; T9 items 2-5 (canvas, vendoring,
37
+ sw.js, build/lint) may run in parallel with backend tracks, but T9 items
38
+ 1/3/6 and all of T9b depend on the T3-T8 API contracts. T10 closes.
39
+
40
+ > **NORMATIVE**: every track below is amended by §"Design-review amendments"
41
+ > at the end of this document. Implementers must read both the track section
42
+ > and its amendments — the amendments win on conflict.
43
+
44
+ ---
45
+
46
+ ### T1 — Truth & safety floor *(small; first because every later track builds on honest primitives)*
47
+
48
+ Owns: `latticeai/api/workspace.py`, `latticeai/core/workspace_os.py`
49
+ (summary/leak only), `latticeai/api/chat.py` (context filter only),
50
+ `latticeai/services/memory_service.py`, `latticeai/core/multi_agent.py` +
51
+ `latticeai/services/agent_runtime.py` (mode labeling only),
52
+ `static/v3/js/views/hybrid-search.js`, `README.md` (claims only).
53
+
54
+ 1. Close by-id authz gaps: snapshot get/area/export/compare + memory delete
55
+ gate through `resolve_read_scope`/`resolve_write_scope` using the record's
56
+ own workspace_id; ownership check on memory delete.
57
+ 2. Strip the unfiltered `workspaces` key from `/workspace/os` (keep the
58
+ membership-filtered `workspace_registry`).
59
+ 3. Fix `build_recent_chat_context` leak: scope to the requesting user's
60
+ conversation (assistant turns only within it).
61
+ 4. Fix `MemoryService.recall` dead graph branch (`results` → `matches`);
62
+ replace hardcoded 0.6/0.5 scores with real normalized lexical scores;
63
+ stop sorting on constants.
64
+ 5. Persist `mode: "simulation"` on every deterministic multi-agent/workflow
65
+ run record; simulated runs stop writing run-derived nodes into the KG.
66
+ 6. Remove the hardcoded fusion-meter values in `hybrid-search.js` (render
67
+ real scores from the API or remove the meters).
68
+ 7. README honesty pass: agent/workflow claims rewritten to match reality
69
+ (full rewrite happens in T10; this removes the falsehoods now).
70
+
71
+ Tests: authz regression tests (cross-workspace access denied), recall returns
72
+ graph results, context isolation test, run records carry mode.
73
+
74
+ ### T2 — Packaging & app factory *(unblocks clean work everywhere)*
75
+
76
+ Owns: root `setup.py`→`setup_wizard.py`, `pyproject.toml`, `requirements.txt`,
77
+ `latticeai/server_app.py` (assembly only), `latticeai/services/app_context.py`,
78
+ `latticeai/api/deps.py`, `latticeai/api/setup.py`, `latticeai/api/chat.py`
79
+ (telegram import only), `codex_telegram_bot.py`, `perm_monitor.py`,
80
+ `knowledge_graph_api.py`, `.github/workflows/*`, ruff config.
81
+
82
+ 1. Rename root `setup.py` → `setup_wizard.py`; update importers
83
+ (`server_app.py:149`, `api/setup.py`); add to `py-modules`; verify wheel
84
+ contains it; add installed-wheel smoke test (build → clean venv → install
85
+ → `import latticeai.server_app` → `/health`) to CI and scripts.
86
+ 2. `create_app(config) -> FastAPI` factory: move import-time singleton
87
+ construction and MLX init out of import scope; build `AppContext`; keep
88
+ module-level names as thin accessors during deprecation (tests import
89
+ them). Router factories accept the context (migrate the worst two:
90
+ chat ~25 kwargs, workspace ~30 kwargs; others opportunistically).
91
+ 3. Decouple telegram: `broadcast_web_chat` becomes a `RealtimeBus` subscriber
92
+ registered only when `ENABLE_TELEGRAM`; `api/chat.py` drops the
93
+ unconditional import.
94
+ 4. Delete dead modules: `codex_telegram_bot.py`, `perm_monitor.py`; remove
95
+ from packaging lists. **`knowledge_graph_api.py` is NOT dead** — it serves
96
+ the `/knowledge-graph/*` data endpoints the v3 SPA consumes: migrate its
97
+ data router into `latticeai/api/knowledge_graph.py` with endpoint-parity
98
+ tests (stats/graph/documents/search/context/neighbors/ingest unchanged);
99
+ its legacy `/graph` page routes move to T9's redirect map. Also owned
100
+ here: relocate `llm_router.py` → `latticeai/models/router.py` and
101
+ `mcp_registry.py` → `latticeai/core/mcp_registry.py` with root shims, and
102
+ update their importers (`api/models.py`, `api/setup.py`,
103
+ `services/model_runtime.py`, `server_app.py`).
104
+ 5. ruff baseline (`[tool.ruff]`, pragmatic select set), fix violations or
105
+ per-file-ignore legacy monoliths; CI gate. Bounded dependency constraints
106
+ in pyproject; `requirements.txt` deleted or generated.
107
+
108
+ Tests: existing suite green; new wheel smoke test; factory produces a working
109
+ app (TestClient).
110
+
111
+ ### T3 — Brain store: decomposition + v2 write-mastering + retrieval
112
+
113
+ Owns: `knowledge_graph.py` (becomes shim), new `latticeai/brain/` package
114
+ (`store.py`, `extraction.py`, `documents.py`, `discovery.py`), `kg_schema.py`
115
+ (moves to `latticeai/brain/schema.py` with root shim), `docs/kg-schema.md`.
116
+
117
+ 1. Mechanical decomposition of `KnowledgeGraphStore` along the seams named in
118
+ the backend audit; root `knowledge_graph.py` becomes a re-export shim.
119
+ No behavior change; full suite green proves it.
120
+ 2. v2 write-mastering flip per §3.1 migration strategy (backup → normalize →
121
+ invert projection direction → equivalence tests). Canonical enums at every
122
+ write site; populate owner/workspace/visibility/created_by/evidence.
123
+ 3. Edge occurrence records (`observed_at`) instead of silent weight-max
124
+ collapse; node `superseded_by`.
125
+ 4. FTS5 index + query path replacing `LIKE` scans; `sqlite-vec` optional
126
+ extra with capability-honest fallback; embedder provisioning stays
127
+ consent-based via setup wizard.
128
+ 5. Wire `graph_curator` promotion rules into ingestion (or delete the module
129
+ — decide by measuring its tests against pipeline reality; no dead code
130
+ survives).
131
+ 6. Regenerate `docs/kg-schema.md` from the enums; delete claims about
132
+ nonexistent APIs.
133
+
134
+ Tests: equivalence suite extended; migration idempotence; FTS5 parity tests;
135
+ scoped-write population tests.
136
+
137
+ ### T4 — One door: ingestion 4/4 + durable conversations + garden absorption
138
+
139
+ Owns: `latticeai/services/ingestion.py`, `latticeai/services/upload_service.py`,
140
+ `latticeai/api/mcp.py`, `latticeai/server_app.py` (save_to_history call only),
141
+ new `latticeai/brain/conversations.py`, `p_reinforce.py`,
142
+ `latticeai/api/garden.py`.
143
+
144
+ 1. New source types (`chat_message`, `upload`, `mcp`, `workspace_event`,
145
+ `note`); rewire the three bypassing write paths through the pipeline;
146
+ provenance coverage metric exposed (`/api/brain/provenance/coverage`).
147
+ 2. `conversations`/`messages` SQLite store (unbounded, redaction on write);
148
+ chat reads/writes it; `chat_history.json` retired to render-cache or
149
+ removed; MemoryService conversation tier reads the store.
150
+ 3. Garden absorption: one-time idempotent import of `~/.ltcai-brain` through
151
+ the pipeline; `/garden` endpoints become brain queries; gardener's
152
+ classifier becomes an ingestion enricher; chat context stops doing the
153
+ O(n) vault rglob. Vault files left untouched on disk.
154
+
155
+ Tests: per-source provenance rows exist; conversation durability past 50;
156
+ garden import idempotence; chat context no longer reads the vault.
157
+
158
+ ### T5 — Memory & Context systems
159
+
160
+ Owns: new `latticeai/brain/memory.py`, new `latticeai/brain/context.py`,
161
+ `latticeai/api/chat.py` (context build), `latticeai/api/memory.py`,
162
+ `latticeai/core/agent.py` (memory_update only),
163
+ `latticeai/services/search_service.py` (consumption only).
164
+
165
+ 1. Memory model: Episodic/Semantic/Experience/Decision records as typed
166
+ nodes (canonical enums from T3) with provenance; consolidation entry
167
+ point (explicit, observable; no magic background jobs yet — the
168
+ consolidation *runner* is a workflow under T7's triggers).
169
+ 2. ContextAssembler: budgeted, ordered, provenance-carrying sections;
170
+ hybrid search replaces LIKE context; workspace memories injected;
171
+ per-section trace exposed to the UI ("why is this in context?").
172
+ 3. Agent learnings flow through the ingestion pipeline as Experience/
173
+ Decision records (no more vault markdown dumps with swallowed errors);
174
+ real runs only.
175
+
176
+ Tests: assembler budget/order/provenance; memories actually retrieved at
177
+ chat time; learnings land as typed nodes.
178
+
179
+ ### T6 — Personal/Org Brain: scoping, identity, transactional state
180
+
181
+ Owns: `latticeai/services/workspace_service.py`, `latticeai/api/search.py`,
182
+ graph read paths in `latticeai/brain/store.py` (scoping joins), new
183
+ `latticeai/core/policy.py`, `latticeai/api/admin.py`, `latticeai/api/auth.py`
184
+ (password policy/PKCE/session hashing), `latticeai/core/sessions.py`,
185
+ `latticeai/core/workspace_os.py` (storage backend), invitations API.
186
+
187
+ 1. Scoped reads everywhere (search/graph/traverse/vector/context); NULL
188
+ workspace = legacy-global compatibility, documented.
189
+ 2. User UUIDs via non-destructive migration; sessions/memberships keyed on
190
+ them; email mutable.
191
+ 3. `core/policy.py` single role→capability map, enforced via router
192
+ dependency; `/admin/roles` serves the now-true policy; invitation flow
193
+ (create/accept/expire).
194
+ 4. Workspace state → SQLite (same DB family), one-time JSON import, caps
195
+ removed, per-operation transactions.
196
+ 5. Auth hardening: session tokens hashed at rest, real password policy,
197
+ PKCE on SSO exchange, delete dead `_sso_states`/`detect_edition` branch.
198
+
199
+ Tests: cross-workspace read denial on every read API; migration assigns
200
+ stable UUIDs idempotently; policy enforcement; truncation gone.
201
+
202
+ ### T7 — Real Act runtimes
203
+
204
+ Owns: `latticeai/core/multi_agent.py`, `latticeai/services/platform_runtime.py`,
205
+ `latticeai/services/agent_runtime.py`, `latticeai/api/agents.py`,
206
+ `latticeai/api/workflow_designer.py`, `latticeai/core/agent_registry.py`,
207
+ new `latticeai/services/run_executor.py` (async engine),
208
+ `latticeai/core/workflow_engine.py` (trigger vocabulary),
209
+ new `latticeai/services/triggers.py`.
210
+
211
+ 1. LLM-backed role runners (planner/critic via `llm_router` prompts;
212
+ executor drives `core/agent.py`); `mode: "llm"` vs `mode: "simulation"`
213
+ persisted; simulation never writes Experience records.
214
+ 2. Workflow tool/skill nodes execute via `dispatch_tool`; `awaiting_approval`
215
+ pause state; plugin capability runners execute or honestly refuse.
216
+ 3. Async run engine: persisted run lifecycle, background workers, cooperative
217
+ cancellation, SSE progress over `core/realtime.py`.
218
+ 4. Per-tool approval gate generalizing `human_in_loop`; `approve()` stops
219
+ auto-approving.
220
+ 5. Triggers: interval/cron scheduler + brain-event subscriptions via hooks;
221
+ trigger-fired runs carry event provenance.
222
+ 6. Registry entries executable (model/prompt/tool allowlist consumed at run
223
+ time); custom agents run for real.
224
+
225
+ Tests: a real LLM run path (mocked router), approval pause/resume,
226
+ cancellation, trigger firing creates runs, simulation labeling end-to-end.
227
+
228
+ ### T8 — Sovereignty & Brain Network
229
+
230
+ Owns: new `latticeai/brain/identity.py`, `latticeai/services/kg_portability.py`,
231
+ new `latticeai/brain/network.py`, new `latticeai/api/network.py`,
232
+ `latticeai/api/portability.py`.
233
+
234
+ 1. Device Ed25519 keypair (file + keyring); fingerprint surfaced in UI/API.
235
+ 2. Signed bundles: detached signature + pubkey in export manifest; import
236
+ verifies; per-workspace export for members (not admin-only).
237
+ 3. Peer registry + pairing (manual pubkey exchange), push/pull signed
238
+ bundles over HTTP (LAN/tailnet), origin-device provenance on imported
239
+ nodes; idempotent content-hash dedup as v1 merge semantics.
240
+
241
+ Tests: sign/verify round-trip, tampered bundle rejected, unpaired peer
242
+ rejected, import provenance recorded.
243
+
244
+ ### T9 — Brain UX *(parallel-safe with T3-T8 after T1)*
245
+
246
+ Owns: `static/v3/**`, `static/*.html` + `static/scripts/**` (deletion),
247
+ `static/sw.js`, `latticeai/api/static_routes.py`, `latticeai/api/workspace.py`
248
+ (onboarding route), `knowledge_graph_api`-served `/graph` route relocation,
249
+ `scripts/lint_v3.mjs`, `scripts/build_v3_assets.mjs`, `STYLE_SYSTEM.md`.
250
+
251
+ 1. IA regroup (Brain · Ask · Capture · Act · Library · System); Knowledge
252
+ Graph = post-login landing view.
253
+ 2. Port the force-directed canvas (zoom/pan/drag/physics) from legacy
254
+ `graph.js` into the v3 explorer.
255
+ 3. Legacy pages deleted; routes 308-redirect into `/app` equivalents;
256
+ onboarding + admin land in `/app`; login rebuilt token-native.
257
+ 4. Vendor Inter + icons locally; remove CDN references; rebuild `sw.js`
258
+ around the v3 manifest.
259
+ 5. Build artifacts ungitted (generated at release); lint_v3 extended (no raw
260
+ hex outside token files; no inline style colors); i18n dictionary (en/ko).
261
+ 6. Update Playwright/visual tests to the v3 surface; retire legacy-page
262
+ suites.
263
+
264
+ Tests: Playwright smoke on /app views, redirect tests, zero CDN URLs in
265
+ shipped HTML/CSS/JS (lint rule), sw precache matches manifest.
266
+
267
+ ### T10 — Release, identity, docs
268
+
269
+ Owns: version files, `scripts/bump_version.py` (new), `README.md`,
270
+ `PROJECT_PRINCIPLES.md`, `ARCHITECTURE.md`, `FEATURE_STATUS.md`,
271
+ `MODEL_POLICY.md`, `KNOWLEDGE_GRAPH.md`, `docs/EDITION_STRATEGY.md`,
272
+ `CHANGELOG.md`, `RELEASE_NOTES_v4.0.0.md`, `package.json` (files list),
273
+ `.gitignore`, `lattice_ai_full_spec.pptx` (delete at HEAD),
274
+ `MANIFEST.in`, release-note consolidation.
275
+
276
+ 1. `bump_version.py` single-source bump; version → 4.0.0 everywhere;
277
+ consistency test still guards.
278
+ 2. Docs rewritten for the Digital Brain identity (constitution in
279
+ PROJECT_PRINCIPLES); FEATURE_STATUS.md regenerated for v4 with the same
280
+ honesty ledger discipline; MODEL_POLICY version fixed; release-notes
281
+ sprawl consolidated (archive old, one current).
282
+ 3. npm `files` slimmed; pptx deleted at HEAD; `.gitignore` covers
283
+ tarballs/logs/venvs; `RELEASE.md` runbook separated from history.
284
+ 4. Full validation: ruff, pytest, `scripts/validate_release_artifacts.py`,
285
+ wheel smoke test, vsix build, npm pack dry-run size check.
286
+ 5. Push branch; RC summary + 13-deliverable final report. STOP for human
287
+ review (no merge, no tag, no publish).
288
+
289
+ ## Design-review amendments (NORMATIVE — bind all tracks)
290
+
291
+ Adversarial review verdicts: 3× approve_with_changes
292
+ (`docs/v4-audit/v4_design_review.json`). Required changes, by track:
293
+
294
+ **T1**
295
+ - Run-record changes are versioned: add `record_schema_version` alongside
296
+ `mode`; simulated runs stop stamping `graph_node_id`. Rewrite the affected
297
+ legacy tests deliberately (`test_agent_platform_maturity`,
298
+ `test_v32_platform`, `test_multi_agent`, `test_workspace_os`) — do not
299
+ discover them broken mid-track.
300
+ - The hybrid-search.js fix must reach the shipped bundle: T1 is granted a
301
+ one-off `scripts/build_v3_assets.mjs` run to regenerate the hashed
302
+ artifact + manifest.
303
+
304
+ **T2**
305
+ - `create_app` acceptance is NOT "TestClient works": the test must assert
306
+ that importing `latticeai.server_app` (or the new factory module) performs
307
+ no side effects — no MLX/GPU init, no singleton construction, no file
308
+ creation under a sandboxed `LATTICEAI_HOME`. A delegating wrapper around
309
+ the old import-time module fails this gate by construction.
310
+ - Wizard-driven embedder provisioning (consent flow) is owned by T2's
311
+ `setup_wizard.py`/`api/setup.py` scope: expose a real provision endpoint
312
+ with explicit user consent, honest progress, and capability re-report.
313
+
314
+ **T3**
315
+ - **Edge identity**: post-flip canonical edges key on
316
+ `UNIQUE(source, target, type)`; migrated legacy rows keep their
317
+ `legacy_type` discriminator. SQLite migration = create-new → copy → swap
318
+ in one transaction, under the automatic pre-flip backup. Test: two
319
+ canonical-typed edges (e.g. MENTIONS + CONTAINS) between the same node
320
+ pair coexist.
321
+ - **Equivalence contract**: byte-equivalence is asserted for pre-flip data
322
+ only; new canonical writes get a separate projection-correctness suite
323
+ (English enum strings on the legacy surface are correct there, not a bug).
324
+ - **Downgrade guard**: set a DB format marker (`PRAGMA user_version` or
325
+ `kg_meta` key) at flip time; v4 refuses to open a newer-format DB than it
326
+ understands; document that v3.6 must not be pointed at a flipped DB and
327
+ provide the restore runbook for the automatic pre-flip backup. The
328
+ migrator is re-entrant, keyed on inspected data state, not a one-time
329
+ stamp. (Same downgrade-guard pattern applies to T4/T6 stores.)
330
+ - **Migrated-row scope**: legacy rows get `visibility=NULL` semantics
331
+ (legacy-global) — the `DEFAULT 'private'` column default must not be
332
+ allowed to privatize pre-v4 shared data to its last writer.
333
+ - **Store write API**: enum normalization is enforced *inside*
334
+ `brain/store.py` write methods (no caller can mint free strings);
335
+ owner/workspace/visibility are parameters defaulting to legacy-global
336
+ NULL — T4 (ingestion) and T6 (scope resolution) progressively supply real
337
+ values; a post-T6 acceptance check reports the % of new writes carrying
338
+ scope via the provenance coverage endpoint.
339
+ - **Decomposition definition of done**: the split follows the class's real
340
+ method clusters — `store.py` (storage + v2 projection), `discovery.py`
341
+ (local roots/audit/watch), `ingest.py` (ingest paths), `provenance.py`,
342
+ `documents.py`, `extraction.py`, plus portability seam; no resulting
343
+ module exceeds ~1,500 lines; a pure mixin-shuffle that recreates the god
344
+ object across files fails review. `local_knowledge_api.py` disposition is
345
+ owned here too (absorb into `brain/discovery.py` + API shim).
346
+ - **FTS5**: gate on `sqlite3` FTS5 availability with the same
347
+ capability-honest fallback as sqlite-vec (the LIKE path survives as
348
+ fallback); use the trigram tokenizer where available so Korean substring
349
+ recall does not regress — add a Korean-recall regression test
350
+ ('프로젝트' must match '프로젝트를').
351
+
352
+ **T4**
353
+ - **Chat history is imported, not dropped**: one-time idempotent import of
354
+ `chat_history.json` into the conversations store; messages lacking
355
+ user/conversation attribution land in a designated `legacy` conversation;
356
+ the `/history` API response contract is preserved (grant: `get_history`,
357
+ ChatService wiring in `server_app.py`, `/history` endpoints). Durability
358
+ test: pre-upgrade messages visible post-cutover.
359
+ - **Garden**: continuous ingestion via the watched-source machinery (see
360
+ architecture §4.2), not a one-time import; API-created notes dual-write
361
+ (brain authoritative, vault markdown mirror); imported vault notes are
362
+ legacy-global scoped. The `graph_curator` wire-or-delete decision moves
363
+ here (it gates concept promotion at ingest time).
364
+ - **Store co-location**: conversations live in the brain DB family covered
365
+ by `kg_portability` backup/restore — extend the backup manifest + restore
366
+ path to enumerate them, with a restore round-trip test.
367
+ - The "chat context stops reading the vault" change in `api/chat.py:368`
368
+ belongs to **T5** (context assembly), not T4.
369
+
370
+ **T5**
371
+ - Token budgeting uses a documented approximation (chars/4) — named as such
372
+ in code and API responses (`approx_tokens`), never presented as a real
373
+ tokenizer count.
374
+
375
+ **T6**
376
+ - **Identity migration scope**: one migration rewrites email→UUID keys
377
+ across `users.json`, workspace state, sessions, AND
378
+ `nodes_v2`/`edges_v2` owner/created_by values (T3/T4 write emails until
379
+ then — the migration maps them). Atomic tmp+rename writes; timestamped
380
+ pre-migration copies of `users.json` and `workspace_os.json`; explicit
381
+ grant over `server_app.py`'s user-store functions (move them into a
382
+ T6-owned module first). Downgrade is a one-way door — say so in the
383
+ migration marker + docs, same pattern as T3.
384
+ - Invitations API lives in new `latticeai/api/invitations.py`.
385
+ - New workspace-state tables join the same backed-up DB family as T4
386
+ (one backup covers the whole brain).
387
+
388
+ **T7**
389
+ - Ownership expands to **full** `workflow_engine.py` and `core/realtime.py`
390
+ (thread-safe publish via `loop.call_soon_threadsafe`).
391
+ - **Suspension model**: the engine returns/raises a `PausedRun` carrying the
392
+ node cursor + JSON-serializable context snapshot; runner exceptions are
393
+ partitioned (`ApprovalRequired` → pause; others → error-and-continue as
394
+ today); resume re-enters at the paused node and **never re-executes
395
+ completed nodes** (explicit test required).
396
+ - **Execution model**: asyncio tasks on the server loop; sync orchestrator/
397
+ tool work via `asyncio.to_thread`; SSE over `/realtime/stream`;
398
+ the honesty boundaries (MLX generate non-interruptible; single inference
399
+ thread serializes agent + chat) are documented and surfaced.
400
+ - **Startup reconciliation**: non-terminal runs → `interrupted` (reason +
401
+ timestamp) before workers start; restart test required.
402
+ - **Missed-trigger policy**: missed interval/cron firings while down are
403
+ skipped with a recorded skip event (no silent gaps, no thundering
404
+ catch-up).
405
+ - **LLM-output failure policy**: when a model responds but the plan/critique
406
+ cannot be parsed, the run FAILS with the raw output preserved in the run
407
+ record — it never silently falls back to fabricated deterministic
408
+ artifacts. Choosing simulation mode is explicit (no model loaded or
409
+ user-requested), never a parse-failure disguise.
410
+
411
+ **T8**
412
+ - Unsigned legacy bundles/backups import fine locally with
413
+ `origin='unsigned-legacy'` provenance; signatures mandatory only on the
414
+ peer path. Test: a v3.6.0-format export imports; a pre-v4 backup restores.
415
+ - Peer-request auth: Ed25519 signature over (body digest + timestamp) against
416
+ the paired key; freshness window + seen-nonce replay protection.
417
+ - Grant: the store's `export_graph_data`/`import_graph_data` functions for
418
+ scope-filtered export + provenance-stamped import.
419
+
420
+ **T9 / T9b (new)**
421
+ - **Capability-complete deletion rule**: a legacy page is deleted only when
422
+ its capabilities exist in `/app` and pass Playwright coverage — the
423
+ redirect map must be capability-complete, not URL-complete. Gap views that
424
+ must be BUILT first: workspace/org management (orgs, members, invitations,
425
+ activation), snapshots/time-machine (list/create/compare/restore),
426
+ activity feed, account profile. Chat parity explicitly includes doc-gen
427
+ sessions, image attach, and file-path injection rendering.
428
+ - **T9b (sequenced after T7/T8)** — surfaces for the new APIs, with
429
+ Playwright coverage: Act runs inbox (live progress, cancel, mode badge,
430
+ approval pause→decide→resume), trigger configuration, System network view
431
+ (device fingerprint, peer registry, pairing), Ask context-trace panel
432
+ ("why is this in context"), Brain provenance-coverage stat. Until T9b
433
+ lands, these capabilities are explicitly labeled API-only in
434
+ FEATURE_STATUS.md — a labeled state, not an omission.
435
+ - **i18n acceptance gate**: all strings in `routes.js`, the shared shell,
436
+ and every NEW v4 view are externalized; a checker script fails the build
437
+ on string literals in those files; remaining legacy-view strings are
438
+ inventoried in FEATURE_STATUS.md as labeled partial coverage.
439
+
440
+ **T10**
441
+ - Env-prefix canonicalization (`LATTICEAI_*` canonical, `LATTICE_*` read as
442
+ fallback aliases in `core/config.py`) and the CLI alias decision
443
+ (`ltcai` canonical, `LTCAI` deprecated) are owned here.
444
+ - Delete the superseded C-queue from the recovery file (replaced by this
445
+ plan) to remove contradictory guidance.
446
+ - Pre-flip migration backups: note in the restore runbook that backups live
447
+ on the same disk (exports dir) — recommend the user copy one off-disk at
448
+ upgrade time; the upgrade flow prints the backup path.
449
+
450
+ ## Execution model
451
+
452
+ Each track runs as its own workflow phase: implementer agent(s) with the
453
+ track's file-ownership list and this plan section as contract → reviewer
454
+ agent (correctness + "no fake functionality" + capability preservation) →
455
+ fix loop → full unit suite → commit. The recovery file is updated at each
456
+ track boundary.
457
+
458
+ ## Risk register
459
+
460
+ - **v2 flip (T3)** is the highest-risk change: mitigations = automatic backup,
461
+ idempotent migrator, equivalence suite, shim layer, and the flip lands as
462
+ its own commit (revertable in isolation).
463
+ - **Legacy frontend deletion (T9)**: redirects + Playwright cover the user
464
+ paths; deletion is one commit (revertable).
465
+ - **Async engine (T7)**: cooperative cancellation only (no thread kill);
466
+ synchronous fallback path retained behind the same API contract.
467
+ - **Garden absorption (T4)**: vault is read-only source; gardener writer
468
+ disabled only after import verifies; original files untouched.
469
+ - **Usage limits**: recovery file discipline; tracks commit independently so
470
+ an interrupted track loses at most its own uncommitted work.
@@ -1,17 +1,18 @@
1
1
  # Lattice AI Architecture
2
2
 
3
- > v3.3.1feature-complete for non-enterprise use cases with a rebuilt `/app`
4
- > visual shell. The agent ecosystem
5
- > (registry, marketplace + templates, workflow agents, autonomous planning),
6
- > the long-term memory platform + manager, and the skills/hooks/tool/MCP
7
- > registries are all operable from `/app`. Enterprise controls remain future
8
- > work.
9
-
10
- Lattice AI is a local-first **AI workspace, AI pipeline platform, Knowledge
11
- Graph platform, and multi-agent workflow platform**. The architecture is
12
- organized around one durable center: the Knowledge Graph. Models, tools,
13
- agents, workflows, and UI modes are replaceable layers that operate on top of
14
- workspace and graph context.
3
+ > v3.6.0**Knowledge Graph First.** Every data source converges into the graph
4
+ > through one unified ingestion pipeline (`latticeai/services/ingestion.py`), with
5
+ > formalized entities/relationships (`docs/kg-schema.md`), browser/web inputs,
6
+ > per-node provenance, and local export/import/backup
7
+ > (`latticeai/services/kg_portability.py`). The agent ecosystem, long-term memory,
8
+ > and skills/hooks/tool/MCP registries are all operable from `/app`. Enterprise
9
+ > controls remain future work.
10
+
11
+ Lattice AI is a local-first **Digital Brain Platform**. The architecture is
12
+ organized around one durable center and the user's asset: the **Knowledge
13
+ Graph**. Models, tools, agents, RAG, workflows, and UI modes are replaceable
14
+ layers that operate as views over graph context. Models are replaceable;
15
+ knowledge is durable.
15
16
 
16
17
  ## Architecture Goals
17
18