ltcai 3.5.0 → 4.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +73 -35
- package/docs/CARRYOVER_AUDIT_v3.6.0.md +61 -0
- package/docs/CHANGELOG.md +32 -0
- package/docs/HANDOVER_v3.6.0.md +46 -0
- package/docs/RUNTIME_HOOK_COVERAGE_v3.6.0.md +49 -0
- package/docs/V4_BRAIN_ARCHITECTURE.md +322 -0
- package/docs/V4_DIGITAL_BRAIN_RECOVERY.md +509 -0
- package/docs/V4_IMPLEMENTATION_PLAN.md +470 -0
- package/docs/architecture.md +13 -12
- package/docs/kg-schema.md +102 -53
- package/docs/privacy.md +18 -2
- package/docs/security-model.md +17 -0
- package/kg_schema.py +139 -10
- package/knowledge_graph.py +874 -26
- package/knowledge_graph_api.py +11 -127
- package/latticeai/__init__.py +1 -1
- package/latticeai/api/admin.py +1 -1
- package/latticeai/api/agents.py +7 -1
- package/latticeai/api/auth.py +27 -4
- package/latticeai/api/browser.py +217 -0
- package/latticeai/api/chat.py +112 -76
- package/latticeai/api/health.py +1 -1
- package/latticeai/api/hooks.py +1 -1
- package/latticeai/api/knowledge_graph.py +146 -0
- package/latticeai/api/local_files.py +1 -1
- package/latticeai/api/mcp.py +23 -11
- package/latticeai/api/memory.py +1 -1
- package/latticeai/api/models.py +1 -1
- package/latticeai/api/network.py +81 -0
- package/latticeai/api/portability.py +93 -0
- package/latticeai/api/realtime.py +1 -1
- package/latticeai/api/search.py +26 -2
- package/latticeai/api/security_dashboard.py +2 -3
- package/latticeai/api/setup.py +2 -2
- package/latticeai/api/static_routes.py +2 -4
- package/latticeai/api/tools.py +3 -0
- package/latticeai/api/workflow_designer.py +46 -0
- package/latticeai/api/workspace.py +71 -49
- package/latticeai/app_factory.py +1710 -0
- package/latticeai/brain/__init__.py +18 -0
- package/latticeai/brain/context.py +213 -0
- package/latticeai/brain/conversations.py +236 -0
- package/latticeai/brain/identity.py +175 -0
- package/latticeai/brain/memory.py +102 -0
- package/latticeai/brain/network.py +205 -0
- package/latticeai/core/agent.py +31 -7
- package/latticeai/core/audit.py +0 -7
- package/latticeai/core/config.py +1 -1
- package/latticeai/core/context_builder.py +1 -2
- package/latticeai/core/enterprise.py +1 -1
- package/latticeai/core/graph_curator.py +2 -2
- package/latticeai/core/marketplace.py +1 -1
- package/latticeai/core/mcp_registry.py +791 -0
- package/latticeai/core/model_compat.py +1 -1
- package/latticeai/core/model_resolution.py +0 -1
- package/latticeai/core/multi_agent.py +238 -4
- package/latticeai/core/security.py +1 -1
- package/latticeai/core/sessions.py +37 -7
- package/latticeai/core/workflow_engine.py +114 -2
- package/latticeai/core/workspace_os.py +58 -10
- package/latticeai/models/__init__.py +7 -0
- package/latticeai/models/router.py +779 -0
- package/latticeai/server_app.py +29 -1504
- package/latticeai/services/agent_runtime.py +1 -0
- package/latticeai/services/app_context.py +75 -14
- package/latticeai/services/ingestion.py +318 -0
- package/latticeai/services/kg_portability.py +207 -0
- package/latticeai/services/memory_service.py +39 -11
- package/latticeai/services/model_runtime.py +2 -5
- package/latticeai/services/platform_runtime.py +100 -23
- package/latticeai/services/search_service.py +17 -8
- package/latticeai/services/tool_dispatch.py +12 -2
- package/latticeai/services/triggers.py +241 -0
- package/latticeai/services/upload_service.py +37 -12
- package/latticeai/services/workspace_service.py +31 -0
- package/llm_router.py +29 -772
- package/ltcai_cli.py +1 -2
- package/mcp_registry.py +25 -788
- package/p_reinforce.py +124 -14
- package/package.json +11 -8
- package/scripts/build_vsix.mjs +72 -0
- package/scripts/bump_version.py +99 -0
- package/scripts/generate_diagrams.py +0 -1
- package/scripts/lint_v3.mjs +82 -18
- package/scripts/validate_release_artifacts.py +0 -1
- package/scripts/wheel_smoke.py +142 -0
- package/server.py +11 -7
- package/setup_wizard.py +1142 -0
- package/static/account.html +2 -4
- package/static/admin.html +3 -5
- package/static/chat.html +3 -6
- package/static/graph.html +2 -4
- package/static/sw.js +81 -52
- package/static/v3/asset-manifest.json +20 -19
- package/static/v3/css/{lattice.base.e4cdd05d.css → lattice.base.49deefb5.css} +1 -1
- package/static/v3/css/lattice.base.css +1 -1
- package/static/v3/css/{lattice.components.9b49d614.css → lattice.components.cde18231.css} +1 -1
- package/static/v3/css/lattice.components.css +1 -1
- package/static/v3/css/{lattice.shell.8fcc9d33.css → lattice.shell.29d36d85.css} +1 -1
- package/static/v3/css/lattice.shell.css +1 -1
- package/static/v3/css/{lattice.tokens.e7018963.css → lattice.tokens.304cbc40.css} +3 -0
- package/static/v3/css/lattice.tokens.css +3 -0
- package/static/v3/css/{lattice.views.22f69117.css → lattice.views.0a18b6c5.css} +2 -2
- package/static/v3/css/lattice.views.css +2 -2
- package/static/v3/index.html +3 -4
- package/static/v3/js/{app.d086489d.js → app.356e6452.js} +1 -1
- package/static/v3/js/core/{api.12b568ad.js → api.7a308b89.js} +39 -1
- package/static/v3/js/core/api.js +38 -0
- package/static/v3/js/core/{routes.d214b399.js → routes.7222343d.js} +22 -22
- package/static/v3/js/core/routes.js +22 -22
- package/static/v3/js/core/{shell.d05266f5.js → shell.a1657f20.js} +4 -4
- package/static/v3/js/core/shell.js +1 -1
- package/static/v3/js/core/{store.34ebd5e6.js → store.204a08b2.js} +1 -1
- package/static/v3/js/core/store.js +1 -1
- package/static/v3/js/views/graph-canvas.17c15d65.js +509 -0
- package/static/v3/js/views/graph-canvas.js +509 -0
- package/static/v3/js/views/{hybrid-search.b22b97e0.js → hybrid-search.2fb63ed9.js} +1 -2
- package/static/v3/js/views/hybrid-search.js +1 -2
- package/static/v3/js/views/knowledge-graph.5e40cbeb.js +509 -0
- package/static/v3/js/views/knowledge-graph.js +326 -54
- package/static/vendor/chart.umd.min.js +20 -0
- package/static/vendor/fonts/inter-latin-300-normal.woff2 +0 -0
- package/static/vendor/fonts/inter-latin-400-normal.woff2 +0 -0
- package/static/vendor/fonts/inter-latin-500-normal.woff2 +0 -0
- package/static/vendor/fonts/inter-latin-600-normal.woff2 +0 -0
- package/static/vendor/fonts/inter-latin-700-normal.woff2 +0 -0
- package/static/vendor/fonts/inter-latin-800-normal.woff2 +0 -0
- package/static/vendor/fonts/inter.css +44 -0
- package/static/vendor/icons/tabler-icons.min.css +4 -0
- package/static/vendor/icons/tabler-icons.woff2 +0 -0
- package/static/vendor/marked.min.js +69 -0
- package/static/workspace.html +2 -2
- package/telegram_bot.py +1 -2
- package/tools/commands.py +4 -2
- package/tools/computer.py +1 -1
- package/tools/documents.py +1 -3
- package/tools/filesystem.py +0 -4
- package/tools/knowledge.py +1 -3
- package/tools/network.py +1 -3
- package/codex_telegram_bot.py +0 -195
- package/docs/assets/v3.4.0/agent-run.png +0 -0
- package/docs/assets/v3.4.0/agents.png +0 -0
- package/docs/assets/v3.4.0/before/chat-before.png +0 -0
- package/docs/assets/v3.4.0/before/files-before.png +0 -0
- package/docs/assets/v3.4.0/chat.png +0 -0
- package/docs/assets/v3.4.0/connect-folder.png +0 -0
- package/docs/assets/v3.4.0/files.png +0 -0
- package/docs/assets/v3.4.0/home.png +0 -0
- package/docs/assets/v3.4.0/hooks-dispatch.png +0 -0
- package/docs/assets/v3.4.0/knowledge-graph.png +0 -0
- package/docs/assets/v3.4.0/local-agent.png +0 -0
- package/docs/assets/v3.4.0/memory.png +0 -0
- package/docs/assets/v3.4.0/settings.png +0 -0
- package/docs/assets/v3.4.0/vision-input.png +0 -0
- package/docs/assets/v3.4.0/workflows.png +0 -0
- package/docs/assets/v3.4.1/e2e_runtime_log.txt +0 -42
- package/docs/assets/v3.4.1/hooks-dispatch.png +0 -0
- package/docs/assets/v3.4.1/local-agent.png +0 -0
- package/docs/images/admin-dashboard.png +0 -0
- package/docs/images/architecture.png +0 -0
- package/docs/images/enterprise.png +0 -0
- package/docs/images/graph.png +0 -0
- package/docs/images/hero.gif +0 -0
- package/docs/images/knowledge-graph.png +0 -0
- package/docs/images/lattice-ai-demo.gif +0 -0
- package/docs/images/lattice-ai-hero.png +0 -0
- package/docs/images/logo.svg +0 -33
- package/docs/images/mobile-responsive.png +0 -0
- package/docs/images/model-recommendation.png +0 -0
- package/docs/images/onboarding.png +0 -0
- package/docs/images/organization.png +0 -0
- package/docs/images/pipeline.png +0 -0
- package/docs/images/screenshot-admin.png +0 -0
- package/docs/images/screenshot-chat.png +0 -0
- package/docs/images/screenshot-graph.png +0 -0
- package/docs/images/skills.png +0 -0
- package/docs/images/workspace-dark.png +0 -0
- package/docs/images/workspace-light.png +0 -0
- package/docs/images/workspace.png +0 -0
- package/requirements.txt +0 -16
- package/static/v3/js/views/knowledge-graph.a14ea7e7.js +0 -237
|
@@ -0,0 +1,470 @@
|
|
|
1
|
+
# Lattice AI v4.0.0 — Implementation Plan
|
|
2
|
+
|
|
3
|
+
Companion to `docs/V4_BRAIN_ARCHITECTURE.md`. Audit evidence in
|
|
4
|
+
`docs/v4-audit/*.json`. Program state in `docs/V4_DIGITAL_BRAIN_RECOVERY.md`.
|
|
5
|
+
|
|
6
|
+
## Ground rules (every track)
|
|
7
|
+
|
|
8
|
+
- Branch: `feat/v4-digital-brain` only. Conventional commits. Commit only
|
|
9
|
+
verified work; one commit (or a few) per track.
|
|
10
|
+
- Gate: `.venv/bin/python -m pytest tests/unit -q` green (baseline 455 →
|
|
11
|
+
grows; zero regressions), plus the new tests each track ships. The 9
|
|
12
|
+
pre-existing integration failures (need a live server) are not the gate but
|
|
13
|
+
must not grow.
|
|
14
|
+
- Python 3.11-compatible syntax (no nested same-quote f-strings). `.venv` is
|
|
15
|
+
3.14 — fine for running, but write for 3.11.
|
|
16
|
+
- **No placeholder code, no demo data, no fabricated numbers, no
|
|
17
|
+
silently-skipped work.** A capability that cannot complete must surface an
|
|
18
|
+
explicit state (`simulation`, `awaiting_approval`, `unavailable`), never a
|
|
19
|
+
fake success.
|
|
20
|
+
- Data safety: every schema migration is preceded by an automatic backup
|
|
21
|
+
(existing `kg_portability` machinery) and is idempotent + re-entrant.
|
|
22
|
+
User data directories (`~/.ltcai`, `~/.ltcai-brain`) are never deleted —
|
|
23
|
+
absorptions import, then deactivate the old writer, leaving source files
|
|
24
|
+
in place.
|
|
25
|
+
- Update `docs/V4_DIGITAL_BRAIN_RECOVERY.md` after every track (status,
|
|
26
|
+
files touched, validation result) and before any foreseeable session limit.
|
|
27
|
+
|
|
28
|
+
## Track sequence and ownership
|
|
29
|
+
|
|
30
|
+
Tracks are sequenced by dependency and run **strictly serially** — several
|
|
31
|
+
files have multiple owners across tracks (`api/chat.py`: T1/T2/T5;
|
|
32
|
+
`server_app.py`: T2/T4/T6; `api/workspace.py`: T1/T9; `brain/store.py`:
|
|
33
|
+
T3/T6/T8), so sequence, not disjointness, is the conflict protection.
|
|
34
|
+
"Owns" below means: *during that track*, only that track touches the file,
|
|
35
|
+
and only within the named scope. T1/T2 unblock everything; T3→T6 are the
|
|
36
|
+
brain spine; T7/T8 build on the spine; T9 items 2-5 (canvas, vendoring,
|
|
37
|
+
sw.js, build/lint) may run in parallel with backend tracks, but T9 items
|
|
38
|
+
1/3/6 and all of T9b depend on the T3-T8 API contracts. T10 closes.
|
|
39
|
+
|
|
40
|
+
> **NORMATIVE**: every track below is amended by §"Design-review amendments"
|
|
41
|
+
> at the end of this document. Implementers must read both the track section
|
|
42
|
+
> and its amendments — the amendments win on conflict.
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
### T1 — Truth & safety floor *(small; first because every later track builds on honest primitives)*
|
|
47
|
+
|
|
48
|
+
Owns: `latticeai/api/workspace.py`, `latticeai/core/workspace_os.py`
|
|
49
|
+
(summary/leak only), `latticeai/api/chat.py` (context filter only),
|
|
50
|
+
`latticeai/services/memory_service.py`, `latticeai/core/multi_agent.py` +
|
|
51
|
+
`latticeai/services/agent_runtime.py` (mode labeling only),
|
|
52
|
+
`static/v3/js/views/hybrid-search.js`, `README.md` (claims only).
|
|
53
|
+
|
|
54
|
+
1. Close by-id authz gaps: snapshot get/area/export/compare + memory delete
|
|
55
|
+
gate through `resolve_read_scope`/`resolve_write_scope` using the record's
|
|
56
|
+
own workspace_id; ownership check on memory delete.
|
|
57
|
+
2. Strip the unfiltered `workspaces` key from `/workspace/os` (keep the
|
|
58
|
+
membership-filtered `workspace_registry`).
|
|
59
|
+
3. Fix `build_recent_chat_context` leak: scope to the requesting user's
|
|
60
|
+
conversation (assistant turns only within it).
|
|
61
|
+
4. Fix `MemoryService.recall` dead graph branch (`results` → `matches`);
|
|
62
|
+
replace hardcoded 0.6/0.5 scores with real normalized lexical scores;
|
|
63
|
+
stop sorting on constants.
|
|
64
|
+
5. Persist `mode: "simulation"` on every deterministic multi-agent/workflow
|
|
65
|
+
run record; simulated runs stop writing run-derived nodes into the KG.
|
|
66
|
+
6. Remove the hardcoded fusion-meter values in `hybrid-search.js` (render
|
|
67
|
+
real scores from the API or remove the meters).
|
|
68
|
+
7. README honesty pass: agent/workflow claims rewritten to match reality
|
|
69
|
+
(full rewrite happens in T10; this removes the falsehoods now).
|
|
70
|
+
|
|
71
|
+
Tests: authz regression tests (cross-workspace access denied), recall returns
|
|
72
|
+
graph results, context isolation test, run records carry mode.
|
|
73
|
+
|
|
74
|
+
### T2 — Packaging & app factory *(unblocks clean work everywhere)*
|
|
75
|
+
|
|
76
|
+
Owns: root `setup.py`→`setup_wizard.py`, `pyproject.toml`, `requirements.txt`,
|
|
77
|
+
`latticeai/server_app.py` (assembly only), `latticeai/services/app_context.py`,
|
|
78
|
+
`latticeai/api/deps.py`, `latticeai/api/setup.py`, `latticeai/api/chat.py`
|
|
79
|
+
(telegram import only), `codex_telegram_bot.py`, `perm_monitor.py`,
|
|
80
|
+
`knowledge_graph_api.py`, `.github/workflows/*`, ruff config.
|
|
81
|
+
|
|
82
|
+
1. Rename root `setup.py` → `setup_wizard.py`; update importers
|
|
83
|
+
(`server_app.py:149`, `api/setup.py`); add to `py-modules`; verify wheel
|
|
84
|
+
contains it; add installed-wheel smoke test (build → clean venv → install
|
|
85
|
+
→ `import latticeai.server_app` → `/health`) to CI and scripts.
|
|
86
|
+
2. `create_app(config) -> FastAPI` factory: move import-time singleton
|
|
87
|
+
construction and MLX init out of import scope; build `AppContext`; keep
|
|
88
|
+
module-level names as thin accessors during deprecation (tests import
|
|
89
|
+
them). Router factories accept the context (migrate the worst two:
|
|
90
|
+
chat ~25 kwargs, workspace ~30 kwargs; others opportunistically).
|
|
91
|
+
3. Decouple telegram: `broadcast_web_chat` becomes a `RealtimeBus` subscriber
|
|
92
|
+
registered only when `ENABLE_TELEGRAM`; `api/chat.py` drops the
|
|
93
|
+
unconditional import.
|
|
94
|
+
4. Delete dead modules: `codex_telegram_bot.py`, `perm_monitor.py`; remove
|
|
95
|
+
from packaging lists. **`knowledge_graph_api.py` is NOT dead** — it serves
|
|
96
|
+
the `/knowledge-graph/*` data endpoints the v3 SPA consumes: migrate its
|
|
97
|
+
data router into `latticeai/api/knowledge_graph.py` with endpoint-parity
|
|
98
|
+
tests (stats/graph/documents/search/context/neighbors/ingest unchanged);
|
|
99
|
+
its legacy `/graph` page routes move to T9's redirect map. Also owned
|
|
100
|
+
here: relocate `llm_router.py` → `latticeai/models/router.py` and
|
|
101
|
+
`mcp_registry.py` → `latticeai/core/mcp_registry.py` with root shims, and
|
|
102
|
+
update their importers (`api/models.py`, `api/setup.py`,
|
|
103
|
+
`services/model_runtime.py`, `server_app.py`).
|
|
104
|
+
5. ruff baseline (`[tool.ruff]`, pragmatic select set), fix violations or
|
|
105
|
+
per-file-ignore legacy monoliths; CI gate. Bounded dependency constraints
|
|
106
|
+
in pyproject; `requirements.txt` deleted or generated.
|
|
107
|
+
|
|
108
|
+
Tests: existing suite green; new wheel smoke test; factory produces a working
|
|
109
|
+
app (TestClient).
|
|
110
|
+
|
|
111
|
+
### T3 — Brain store: decomposition + v2 write-mastering + retrieval
|
|
112
|
+
|
|
113
|
+
Owns: `knowledge_graph.py` (becomes shim), new `latticeai/brain/` package
|
|
114
|
+
(`store.py`, `extraction.py`, `documents.py`, `discovery.py`), `kg_schema.py`
|
|
115
|
+
(moves to `latticeai/brain/schema.py` with root shim), `docs/kg-schema.md`.
|
|
116
|
+
|
|
117
|
+
1. Mechanical decomposition of `KnowledgeGraphStore` along the seams named in
|
|
118
|
+
the backend audit; root `knowledge_graph.py` becomes a re-export shim.
|
|
119
|
+
No behavior change; full suite green proves it.
|
|
120
|
+
2. v2 write-mastering flip per §3.1 migration strategy (backup → normalize →
|
|
121
|
+
invert projection direction → equivalence tests). Canonical enums at every
|
|
122
|
+
write site; populate owner/workspace/visibility/created_by/evidence.
|
|
123
|
+
3. Edge occurrence records (`observed_at`) instead of silent weight-max
|
|
124
|
+
collapse; node `superseded_by`.
|
|
125
|
+
4. FTS5 index + query path replacing `LIKE` scans; `sqlite-vec` optional
|
|
126
|
+
extra with capability-honest fallback; embedder provisioning stays
|
|
127
|
+
consent-based via setup wizard.
|
|
128
|
+
5. Wire `graph_curator` promotion rules into ingestion (or delete the module
|
|
129
|
+
— decide by measuring its tests against pipeline reality; no dead code
|
|
130
|
+
survives).
|
|
131
|
+
6. Regenerate `docs/kg-schema.md` from the enums; delete claims about
|
|
132
|
+
nonexistent APIs.
|
|
133
|
+
|
|
134
|
+
Tests: equivalence suite extended; migration idempotence; FTS5 parity tests;
|
|
135
|
+
scoped-write population tests.
|
|
136
|
+
|
|
137
|
+
### T4 — One door: ingestion 4/4 + durable conversations + garden absorption
|
|
138
|
+
|
|
139
|
+
Owns: `latticeai/services/ingestion.py`, `latticeai/services/upload_service.py`,
|
|
140
|
+
`latticeai/api/mcp.py`, `latticeai/server_app.py` (save_to_history call only),
|
|
141
|
+
new `latticeai/brain/conversations.py`, `p_reinforce.py`,
|
|
142
|
+
`latticeai/api/garden.py`.
|
|
143
|
+
|
|
144
|
+
1. New source types (`chat_message`, `upload`, `mcp`, `workspace_event`,
|
|
145
|
+
`note`); rewire the three bypassing write paths through the pipeline;
|
|
146
|
+
provenance coverage metric exposed (`/api/brain/provenance/coverage`).
|
|
147
|
+
2. `conversations`/`messages` SQLite store (unbounded, redaction on write);
|
|
148
|
+
chat reads/writes it; `chat_history.json` retired to render-cache or
|
|
149
|
+
removed; MemoryService conversation tier reads the store.
|
|
150
|
+
3. Garden absorption: one-time idempotent import of `~/.ltcai-brain` through
|
|
151
|
+
the pipeline; `/garden` endpoints become brain queries; gardener's
|
|
152
|
+
classifier becomes an ingestion enricher; chat context stops doing the
|
|
153
|
+
O(n) vault rglob. Vault files left untouched on disk.
|
|
154
|
+
|
|
155
|
+
Tests: per-source provenance rows exist; conversation durability past 50;
|
|
156
|
+
garden import idempotence; chat context no longer reads the vault.
|
|
157
|
+
|
|
158
|
+
### T5 — Memory & Context systems
|
|
159
|
+
|
|
160
|
+
Owns: new `latticeai/brain/memory.py`, new `latticeai/brain/context.py`,
|
|
161
|
+
`latticeai/api/chat.py` (context build), `latticeai/api/memory.py`,
|
|
162
|
+
`latticeai/core/agent.py` (memory_update only),
|
|
163
|
+
`latticeai/services/search_service.py` (consumption only).
|
|
164
|
+
|
|
165
|
+
1. Memory model: Episodic/Semantic/Experience/Decision records as typed
|
|
166
|
+
nodes (canonical enums from T3) with provenance; consolidation entry
|
|
167
|
+
point (explicit, observable; no magic background jobs yet — the
|
|
168
|
+
consolidation *runner* is a workflow under T7's triggers).
|
|
169
|
+
2. ContextAssembler: budgeted, ordered, provenance-carrying sections;
|
|
170
|
+
hybrid search replaces LIKE context; workspace memories injected;
|
|
171
|
+
per-section trace exposed to the UI ("why is this in context?").
|
|
172
|
+
3. Agent learnings flow through the ingestion pipeline as Experience/
|
|
173
|
+
Decision records (no more vault markdown dumps with swallowed errors);
|
|
174
|
+
real runs only.
|
|
175
|
+
|
|
176
|
+
Tests: assembler budget/order/provenance; memories actually retrieved at
|
|
177
|
+
chat time; learnings land as typed nodes.
|
|
178
|
+
|
|
179
|
+
### T6 — Personal/Org Brain: scoping, identity, transactional state
|
|
180
|
+
|
|
181
|
+
Owns: `latticeai/services/workspace_service.py`, `latticeai/api/search.py`,
|
|
182
|
+
graph read paths in `latticeai/brain/store.py` (scoping joins), new
|
|
183
|
+
`latticeai/core/policy.py`, `latticeai/api/admin.py`, `latticeai/api/auth.py`
|
|
184
|
+
(password policy/PKCE/session hashing), `latticeai/core/sessions.py`,
|
|
185
|
+
`latticeai/core/workspace_os.py` (storage backend), invitations API.
|
|
186
|
+
|
|
187
|
+
1. Scoped reads everywhere (search/graph/traverse/vector/context); NULL
|
|
188
|
+
workspace = legacy-global compatibility, documented.
|
|
189
|
+
2. User UUIDs via non-destructive migration; sessions/memberships keyed on
|
|
190
|
+
them; email mutable.
|
|
191
|
+
3. `core/policy.py` single role→capability map, enforced via router
|
|
192
|
+
dependency; `/admin/roles` serves the now-true policy; invitation flow
|
|
193
|
+
(create/accept/expire).
|
|
194
|
+
4. Workspace state → SQLite (same DB family), one-time JSON import, caps
|
|
195
|
+
removed, per-operation transactions.
|
|
196
|
+
5. Auth hardening: session tokens hashed at rest, real password policy,
|
|
197
|
+
PKCE on SSO exchange, delete dead `_sso_states`/`detect_edition` branch.
|
|
198
|
+
|
|
199
|
+
Tests: cross-workspace read denial on every read API; migration assigns
|
|
200
|
+
stable UUIDs idempotently; policy enforcement; truncation gone.
|
|
201
|
+
|
|
202
|
+
### T7 — Real Act runtimes
|
|
203
|
+
|
|
204
|
+
Owns: `latticeai/core/multi_agent.py`, `latticeai/services/platform_runtime.py`,
|
|
205
|
+
`latticeai/services/agent_runtime.py`, `latticeai/api/agents.py`,
|
|
206
|
+
`latticeai/api/workflow_designer.py`, `latticeai/core/agent_registry.py`,
|
|
207
|
+
new `latticeai/services/run_executor.py` (async engine),
|
|
208
|
+
`latticeai/core/workflow_engine.py` (trigger vocabulary),
|
|
209
|
+
new `latticeai/services/triggers.py`.
|
|
210
|
+
|
|
211
|
+
1. LLM-backed role runners (planner/critic via `llm_router` prompts;
|
|
212
|
+
executor drives `core/agent.py`); `mode: "llm"` vs `mode: "simulation"`
|
|
213
|
+
persisted; simulation never writes Experience records.
|
|
214
|
+
2. Workflow tool/skill nodes execute via `dispatch_tool`; `awaiting_approval`
|
|
215
|
+
pause state; plugin capability runners execute or honestly refuse.
|
|
216
|
+
3. Async run engine: persisted run lifecycle, background workers, cooperative
|
|
217
|
+
cancellation, SSE progress over `core/realtime.py`.
|
|
218
|
+
4. Per-tool approval gate generalizing `human_in_loop`; `approve()` stops
|
|
219
|
+
auto-approving.
|
|
220
|
+
5. Triggers: interval/cron scheduler + brain-event subscriptions via hooks;
|
|
221
|
+
trigger-fired runs carry event provenance.
|
|
222
|
+
6. Registry entries executable (model/prompt/tool allowlist consumed at run
|
|
223
|
+
time); custom agents run for real.
|
|
224
|
+
|
|
225
|
+
Tests: a real LLM run path (mocked router), approval pause/resume,
|
|
226
|
+
cancellation, trigger firing creates runs, simulation labeling end-to-end.
|
|
227
|
+
|
|
228
|
+
### T8 — Sovereignty & Brain Network
|
|
229
|
+
|
|
230
|
+
Owns: new `latticeai/brain/identity.py`, `latticeai/services/kg_portability.py`,
|
|
231
|
+
new `latticeai/brain/network.py`, new `latticeai/api/network.py`,
|
|
232
|
+
`latticeai/api/portability.py`.
|
|
233
|
+
|
|
234
|
+
1. Device Ed25519 keypair (file + keyring); fingerprint surfaced in UI/API.
|
|
235
|
+
2. Signed bundles: detached signature + pubkey in export manifest; import
|
|
236
|
+
verifies; per-workspace export for members (not admin-only).
|
|
237
|
+
3. Peer registry + pairing (manual pubkey exchange), push/pull signed
|
|
238
|
+
bundles over HTTP (LAN/tailnet), origin-device provenance on imported
|
|
239
|
+
nodes; idempotent content-hash dedup as v1 merge semantics.
|
|
240
|
+
|
|
241
|
+
Tests: sign/verify round-trip, tampered bundle rejected, unpaired peer
|
|
242
|
+
rejected, import provenance recorded.
|
|
243
|
+
|
|
244
|
+
### T9 — Brain UX *(parallel-safe with T3-T8 after T1)*
|
|
245
|
+
|
|
246
|
+
Owns: `static/v3/**`, `static/*.html` + `static/scripts/**` (deletion),
|
|
247
|
+
`static/sw.js`, `latticeai/api/static_routes.py`, `latticeai/api/workspace.py`
|
|
248
|
+
(onboarding route), `knowledge_graph_api`-served `/graph` route relocation,
|
|
249
|
+
`scripts/lint_v3.mjs`, `scripts/build_v3_assets.mjs`, `STYLE_SYSTEM.md`.
|
|
250
|
+
|
|
251
|
+
1. IA regroup (Brain · Ask · Capture · Act · Library · System); Knowledge
|
|
252
|
+
Graph = post-login landing view.
|
|
253
|
+
2. Port the force-directed canvas (zoom/pan/drag/physics) from legacy
|
|
254
|
+
`graph.js` into the v3 explorer.
|
|
255
|
+
3. Legacy pages deleted; routes 308-redirect into `/app` equivalents;
|
|
256
|
+
onboarding + admin land in `/app`; login rebuilt token-native.
|
|
257
|
+
4. Vendor Inter + icons locally; remove CDN references; rebuild `sw.js`
|
|
258
|
+
around the v3 manifest.
|
|
259
|
+
5. Build artifacts ungitted (generated at release); lint_v3 extended (no raw
|
|
260
|
+
hex outside token files; no inline style colors); i18n dictionary (en/ko).
|
|
261
|
+
6. Update Playwright/visual tests to the v3 surface; retire legacy-page
|
|
262
|
+
suites.
|
|
263
|
+
|
|
264
|
+
Tests: Playwright smoke on /app views, redirect tests, zero CDN URLs in
|
|
265
|
+
shipped HTML/CSS/JS (lint rule), sw precache matches manifest.
|
|
266
|
+
|
|
267
|
+
### T10 — Release, identity, docs
|
|
268
|
+
|
|
269
|
+
Owns: version files, `scripts/bump_version.py` (new), `README.md`,
|
|
270
|
+
`PROJECT_PRINCIPLES.md`, `ARCHITECTURE.md`, `FEATURE_STATUS.md`,
|
|
271
|
+
`MODEL_POLICY.md`, `KNOWLEDGE_GRAPH.md`, `docs/EDITION_STRATEGY.md`,
|
|
272
|
+
`CHANGELOG.md`, `RELEASE_NOTES_v4.0.0.md`, `package.json` (files list),
|
|
273
|
+
`.gitignore`, `lattice_ai_full_spec.pptx` (delete at HEAD),
|
|
274
|
+
`MANIFEST.in`, release-note consolidation.
|
|
275
|
+
|
|
276
|
+
1. `bump_version.py` single-source bump; version → 4.0.0 everywhere;
|
|
277
|
+
consistency test still guards.
|
|
278
|
+
2. Docs rewritten for the Digital Brain identity (constitution in
|
|
279
|
+
PROJECT_PRINCIPLES); FEATURE_STATUS.md regenerated for v4 with the same
|
|
280
|
+
honesty ledger discipline; MODEL_POLICY version fixed; release-notes
|
|
281
|
+
sprawl consolidated (archive old, one current).
|
|
282
|
+
3. npm `files` slimmed; pptx deleted at HEAD; `.gitignore` covers
|
|
283
|
+
tarballs/logs/venvs; `RELEASE.md` runbook separated from history.
|
|
284
|
+
4. Full validation: ruff, pytest, `scripts/validate_release_artifacts.py`,
|
|
285
|
+
wheel smoke test, vsix build, npm pack dry-run size check.
|
|
286
|
+
5. Push branch; RC summary + 13-deliverable final report. STOP for human
|
|
287
|
+
review (no merge, no tag, no publish).
|
|
288
|
+
|
|
289
|
+
## Design-review amendments (NORMATIVE — bind all tracks)
|
|
290
|
+
|
|
291
|
+
Adversarial review verdicts: 3× approve_with_changes
|
|
292
|
+
(`docs/v4-audit/v4_design_review.json`). Required changes, by track:
|
|
293
|
+
|
|
294
|
+
**T1**
|
|
295
|
+
- Run-record changes are versioned: add `record_schema_version` alongside
|
|
296
|
+
`mode`; simulated runs stop stamping `graph_node_id`. Rewrite the affected
|
|
297
|
+
legacy tests deliberately (`test_agent_platform_maturity`,
|
|
298
|
+
`test_v32_platform`, `test_multi_agent`, `test_workspace_os`) — do not
|
|
299
|
+
discover them broken mid-track.
|
|
300
|
+
- The hybrid-search.js fix must reach the shipped bundle: T1 is granted a
|
|
301
|
+
one-off `scripts/build_v3_assets.mjs` run to regenerate the hashed
|
|
302
|
+
artifact + manifest.
|
|
303
|
+
|
|
304
|
+
**T2**
|
|
305
|
+
- `create_app` acceptance is NOT "TestClient works": the test must assert
|
|
306
|
+
that importing `latticeai.server_app` (or the new factory module) performs
|
|
307
|
+
no side effects — no MLX/GPU init, no singleton construction, no file
|
|
308
|
+
creation under a sandboxed `LATTICEAI_HOME`. A delegating wrapper around
|
|
309
|
+
the old import-time module fails this gate by construction.
|
|
310
|
+
- Wizard-driven embedder provisioning (consent flow) is owned by T2's
|
|
311
|
+
`setup_wizard.py`/`api/setup.py` scope: expose a real provision endpoint
|
|
312
|
+
with explicit user consent, honest progress, and capability re-report.
|
|
313
|
+
|
|
314
|
+
**T3**
|
|
315
|
+
- **Edge identity**: post-flip canonical edges key on
|
|
316
|
+
`UNIQUE(source, target, type)`; migrated legacy rows keep their
|
|
317
|
+
`legacy_type` discriminator. SQLite migration = create-new → copy → swap
|
|
318
|
+
in one transaction, under the automatic pre-flip backup. Test: two
|
|
319
|
+
canonical-typed edges (e.g. MENTIONS + CONTAINS) between the same node
|
|
320
|
+
pair coexist.
|
|
321
|
+
- **Equivalence contract**: byte-equivalence is asserted for pre-flip data
|
|
322
|
+
only; new canonical writes get a separate projection-correctness suite
|
|
323
|
+
(English enum strings on the legacy surface are correct there, not a bug).
|
|
324
|
+
- **Downgrade guard**: set a DB format marker (`PRAGMA user_version` or
|
|
325
|
+
`kg_meta` key) at flip time; v4 refuses to open a newer-format DB than it
|
|
326
|
+
understands; document that v3.6 must not be pointed at a flipped DB and
|
|
327
|
+
provide the restore runbook for the automatic pre-flip backup. The
|
|
328
|
+
migrator is re-entrant, keyed on inspected data state, not a one-time
|
|
329
|
+
stamp. (Same downgrade-guard pattern applies to T4/T6 stores.)
|
|
330
|
+
- **Migrated-row scope**: legacy rows get `visibility=NULL` semantics
|
|
331
|
+
(legacy-global) — the `DEFAULT 'private'` column default must not be
|
|
332
|
+
allowed to privatize pre-v4 shared data to its last writer.
|
|
333
|
+
- **Store write API**: enum normalization is enforced *inside*
|
|
334
|
+
`brain/store.py` write methods (no caller can mint free strings);
|
|
335
|
+
owner/workspace/visibility are parameters defaulting to legacy-global
|
|
336
|
+
NULL — T4 (ingestion) and T6 (scope resolution) progressively supply real
|
|
337
|
+
values; a post-T6 acceptance check reports the % of new writes carrying
|
|
338
|
+
scope via the provenance coverage endpoint.
|
|
339
|
+
- **Decomposition definition of done**: the split follows the class's real
|
|
340
|
+
method clusters — `store.py` (storage + v2 projection), `discovery.py`
|
|
341
|
+
(local roots/audit/watch), `ingest.py` (ingest paths), `provenance.py`,
|
|
342
|
+
`documents.py`, `extraction.py`, plus portability seam; no resulting
|
|
343
|
+
module exceeds ~1,500 lines; a pure mixin-shuffle that recreates the god
|
|
344
|
+
object across files fails review. `local_knowledge_api.py` disposition is
|
|
345
|
+
owned here too (absorb into `brain/discovery.py` + API shim).
|
|
346
|
+
- **FTS5**: gate on `sqlite3` FTS5 availability with the same
|
|
347
|
+
capability-honest fallback as sqlite-vec (the LIKE path survives as
|
|
348
|
+
fallback); use the trigram tokenizer where available so Korean substring
|
|
349
|
+
recall does not regress — add a Korean-recall regression test
|
|
350
|
+
('프로젝트' must match '프로젝트를').
|
|
351
|
+
|
|
352
|
+
**T4**
|
|
353
|
+
- **Chat history is imported, not dropped**: one-time idempotent import of
|
|
354
|
+
`chat_history.json` into the conversations store; messages lacking
|
|
355
|
+
user/conversation attribution land in a designated `legacy` conversation;
|
|
356
|
+
the `/history` API response contract is preserved (grant: `get_history`,
|
|
357
|
+
ChatService wiring in `server_app.py`, `/history` endpoints). Durability
|
|
358
|
+
test: pre-upgrade messages visible post-cutover.
|
|
359
|
+
- **Garden**: continuous ingestion via the watched-source machinery (see
|
|
360
|
+
architecture §4.2), not a one-time import; API-created notes dual-write
|
|
361
|
+
(brain authoritative, vault markdown mirror); imported vault notes are
|
|
362
|
+
legacy-global scoped. The `graph_curator` wire-or-delete decision moves
|
|
363
|
+
here (it gates concept promotion at ingest time).
|
|
364
|
+
- **Store co-location**: conversations live in the brain DB family covered
|
|
365
|
+
by `kg_portability` backup/restore — extend the backup manifest + restore
|
|
366
|
+
path to enumerate them, with a restore round-trip test.
|
|
367
|
+
- The "chat context stops reading the vault" change in `api/chat.py:368`
|
|
368
|
+
belongs to **T5** (context assembly), not T4.
|
|
369
|
+
|
|
370
|
+
**T5**
|
|
371
|
+
- Token budgeting uses a documented approximation (chars/4) — named as such
|
|
372
|
+
in code and API responses (`approx_tokens`), never presented as a real
|
|
373
|
+
tokenizer count.
|
|
374
|
+
|
|
375
|
+
**T6**
|
|
376
|
+
- **Identity migration scope**: one migration rewrites email→UUID keys
|
|
377
|
+
across `users.json`, workspace state, sessions, AND
|
|
378
|
+
`nodes_v2`/`edges_v2` owner/created_by values (T3/T4 write emails until
|
|
379
|
+
then — the migration maps them). Atomic tmp+rename writes; timestamped
|
|
380
|
+
pre-migration copies of `users.json` and `workspace_os.json`; explicit
|
|
381
|
+
grant over `server_app.py`'s user-store functions (move them into a
|
|
382
|
+
T6-owned module first). Downgrade is a one-way door — say so in the
|
|
383
|
+
migration marker + docs, same pattern as T3.
|
|
384
|
+
- Invitations API lives in new `latticeai/api/invitations.py`.
|
|
385
|
+
- New workspace-state tables join the same backed-up DB family as T4
|
|
386
|
+
(one backup covers the whole brain).
|
|
387
|
+
|
|
388
|
+
**T7**
|
|
389
|
+
- Ownership expands to **full** `workflow_engine.py` and `core/realtime.py`
|
|
390
|
+
(thread-safe publish via `loop.call_soon_threadsafe`).
|
|
391
|
+
- **Suspension model**: the engine returns/raises a `PausedRun` carrying the
|
|
392
|
+
node cursor + JSON-serializable context snapshot; runner exceptions are
|
|
393
|
+
partitioned (`ApprovalRequired` → pause; others → error-and-continue as
|
|
394
|
+
today); resume re-enters at the paused node and **never re-executes
|
|
395
|
+
completed nodes** (explicit test required).
|
|
396
|
+
- **Execution model**: asyncio tasks on the server loop; sync orchestrator/
|
|
397
|
+
tool work via `asyncio.to_thread`; SSE over `/realtime/stream`;
|
|
398
|
+
the honesty boundaries (MLX generate non-interruptible; single inference
|
|
399
|
+
thread serializes agent + chat) are documented and surfaced.
|
|
400
|
+
- **Startup reconciliation**: non-terminal runs → `interrupted` (reason +
|
|
401
|
+
timestamp) before workers start; restart test required.
|
|
402
|
+
- **Missed-trigger policy**: missed interval/cron firings while down are
|
|
403
|
+
skipped with a recorded skip event (no silent gaps, no thundering
|
|
404
|
+
catch-up).
|
|
405
|
+
- **LLM-output failure policy**: when a model responds but the plan/critique
|
|
406
|
+
cannot be parsed, the run FAILS with the raw output preserved in the run
|
|
407
|
+
record — it never silently falls back to fabricated deterministic
|
|
408
|
+
artifacts. Choosing simulation mode is explicit (no model loaded or
|
|
409
|
+
user-requested), never a parse-failure disguise.
|
|
410
|
+
|
|
411
|
+
**T8**
|
|
412
|
+
- Unsigned legacy bundles/backups import fine locally with
|
|
413
|
+
`origin='unsigned-legacy'` provenance; signatures mandatory only on the
|
|
414
|
+
peer path. Test: a v3.6.0-format export imports; a pre-v4 backup restores.
|
|
415
|
+
- Peer-request auth: Ed25519 signature over (body digest + timestamp) against
|
|
416
|
+
the paired key; freshness window + seen-nonce replay protection.
|
|
417
|
+
- Grant: the store's `export_graph_data`/`import_graph_data` functions for
|
|
418
|
+
scope-filtered export + provenance-stamped import.
|
|
419
|
+
|
|
420
|
+
**T9 / T9b (new)**
|
|
421
|
+
- **Capability-complete deletion rule**: a legacy page is deleted only when
|
|
422
|
+
its capabilities exist in `/app` and pass Playwright coverage — the
|
|
423
|
+
redirect map must be capability-complete, not URL-complete. Gap views that
|
|
424
|
+
must be BUILT first: workspace/org management (orgs, members, invitations,
|
|
425
|
+
activation), snapshots/time-machine (list/create/compare/restore),
|
|
426
|
+
activity feed, account profile. Chat parity explicitly includes doc-gen
|
|
427
|
+
sessions, image attach, and file-path injection rendering.
|
|
428
|
+
- **T9b (sequenced after T7/T8)** — surfaces for the new APIs, with
|
|
429
|
+
Playwright coverage: Act runs inbox (live progress, cancel, mode badge,
|
|
430
|
+
approval pause→decide→resume), trigger configuration, System network view
|
|
431
|
+
(device fingerprint, peer registry, pairing), Ask context-trace panel
|
|
432
|
+
("why is this in context"), Brain provenance-coverage stat. Until T9b
|
|
433
|
+
lands, these capabilities are explicitly labeled API-only in
|
|
434
|
+
FEATURE_STATUS.md — a labeled state, not an omission.
|
|
435
|
+
- **i18n acceptance gate**: all strings in `routes.js`, the shared shell,
|
|
436
|
+
and every NEW v4 view are externalized; a checker script fails the build
|
|
437
|
+
on string literals in those files; remaining legacy-view strings are
|
|
438
|
+
inventoried in FEATURE_STATUS.md as labeled partial coverage.
|
|
439
|
+
|
|
440
|
+
**T10**
|
|
441
|
+
- Env-prefix canonicalization (`LATTICEAI_*` canonical, `LATTICE_*` read as
|
|
442
|
+
fallback aliases in `core/config.py`) and the CLI alias decision
|
|
443
|
+
(`ltcai` canonical, `LTCAI` deprecated) are owned here.
|
|
444
|
+
- Delete the superseded C-queue from the recovery file (replaced by this
|
|
445
|
+
plan) to remove contradictory guidance.
|
|
446
|
+
- Pre-flip migration backups: note in the restore runbook that backups live
|
|
447
|
+
on the same disk (exports dir) — recommend the user copy one off-disk at
|
|
448
|
+
upgrade time; the upgrade flow prints the backup path.
|
|
449
|
+
|
|
450
|
+
## Execution model
|
|
451
|
+
|
|
452
|
+
Each track runs as its own workflow phase: implementer agent(s) with the
|
|
453
|
+
track's file-ownership list and this plan section as contract → reviewer
|
|
454
|
+
agent (correctness + "no fake functionality" + capability preservation) →
|
|
455
|
+
fix loop → full unit suite → commit. The recovery file is updated at each
|
|
456
|
+
track boundary.
|
|
457
|
+
|
|
458
|
+
## Risk register
|
|
459
|
+
|
|
460
|
+
- **v2 flip (T3)** is the highest-risk change: mitigations = automatic backup,
|
|
461
|
+
idempotent migrator, equivalence suite, shim layer, and the flip lands as
|
|
462
|
+
its own commit (revertable in isolation).
|
|
463
|
+
- **Legacy frontend deletion (T9)**: redirects + Playwright cover the user
|
|
464
|
+
paths; deletion is one commit (revertable).
|
|
465
|
+
- **Async engine (T7)**: cooperative cancellation only (no thread kill);
|
|
466
|
+
synchronous fallback path retained behind the same API contract.
|
|
467
|
+
- **Garden absorption (T4)**: vault is read-only source; gardener writer
|
|
468
|
+
disabled only after import verifies; original files untouched.
|
|
469
|
+
- **Usage limits**: recovery file discipline; tracks commit independently so
|
|
470
|
+
an interrupted track loses at most its own uncommitted work.
|
package/docs/architecture.md
CHANGED
|
@@ -1,17 +1,18 @@
|
|
|
1
1
|
# Lattice AI Architecture
|
|
2
2
|
|
|
3
|
-
> v3.
|
|
4
|
-
>
|
|
5
|
-
>
|
|
6
|
-
>
|
|
7
|
-
>
|
|
8
|
-
>
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
organized around one durable center
|
|
13
|
-
agents, workflows, and UI modes are replaceable
|
|
14
|
-
|
|
3
|
+
> v3.6.0 — **Knowledge Graph First.** Every data source converges into the graph
|
|
4
|
+
> through one unified ingestion pipeline (`latticeai/services/ingestion.py`), with
|
|
5
|
+
> formalized entities/relationships (`docs/kg-schema.md`), browser/web inputs,
|
|
6
|
+
> per-node provenance, and local export/import/backup
|
|
7
|
+
> (`latticeai/services/kg_portability.py`). The agent ecosystem, long-term memory,
|
|
8
|
+
> and skills/hooks/tool/MCP registries are all operable from `/app`. Enterprise
|
|
9
|
+
> controls remain future work.
|
|
10
|
+
|
|
11
|
+
Lattice AI is a local-first **Digital Brain Platform**. The architecture is
|
|
12
|
+
organized around one durable center and the user's asset: the **Knowledge
|
|
13
|
+
Graph**. Models, tools, agents, RAG, workflows, and UI modes are replaceable
|
|
14
|
+
layers that operate as views over graph context. Models are replaceable;
|
|
15
|
+
knowledge is durable.
|
|
15
16
|
|
|
16
17
|
## Architecture Goals
|
|
17
18
|
|