ltcai 2.2.2 → 3.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (78) hide show
  1. package/README.md +66 -27
  2. package/codex_telegram_bot.py +6 -2
  3. package/docs/CHANGELOG.md +154 -0
  4. package/docs/V3_BACKEND_ARCHITECTURE.md +138 -0
  5. package/docs/V3_FRONTEND.md +136 -0
  6. package/knowledge_graph.py +649 -21
  7. package/latticeai/__init__.py +1 -1
  8. package/latticeai/api/admin.py +47 -0
  9. package/latticeai/api/agents.py +54 -31
  10. package/latticeai/api/auth.py +1 -1
  11. package/latticeai/api/chat.py +10 -2
  12. package/latticeai/api/search.py +236 -0
  13. package/latticeai/api/static_routes.py +21 -2
  14. package/latticeai/core/config.py +16 -0
  15. package/latticeai/core/embedding_providers.py +502 -0
  16. package/latticeai/core/local_embeddings.py +86 -0
  17. package/latticeai/core/logging_safety.py +62 -0
  18. package/latticeai/core/workspace_os.py +1 -1
  19. package/latticeai/server_app.py +49 -1
  20. package/latticeai/services/agent_runtime.py +245 -0
  21. package/latticeai/services/search_service.py +346 -0
  22. package/package.json +8 -4
  23. package/static/account.html +9 -4
  24. package/static/activity.html +4 -4
  25. package/static/admin.html +8 -3
  26. package/static/agents.html +4 -4
  27. package/static/chat.html +16 -11
  28. package/static/css/reference/account.css +439 -0
  29. package/static/css/reference/admin.css +610 -0
  30. package/static/css/reference/base.css +1658 -0
  31. package/static/{lattice-reference.css → css/reference/chat.css} +271 -3633
  32. package/static/css/reference/graph.css +1016 -0
  33. package/static/css/responsive.css +248 -1
  34. package/static/css/tokens.css +132 -126
  35. package/static/favicon.ico +0 -0
  36. package/static/graph.html +9 -4
  37. package/static/manifest.json +3 -3
  38. package/static/platform.css +1 -1
  39. package/static/plugins.html +4 -4
  40. package/static/scripts/account.js +4 -4
  41. package/static/scripts/chat.js +227 -77
  42. package/static/scripts/workspace.js +78 -0
  43. package/static/sw.js +5 -3
  44. package/static/v3/css/lattice.base.css +128 -0
  45. package/static/v3/css/lattice.components.css +447 -0
  46. package/static/v3/css/lattice.shell.css +407 -0
  47. package/static/v3/css/lattice.tokens.css +132 -0
  48. package/static/v3/css/lattice.views.css +277 -0
  49. package/static/v3/index.html +40 -0
  50. package/static/v3/js/app.js +26 -0
  51. package/static/v3/js/core/api.js +327 -0
  52. package/static/v3/js/core/components.js +215 -0
  53. package/static/v3/js/core/dom.js +148 -0
  54. package/static/v3/js/core/fixtures.js +171 -0
  55. package/static/v3/js/core/router.js +37 -0
  56. package/static/v3/js/core/routes.js +73 -0
  57. package/static/v3/js/core/shell.js +363 -0
  58. package/static/v3/js/core/store.js +113 -0
  59. package/static/v3/js/views/admin-audit.js +185 -0
  60. package/static/v3/js/views/admin-permissions.js +178 -0
  61. package/static/v3/js/views/admin-policies.js +103 -0
  62. package/static/v3/js/views/admin-private-vpc.js +138 -0
  63. package/static/v3/js/views/admin-security.js +181 -0
  64. package/static/v3/js/views/admin-users.js +168 -0
  65. package/static/v3/js/views/agents.js +194 -0
  66. package/static/v3/js/views/chat.js +450 -0
  67. package/static/v3/js/views/files.js +180 -0
  68. package/static/v3/js/views/home.js +119 -0
  69. package/static/v3/js/views/hybrid-search.js +195 -0
  70. package/static/v3/js/views/knowledge-graph.js +238 -0
  71. package/static/v3/js/views/models.js +247 -0
  72. package/static/v3/js/views/my-computer.js +237 -0
  73. package/static/v3/js/views/pipeline.js +161 -0
  74. package/static/v3/js/views/settings.js +258 -0
  75. package/static/workflows.html +4 -4
  76. package/static/workspace.css +408 -14
  77. package/static/workspace.html +43 -24
  78. package/telegram_bot.py +18 -14
package/README.md CHANGED
@@ -3,10 +3,11 @@
3
3
 
4
4
  # Lattice AI
5
5
 
6
- **Local-first AI workspace for knowledge graphs, AI pipelines, and multi-agent coding workflows.**
6
+ **Lattice AI v3 Local-First AI Workspace Platform.**
7
7
 
8
- Plan, execute, review, and remember work across local models, cloud models,
9
- files, and team workflows.
8
+ Work across Personal and Organization workspaces with Knowledge Graph,
9
+ Vector Index, Hybrid Search, Native Chat, agents, files, models, and
10
+ Basic / Advanced / Admin modes.
10
11
  </div>
11
12
 
12
13
  <div align="center">
@@ -15,7 +16,7 @@
15
16
  [![npm version](https://img.shields.io/npm/v/ltcai?label=npm)](https://www.npmjs.com/package/ltcai)
16
17
  [![VS Code Marketplace](https://vsmarketplacebadges.dev/version-short/parktaesoo.ltcai.svg)](https://marketplace.visualstudio.com/items?itemName=parktaesoo.ltcai)
17
18
  [![Open VSX](https://img.shields.io/open-vsx/v/parktaesoo/ltcai?label=Open%20VSX)](https://open-vsx.org/extension/parktaesoo/ltcai)
18
- [![GitHub release](https://img.shields.io/github/v/release/TaeSooPark-PTS/LatticeAI?label=GitHub%20release)](https://github.com/TaeSooPark-PTS/LatticeAI/releases/tag/v2.2.2)
19
+ [![GitHub release](https://img.shields.io/github/v/release/TaeSooPark-PTS/LatticeAI?label=GitHub%20release)](https://github.com/TaeSooPark-PTS/LatticeAI/releases/tag/v3.0.0)
19
20
  [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
20
21
  [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/)
21
22
  [![VS Code extension](https://img.shields.io/badge/VS%20Code-extension-blue?logo=visualstudiocode)](https://marketplace.visualstudio.com/items?itemName=parktaesoo.ltcai)
@@ -48,7 +49,7 @@ Install the coding extension:
48
49
 
49
50
  - [VS Code Marketplace: parktaesoo.ltcai](https://marketplace.visualstudio.com/items?itemName=parktaesoo.ltcai)
50
51
  - [Open VSX: parktaesoo.ltcai](https://open-vsx.org/extension/parktaesoo/ltcai)
51
- - [GitHub Release v2.2.2](https://github.com/TaeSooPark-PTS/LatticeAI/releases/tag/v2.2.2)
52
+ - [GitHub Release v3.0.0](https://github.com/TaeSooPark-PTS/LatticeAI/releases/tag/v3.0.0)
52
53
 
53
54
  ## Quick Start
54
55
 
@@ -61,7 +62,7 @@ LTCAI
61
62
  Then open:
62
63
 
63
64
  ```text
64
- http://127.0.0.1:4825
65
+ http://127.0.0.1:4825/app
65
66
  ```
66
67
 
67
68
  Development checkout:
@@ -81,9 +82,14 @@ npm run build
81
82
 
82
83
  ## What Is Lattice AI?
83
84
 
84
- Lattice AI is a local-first AI workspace for people and teams who want their
85
- files, models, graph context, and agent workflows in one place.
85
+ Lattice AI v3 is a local-first AI workspace platform for people and teams who
86
+ want their files, models, graph context, retrieval, and agent workflows in one
87
+ place.
86
88
 
89
+ - **Primary app shell**: `/app` is the default product experience with Native
90
+ Chat, Knowledge Graph, Hybrid Search, Files, Pipeline, Agents, Models, My
91
+ Computer, Settings, and Admin areas. Legacy `/chat` remains available as a
92
+ rollback/debug path.
87
93
  - **Local-first AI Workspace**: work starts on your machine, with local data and
88
94
  workspace state by default.
89
95
  - **AI Pipeline Platform**: plan, execute, review, retry, and replay work across
@@ -94,7 +100,9 @@ files, models, graph context, and agent workflows in one place.
94
100
  - **Multi-Agent Workflow Platform**: agents hand off structured context, review
95
101
  work, retry with reasons, and keep timelines inspectable.
96
102
  - **Personal / Organization Workspace**: move between personal work and team
97
- workspaces with role-aware views.
103
+ workspaces with role-aware views and Basic / Advanced / Admin modes.
104
+ - **Vector Index and Hybrid Search**: local vector rows are derived from the
105
+ Knowledge Graph and fused with keyword and graph signals.
98
106
  - **Local Model Management**: choose current multimodal local models with source
99
107
  disclosure, hardware-aware recommendations, and cloud fallback options.
100
108
  - **SSO for teams**: organization workspaces can be paired with Okta or
@@ -114,25 +122,23 @@ and disconnected automations. Lattice AI keeps those parts together:
114
122
  - multi-agent workflows leave behind replayable plans, reviews, retries, and
115
123
  outcomes.
116
124
 
117
- ## v2.2.2 QA & Stability Highlights
125
+ ## v3.0.0 Highlights
118
126
 
119
- A no-features stabilization release that hardens the v2.2.x UI. Every fix keeps
120
- the existing design-token structure and adds no `!important`.
127
+ Lattice AI v3.0.0 makes `/app` the primary workspace shell and ships the v3
128
+ backend retrieval stack together with the native frontend.
121
129
 
122
- - Mobile hamburger navigation on the Knowledge Graph and Admin pages is
123
- reachable again (a CSS source-order bug had kept the toggles hidden).
124
- - Admin top-bar actions (Refresh, Logout) are clickable again — a graph-only
125
- toolbar rule that floated over the header was scoped back off the graph page.
126
- - Removed latent horizontal overflow on the Workspace page.
127
- - Verified across the full viewport matrix (375px phone 3440px ultrawide):
128
- light/dark theme parity, button hit-testing, no horizontal scroll, mobile
129
- drawer open/close, and Escape-to-close all covered by an expanded
130
- Playwright suite.
131
-
132
- Carried forward from v2.2.1: mobile-first responsive layout, design-token
133
- light/dark themes, keyboard-safe chat composer, Knowledge Graph responsive UX,
134
- Admin mobile card layout, drag-and-drop file attachment, and model-card source
135
- disclosure.
130
+ - Native v3 Chat lives inside `/app#/chat` and streams through the real
131
+ `POST /chat` backend while showing friendly setup guidance when no model is
132
+ loaded.
133
+ - Knowledge Graph, Vector Index, and Hybrid Search are first-class retrieval
134
+ surfaces. Hybrid results show keyword, local vector, and graph scores.
135
+ - Personal and Organization workspaces, plus Basic / Advanced / Admin modes,
136
+ are built into the shell.
137
+ - Legacy `/chat` remains reachable for rollback and debugging.
138
+ - The default embedding signal is `lattice-local-hash-v1`, a deterministic
139
+ local fallback. It is not described as a production semantic embedding model;
140
+ future providers may include Ollama, MLX, OpenAI-compatible endpoints, and
141
+ other local embedding runtimes.
136
142
 
137
143
  ## Screenshots
138
144
 
@@ -172,6 +178,32 @@ files / documents / images / screenshots / conversations / decisions
172
178
 
173
179
  The graph keeps useful workspace context available even when you change models.
174
180
 
181
+ ## v3 Backend Retrieval
182
+
183
+ The v3 backend adds a local-first retrieval stack that combines the Knowledge
184
+ Graph, a SQLite vector index, and hybrid result fusion. It preserves existing
185
+ graph data while adding derived vector rows that can be rebuilt at any time.
186
+
187
+ Embedding status: the current default is `lattice-local-hash-v1`, a
188
+ deterministic local fallback embedder for indexing and tests. It provides a
189
+ stable vector signal without downloads or cloud calls; it is not a production
190
+ semantic embedding model. Future provider support may include Ollama, MLX,
191
+ OpenAI-compatible providers, and other local embedding runtimes.
192
+
193
+ Core API contracts:
194
+
195
+ - `POST /api/search/hybrid`
196
+ - `GET /api/search/keyword?q=...`
197
+ - `GET /api/search/vector?q=...`
198
+ - `GET /api/graph`
199
+ - `GET /api/graph/node?node_id=...`
200
+ - `GET /api/graph/relationship`
201
+ - `GET /api/index/status`
202
+ - `POST /api/index/rebuild`
203
+
204
+ See [docs/V3_BACKEND_ARCHITECTURE.md](docs/V3_BACKEND_ARCHITECTURE.md) for the
205
+ storage model, search model, migration behavior, and API response shape.
206
+
175
207
  ## Local Model Policy
176
208
 
177
209
  Lattice AI recommends current-generation multimodal models for local use and
@@ -227,7 +259,14 @@ Core areas:
227
259
 
228
260
  | Version | Theme |
229
261
  | --- | --- |
230
- | **2.2.2** | Frontend QA stabilization mobile nav, admin actions, overflow fixes, and expanded visual tests |
262
+ | **3.0.1** | Release-blocker remediationprovider-backed embeddings (Hash/MLX/Ollama/OpenAI/Custom), unified AgentRuntime boundary, every v3 surface connected or clearly unavailable |
263
+ | 3.0.0 | v3 local-first AI workspace platform — `/app`, Native Chat, Knowledge Graph, Vector Index, Hybrid Search, workspace modes |
264
+ | 2.2.7 | Visual system stabilization — cohesive dark/light screens, crisp chat composer, dark graph canvas, Workspace OS polish |
265
+ | 2.2.6 | Token-native CSS foundation |
266
+ | 2.2.5 | Release hygiene hotfix — dark overlays, modal stack, cache-busting, favicon, and Telegram log masking |
267
+ | 2.2.4 | Chat dark-mode completion |
268
+ | 2.2.3 | Frontend stability and UX fixes |
269
+ | 2.2.2 | Frontend QA stabilization — mobile nav, admin actions, overflow fixes, and expanded visual tests |
231
270
  | 2.2.1 | Frontend and UX overhaul for responsive workspace, themes, graph UX, admin reflow, and file attachment |
232
271
  | 2.2.0 | Multimodal-first Knowledge Graph and local model source disclosure |
233
272
  | 2.1.0 | Multi-agent workflow maturity |
@@ -7,6 +7,10 @@ from pathlib import Path
7
7
  import httpx
8
8
  from openai import AsyncOpenAI
9
9
 
10
+ from latticeai.core.logging_safety import install_sensitive_log_filter, safe_log_text
11
+
12
+ install_sensitive_log_filter()
13
+
10
14
 
11
15
  def load_env_file(path=".env"):
12
16
  env_path = Path(path)
@@ -72,7 +76,7 @@ async def get_updates(client, offset=None):
72
76
  res = await client.get(f"{TELEGRAM_API_URL}/getUpdates", params=params, timeout=35)
73
77
  return res.json()
74
78
  except Exception as exc:
75
- logger.error("Telegram update failed: %s", exc)
79
+ logger.error("Telegram update failed: %s", safe_log_text(exc))
76
80
  return None
77
81
 
78
82
 
@@ -161,7 +165,7 @@ async def handle_message(client, chat_id, text):
161
165
  answer = await ask_codex(chat_id, text)
162
166
  except Exception as exc:
163
167
  logger.exception("OpenAI request failed")
164
- answer = f"OpenAI 요청 실패: {exc}"
168
+ answer = f"OpenAI 요청 실패: {safe_log_text(exc)}"
165
169
  await send_message(client, chat_id, answer)
166
170
 
167
171
 
package/docs/CHANGELOG.md CHANGED
@@ -1,5 +1,159 @@
1
1
  # Changelog
2
2
 
3
+ ## [3.0.0] - 2026-06-07
4
+
5
+ > v3 — Local-first AI Workspace Platform. The hybrid-search
6
+ > backend and the token-native `/app` workspace shell now ship together: the
7
+ > shell's adapters call the real v3 retrieval APIs, and Chat is a first-class
8
+ > native view (no link-out to the legacy page). Legacy `/chat` remains available
9
+ > as a rollback/debug path.
10
+
11
+ ### Added — Backend retrieval
12
+
13
+ - **Hybrid search API** — added `/api/search/hybrid`, `/api/search/keyword`,
14
+ `/api/search/vector`, `/api/graph`, `/api/graph/node`,
15
+ `/api/graph/relationship`, `/api/index/status`, and `/api/index/rebuild`.
16
+ - **SQLite vector index** — added local deterministic embeddings,
17
+ `vector_embeddings`, and `vector_index_operations` for incremental indexing,
18
+ rebuilds, and status monitoring.
19
+ - **Embedding status** — the default `lattice-local-hash-v1` embedder is a
20
+ deterministic local fallback, not a production semantic embedding model.
21
+ Future providers may include Ollama, MLX, OpenAI-compatible providers, and
22
+ other local embedding runtimes.
23
+ - **Graph retrieval helpers** — added node lookup, relationship search, bounded
24
+ traversal, neighbor expansion, and service-level result fusion.
25
+ - **Backend architecture doc** — added `docs/V3_BACKEND_ARCHITECTURE.md` with
26
+ storage, search, API, and migration details.
27
+
28
+ ### Added — Native app shell (`/app`)
29
+
30
+ - **Unified app shell** (`static/v3/`) — nav rail, command palette (⌘K),
31
+ workspace switcher (Personal/Organization), and mode switcher
32
+ (Basic/Advanced/Admin); hash-routed views for every primary and admin area
33
+ (Home, Chat, Knowledge Graph, Hybrid Search, Files, Pipeline, Agents, Models,
34
+ My Computer, Settings, and Admin · Users/Permissions/Audit/Security/Policies/
35
+ Private VPC).
36
+ - **Native Chat view** — a first-class 3-pane chat (conversations · thread ·
37
+ retrieval context) wired to the real backend (`POST /chat` SSE + `/history/*`)
38
+ with streaming, empty/error/loading states; surfaces Knowledge Graph, Vector,
39
+ Hybrid Search, and indexed-file context per answer. The legacy `/chat` page
40
+ stays reachable but is no longer the primary chat experience.
41
+ - **Primary entry behavior** — `/app` is the product entry after login and SSO;
42
+ the PWA manifest starts at `/app`.
43
+ - **Retrieval identity** — Knowledge Graph + Vector Index + Hybrid Search are
44
+ surfaced as a first-class "retrieval lattice" on Home and a live index chip.
45
+ - **Token-native design system** — `static/v3/css/lattice.*.css` built on top of
46
+ `tokens.css` with no dependency on the legacy override layers; full light/dark
47
+ and desktop/tablet/mobile support.
48
+ - **Integration adapters** — `static/v3/js/core/api.js` calls the real v3
49
+ endpoints and degrades to clearly-badged sample data; no backend logic in the UI.
50
+
51
+ ### Validation
52
+
53
+ - Backend coverage includes v3 indexing, migration status, vector retrieval,
54
+ graph relationship traversal, hybrid result fusion, and API contract tests.
55
+ - Frontend coverage: `tests/visual/v3.spec.js` and `scripts/lint_v3.mjs` (wired
56
+ into `npm run lint`); see `docs/V3_FRONTEND.md` for IA + design decisions.
57
+ - Release preparation builds exact `3.0.0` Python, npm, and VSIX artifacts.
58
+ Package-store publication remains manual and is not triggered by pushing the
59
+ release tag.
60
+
61
+ > Frontend Product Shell Redesign — workspace navigation, auth entry, and shared
62
+ > product surfaces were realigned around the local-first AI workspace model
63
+ > without changing backend contracts.
64
+
65
+ ### Changed
66
+
67
+ - **Workspace IA** — the workspace shell now separates primary user workflows,
68
+ admin controls, and runtime tooling, with Basic, Advanced, and Admin modes.
69
+ - **Navigation** — Chat and Workspace navigation now use consistent labels for
70
+ Home, Chat, Knowledge Graph, Files, Pipeline, My Computer, Search, and
71
+ organization administration.
72
+ - **Design tokens** — shared product surfaces moved away from the prior
73
+ lavender-heavy treatment toward neutral work surfaces with blue, teal, and
74
+ amber accents.
75
+ - **Auth surface** — account screens use the same token-native product shell as
76
+ the workspace experience and hide decorative background elements.
77
+
78
+ ### Validation
79
+
80
+ - Frontend validation includes lint, Python checks, browser-rendered workspace
81
+ smoke checks, and Playwright visual regression coverage.
82
+ - Production build output was intentionally not generated for this frontend-only
83
+ redesign pass.
84
+
85
+ ## [2.2.7] - 2026-06-05
86
+
87
+ > Visual Stabilization Release — browser-rendered screens were reviewed and
88
+ > polished until Chat, onboarding, graph, Workspace OS, and operational panels
89
+ > felt like one product.
90
+
91
+ ### Fixed
92
+
93
+ - **Chat composer haze** — removed the dark-mode white/milky bottom composer
94
+ effect and the legacy inner textarea border; the shell now owns the focus
95
+ state and the attachment/send controls stay readable.
96
+ - **Knowledge Graph canvas** — replaced the washed-out light graph work surface
97
+ with an intentional dark canvas treatment.
98
+ - **Workspace OS dark surfaces** — relationship/list cards, inputs, tags,
99
+ health cards, and capability cards no longer fall back to hardcoded white.
100
+ - **Onboarding/modals** — workspace select, PC environment analysis,
101
+ recommendation result, auto setup, mode select, pipeline, My Computer,
102
+ profile, settings, Private VPC, and model-state panels now share the same
103
+ dark panel language.
104
+ - **Account dark contrast** — account/register logo text, inputs, and window
105
+ controls remain readable in dark mode.
106
+
107
+ ### Changed
108
+
109
+ - **Cache-busting** — all versioned frontend assets now use `?v=2.2.7`,
110
+ including `/static/scripts/chat.js?v=2.2.7`.
111
+ - **Version sync** — Python package, npm package, VS Code extension, Workspace
112
+ OS, lockfiles, and runtime metadata aligned at `2.2.7`.
113
+
114
+ ### Validation
115
+
116
+ - Release target includes Python compile/pytest, npm lint/typecheck/test/build,
117
+ Python build + twine check, npm pack, VSIX package, and Playwright visual QA.
118
+ - Package-store publishing remains manual; release artifacts are version-scoped
119
+ and must use exact `2.2.7` filenames.
120
+
121
+ ## [2.2.5] - 2026-06-04
122
+
123
+ > Release Hygiene Hotfix — dark-mode overlay clarity, modal state protection,
124
+ > static asset version alignment, favicon routing, and Telegram log redaction.
125
+
126
+ ### Added
127
+
128
+ - **Modal manager** — Chat overlays now share one blocking-modal controller with
129
+ Escape close, backdrop close, pagehide/navigation cleanup, and body scroll-lock
130
+ restoration.
131
+ - **Favicon asset** — `static/favicon.ico` is packaged and served by
132
+ `/favicon.ico`.
133
+ - **Sensitive-log helper** — Telegram bot tokens are normalized to
134
+ `bot123:REDACTED` before HTTP, exception, or response text reaches logs.
135
+ - **Validation coverage** — unit tests for token masking and static release
136
+ hygiene, plus Playwright checks for modal stack behavior and favicon response.
137
+
138
+ ### Changed
139
+
140
+ - **Overlay theme tokens** — full-screen overlays use `--overlay-scrim` and no
141
+ blur-heavy backdrop, preventing washed-out dark-mode UI.
142
+ - **Surface token coverage** — modal, drawer, file manager, My Computer,
143
+ onboarding, model switcher, pipeline, and admin surfaces are remapped to
144
+ semantic tokens (`--modal`, `--surface`, `--surface-elevated`, `--input`).
145
+ - **Cache-busting** — all versioned frontend assets now use `?v=2.2.5`, including
146
+ `/static/scripts/chat.js?v=2.2.5`.
147
+ - **Version sync** — Python package, npm package, VS Code extension, Workspace
148
+ OS, lockfiles, and runtime metadata aligned at `2.2.5`.
149
+
150
+ ### Validation
151
+
152
+ - Release target includes Python compile/pytest, npm lint/typecheck/test/build,
153
+ Python build + twine check, npm pack, VSIX package, and Playwright visual QA.
154
+ - Package-store publishing remains manual; release artifacts are version-scoped
155
+ and must use exact `2.2.5` filenames.
156
+
3
157
  ## [2.2.1] - 2026-06-04
4
158
 
5
159
  > Frontend / UX Overhaul Release — Lattice AI keeps feature behavior stable
@@ -0,0 +1,138 @@
1
+ # Lattice AI v3 Backend Architecture
2
+
3
+ Lattice AI v3 keeps the product local-first and knowledge-centric by combining
4
+ three retrieval layers inside the backend:
5
+
6
+ ```text
7
+ User Data
8
+ -> Chunking
9
+ -> Local Embeddings
10
+ -> SQLite Vector Index
11
+
12
+ User Data
13
+ -> Entity Extraction
14
+ -> Relationship Extraction
15
+ -> SQLite Knowledge Graph
16
+
17
+ Search
18
+ -> Hybrid Retrieval
19
+ -> Keyword
20
+ -> Vector
21
+ -> Graph
22
+ -> Unified Results
23
+ ```
24
+
25
+ ## Storage Model
26
+
27
+ The authoritative knowledge store remains `knowledge_graph.py` on SQLite.
28
+ Existing legacy graph rows are preserved, with the normalized v2 projection kept
29
+ as a derived read surface.
30
+
31
+ - `nodes`: entities, documents, folders, conversations, tasks, decisions, and
32
+ chunk nodes.
33
+ - `edges`: directed relationships with type, weight, metadata, and timestamps.
34
+ - `chunks`: extracted text chunks linked back to the source node.
35
+ - `knowledge_sources` and `local_file_index`: approved local folder indexing
36
+ state, source metadata, and incremental scan bookkeeping.
37
+ - `vector_embeddings`: derived local vector index rows for non-chunk nodes and
38
+ full chunk text.
39
+ - `vector_index_operations`: rebuild/incremental indexing history and status.
40
+
41
+ Vector rows are derived data. A full rebuild may delete and recreate vector
42
+ rows, but it does not mutate source documents, nodes, relationships, chunks, or
43
+ local file records.
44
+
45
+ ## Embeddings
46
+
47
+ The default embedding model is `lattice-local-hash-v1`, a deterministic local
48
+ feature-hashing embedder in `latticeai/core/local_embeddings.py`.
49
+
50
+ It provides:
51
+
52
+ - no cloud dependency,
53
+ - no model download requirement,
54
+ - deterministic output for tests and migrations,
55
+ - normalized vectors for cosine similarity by dot product,
56
+ - a stable interface for replacing the implementation with a local model
57
+ runtime later.
58
+
59
+ It is a fallback vector signal, not a production semantic embedding model.
60
+ Future provider support may include Ollama, MLX, OpenAI-compatible providers,
61
+ and other local embedding providers behind the same vector-index interface.
62
+
63
+ ## Search Model
64
+
65
+ `latticeai/services/search_service.py` composes the retrieval layers.
66
+
67
+ - Keyword search uses the existing graph text search over title, summary, and
68
+ metadata.
69
+ - Vector search embeds the query locally and ranks SQLite vector rows by
70
+ similarity.
71
+ - Graph search performs direct graph matching, relationship search, and bounded
72
+ neighbor expansion.
73
+ - Hybrid search fuses the three result streams with configurable weights and
74
+ returns one UI-ready result shape with per-source scores.
75
+
76
+ Default hybrid weights:
77
+
78
+ ```json
79
+ {
80
+ "keyword": 0.35,
81
+ "vector": 0.40,
82
+ "graph": 0.25
83
+ }
84
+ ```
85
+
86
+ ## API Contracts
87
+
88
+ The v3 backend exposes frontend-ready routes under `/api` while preserving all
89
+ existing `/knowledge-graph/...` compatibility routes.
90
+
91
+ ```text
92
+ POST /api/search/hybrid
93
+ GET /api/search/hybrid?q=...
94
+ POST /api/search/keyword
95
+ GET /api/search/keyword?q=...
96
+ POST /api/search/vector
97
+ GET /api/search/vector?q=...
98
+ GET /api/graph
99
+ GET /api/graph/node?node_id=...
100
+ POST /api/graph/node
101
+ GET /api/graph/relationship
102
+ POST /api/graph/relationship
103
+ GET /api/index/status
104
+ POST /api/index/rebuild
105
+ ```
106
+
107
+ Results include stable fields for UI consumption:
108
+
109
+ - `id`
110
+ - `node_id`
111
+ - `item_type`
112
+ - `type`
113
+ - `title`
114
+ - `summary`
115
+ - `score`
116
+ - `rank`
117
+ - `sources`
118
+ - `source_scores`
119
+ - `metadata`
120
+ - `updated_at`
121
+ - optional `graph_context`
122
+
123
+ ## Migration Impact
124
+
125
+ The migration is additive and non-destructive:
126
+
127
+ - new SQLite tables are created with `CREATE TABLE IF NOT EXISTS`,
128
+ - existing graph and local file data is preserved,
129
+ - normalized v2 graph projection remains a derived compatibility layer,
130
+ - vector rows can be rebuilt from existing graph/chunk rows,
131
+ - `GET /api/index/status` reports missing or stale vector items,
132
+ - `POST /api/index/rebuild` repairs the derived index.
133
+
134
+ ## Backend Ownership
135
+
136
+ This architecture deliberately avoids frontend shell redesign. The backend owns
137
+ storage, indexing, retrieval, ranking, and API contracts; the UI can consume the
138
+ new `/api` routes without requiring visual system changes.
@@ -0,0 +1,136 @@
1
+ # Lattice AI v3 — Frontend Product Shell
2
+
3
+ > A token-native single-page workspace shell for Lattice AI v3. It ships as the
4
+ > primary product experience and calls the real v3 retrieval/chat APIs while
5
+ > keeping graceful, clearly-badged sample states for unavailable local services.
6
+
7
+ Entry point: **`/app`** (served by `latticeai/api/static_routes.py` →
8
+ `static/v3/index.html`). Login, auto-login after registration, SSO callback, and
9
+ the PWA manifest land on `/app`. The legacy multi-page screens (`/workspace`,
10
+ `/chat`, `/graph`, `/admin`, …) remain reachable; `/chat` is the rollback/debug
11
+ path for the native v3 Chat view.
12
+
13
+ ---
14
+
15
+ ## Product model
16
+
17
+ | Concept | Surface |
18
+ | --- | --- |
19
+ | **Personal / Organization workspace** | Scope switcher in the rail (`X-Workspace-Id` header is sent on every adapter call). |
20
+ | **Basic / Advanced / Admin mode** | Mode switcher in the topbar. Mode gates which rail items appear and how dense each view is. |
21
+ | **Retrieval lattice** | The signature identity: **Knowledge Graph + Vector Index + Hybrid Search**, surfaced on Home and as a live 3-dot index chip in the topbar. |
22
+
23
+ Mode gating (`MODE_RANK` in `core/routes.js`): Basic ⊂ Advanced ⊂ Admin.
24
+ Deep-linking into an `admin/*` route auto-promotes the shell to Admin mode.
25
+
26
+ ---
27
+
28
+ ## Information architecture
29
+
30
+ One declarative table (`static/v3/js/core/routes.js`) drives the nav rail, the
31
+ command palette, the router, breadcrumbs, and lazy view loading.
32
+
33
+ ```
34
+ Workspace Home · Chat
35
+ Retrieval Knowledge Graph · Hybrid Search
36
+ Data Files · Pipeline*
37
+ Compute Agents* · Models · My Computer*
38
+ System Settings
39
+ Administration Users · Permissions · Audit Logs · Security · Policies · Private VPC (admin mode)
40
+ ```
41
+
42
+ `*` = revealed in Advanced mode and above. Routes are hash-based
43
+ (`#/knowledge-graph`, `#/admin/users`) so the SPA needs no server rewrites.
44
+
45
+ ---
46
+
47
+ ## Design system
48
+
49
+ Token-native, layered on the existing **single color source**
50
+ (`static/css/tokens.css`, `data-lt-theme` light/dark). No legacy override layers
51
+ (`responsive.css` / `workspace.css` / `platform.css`) are loaded by v3.
52
+
53
+ | File | Responsibility |
54
+ | --- | --- |
55
+ | `css/lattice.tokens.css` | Structural tokens — type scale, spacing (4pt), radii, elevation, motion, z-index, layout rails, pillar accents, lattice mesh. |
56
+ | `css/lattice.base.css` | Reset, element defaults, the signature lattice backdrop, a11y utilities. |
57
+ | `css/lattice.components.css` | Primitive vocabulary — cards, panels, buttons, inputs, pills, stats, tables, tabs, switches, meters, empty/loading/error states, toasts. |
58
+ | `css/lattice.shell.css` | App chrome — rail, topbar, command palette, mobile drawer, retrieval pillars. |
59
+ | `css/lattice.views.css` | Bespoke view layouts — hero, graph canvas, search results, chat, files, pipeline, admin. |
60
+
61
+ **Rule:** component CSS uses only `var(--token)`; no per-component hardcoded
62
+ themed colors and no `!important`. Both themes flow from the token values.
63
+
64
+ **Identity:** a crystalline *lattice* mark, a faint node-and-edge mesh backdrop,
65
+ and a three-pillar retrieval motif (graph = blue, vector = teal, hybrid =
66
+ magenta). Deliberately **not** a clone of ChatGPT / Claude / Cursor / VS Code /
67
+ Notion / Obsidian.
68
+
69
+ ---
70
+
71
+ ## Shell + view contract
72
+
73
+ `core/shell.js` builds the chrome and renders views. Each view is a lazy ES
74
+ module under `js/views/` exporting:
75
+
76
+ ```js
77
+ export async function render(ctx) { /* … */ return singleDomNode; }
78
+ // optional: export const layout = "flush"; // full-bleed (chat)
79
+ ```
80
+
81
+ `ctx = { h, icon, api, store, c, route, params, navigate, toast }`
82
+
83
+ - `h()` — dependency-free hyperscript (`core/dom.js`); child strings are
84
+ auto-escaped (no injection surface).
85
+ - `c` — component factories (`core/components.js`).
86
+ - `store` — observable app state (theme / mode / workspace), persisted to
87
+ `localStorage` (`core/store.js`).
88
+
89
+ ---
90
+
91
+ ## Integration contract
92
+
93
+ `core/api.js` is the only transport layer. Every call hits the **real** endpoint
94
+ first and degrades to a clearly-badged sample payload (`core/fixtures.js`) when
95
+ the endpoint is absent — returning `{ ok, status, data, source }` where `source`
96
+ is `"live"` or `"placeholder"`. The UI always renders a **Sample data** badge for
97
+ placeholder responses, so nothing fake is presented as backend output.
98
+
99
+ Live surfaces wired in v3.0.0:
100
+
101
+ | Adapter | Endpoint | Fallback |
102
+ | --- | --- | --- |
103
+ | `api.indexStatus()` | `GET /api/index/status` | sample KG/vector/hybrid pipeline state |
104
+ | `api.graph(params)` | `GET /api/graph` → `GET /knowledge-graph/graph` | sample mesh |
105
+ | `api.hybridSearch(q, opts)` | `POST /api/search/hybrid` | sample fused results |
106
+ | `api.streamChat(body)` | `POST /chat` SSE | sample stream only when endpoint is unavailable |
107
+
108
+ No backend retrieval logic is implemented in the frontend — only transport,
109
+ normalization, clear provenance badges, and graceful fallback. When the live
110
+ chat backend reports `no_model_loaded`, v3 Chat shows a setup message instead of
111
+ falling back to sample generation.
112
+
113
+ Embedding disclosure: the current default vector signal is
114
+ `lattice-local-hash-v1`, a deterministic local fallback. The UI avoids calling
115
+ it a production semantic embedding model.
116
+
117
+ ---
118
+
119
+ ## Responsive strategy
120
+
121
+ - **Desktop (≥1101px):** full 268px rail + content.
122
+ - **Tablet (761–1100px):** rail collapses to a 76px icon rail.
123
+ - **Mobile (≤760px):** rail becomes an off-canvas drawer (hamburger in the
124
+ topbar); grids collapse to single column; the chat context rail hides.
125
+
126
+ Plus: `prefers-color-scheme` following, `prefers-reduced-motion` honored, visible
127
+ focus rings, skip link, keyboard command palette (⌘K / Ctrl-K).
128
+
129
+ ---
130
+
131
+ ## Validation
132
+
133
+ - `npm run lint` (extended to cover `static/v3/**` via `scripts/lint_v3.mjs`).
134
+ - `npm run test:visual` (`tests/visual/v3.spec.js` against the mock server, which
135
+ serves `/app` and mocks the future API surfaces).
136
+ - Browser-rendered smoke checks of every route in light and dark themes.