ltcai 3.5.0 → 4.0.0

package/README.md

@@ -3,10 +3,11 @@
 
   # Lattice AI
 
-  **Local-first AI workspace for your files, chats, knowledge, models, and agents.**
+  **A local-first Digital Brain Platform. Your Knowledge Graph is the durable asset; models just read it.**
 
-  Keep your work context on your own machine. Connect documents, conversations,
-  local models, graph memory, and agent workflows in one self-hosted workspace.
+  Every source — files, folders, web pages, browser tabs — converges into one
+  Knowledge Graph on your own machine. Connect models, agents, and search to that
+  graph instead of placing your work inside any single model.
 </div>
 
 <div align="center">
@@ -24,10 +25,21 @@
 
 ![Lattice AI — local-first AI workspace home](docs/assets/v3.4.0/home.png)
 
-> **Lattice AI is a self-hosted AI workspace that keeps your files, chats, knowledge, local models, and agents together on your own machine.**
+> **Lattice AI is not a model-personalization system. It is a Digital Brain Platform.**
+> The Knowledge Graph is your durable asset. **Models are replaceable. Knowledge is durable.**
 
-It isn't another chat window. It's a workspace built around your work — local-first
-by default, cloud only when you choose.
+It isn't another chat window, and it isn't a way to "fine-tune a model on you." The
+purpose of Lattice AI is to **connect models to your Knowledge Graph** — your digital
+brain — not to place you inside a model. AI reads your knowledge; you own it.
+
+- **Models are replaceable.** Swap MLX, Ollama, LM Studio, or a cloud LLM at will.
+- **Agents, RAG, and the UI are replaceable.** They are implementations, not the asset.
+- **Your Knowledge Graph is durable.** It outlives every model and is yours to export,
+  import, and back up locally — no cloud required.
+
+Local-first by default; cloud only when you choose. (The Vercel site is a
+landing/download/demo surface only — never the runtime. Lattice AI runs on your
+machine over local SQLite.)
 
 ## Why install Lattice AI?
 
@@ -44,8 +56,10 @@ workspace around the work itself:
   LM Studio; nothing leaves your machine unless you opt in.
 - **Use cloud models only when you choose** — bring an API key for cloud LLMs
   when you want them, not by default.
-- **Automate with agents you can inspect** — workflows leave behind plans,
-  reviews, retries, and results you can replay.
+- **Inspect every agent run** — the built-in runner persists plans, reviews,
+  retries, and results as replayable run records. Today that runner is
+  deterministic and does not call a model; LLM-driven execution is the v4
+  runtime, in progress.
 
 Lattice AI is not a clone of ChatGPT, Claude, Cursor, Obsidian, or Notion. It
 sits in a different place: a **workspace** that ties local/self-hosted AI, your
@@ -61,8 +75,8 @@ and workflows together — and runs on your own hardware.
   Knowledge Graph.
 - Run local models through MLX, Ollama, or LM Studio, and use cloud LLMs only when
   you want to.
-- Create repeatable agent workflows for research, coding, analysis, and
-  documentation.
+- Define agent workflows and replay their run records step by step (execution
+  is deterministic scaffolding today — it does not yet call a model).
 - Separate personal work from organization work.
 - Switch between Basic, Advanced, and Admin modes depending on your role.
 
@@ -96,12 +110,14 @@ chat and hybrid search.
 The Knowledge Graph shows how files, decisions, conversations, and entities
 connect — context that stays useful even when you switch models.
 
-### Run agent workflows
+### Inspect agent run records
 
 ![Lattice AI agent run with roles, logs, review, and retry](docs/assets/v3.4.0/agent-run.png)
 
-Agents turn a goal into an inspectable run — roles, logs, review, and retry — that
-you can read back step by step.
+The agent runner turns a goal into an inspectable, replayable run record — roles,
+logs, review, and retry — that you can read back step by step. The runner shown
+here is deterministic and LLM-free; real model-driven execution is the v4
+runtime, currently in progress.
 
 ### Extend with hooks and the local runtime
 
@@ -187,32 +203,51 @@ npm run dev
 
 ## Latest Release
 
-### v3.4.1 — Runtime Completion
-
-- Full hooks lifecycle across HTTP, agent, workflow, upload, and indexing paths.
-- Real Local Agent probes instead of hardcoded readiness.
-- Connect Folder verified end-to-end.
-- Folder Watch verified, including restore after restart.
-
-See [RELEASE_NOTES_v3.4.1.md](RELEASE_NOTES_v3.4.1.md) and
+### v3.6.0 — Knowledge Graph First
+
+- **Unified ingestion pipeline** — one entrypoint normalizes every source
+  (file, folder, web URL, browser tab, text/markdown/code) into the graph,
+  idempotent by content hash and bracketed by the `pre_tool`/`post_tool` hook
+  lifecycle.
+- **Formalized entity/relationship model** — first-class `Source`, `Repository`,
+  `Meeting`, `Organization`, `Workflow`, `Agent` entities and `indexed_from`,
+  `modified_by`, `belongs_to_project`, `part_of`, `discussed_in`, `decided_by`,
+  `generated_by`, `used_by_agent` relationships.
+- **Browser & web ingestion** — `POST /api/browser/read-url` and
+  `/ingest-current-tab`, plus a local Manifest V3 extension that posts only to
+  `127.0.0.1`.
+- **Portability** — local Knowledge Graph export/import (versioned JSON) and
+  binary backup/restore (DB + blobs, integrity-checked). No cloud required.
+- **Provenance** — every node records where it came from; a queryable audit trail
+  makes the graph explainable.
+- **Knowledge Graph as the primary surface** — the view becomes your digital brain
+  with Status, Sources, Capture, and Backup tabs.
+
+See [RELEASE_NOTES_v3.6.0.md](RELEASE_NOTES_v3.6.0.md),
+[docs/kg-schema.md](docs/kg-schema.md),
+[docs/RUNTIME_HOOK_COVERAGE_v3.6.0.md](docs/RUNTIME_HOOK_COVERAGE_v3.6.0.md), and
 [FEATURE_STATUS.md](FEATURE_STATUS.md).
 
-## How it works
+## How it works — every source converges into the graph
+
+As of v3.6.0, all data sources flow through **one unified ingestion pipeline** into
+the Knowledge Graph — no source bypasses it, none becomes an isolated silo:
 
 ```text
-files / chats / notes / images / decisions
-  -> workspace memory
-  -> knowledge graph
-  -> hybrid search
-  -> chat / agents / workflows
-  -> reusable outputs
+source (file · folder · PDF · web URL · browser tab · text)
+  -> extraction -> normalization -> content hash (idempotent)
+  -> chunking -> entity detection -> relationship detection -> embedding
+  -> Knowledge Graph  (Source -[indexed_from]- content -[contains]- chunks)
+  -> RAG / agents / memory / hybrid search
 ```
 
-- Your content stays on your machine and becomes durable workspace memory.
-- Memory is organized into a knowledge graph of entities and relationships.
-- Hybrid search fuses keyword, vector, and graph signals over that context.
-- Chat, agents, and workflows draw on the same grounded context.
-- Outputs — documents, analysis, and decisions — feed back into the workspace.
+- **Every node is explainable.** Each ingested item carries provenance — where it
+  came from, when, how it was processed, whether it was embedded or linked.
+- **The graph is the asset.** Memory, search, and agents are views over it; models
+  read it. Swap a model and your knowledge is unchanged.
+- **Portable, no cloud.** Export/import the graph as JSON, or take a full local
+  binary backup (DB + blobs) and restore it.
+- **Local-first protects the graph.** It lives in local SQLite on your machine.
 
 For the deeper design, see [ARCHITECTURE.md](ARCHITECTURE.md) and
 [docs/architecture.md](docs/architecture.md).
@@ -247,16 +282,19 @@ For the deeper design, see [ARCHITECTURE.md](ARCHITECTURE.md) and
 ### Releases
 
 - [RELEASE_NOTES.md](RELEASE_NOTES.md) — current release notes
+- [RELEASE_NOTES_v3.6.0.md](RELEASE_NOTES_v3.6.0.md)
+- [RELEASE_NOTES_v3.5.0.md](RELEASE_NOTES_v3.5.0.md)
 - [RELEASE_NOTES_v3.4.1.md](RELEASE_NOTES_v3.4.1.md)
 - [RELEASE_NOTES_v3.4.0.md](RELEASE_NOTES_v3.4.0.md)
-- [RELEASE_NOTES_v3.3.0.md](RELEASE_NOTES_v3.3.0.md)
 - [CHANGELOG.md](CHANGELOG.md) and [docs/CHANGELOG.md](docs/CHANGELOG.md)
 
 ## Release History
 
 | Version | Theme |
 | --- | --- |
-| **3.4.1** | Runtime completion — full hooks lifecycle, real Local Agent probes, Connect Folder and Folder Watch verified end-to-end |
+| **3.6.0** | Knowledge Graph First — unified ingestion pipeline, formalized entity/relationship model, browser/web ingestion, local export/import/backup, provenance, KG as the primary surface |
+| 3.5.0 | Foundation stabilization & verification — OIDC verifier, trusted-proxy gating, runtime hook coverage, `tools/` package, reproducible artifacts |
+| 3.4.1 | Runtime completion — full hooks lifecycle, real Local Agent probes, Connect Folder and Folder Watch verified end-to-end |
 | 3.4.0 | Platform completion — hooks execution, uploads in Files, vision image input, agent run trigger, on-device Local Agent / Connect Folder / Folder Watch |
 | 3.3.1 | Visual product rebuild — rebuilt `/app` shell, Basic/Advanced/Admin navigation, refreshed design system |
 | **3.3.0** | Product quality & honesty release — evidence-based feature audit, single-source version truth, working document upload, documented design system |

package/docs/CARRYOVER_AUDIT_v3.6.0.md

@@ -0,0 +1,61 @@
+# v3.5.0 → v3.6.0 Carry-Over Audit
+
+**Date:** 2026-06-09
+**Baseline:** v3.5.0 (tag published, GitHub Release live, CI + Visual Smoke green on `main`)
+**Purpose:** Classify every open v3.5.0 carry-over risk as **blocking**, **non-blocking**, or
+**obsolete** before starting v3.6.0 Knowledge Graph First work.
+
+## Headline
+
+**No carry-over item blocks v3.6.0.** v3.5.0 was a stabilization/verification release that added no
+product surface. Every documented limitation is either an intentional, honestly-labeled scope
+boundary or a closed issue. KG work can start immediately on the existing local SQLite store.
+
+## Verified baseline state
+
+| Check | Result | Evidence |
+|---|---|---|
+| `v3.5.0` git tag | exists | `git tag` |
+| GitHub Release `v3.5.0` | published, not draft | `gh release view v3.5.0` |
+| CI on `main` | success | `gh run list --branch main` (run 27155690240) |
+| Visual Smoke on `main` | success | run 27155690270 |
+| VSIX reproducibility fix | merged | commit `78deb95` |
+
+## Classified carry-over items
+
+### Blocking
+*(none)*
+
+### Non-blocking
+
+| Item | Why it does not block | Evidence |
+|---|---|---|
+| OIDC is RSA-only (RS256/384/512); ES*/HS*/`alg:none` rejected fail-closed | v3.6.0 KG work does not touch the SSO callback. EC support is additive when a provider needs it. | `latticeai/core/oidc.py:36` |
+| Memory/KG maintenance endpoints (`/api/memory/{prune,compact,rebuild,clear}`) sit outside `pre_tool`/`post_tool` | Intentional, documented decision; these ops carry their own audit events. v3.6.0 follows the same convention and adds a coverage row rather than routing maintenance through `dispatch_tool`. | `docs/RUNTIME_HOOK_COVERAGE_v3.5.0.md:47` |
+| Knowledge Graph is config-dependent on `LATTICEAI_ENABLE_GRAPH`, backed by a large `knowledge_graph.py` | This is the surface v3.6.0 builds on. Size/coupling is refactor context, addressed additively (new service modules, not a rewrite). | `FEATURE_STATUS.md:257-263` |
+| Memory project/graph/vector tiers PARTIAL; prune/clear API-only (no UI) | KG-adjacent tiers v3.6.0 completes; honestly labeled, not broken. v3.6.0 adds the ingestion/export/provenance UI. | `FEATURE_STATUS.md:280-286` |
+| Hybrid-search "fusion" explainer renders illustrative bars | Cosmetic placeholder; orthogonal to KG-first ingestion work. | `FEATURE_STATUS.md:249-251` |
+| Chat grounding chips set state but `ChatRequest` drops them | Wiring grounding into generation is a natural KG-first feature; additive, not a blocker. | `FEATURE_STATUS.md:131-138` |
+| CI action versions inconsistent (`ci.yml` `checkout@v4` vs `release.yml` `@v5`) | None are currently deprecated. Aligning is hygiene; addressed opportunistically. | `.github/workflows/ci.yml` |
+| Live MCP tool calls + VLM inference PARTIAL (env/model dependent) | Honestly badged; orthogonal to KG-first work. | `FEATURE_STATUS.md:169-178,392` |
+
+### Obsolete
+
+| Item | Why it is closed |
+|---|---|
+| "Hooks registered but not executing" (v3.3.0 issue) | v3.4.1 added real runners for all built-ins; v3.5.0 closed the last tool-path bypasses at 100% coverage. |
+| Legacy `/account` `/admin` glassmorphism | Blur removed in v3.5.0 (`account.css:120`, `admin.css:48`). Remaining "not restyled" note is a deliberate scope boundary — these pages are outside the v3 SPA view set. |
+| Vercel deployment returns HTTP 500 | Settled posture: Vercel is **landing/download/demo only, never runtime**. Lattice AI is local-first; the KG runs on local SQLite. Do not present any Vercel URL as a product surface. |
+| CI syntax-gate staleness | Fixed in v3.5.0 by `scripts/check_python.py` (discover-based via `rglob`); new v3.6.0 modules are covered automatically with zero maintenance. |
+
+## v3.6.0 posture decisions taken from this audit
+
+1. **Build additively on the existing store.** Mirror the proven `tools.py → tools/` decomposition
+   pattern: preserve the import surface, add focused modules (`services/ingestion.py`,
+   `services/kg_portability.py`, `api/browser.py`, `api/portability.py`) rather than rewriting
+   `knowledge_graph.py`.
+2. **Route new ingestion paths through `dispatch_tool`** so `pre_tool`/`post_tool` fire — this is the
+   one v3.5.0 gap (no ingest path fired hooks). Maintenance ops stay audit-only by the documented
+   convention.
+3. **Keep Vercel landing-only and OIDC RSA-only** — settled, out of v3.6.0 scope.
+4. **Leave legacy `/account` `/admin` pages alone** — out of the local-first KG scope.

package/docs/CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## [3.6.0] - 2026-06-10
+
+> Knowledge Graph First. The Knowledge Graph becomes the primary architecture:
+> every data source converges into it through one unified ingestion pipeline, with
+> formalized entities/relationships, browser/web inputs, local portability, and
+> per-node provenance. Lattice AI is a Digital Brain Platform — the graph is the
+> durable asset; models read it and are replaceable.
+
+### Added
+
+- Unified ingestion pipeline (`latticeai/services/ingestion.py`): one entrypoint
+  for files, folders, web URLs, browser tabs, and text — idempotent by content
+  hash, bracketed by `pre_tool`/`post_tool`.
+- Knowledge Graph entities `Source`/`Repository`/`Meeting`/`Organization`/
+  `Workflow`/`Agent` and relationships `indexed_from`/`modified_by`/
+  `belongs_to_project`/`part_of`/`discussed_in`/`decided_by`/`generated_by`/
+  `used_by_agent` (additive, lossless `from_legacy`).
+- Browser & web ingestion routes (`/api/browser/read-url`, `/ingest-current-tab`)
+  and a Manifest V3 extension scaffold that posts only to `127.0.0.1`.
+- Knowledge Graph export/import (versioned JSON) and binary backup/restore
+  (`latticeai/services/kg_portability.py`,
+  `/api/knowledge-graph/{export,import,backup,restore,portability,provenance}`).
+- Provenance trail (`ingestion_provenance` table + query API) — every node is
+  explainable.
+- Knowledge Graph UI tabs: Status, Sources, Capture, Backup.
+
+### Changed
+
+- KG ingestion now fires the tool hook lifecycle (closes the v3.5.0 gap);
+  coverage documented in `docs/RUNTIME_HOOK_COVERAGE_v3.6.0.md`.
+- README repositioned as a Digital Brain Platform; Vercel remains landing-only.
+
 ## [3.3.1] - 2026-06-08
 
 > v3.3.1 — Visual Product Rebuild. The `/app` frontend keeps the same runtime

package/docs/HANDOVER_v3.6.0.md

@@ -0,0 +1,46 @@
+# Lattice AI v3.6.0 — Completion Record (Knowledge Graph First)
+
+**Status: ✅ RELEASED — 2026-06-10.** All planned scopes implemented, tested,
+committed, pushed, CI green, tagged, and published. No external publish/deploy.
+
+- **Tag:** `v3.6.0` → commit `3c85675`
+- **GitHub Release:** https://github.com/TaeSooPark-PTS/LatticeAI/releases/tag/v3.6.0
+  (published, not draft; assets: `ltcai-3.6.0-py3-none-any.whl`,
+  `ltcai-3.6.0.tar.gz`, `ltcai-3.6.0.tgz`, `ltcai-3.6.0.vsix`)
+- **CI:** main CI ✓, Visual Smoke ✓, release.yml (tag) ✓
+- **Tests:** unit 455 passing · lint 64/64 · check:python 153 (3.11/3.12/3.14) ·
+  release artifacts built + validated
+
+## Commits (v3.5.0 → v3.6.0)
+
+| Commit | Scope |
+|---|---|
+| `baa2bf6` | chore(audit) — v3.5.0 carry-over (0 blocking) |
+| `5a6a7d4` | feat(kg) — entities/relationships schema |
+| `135e81a` | feat(kg) — unified ingestion pipeline + provenance |
+| `b548885` | feat(browser) — browser/web ingestion + MV3 extension |
+| `39a7a0c` | feat(kg) — export/import/backup/restore |
+| `21cfb97` | fix(runtime) — hook coverage for ingestion paths |
+| `7009e39` | fix(ui) — Knowledge Graph as primary surface |
+| `fa89a84` | docs(philosophy) — Digital Brain Platform rewrite |
+| `aa011a5` | release: v3.6.0 (version bump) |
+| `3c85675` | fix(ci) — 3.11-compatible f-string (PEP 701 quote reuse) |
+
+## Carry-over audit result
+
+Zero blocking items. Settled postures preserved: Vercel landing-only, OIDC
+RSA-only, legacy `/account` `/admin` out of scope. The one honest v3.5.0 gap (KG
+ingestion not firing tool hooks) is **closed**. Full detail:
+`docs/CARRYOVER_AUDIT_v3.6.0.md`.
+
+## Key facts for future work
+
+- New seams: `latticeai/services/ingestion.py` (single write-side entrypoint),
+  `latticeai/services/kg_portability.py`, `latticeai/api/browser.py`,
+  `latticeai/api/portability.py`, provenance in `knowledge_graph.py`.
+- **Gotcha:** PEP 701 f-string quote reuse (`f'{x or ''}'`) compiles on 3.12+ but
+  is a SyntaxError on 3.11 — always run `python3.11 scripts/check_python.py`
+  before pushing (CI tests on 3.11 + 3.12).
+- Version canonical: `WORKSPACE_OS_VERSION`; mirrors enforced by
+  `test_version_consistency.py`.
+- Local tests need `.venv/bin/python` (system `python3` lacks fastapi).

package/docs/RUNTIME_HOOK_COVERAGE_v3.6.0.md

@@ -0,0 +1,49 @@
+# Runtime Hook Coverage — v3.6.0
+
+v3.6.0 makes the Knowledge Graph the primary architecture and adds new
+**ingestion** paths (web URL, browser tab, unified text/file pipeline) plus
+**portability** ops (export/import/backup/restore). This doc extends
+[`RUNTIME_HOOK_COVERAGE_v3.5.0.md`](./RUNTIME_HOOK_COVERAGE_v3.5.0.md) and records
+that the new data-mutating paths run through the unified lifecycle.
+
+The single tool path is `dispatch_tool(hooks, name, args, run_fn)` in
+`latticeai/core/hooks.py` (`pre_tool → execute → post_tool`). v3.5.0's one honest
+gap was that **KG ingestion did not fire hooks**. v3.6.0 closes it: every source
+now flows through `IngestionPipeline.ingest` (`latticeai/services/ingestion.py`),
+which wraps the store write in `dispatch_tool(..., source="ingestion")`.
+
+**Result.** All v3.5.0 coverage is preserved (no regression), and every new
+v3.6.0 ingestion path is covered. Portability admin/maintenance ops follow the
+v3.5.0 convention for service maintenance (own audit events), documented below.
+
+## New v3.6.0 execution paths
+
+| Entrypoint | Execution | Lifecycle path | pre fired | post fired | Test |
+|---|---|---|---|---|---|
+| `IngestionPipeline.ingest` (any source) | `ingest_source` / `ingest_document` | `dispatch_tool(name="kg_ingest.<type>", source="ingestion")` | yes (`pre_tool`) | yes (`post_tool`) | `test_ingestion_pipeline` |
+| `POST /api/browser/read-url` | fetch URL → `pipeline.ingest` (web_url) | pipeline → `dispatch_tool` | yes | yes | `test_browser_ingestion`, `test_runtime_coverage_v36` |
+| `POST /api/browser/ingest-current-tab` | sanitize → `pipeline.ingest` (browser_tab) | pipeline → `dispatch_tool` | yes | yes | `test_browser_ingestion`, `test_runtime_coverage_v36` |
+| Local file / upload via pipeline | `ingest_document` | pipeline → `dispatch_tool` | yes | yes | `test_ingestion_pipeline` |
+| Provenance write per ingestion | `record_provenance` | inside the bracketed `dispatch_tool` run_fn | (bracketed) | (bracketed) | `test_ingestion_pipeline` |
+
+A blocking `pre_tool` hook makes ingestion return `status="blocked"` (the
+`PermissionError` from `dispatch_tool` is caught and surfaced honestly), exactly
+mirroring how a blocked tool call is handled — verified in
+`test_runtime_coverage_v36` and `test_ingestion_pipeline`.
+
+## Intentionally outside the tool lifecycle (documented, not gaps)
+
+| Entrypoint | Why not `pre_tool`/`post_tool` |
+|---|---|
+| `POST /api/knowledge-graph/{export,export-file,backup,restore,import}` | Admin **portability/maintenance** operations over the whole machine-global graph, not agent-vocabulary tools. They are admin-gated (`require_admin`) and recorded via the platform audit trail — same convention as the v3.5.0 memory maintenance ops (`prune/compact/rebuild/clear`). Wrapping a whole-store backup in `pre_tool` would misrepresent it as a per-action agent tool. |
+| `read_document` inside upload (`upload_service.py`) | Already inside the `pre_upload`→`post_upload` lifecycle (unchanged from v3.5.0). |
+| `POST /api/memory/{prune,compact,rebuild,clear}` | Unchanged from v3.5.0 — service maintenance with own audit events. |
+| Read-only KG reads (`/knowledge-graph/{stats,graph,search,...}`, `/api/knowledge-graph/portability` status) | Execute no mutation — nothing to gate. |
+
+## Summary
+
+- v3.5.0 coverage of tool/agent execution paths: **100%, preserved** (no regression).
+- v3.6.0 gap closed: **KG ingestion now fires `pre_tool`/`post_tool`** via the unified pipeline (the one honest carry-over note from v3.5.0).
+- New ingestion paths (unified pipeline, `read-url`, `ingest-current-tab`): **covered**.
+- Portability admin ops: **documented as audit-gated maintenance**, consistent with the v3.5.0 convention — not bypasses.
+- Coverage of discovered mutating ingestion paths: **100%**.