ltcai 9.6.0 → 9.8.0

package/docs/PERFORMANCE.md

@@ -0,0 +1,106 @@
+# Knowledge Graph Performance Baseline
+
+Synthetic performance and memory baseline for `KnowledgeGraphStore`
+(`lattice_brain/graph/`). Measured with `scripts/profile_kg.py`.
+
+**This is a synthetic baseline, not a real-workload benchmark.** Use it for
+release-over-release regression comparison, not as a marketing number.
+
+## Methodology
+
+- `scripts/profile_kg.py` builds a throwaway store the same way the unit tests
+  do (`KnowledgeGraphStore(tmp/graph.sqlite, tmp/blobs)`) and ingests N
+  synthetic text sources through the real `ingest_source()` pipeline
+  (chunking, rule-based concept/triple extraction, provenance edges).
+- Corpus: fixed-vocabulary English + Korean text, seeded RNG (`--seed`,
+  default 42), ~60% short (~40 words) / 30% medium (~160) / 10% long (~520)
+  documents with recurring entities so concept nodes are shared and edges form.
+- Fully offline: no LLM router is registered (extraction takes the rules
+  path) and the embedder is the built-in deterministic hash model. No network,
+  no model downloads.
+- Timing: `time.perf_counter()` per call; p50/p95/mean over all calls in a
+  phase. Memory: `tracemalloc` peak, reset per phase.
+- Caveat: tracemalloc runs for the whole profile and roughly doubles Python
+  allocation cost, so absolute timings are conservative upper bounds.
+- Caveat: the synthetic vocabulary is small, so FTS selectivity and concept
+  density differ from real user data; every tenth query is a guaranteed miss
+  to avoid measuring only warm hits.
+
+Run it:
+
+```bash
+.venv/bin/python scripts/profile_kg.py                 # 5000 sources, 50 queries
+.venv/bin/python scripts/profile_kg.py --nodes 500     # quick run (~10 s)
+.venv/bin/python scripts/profile_kg.py --json          # machine-readable
+```
+
+## Measured baseline (2026-07-20, v9.6.0 working tree)
+
+Apple Silicon (darwin), Python 3.x from `.venv`, SQLite with FTS5 trigram
+available, tracemalloc enabled.
+
+### 500 sources (quick profile, ~9 s total)
+
+Graph produced: 2,019 nodes / 6,546 edges, 21.2 MB SQLite file.
+
+| phase                | calls | p50 ms | p95 ms | wall s | peak MB |
+|----------------------|------:|-------:|-------:|-------:|--------:|
+| ingest_source        |   500 |   6.9  |  17.7  |   4.1  |   0.6   |
+| search               |    50 |   1.2  |   2.5  |   0.07 |   0.6   |
+| context_for_query    |    50 |   1.0  |   2.4  |   0.06 |   0.5   |
+| neighbors            |    50 |   0.35 |   0.45 |   0.02 |   0.6   |
+| traverse (depth 2)   |    50 |   4.8  |   5.2  |   0.24 |   1.1   |
+| stats                |     5 |   1.5  |   1.7  |   0.01 |   0.4   |
+| rebuild_vector_index |     1 |  1319  |  1319  |   1.3  |   5.0   |
+| vector_search        |    10 |  287   |  294   |   2.9  |  14.4   |
+
+Ingestion throughput: **~122 items/sec** (412 KB corpus).
+
+### 5000 sources (default profile, ~2.5 min total)
+
+Graph produced: 18,426 nodes / 62,534 edges, 198.8 MB SQLite file
+(4.26 MB corpus). Measured while other processes shared the CPU, so treat as
+an upper bound.
+
+| phase                | calls | p50 ms | p95 ms | wall s | peak MB |
+|----------------------|------:|-------:|-------:|-------:|--------:|
+| ingest_source        |  5000 |  20.1  |  54.6  | 119.9  |   1.4   |
+| search               |    50 |   4.3  |  19.8  |   0.40 |   1.2   |
+| context_for_query    |    50 |   4.1  |  18.5  |   0.40 |   1.2   |
+| neighbors            |    50 |   0.39 |   0.50 |   0.03 |   1.2   |
+| traverse (depth 2)   |    50 |  12.2  |  15.8  |   0.64 |   1.7   |
+| stats                |     5 |  13.1  |  13.5  |   0.07 |   1.1   |
+| rebuild_vector_index |     1 | 14624  | 14624  |  14.6  |  45.6   |
+| vector_search        |    10 |  1732  |  1774  |  17.4  |  83.5   |
+
+Ingestion throughput: **~42 items/sec** at this scale.
+
+Scaling signal (500 → 5000 sources, 10x): per-item ingest p50 grew ~3x
+(6.9 ms → 20.1 ms) — ingestion cost is superlinear in DB size (FTS triggers,
+dedup lookups, edge upserts against growing tables). `vector_search` p50 grew
+~6x (287 ms → 1732 ms), consistent with its O(index size) brute-force scan.
+Keyword search p50 grew ~3.6x but stays under 5 ms.
+
+Regenerate with `.venv/bin/python scripts/profile_kg.py` after graph-layer
+changes and compare against these tables.
+
+## Observations
+
+- Keyword `search()` / `context_for_query()` stay low-millisecond thanks to
+  the FTS5 trigram index; they are not the scaling bottleneck.
+- `traverse(depth=2)` cost grows with edge fan-out (synthetic corpus creates
+  dense shared-concept hubs); p95 is ~5 ms at 500 sources and ~16 ms at 5000.
+- `vector_search()` is a brute-force scan over every stored embedding
+  (O(index size) per query). It is the dominant cost at scale and the first
+  candidate for an ANN/pruning optimization if vector recall becomes a hot
+  path. The profiler caps vector queries at 10 for this reason.
+- `rebuild_vector_index(full=True)` embeds documents and chunks with the hash
+  embedder; with a real embedding provider expect this phase to be slower by
+  the provider's per-call latency times the item count.
+
+## Regression workflow
+
+1. Before a graph-layer change: `.venv/bin/python scripts/profile_kg.py --nodes 500 --json > /tmp/kg-before.json`
+2. After the change: same command to `/tmp/kg-after.json`.
+3. Compare phase p50/p95 and `db_size_mb`; investigate anything that moved
+   more than ~20% beyond run-to-run noise.

package/docs/TRUST_MODEL.md

@@ -1,6 +1,6 @@
 # Lattice AI Trust Model
 
-Current release: **9.6.0 — Trusted Agent Loop**.
+Current release: **9.8.0 — Honest Knowledge Pipeline**.
 
 Lattice AI is local-first, explicit about external communication, and honest
 when a capability is unavailable.

package/docs/WHY_LATTICE.md

@@ -1,6 +1,6 @@
 # Why Lattice AI Exists
 
-Current release: **9.6.0 — Trusted Agent Loop**.
+Current release: **9.8.0 — Honest Knowledge Pipeline**.
 
 **Lattice AI is a local-first Digital Brain that keeps your knowledge durable
 across any AI model.**

package/docs/kg-schema.md

@@ -1,6 +1,6 @@
 # Knowledge Graph Schema
 
-Current release: **9.6.0 — Trusted Agent Loop**.
+Current release: **9.8.0 — Honest Knowledge Pipeline**.
 
 명세 출처: `lattice_ai_full_spec.pptx` 슬라이드 20·21·22
 구현: `kg_schema.py`

package/kg_schema.py

@@ -1,3 +1,13 @@
 """Compatibility shim for the v4 Knowledge Graph schema."""
 
-from lattice_brain.graph.schema import *  # noqa: F403,F401
+import warnings as _warnings
+
+_warnings.warn(
+    "Importing 'kg_schema' from the repository root is deprecated; "
+    "use 'from lattice_brain.graph.schema import ...' instead. "
+    "The root shim will be removed in a future major release.",
+    DeprecationWarning,
+    stacklevel=2,
+)
+
+from lattice_brain.graph.schema import *  # noqa: F403,F401,E402

package/knowledge_graph.py

@@ -4,7 +4,17 @@ The implementation now lives under :mod:`lattice_brain`. Root imports are
 kept for older integrations and tests.
 """
 
-from lattice_brain.graph._kg_common import (  # noqa: F401
+import warnings as _warnings
+
+_warnings.warn(
+    "Importing 'knowledge_graph' from the repository root is deprecated; "
+    "use 'from lattice_brain.graph.store import KnowledgeGraphStore' instead. "
+    "The root shim will be removed in a future major release.",
+    DeprecationWarning,
+    stacklevel=2,
+)
+
+from lattice_brain.graph._kg_common import (  # noqa: F401,E402
     EDGE_VERB,
     GRAPH_SCHEMA_VERSION,
     LOCAL_CODE_EXTENSIONS,
@@ -24,7 +34,7 @@ from lattice_brain.graph._kg_common import (  # noqa: F401
     _slug,
     set_llm_router,
 )
-from lattice_brain.graph.store import KnowledgeGraphStore
+from lattice_brain.graph.store import KnowledgeGraphStore  # noqa: E402
 
 __all__ = [
     "KnowledgeGraphStore",

package/knowledge_graph_api.py

@@ -6,7 +6,17 @@ importable for the deprecation window and will be removed in a future major
 release.
 """
 
-from latticeai.api.knowledge_graph import (  # noqa: F401
+import warnings as _warnings
+
+_warnings.warn(
+    "Importing 'knowledge_graph_api' from the repository root is deprecated; "
+    "use 'from latticeai.api.knowledge_graph import create_knowledge_graph_router' instead. "
+    "The root shim will be removed in a future major release.",
+    DeprecationWarning,
+    stacklevel=2,
+)
+
+from latticeai.api.knowledge_graph import (  # noqa: F401,E402
     KnowledgeGraphIngestRequest,
     create_knowledge_graph_router,
 )

package/lattice_brain/__init__.py

@@ -26,7 +26,7 @@ from .storage import (
     storage_from_env,
 )
 
-__version__ = "9.6.0"
+__version__ = "9.8.0"
 
 __all__ = [
     "AgentRuntime",