memory-kg 0.5.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. memory_kg-0.5.2/LICENSE +94 -0
  2. memory_kg-0.5.2/PKG-INFO +304 -0
  3. memory_kg-0.5.2/README.md +245 -0
  4. memory_kg-0.5.2/pyproject.toml +221 -0
  5. memory_kg-0.5.2/src/memory_kg/__init__.py +38 -0
  6. memory_kg-0.5.2/src/memory_kg/__main__.py +6 -0
  7. memory_kg-0.5.2/src/memory_kg/app.py +433 -0
  8. memory_kg-0.5.2/src/memory_kg/chunker.py +630 -0
  9. memory_kg-0.5.2/src/memory_kg/cli/__init__.py +0 -0
  10. memory_kg-0.5.2/src/memory_kg/cli/cmd_analyze.py +69 -0
  11. memory_kg-0.5.2/src/memory_kg/cli/cmd_build.py +493 -0
  12. memory_kg-0.5.2/src/memory_kg/cli/cmd_hooks.py +364 -0
  13. memory_kg-0.5.2/src/memory_kg/cli/cmd_mcp.py +69 -0
  14. memory_kg-0.5.2/src/memory_kg/cli/cmd_model.py +54 -0
  15. memory_kg-0.5.2/src/memory_kg/cli/cmd_pipeline.py +352 -0
  16. memory_kg-0.5.2/src/memory_kg/cli/cmd_query.py +187 -0
  17. memory_kg-0.5.2/src/memory_kg/cli/cmd_semantic_analyze.py +69 -0
  18. memory_kg-0.5.2/src/memory_kg/cli/cmd_snapshot.py +393 -0
  19. memory_kg-0.5.2/src/memory_kg/cli/cmd_viz.py +76 -0
  20. memory_kg-0.5.2/src/memory_kg/cli/group.py +22 -0
  21. memory_kg-0.5.2/src/memory_kg/cli/main.py +39 -0
  22. memory_kg-0.5.2/src/memory_kg/cli/options.py +50 -0
  23. memory_kg-0.5.2/src/memory_kg/config.py +44 -0
  24. memory_kg-0.5.2/src/memory_kg/doc_kg.code-workspace +8 -0
  25. memory_kg-0.5.2/src/memory_kg/embedder_worker.py +273 -0
  26. memory_kg-0.5.2/src/memory_kg/entry_chunk.py +117 -0
  27. memory_kg-0.5.2/src/memory_kg/graph.py +171 -0
  28. memory_kg-0.5.2/src/memory_kg/index.py +671 -0
  29. memory_kg-0.5.2/src/memory_kg/kg.py +760 -0
  30. memory_kg-0.5.2/src/memory_kg/manifold.py +284 -0
  31. memory_kg-0.5.2/src/memory_kg/mcp_server.py +170 -0
  32. memory_kg-0.5.2/src/memory_kg/memorykg.py +645 -0
  33. memory_kg-0.5.2/src/memory_kg/memorykg_semantic_analysis.py +841 -0
  34. memory_kg-0.5.2/src/memory_kg/memorykg_thorough_analysis.py +416 -0
  35. memory_kg-0.5.2/src/memory_kg/pipeline.py +470 -0
  36. memory_kg-0.5.2/src/memory_kg/relations.py +128 -0
  37. memory_kg-0.5.2/src/memory_kg/sampler.py +340 -0
  38. memory_kg-0.5.2/src/memory_kg/semantic_builder.py +261 -0
  39. memory_kg-0.5.2/src/memory_kg/semantic_extractor.py +246 -0
  40. memory_kg-0.5.2/src/memory_kg/semantic_primitives.py +104 -0
  41. memory_kg-0.5.2/src/memory_kg/snapshots.py +483 -0
  42. memory_kg-0.5.2/src/memory_kg/store.py +454 -0
  43. memory_kg-0.5.2/src/memory_kg/topics.py +335 -0
@@ -0,0 +1,94 @@
1
+ Elastic License 2.0
2
+
3
+ URL: https://www.elastic.co/licensing/elastic-license
4
+
5
+ ## Acceptance
6
+
7
+ By using the software, you agree to all of the terms and conditions below.
8
+
9
+ ## Copyright License
10
+
11
+ The licensor grants you a non-exclusive, royalty-free, worldwide,
12
+ non-sublicensable, non-transferable license to use, copy, distribute, make
13
+ available, and prepare derivative works of the software, in each case subject to
14
+ the limitations and conditions below.
15
+
16
+ ## Limitations
17
+
18
+ **You may not provide the software to third parties as a hosted or managed
19
+ service, where the service provides users with access to any substantial set of
20
+ the features or functionality of the software.**
21
+
22
+ You may not move, change, disable, or circumvent the license key functionality
23
+ in the software, and you may not remove or obscure any functionality in the
24
+ software that is protected by the license key.
25
+
26
+ You may not alter, remove, or obscure any licensing, copyright, or other notices
27
+ of the licensor in the software. Any use of the licensor's trademarks is subject
28
+ to applicable law.
29
+
30
+ ## Patents
31
+
32
+ The licensor grants you a license, under any patent claims the licensor can
33
+ license, or becomes able to license, to make, have made, use, sell, offer for
34
+ sale, import and have imported the software, in each case subject to the
35
+ limitations and conditions in this license. This license does not cover any
36
+ patent claims that you cause to be infringed by modifications or additions to the
37
+ software. If you or your company make any written claim that the software
38
+ infringes or contributes to infringement of any patent, your patent license for
39
+ the software granted under these terms ends immediately. If your company makes
40
+ such a claim, your patent license ends immediately for work on behalf of your
41
+ company.
42
+
43
+ ## Notices
44
+
45
+ You must ensure that anyone who gets a copy of any part of the software from you
46
+ also gets a copy of these terms or the URL for them above, as well as copies of
47
+ any plain-text lines beginning with "Required Notice:" that the licensor provided
48
+ with the software. For example:
49
+
50
+ Required Notice: Copyright (c) 2026 Eric G. Suchanek, PhD
51
+
52
+ ## No Other Rights
53
+
54
+ These terms do not imply any other licenses not expressly granted in this
55
+ license.
56
+
57
+ ## Termination
58
+
59
+ If you use the software in violation of these terms, such use is not licensed,
60
+ and your licenses will automatically terminate. If the licensor provides you with
61
+ a notice of your violation, and you cease all violation of this license no later
62
+ than 30 days after you receive that notice, your licenses will be reinstated
63
+ retroactively. However, if you violate these terms after such reinstatement, any
64
+ additional violation of these terms will cause your licenses to terminate
65
+ automatically and permanently.
66
+
67
+ ## No Liability
68
+
69
+ *As far as the law allows, the software comes as is, without any warranty or
70
+ condition, and the licensor will not be liable to you for any damages arising out
71
+ of these terms or the use or nature of the software, under any kind of legal
72
+ claim.*
73
+
74
+ ## Definitions
75
+
76
+ The **licensor** is the entity offering these terms, and the **software** is the
77
+ software the licensor makes available under these terms, including any portion of
78
+ it.
79
+
80
+ **You** refers to the individual or entity agreeing to these terms.
81
+
82
+ **Your company** is any legal entity, sole proprietorship, or other kind of
83
+ organization that you work for, plus all organizations that have control over,
84
+ are under the control of, or are under common control with that organization.
85
+ Control means ownership of substantially all the assets of an entity, or the
86
+ power to direct its management and policies by vote, contract, or otherwise.
87
+ Control can be direct or indirect.
88
+
89
+ **Your licenses** are all the licenses granted to you for the software under
90
+ these terms.
91
+
92
+ **Use** means anything you do with the software requiring one of your licenses.
93
+
94
+ **Trademark** means trademarks, service marks, and similar rights.
@@ -0,0 +1,304 @@
1
+ Metadata-Version: 2.4
2
+ Name: memory-kg
3
+ Version: 0.5.2
4
+ Summary: A tool to build a semantically searchable knowledge graph from memories
5
+ License-Expression: Elastic-2.0
6
+ License-File: LICENSE
7
+ Keywords: knowledge-graph,document-analysis,markdown,lancedb,sqlite,semantic-search
8
+ Author: Eric G. Suchanek, PhD
9
+ Author-email: suchanek@flux-frontiers.com
10
+ Requires-Python: >=3.12,<3.14
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
14
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Provides-Extra: all
19
+ Provides-Extra: dev
20
+ Provides-Extra: viz
21
+ Requires-Dist: click (>=8.1.0,<9)
22
+ Requires-Dist: detect-secrets (>=1.5.0) ; extra == "all"
23
+ Requires-Dist: detect-secrets (>=1.5.0) ; extra == "dev"
24
+ Requires-Dist: einops (>=0.8.2)
25
+ Requires-Dist: kgmodule-utils (>=0.4.4)
26
+ Requires-Dist: lancedb (>=0.29.0)
27
+ Requires-Dist: markdown-it-py (>=3.0.0)
28
+ Requires-Dist: mcp (>=1.0.0)
29
+ Requires-Dist: numpy (>=1.24.0)
30
+ Requires-Dist: pandas (>=2.0.0)
31
+ Requires-Dist: pdoc (>=14.0.0) ; extra == "all"
32
+ Requires-Dist: pdoc (>=14.0.0) ; extra == "dev"
33
+ Requires-Dist: plotly (>=5.14.0) ; extra == "all"
34
+ Requires-Dist: plotly (>=5.14.0) ; extra == "viz"
35
+ Requires-Dist: pre-commit (>=4.5.1) ; extra == "all"
36
+ Requires-Dist: pre-commit (>=4.5.1) ; extra == "dev"
37
+ Requires-Dist: pytest (>=8.0.0) ; extra == "all"
38
+ Requires-Dist: pytest (>=8.0.0) ; extra == "dev"
39
+ Requires-Dist: pytest-cov (>=5.0.0) ; extra == "all"
40
+ Requires-Dist: pytest-cov (>=5.0.0) ; extra == "dev"
41
+ Requires-Dist: pyvis (>=0.3.2) ; extra == "all"
42
+ Requires-Dist: pyvis (>=0.3.2) ; extra == "viz"
43
+ Requires-Dist: pyyaml (>=6.0.0)
44
+ Requires-Dist: rich (>=14.3.3,<15)
45
+ Requires-Dist: ruff (>=0.4.0) ; extra == "all"
46
+ Requires-Dist: ruff (>=0.4.0) ; extra == "dev"
47
+ Requires-Dist: safetensors (>=0.5.0)
48
+ Requires-Dist: sentence-transformers (>=5.4.1)
49
+ Requires-Dist: streamlit (>=1.35.0) ; extra == "all"
50
+ Requires-Dist: streamlit (>=1.35.0) ; extra == "viz"
51
+ Requires-Dist: torch (>=2.5.1)
52
+ Requires-Dist: transformers (>=4.40.0,<4.57)
53
+ Requires-Dist: ty (>=0.0.41) ; extra == "all"
54
+ Requires-Dist: ty (>=0.0.41) ; extra == "dev"
55
+ Project-URL: Homepage, https://github.com/Flux-Frontiers/memory_kg
56
+ Project-URL: Repository, https://github.com/Flux-Frontiers/memory_kg
57
+ Description-Content-Type: text/markdown
58
+
59
+ [![CI](https://github.com/Flux-Frontiers/memory_kg/actions/workflows/publish.yml/badge.svg)](https://github.com/Flux-Frontiers/memory_kg/actions/workflows/publish.yml)
60
+ [![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue.svg)](https://www.python.org/)
61
+ [![License: Elastic-2.0](https://img.shields.io/badge/License-Elastic%202.0-blue.svg)](https://www.elastic.co/licensing/elastic-license)
62
+ [![Version](https://img.shields.io/badge/version-0.5.2-blue.svg)](https://github.com/Flux-Frontiers/memory_kg/releases)
63
+ [![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
64
+ [![DOI](https://zenodo.org/badge/1205364687.svg)](https://zenodo.org/badge/latestdoi/1205364687)
65
+
66
+ **MemoryKG** — A Hybrid Knowledge Graph for Conversational Memory and Document Corpora
67
+
68
+ *Author: Eric G. Suchanek, PhD — Flux-Frontiers, Liberty TWP, OH*
69
+
70
+ ---
71
+
72
+ ## TL;DR
73
+
74
+ MemoryKG achieves **100% retrieval recall on the ConvoMem benchmark — every evidence message found, on every question, across 17,463 items** spanning six evidence categories and four evidence tiers (1–4 messages). No LLM, no API key, no cloud inference at any stage. This is the largest non-LLM evaluation on ConvoMem reported. Full write-up: [`benchmarks/convomem/convomem_article.pdf`](benchmarks/convomem/convomem_article.pdf).
75
+
76
+ Recall is measured by substring containment in the top-10 retrieved nodes: an evidence message counts as found if its text appears verbatim in (or contains) any retrieved node — lenient toward retrieval, but it cannot be fooled by paraphrase.
77
+
78
+ On the LongMemEval-S benchmark, MemoryKG is **tied for the top LLM-free score** — 98.4% Recall@5, 99.4% Recall@10, 0.943 NDCG@10. It matches MemoryPalace's best LLM-free results (hybrid v4 held-out and hybrid v2) and beats every other LLM-free baseline. Three LLM-augmented systems rank higher at R@5 (MemoryPalace v4 + Haiku at 100%, MemoryPalace v3 + Haiku rerank at 99.4%, Supermemory ASMR at ~99%); MemoryKG narrows that gap without paying the inference cost. Full write-up: [`benchmarks/longmemeval/longmemeval_article.pdf`](benchmarks/longmemeval/longmemeval_article.pdf).
79
+
80
+ | System | LongMemEval R@5 | LLM at query time | Cost / query |
81
+ |---|--:|---|--:|
82
+ | MemoryPalace hybrid v4 + Haiku (500q) | 100% | Yes (Claude Haiku) | $$ |
83
+ | MemoryPalace hybrid v4 held-out (450q) | 98.4% | None | $0 |
84
+ | **MemoryKG (this work)** | **98.4%** | **None** | **$0** |
85
+ | MemoryPalace hybrid v3 + Haiku rerank | 99.4% | Yes (Claude Haiku) | $$ |
86
+ | Supermemory ASMR | ~99% | Yes (undisclosed) | $$ |
87
+ | MemoryPalace hybrid v2 | 98.4% | None | $0 |
88
+ | Mastra | 94.9% | Yes (GPT-5-mini) | $$ |
89
+ | MemoryPalace raw ChromaDB | 96.6% | None | $0 |
90
+ | Hindsight | 91.4% | Yes (Gemini-3) | $$ |
91
+ | Supermemory (production) | ~85% | Yes (undisclosed) | $$ |
92
+ | Stella (dense retriever) | ~85% | None | $0 |
93
+ | BM25 (sparse baseline) | ~70% | None | $0 |
94
+
95
+ With the sibling boost enabled on LongMemEval, **recall_all@10 reaches 98.6%** — meaning MemoryKG retrieves *every* required session for 493 of 500 questions without any LLM. No published system reports this metric; we track it because multi-session coverage is the real test of memory completeness.
96
+
97
+ The field has been over-engineering retrieval. A graph-augmented index with correct search-space scoping matches the best LLM-free result in the field at a fraction of the complexity.
98
+
99
+ ---
100
+
101
+ ## Why It Works
102
+
103
+ Most "memory" systems flatten a session into a single embedding and lean on an LLM at query time to rerank what they retrieve. MemoryKG does the opposite: it preserves session structure as a typed graph, then uses that structure as the ranking signal.
104
+
105
+ 1. **Finer granularity.** Sessions are chunked by heading, not embedded as 2,000-word blobs. A 150-word chunk about "Dr. Chen's appointment" is dramatically more discriminative than the session it lives in.
106
+ 2. **Structural expansion.** A `HAS_TOPIC` or `MENTIONS_ENTITY` edge from a weakly-matching chunk surfaces strongly-linked neighbors that pure cosine similarity never finds.
107
+ 3. **Score-first ranking.** Graph proximity breaks ties *within* a vector-quality band — never across one. Good seeds get amplified; bad seeds don't get rescued.
108
+ 4. **Kind-aware ranking.** Chunk matches outrank entity stubs outrank synthetic topic summaries. Flat vector stores treat every document equally.
109
+ 5. **Search-space scoping.** When the benchmark defines a per-question candidate pool, MemoryKG honours it (`haystack_files=...`). This was the +11 pp fix that narrowed the gap to the inference-based leaderboard.
110
+
111
+ **No LLM. No API key. No cloud round-trip. Runs on Apple Silicon (MPS), CUDA, or CPU.**
112
+
113
+ ---
114
+
115
+ ## What MemoryKG Is
116
+
117
+ A **deterministic, explainable knowledge graph** built from conversational logs and document corpora (Markdown, plain text). MemoryKG semantically chunks text, extracts topics/entities/keywords, links them through typed edges, stores everything in SQLite, and adds a LanceDB vector index as an *acceleration layer* — not the source of truth.
118
+
119
+ Structure is treated as ground truth. Semantic search is a tool, not the system. The result is a searchable, auditable representation that supports precise navigation, source-grounded passage extraction, and downstream LLM reasoning — a practical foundation for **Knowledge-Graph RAG (KGRAG)**.
120
+
121
+ MemoryKG shares its architecture with [PyCodeKG](https://github.com/Flux-Frontiers/code_kg) (Python codebases) and [DocKG](https://github.com/Flux-Frontiers/doc_kg) (general document corpora).
122
+
123
+ ---
124
+
125
+ ## Features
126
+
127
+ - **Semantic chunking** — Multiple strategies: `heading` (one chunk per `## Section`), `fixed`, `sentence_group`, `semantic` (embedding-boundary detection)
128
+ - **Deterministic knowledge graph** — SQLite-backed canonical store with typed nodes and provenance-tracked edges
129
+ - **Relation extraction** — Topics, named entities, keywords; co-occurrence and similarity edges built automatically
130
+ - **Hybrid query model** — Semantic seeding (LanceDB) + structural expansion (graph traversal) + score-first ranking
131
+ - **Haystack-scoped search** — Restrict vector seeding to a per-question candidate pool for benchmark-grade precision
132
+ - **Passage packing** — Source-grounded text passages with headings, ready to paste into an LLM prompt
133
+ - **Coverage analysis & temporal snapshots** — Per-document metrics, hot chunks, orphan detection, version-over-version diffs
134
+ - **Parallel ingestion** — `--workers N` parallel Phase 1 parsing for large corpora
135
+ - **MCP server** — Four tools for AI agent integration (`graph_stats`, `query_docs`, `pack_docs`, `get_node`)
136
+ - **Streamlit web app** — Interactive graph browser, hybrid query UI, and passage pack explorer
137
+
138
+ ---
139
+
140
+ ## Quick Start
141
+
142
+ ```bash
143
+ # Index a corpus (SQLite + LanceDB in one step; wipe is the default)
144
+ memorykg build --repo docs/
145
+
146
+ # Natural-language query — returns ranked chunks
147
+ memorykg query "authentication flow"
148
+
149
+ # Source-grounded passage pack — paste straight into an LLM prompt
150
+ memorykg pack "configuration reference" --fmt md --out context.md
151
+ ```
152
+
153
+ ---
154
+
155
+ ## Installation
156
+
157
+ ```bash
158
+ pip install 'memory-kg @ git+https://github.com/Flux-Frontiers/memory_kg.git'
159
+ ```
160
+
161
+ See [docs/installation.md](docs/installation.md) for editable installs, dev setup, and offline model caching.
162
+
163
+ ---
164
+
165
+ ## Usage
166
+
167
+ ### Build the knowledge graph
168
+
169
+ ```bash
170
+ # Full pipeline: parse → SQLite graph → LanceDB index (wipe is default)
171
+ memorykg build --repo docs/
172
+
173
+ # Granular steps for large corpora
174
+ memorykg build-graph --repo docs/ # SQLite only
175
+ memorykg build-index # LanceDB from existing SQLite
176
+
177
+ # Incremental update — keep existing data
178
+ memorykg build --repo docs/ --update
179
+
180
+ # Parallelise Phase 1 parsing
181
+ memorykg build --repo docs/ --workers 8
182
+
183
+ # Exclude directories
184
+ memorykg build --repo docs/ --exclude-dir archive --exclude-dir vendor
185
+ ```
186
+
187
+ ### Query and pack passages
188
+
189
+ ```bash
190
+ # Hybrid query — semantic seed + graph expansion
191
+ memorykg query "deployment configuration"
192
+
193
+ # Tune top-K and expansion hops
194
+ memorykg query "API authentication" --k 12 --hop 2
195
+
196
+ # Pack as Markdown for LLM context injection
197
+ memorykg pack "error handling strategies" --fmt md --out context.md
198
+ ```
199
+
200
+ ### Analyze, snapshot, visualize
201
+
202
+ ```bash
203
+ memorykg analyze --repo docs/ # corpus health report
204
+ memorykg snapshot save 0.4.1 && memorykg snapshot diff 0.4.0 0.4.1
205
+ memorykg viz # Streamlit graph browser
206
+ memorykg mcp --repo docs/ # MCP server for AI agents
207
+ ```
208
+
209
+ See [docs/cli-reference.md](docs/cli-reference.md) for every flag.
210
+
211
+ ---
212
+
213
+ ## Reproducing the Benchmarks
214
+
215
+ ### LongMemEval-S — 98.4% R@5, 99.4% R@10
216
+
217
+ Full write-up: [`benchmarks/longmemeval/longmemeval_article.pdf`](benchmarks/longmemeval/longmemeval_article.pdf)
218
+
219
+ ```bash
220
+ # 1. Install
221
+ poetry install
222
+
223
+ # 2. Download LongMemEval-S
224
+ mkdir -p /tmp/longmemeval-data
225
+ curl -fsSL -o /tmp/longmemeval-data/longmemeval_s_cleaned.json \
226
+ https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json
227
+
228
+ # 3. Build the corpus + KG (BGE-small-en-v1.5, heading chunks)
229
+ poetry run python3 benchmarks/longmemeval/longmemeval_memkg.py prepare \
230
+ /tmp/longmemeval-data/longmemeval_s_cleaned.json \
231
+ --wipe --chunk-strategy heading
232
+
233
+ # 4. Run evaluation (haystack filter and k=50 are now defaults)
234
+ poetry run python3 benchmarks/longmemeval/longmemeval_memkg.py run \
235
+ /tmp/longmemeval-data/longmemeval_s_cleaned.json \
236
+ --out benchmarks/longmemeval/results_bge_haystack.jsonl
237
+
238
+ # Expected: R@5=98.4% R@10=99.4% NDCG@10=0.943 Misses@10=3
239
+ ```
240
+
241
+ ### ConvoMem — 100% Recall Across 17,463 Items
242
+
243
+ Full write-up: [`benchmarks/convomem/convomem_article.pdf`](benchmarks/convomem/convomem_article.pdf)
244
+
245
+ ```bash
246
+ # Run all four evidence tiers (top-10, hop=1, BGE-small-en-v1.5)
247
+ poetry run python3 benchmarks/convomem/convomem_bench.py --limit 1000 --tier 1
248
+ poetry run python3 benchmarks/convomem/convomem_bench.py --limit 1000 --tier 2
249
+ poetry run python3 benchmarks/convomem/convomem_bench.py --limit 1000 --tier 3
250
+ poetry run python3 benchmarks/convomem/convomem_bench.py --limit 1000 --tier 4
251
+
252
+ # Expected: 100% retrieval recall on every category × tier (17,463 items, ~20 min)
253
+ ```
254
+
255
+ **Hardware tested:** Apple M5 Max MacBook Pro, 64 GB RAM. Also runs on CUDA and pure CPU (`MEMORYKG_DEVICE=cpu`).
256
+
257
+ ---
258
+
259
+ ## Documentation
260
+
261
+ | Doc | Contents |
262
+ |---|---|
263
+ | [docs/installation.md](docs/installation.md) | Detailed install, dev setup, entry points, config |
264
+ | [docs/cli-reference.md](docs/cli-reference.md) | Full CLI reference with all options |
265
+ | [docs/ingestion.md](docs/ingestion.md) | Build pipeline architecture, **node kinds & edge types** |
266
+ | [docs/python-api.md](docs/python-api.md) | `MemoryKG` class — build, query, haystack-scoping, passage packing |
267
+ | [docs/MCP.md](docs/MCP.md) | MCP server setup (Claude Code, Copilot, Claude Desktop, Cline) |
268
+ | [docs/CHEATSHEET.md](docs/CHEATSHEET.md) | MCP tool query patterns and examples |
269
+ | [docs/SNAPSHOTS.md](docs/SNAPSHOTS.md) | Snapshot workflow and diff guide |
270
+ | [benchmarks/BENCHMARKS.md](benchmarks/BENCHMARKS.md) | Full LongMemEval progression (75.8% → 98.4%), recall_all analysis, integrity notes |
271
+ | [benchmarks/longmemeval/longmemeval_article.pdf](benchmarks/longmemeval/longmemeval_article.pdf) | LongMemEval-S report (PDF): 98.4% R@5, 99.4% R@10, 0.943 NDCG@10 |
272
+ | [benchmarks/convomem/convomem_article.pdf](benchmarks/convomem/convomem_article.pdf) | ConvoMem report (PDF): 100% retrieval recall across 17,463 items |
273
+
274
+ ---
275
+
276
+ ## Citation
277
+
278
+ If you use MemoryKG in your research or project, please cite it:
279
+
280
+ [![DOI](https://zenodo.org/badge/1205364687.svg)](https://zenodo.org/badge/latestdoi/1205364687)
281
+
282
+ **APA**
283
+
284
+ > Suchanek, E. G. (2026). *MemoryKG: Hybrid Semantic-Graph Knowledge Base for Conversational Memory* (Version 0.5.2) [Software]. Flux-Frontiers. https://github.com/Flux-Frontiers/memory_kg
285
+
286
+ **BibTeX**
287
+
288
+ ```bibtex
289
+ @software{suchanek_memory_kg,
290
+ author = {Suchanek, Eric G.},
291
+ title = {{MemoryKG}: Hybrid Semantic-Graph Knowledge Base for Conversational Memory},
292
+ version = {0.5.2},
293
+ year = {2026},
294
+ publisher = {Flux-Frontiers},
295
+ url = {https://github.com/Flux-Frontiers/memory_kg},
296
+ doi = {10.5281/zenodo.TBD},
297
+ }
298
+ ```
299
+ ---
300
+
301
+ ## License
302
+
303
+ [Elastic License 2.0](LICENSE) — free for non-commercial and internal use; commercial hosting or redistribution requires a license from Flux-Frontiers.
304
+
@@ -0,0 +1,245 @@
1
+ [![CI](https://github.com/Flux-Frontiers/memory_kg/actions/workflows/publish.yml/badge.svg)](https://github.com/Flux-Frontiers/memory_kg/actions/workflows/publish.yml)
2
+ [![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13-blue.svg)](https://www.python.org/)
3
+ [![License: Elastic-2.0](https://img.shields.io/badge/License-Elastic%202.0-blue.svg)](https://www.elastic.co/licensing/elastic-license)
4
+ [![Version](https://img.shields.io/badge/version-0.5.2-blue.svg)](https://github.com/Flux-Frontiers/memory_kg/releases)
5
+ [![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)
6
+ [![DOI](https://zenodo.org/badge/1205364687.svg)](https://zenodo.org/badge/latestdoi/1205364687)
7
+
8
+ **MemoryKG** — A Hybrid Knowledge Graph for Conversational Memory and Document Corpora
9
+
10
+ *Author: Eric G. Suchanek, PhD — Flux-Frontiers, Liberty TWP, OH*
11
+
12
+ ---
13
+
14
+ ## TL;DR
15
+
16
+ MemoryKG achieves **100% retrieval recall on the ConvoMem benchmark — every evidence message found, on every question, across 17,463 items** spanning six evidence categories and four evidence tiers (1–4 messages). No LLM, no API key, no cloud inference at any stage. This is the largest non-LLM evaluation on ConvoMem reported. Full write-up: [`benchmarks/convomem/convomem_article.pdf`](benchmarks/convomem/convomem_article.pdf).
17
+
18
+ Recall is measured by substring containment in the top-10 retrieved nodes: an evidence message counts as found if its text appears verbatim in (or contains) any retrieved node — lenient toward retrieval, but it cannot be fooled by paraphrase.
19
+
20
+ On the LongMemEval-S benchmark, MemoryKG is **tied for the top LLM-free score** — 98.4% Recall@5, 99.4% Recall@10, 0.943 NDCG@10. It matches MemoryPalace's best LLM-free results (hybrid v4 held-out and hybrid v2) and beats every other LLM-free baseline. Three LLM-augmented systems rank higher at R@5 (MemoryPalace v4 + Haiku at 100%, MemoryPalace v3 + Haiku rerank at 99.4%, Supermemory ASMR at ~99%); MemoryKG narrows that gap without paying the inference cost. Full write-up: [`benchmarks/longmemeval/longmemeval_article.pdf`](benchmarks/longmemeval/longmemeval_article.pdf).
21
+
22
+ | System | LongMemEval R@5 | LLM at query time | Cost / query |
23
+ |---|--:|---|--:|
24
+ | MemoryPalace hybrid v4 + Haiku (500q) | 100% | Yes (Claude Haiku) | $$ |
25
+ | MemoryPalace hybrid v4 held-out (450q) | 98.4% | None | $0 |
26
+ | **MemoryKG (this work)** | **98.4%** | **None** | **$0** |
27
+ | MemoryPalace hybrid v3 + Haiku rerank | 99.4% | Yes (Claude Haiku) | $$ |
28
+ | Supermemory ASMR | ~99% | Yes (undisclosed) | $$ |
29
+ | MemoryPalace hybrid v2 | 98.4% | None | $0 |
30
+ | Mastra | 94.9% | Yes (GPT-5-mini) | $$ |
31
+ | MemoryPalace raw ChromaDB | 96.6% | None | $0 |
32
+ | Hindsight | 91.4% | Yes (Gemini-3) | $$ |
33
+ | Supermemory (production) | ~85% | Yes (undisclosed) | $$ |
34
+ | Stella (dense retriever) | ~85% | None | $0 |
35
+ | BM25 (sparse baseline) | ~70% | None | $0 |
36
+
37
+ With the sibling boost enabled on LongMemEval, **recall_all@10 reaches 98.6%** — meaning MemoryKG retrieves *every* required session for 493 of 500 questions without any LLM. No published system reports this metric; we track it because multi-session coverage is the real test of memory completeness.
38
+
39
+ The field has been over-engineering retrieval. A graph-augmented index with correct search-space scoping matches the best LLM-free result in the field at a fraction of the complexity.
40
+
41
+ ---
42
+
43
+ ## Why It Works
44
+
45
+ Most "memory" systems flatten a session into a single embedding and lean on an LLM at query time to rerank what they retrieve. MemoryKG does the opposite: it preserves session structure as a typed graph, then uses that structure as the ranking signal.
46
+
47
+ 1. **Finer granularity.** Sessions are chunked by heading, not embedded as 2,000-word blobs. A 150-word chunk about "Dr. Chen's appointment" is dramatically more discriminative than the session it lives in.
48
+ 2. **Structural expansion.** A `HAS_TOPIC` or `MENTIONS_ENTITY` edge from a weakly-matching chunk surfaces strongly-linked neighbors that pure cosine similarity never finds.
49
+ 3. **Score-first ranking.** Graph proximity breaks ties *within* a vector-quality band — never across one. Good seeds get amplified; bad seeds don't get rescued.
50
+ 4. **Kind-aware ranking.** Chunk matches outrank entity stubs outrank synthetic topic summaries. Flat vector stores treat every document equally.
51
+ 5. **Search-space scoping.** When the benchmark defines a per-question candidate pool, MemoryKG honours it (`haystack_files=...`). This was the +11 pp fix that narrowed the gap to the inference-based leaderboard.
52
+
53
+ **No LLM. No API key. No cloud round-trip. Runs on Apple Silicon (MPS), CUDA, or CPU.**
54
+
55
+ ---
56
+
57
+ ## What MemoryKG Is
58
+
59
+ A **deterministic, explainable knowledge graph** built from conversational logs and document corpora (Markdown, plain text). MemoryKG semantically chunks text, extracts topics/entities/keywords, links them through typed edges, stores everything in SQLite, and adds a LanceDB vector index as an *acceleration layer* — not the source of truth.
60
+
61
+ Structure is treated as ground truth. Semantic search is a tool, not the system. The result is a searchable, auditable representation that supports precise navigation, source-grounded passage extraction, and downstream LLM reasoning — a practical foundation for **Knowledge-Graph RAG (KGRAG)**.
62
+
63
+ MemoryKG shares its architecture with [PyCodeKG](https://github.com/Flux-Frontiers/code_kg) (Python codebases) and [DocKG](https://github.com/Flux-Frontiers/doc_kg) (general document corpora).
64
+
65
+ ---
66
+
67
+ ## Features
68
+
69
+ - **Semantic chunking** — Multiple strategies: `heading` (one chunk per `## Section`), `fixed`, `sentence_group`, `semantic` (embedding-boundary detection)
70
+ - **Deterministic knowledge graph** — SQLite-backed canonical store with typed nodes and provenance-tracked edges
71
+ - **Relation extraction** — Topics, named entities, keywords; co-occurrence and similarity edges built automatically
72
+ - **Hybrid query model** — Semantic seeding (LanceDB) + structural expansion (graph traversal) + score-first ranking
73
+ - **Haystack-scoped search** — Restrict vector seeding to a per-question candidate pool for benchmark-grade precision
74
+ - **Passage packing** — Source-grounded text passages with headings, ready to paste into an LLM prompt
75
+ - **Coverage analysis & temporal snapshots** — Per-document metrics, hot chunks, orphan detection, version-over-version diffs
76
+ - **Parallel ingestion** — `--workers N` parallel Phase 1 parsing for large corpora
77
+ - **MCP server** — Four tools for AI agent integration (`graph_stats`, `query_docs`, `pack_docs`, `get_node`)
78
+ - **Streamlit web app** — Interactive graph browser, hybrid query UI, and passage pack explorer
79
+
80
+ ---
81
+
82
+ ## Quick Start
83
+
84
+ ```bash
85
+ # Index a corpus (SQLite + LanceDB in one step; wipe is the default)
86
+ memorykg build --repo docs/
87
+
88
+ # Natural-language query — returns ranked chunks
89
+ memorykg query "authentication flow"
90
+
91
+ # Source-grounded passage pack — paste straight into an LLM prompt
92
+ memorykg pack "configuration reference" --fmt md --out context.md
93
+ ```
94
+
95
+ ---
96
+
97
+ ## Installation
98
+
99
+ ```bash
100
+ pip install 'memory-kg @ git+https://github.com/Flux-Frontiers/memory_kg.git'
101
+ ```
102
+
103
+ See [docs/installation.md](docs/installation.md) for editable installs, dev setup, and offline model caching.
104
+
105
+ ---
106
+
107
+ ## Usage
108
+
109
+ ### Build the knowledge graph
110
+
111
+ ```bash
112
+ # Full pipeline: parse → SQLite graph → LanceDB index (wipe is default)
113
+ memorykg build --repo docs/
114
+
115
+ # Granular steps for large corpora
116
+ memorykg build-graph --repo docs/ # SQLite only
117
+ memorykg build-index # LanceDB from existing SQLite
118
+
119
+ # Incremental update — keep existing data
120
+ memorykg build --repo docs/ --update
121
+
122
+ # Parallelise Phase 1 parsing
123
+ memorykg build --repo docs/ --workers 8
124
+
125
+ # Exclude directories
126
+ memorykg build --repo docs/ --exclude-dir archive --exclude-dir vendor
127
+ ```
128
+
129
+ ### Query and pack passages
130
+
131
+ ```bash
132
+ # Hybrid query — semantic seed + graph expansion
133
+ memorykg query "deployment configuration"
134
+
135
+ # Tune top-K and expansion hops
136
+ memorykg query "API authentication" --k 12 --hop 2
137
+
138
+ # Pack as Markdown for LLM context injection
139
+ memorykg pack "error handling strategies" --fmt md --out context.md
140
+ ```
141
+
142
+ ### Analyze, snapshot, visualize
143
+
144
+ ```bash
145
+ memorykg analyze --repo docs/ # corpus health report
146
+ memorykg snapshot save 0.4.1 && memorykg snapshot diff 0.4.0 0.4.1
147
+ memorykg viz # Streamlit graph browser
148
+ memorykg mcp --repo docs/ # MCP server for AI agents
149
+ ```
150
+
151
+ See [docs/cli-reference.md](docs/cli-reference.md) for every flag.
152
+
153
+ ---
154
+
155
+ ## Reproducing the Benchmarks
156
+
157
+ ### LongMemEval-S — 98.4% R@5, 99.4% R@10
158
+
159
+ Full write-up: [`benchmarks/longmemeval/longmemeval_article.pdf`](benchmarks/longmemeval/longmemeval_article.pdf)
160
+
161
+ ```bash
162
+ # 1. Install
163
+ poetry install
164
+
165
+ # 2. Download LongMemEval-S
166
+ mkdir -p /tmp/longmemeval-data
167
+ curl -fsSL -o /tmp/longmemeval-data/longmemeval_s_cleaned.json \
168
+ https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json
169
+
170
+ # 3. Build the corpus + KG (BGE-small-en-v1.5, heading chunks)
171
+ poetry run python3 benchmarks/longmemeval/longmemeval_memkg.py prepare \
172
+ /tmp/longmemeval-data/longmemeval_s_cleaned.json \
173
+ --wipe --chunk-strategy heading
174
+
175
+ # 4. Run evaluation (haystack filter and k=50 are now defaults)
176
+ poetry run python3 benchmarks/longmemeval/longmemeval_memkg.py run \
177
+ /tmp/longmemeval-data/longmemeval_s_cleaned.json \
178
+ --out benchmarks/longmemeval/results_bge_haystack.jsonl
179
+
180
+ # Expected: R@5=98.4% R@10=99.4% NDCG@10=0.943 Misses@10=3
181
+ ```
182
+
183
+ ### ConvoMem — 100% Recall Across 17,463 Items
184
+
185
+ Full write-up: [`benchmarks/convomem/convomem_article.pdf`](benchmarks/convomem/convomem_article.pdf)
186
+
187
+ ```bash
188
+ # Run all four evidence tiers (top-10, hop=1, BGE-small-en-v1.5)
189
+ poetry run python3 benchmarks/convomem/convomem_bench.py --limit 1000 --tier 1
190
+ poetry run python3 benchmarks/convomem/convomem_bench.py --limit 1000 --tier 2
191
+ poetry run python3 benchmarks/convomem/convomem_bench.py --limit 1000 --tier 3
192
+ poetry run python3 benchmarks/convomem/convomem_bench.py --limit 1000 --tier 4
193
+
194
+ # Expected: 100% retrieval recall on every category × tier (17,463 items, ~20 min)
195
+ ```
196
+
197
+ **Hardware tested:** Apple M5 Max MacBook Pro, 64 GB RAM. Also runs on CUDA and pure CPU (`MEMORYKG_DEVICE=cpu`).
198
+
199
+ ---
200
+
201
+ ## Documentation
202
+
203
+ | Doc | Contents |
204
+ |---|---|
205
+ | [docs/installation.md](docs/installation.md) | Detailed install, dev setup, entry points, config |
206
+ | [docs/cli-reference.md](docs/cli-reference.md) | Full CLI reference with all options |
207
+ | [docs/ingestion.md](docs/ingestion.md) | Build pipeline architecture, **node kinds & edge types** |
208
+ | [docs/python-api.md](docs/python-api.md) | `MemoryKG` class — build, query, haystack-scoping, passage packing |
209
+ | [docs/MCP.md](docs/MCP.md) | MCP server setup (Claude Code, Copilot, Claude Desktop, Cline) |
210
+ | [docs/CHEATSHEET.md](docs/CHEATSHEET.md) | MCP tool query patterns and examples |
211
+ | [docs/SNAPSHOTS.md](docs/SNAPSHOTS.md) | Snapshot workflow and diff guide |
212
+ | [benchmarks/BENCHMARKS.md](benchmarks/BENCHMARKS.md) | Full LongMemEval progression (75.8% → 98.4%), recall_all analysis, integrity notes |
213
+ | [benchmarks/longmemeval/longmemeval_article.pdf](benchmarks/longmemeval/longmemeval_article.pdf) | LongMemEval-S report (PDF): 98.4% R@5, 99.4% R@10, 0.943 NDCG@10 |
214
+ | [benchmarks/convomem/convomem_article.pdf](benchmarks/convomem/convomem_article.pdf) | ConvoMem report (PDF): 100% retrieval recall across 17,463 items |
215
+
216
+ ---
217
+
218
+ ## Citation
219
+
220
+ If you use MemoryKG in your research or project, please cite it:
221
+
222
+ [![DOI](https://zenodo.org/badge/1205364687.svg)](https://zenodo.org/badge/latestdoi/1205364687)
223
+
224
+ **APA**
225
+
226
+ > Suchanek, E. G. (2026). *MemoryKG: Hybrid Semantic-Graph Knowledge Base for Conversational Memory* (Version 0.5.2) [Software]. Flux-Frontiers. https://github.com/Flux-Frontiers/memory_kg
227
+
228
+ **BibTeX**
229
+
230
+ ```bibtex
231
+ @software{suchanek_memory_kg,
232
+ author = {Suchanek, Eric G.},
233
+ title = {{MemoryKG}: Hybrid Semantic-Graph Knowledge Base for Conversational Memory},
234
+ version = {0.5.2},
235
+ year = {2026},
236
+ publisher = {Flux-Frontiers},
237
+ url = {https://github.com/Flux-Frontiers/memory_kg},
238
+ doi = {10.5281/zenodo.TBD},
239
+ }
240
+ ```
241
+ ---
242
+
243
+ ## License
244
+
245
+ [Elastic License 2.0](LICENSE) — free for non-commercial and internal use; commercial hosting or redistribution requires a license from Flux-Frontiers.