memxcore 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- memxcore-0.1.0/PKG-INFO +359 -0
- memxcore-0.1.0/README.md +321 -0
- memxcore-0.1.0/memxcore/__init__.py +3 -0
- memxcore-0.1.0/memxcore/cli.py +692 -0
- memxcore-0.1.0/memxcore/core/__init__.py +8 -0
- memxcore-0.1.0/memxcore/core/benchmark.py +285 -0
- memxcore-0.1.0/memxcore/core/bm25.py +181 -0
- memxcore-0.1.0/memxcore/core/compaction.py +598 -0
- memxcore-0.1.0/memxcore/core/knowledge_graph.py +207 -0
- memxcore-0.1.0/memxcore/core/memory_manager.py +716 -0
- memxcore-0.1.0/memxcore/core/mining.py +254 -0
- memxcore-0.1.0/memxcore/core/paths.py +57 -0
- memxcore-0.1.0/memxcore/core/rag.py +233 -0
- memxcore-0.1.0/memxcore/core/utils.py +192 -0
- memxcore-0.1.0/memxcore/core/watcher.py +121 -0
- memxcore-0.1.0/memxcore/hooks/__init__.py +0 -0
- memxcore-0.1.0/memxcore/hooks/auto_remember.py +292 -0
- memxcore-0.1.0/memxcore/hooks/user_prompt_submit.py +235 -0
- memxcore-0.1.0/memxcore/mcp_server.py +272 -0
- memxcore-0.1.0/memxcore/server.py +98 -0
- memxcore-0.1.0/memxcore/tests/test_basic.py +518 -0
- memxcore-0.1.0/memxcore.egg-info/PKG-INFO +359 -0
- memxcore-0.1.0/memxcore.egg-info/SOURCES.txt +27 -0
- memxcore-0.1.0/memxcore.egg-info/dependency_links.txt +1 -0
- memxcore-0.1.0/memxcore.egg-info/entry_points.txt +2 -0
- memxcore-0.1.0/memxcore.egg-info/requires.txt +25 -0
- memxcore-0.1.0/memxcore.egg-info/top_level.txt +1 -0
- memxcore-0.1.0/pyproject.toml +67 -0
- memxcore-0.1.0/setup.cfg +4 -0
memxcore-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,359 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: memxcore
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Persistent memory system for AI coding assistants
|
|
5
|
+
Author: MemXCore Contributors
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/Danny0218/memxcore
|
|
8
|
+
Project-URL: Repository, https://github.com/Danny0218/memxcore
|
|
9
|
+
Keywords: ai,memory,mcp,claude,rag
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
16
|
+
Classifier: Topic :: Software Development :: Libraries
|
|
17
|
+
Requires-Python: >=3.11
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
Requires-Dist: pyyaml>=6.0
|
|
20
|
+
Requires-Dist: pydantic>=2.0
|
|
21
|
+
Requires-Dist: litellm>=1.0.0
|
|
22
|
+
Requires-Dist: mcp>=1.0.0
|
|
23
|
+
Provides-Extra: rag
|
|
24
|
+
Requires-Dist: chromadb>=0.5.0; extra == "rag"
|
|
25
|
+
Requires-Dist: sentence-transformers>=3.0.0; extra == "rag"
|
|
26
|
+
Provides-Extra: server
|
|
27
|
+
Requires-Dist: fastapi>=0.100.0; extra == "server"
|
|
28
|
+
Requires-Dist: uvicorn[standard]>=0.20.0; extra == "server"
|
|
29
|
+
Provides-Extra: watch
|
|
30
|
+
Requires-Dist: watchdog>=4.0.0; extra == "watch"
|
|
31
|
+
Provides-Extra: bm25
|
|
32
|
+
Requires-Dist: rank-bm25>=0.2.2; extra == "bm25"
|
|
33
|
+
Provides-Extra: all
|
|
34
|
+
Requires-Dist: memxcore[bm25,rag,server,watch]; extra == "all"
|
|
35
|
+
Provides-Extra: dev
|
|
36
|
+
Requires-Dist: pytest>=7.0; extra == "dev"
|
|
37
|
+
Requires-Dist: pytest-cov>=4.0; extra == "dev"
|
|
38
|
+
|
|
39
|
+
# MemXCore
|
|
40
|
+
|
|
41
|
+
**English** | [中文](README.zh-CN.md)
|
|
42
|
+
|
|
43
|
+
Persistent memory system for AI coding assistants. Stores and retrieves memories across sessions via MCP (Model Context Protocol), with semantic search powered by ChromaDB + sentence-transformers.
|
|
44
|
+
|
|
45
|
+
Works with **Claude Code**, **Cursor**, **Gemini CLI**, and any MCP-compatible tool.
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Architecture
|
|
50
|
+
|
|
51
|
+
```
|
|
52
|
+
RECENT.md <- L0: raw append log (WAL). Every remember() lands here.
|
|
53
|
+
archive/<cat>.md <- L1: distilled long-term memory, per category, with YAML front matter
|
|
54
|
+
USER.md <- L2: permanent facts (remember(permanent=True)), never compacted
|
|
55
|
+
chroma/ <- RAG vector index (rebuilt from archive/ files, lossy-safe)
|
|
56
|
+
index.json <- Keyword search index (tags + summaries from archive/ front matter)
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
**Search tiers (in order):**
|
|
60
|
+
1. **RAG semantic search** (ChromaDB + BM25, RRF fusion) — most accurate, requires `chromadb` + `sentence-transformers`
|
|
61
|
+
2. **LLM relevance judgment** — Claude reads all facts and picks relevant ones, requires API key
|
|
62
|
+
3. **Keyword fallback** — always available, scans archive files directly
|
|
63
|
+
|
|
64
|
+
**Write flow:**
|
|
65
|
+
1. `remember(text)` -> append to RECENT.md
|
|
66
|
+
2. When RECENT.md exceeds token threshold -> LLM distills into structured facts
|
|
67
|
+
3. Each fact -> written to `archive/<category>.md` + upserted into ChromaDB (dual-write)
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
## Quick Start
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
pip install memxcore # core
|
|
75
|
+
pip install 'memxcore[rag]' # + semantic search (recommended)
|
|
76
|
+
pip install 'memxcore[all]' # everything
|
|
77
|
+
|
|
78
|
+
# Or from source:
|
|
79
|
+
pip install -r memxcore/requirements.txt
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
**Check your setup:**
|
|
83
|
+
```bash
|
|
84
|
+
memxcore doctor
|
|
85
|
+
# or: python -m memxcore.cli doctor
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
**Requirements:** Python 3.11+
|
|
89
|
+
|
|
90
|
+
**LLM API key** (any provider supported via [litellm](https://docs.litellm.ai/)):
|
|
91
|
+
```bash
|
|
92
|
+
# Pick one:
|
|
93
|
+
export ANTHROPIC_API_KEY=sk-ant-... # Anthropic (direct API)
|
|
94
|
+
export ANTHROPIC_AUTH_TOKEN=xxx # Anthropic (gateway/proxy, with ANTHROPIC_BASE_URL)
|
|
95
|
+
export OPENAI_API_KEY=sk-... # OpenAI
|
|
96
|
+
export GEMINI_API_KEY=... # Google Gemini
|
|
97
|
+
# Or use Ollama for fully local/offline: no key needed, just set model in config.yaml
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Without any API key, compaction falls back to basic mode (truncation instead of LLM distillation).
|
|
101
|
+
|
|
102
|
+
> **Important: Compaction is not automatic.**
|
|
103
|
+
> MemXCore does not detect session boundaries. If compaction is never triggered,
|
|
104
|
+
> memories accumulate in RECENT.md and are not distilled into long-term storage.
|
|
105
|
+
>
|
|
106
|
+
> **Recommended setup** (pick one):
|
|
107
|
+
> - Use the provided **Stop hook** (`memxcore.hooks.auto_remember`) — extracts facts after each response and triggers compaction automatically.
|
|
108
|
+
> - Call `compact(force=True)` via the MCP tool at the end of each session.
|
|
109
|
+
> - Run `memxcore compact` from the command line as a cron job or manual step.
|
|
110
|
+
>
|
|
111
|
+
> See the [Integration](#integration-claude-code) section for hook setup examples.
|
|
112
|
+
|
|
113
|
+
**Workspace resolution:**
|
|
114
|
+
|
|
115
|
+
By default, MemXCore resolves the workspace from the module's install location (suitable for development / editable installs). For pip-installed usage, set `MEMXCORE_WORKSPACE` to your project root:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
export MEMXCORE_WORKSPACE=/path/to/your/project
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Or in your MCP server config:
|
|
122
|
+
```json
|
|
123
|
+
{
|
|
124
|
+
"env": { "MEMXCORE_WORKSPACE": "/path/to/your/project" }
|
|
125
|
+
}
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## Configuration (`config.yaml`)
|
|
131
|
+
|
|
132
|
+
Located at `memxcore/config.yaml`. All fields are optional — sensible defaults apply.
|
|
133
|
+
|
|
134
|
+
```yaml
|
|
135
|
+
compaction:
|
|
136
|
+
strategy: llm # 'llm' = LLM distills to structured facts (via litellm)
|
|
137
|
+
# 'basic' = truncate first 50 lines to general.md
|
|
138
|
+
threshold_tokens: 2000 # Trigger compaction when RECENT.md exceeds this
|
|
139
|
+
min_entries: 3 # Don't compact if fewer than N entries in RECENT.md
|
|
140
|
+
check_interval: 5 # Check token count every N remember() calls (reduces I/O)
|
|
141
|
+
stale_minutes: 10 # Auto-compact on search() if RECENT.md older than this
|
|
142
|
+
|
|
143
|
+
llm:
|
|
144
|
+
model: anthropic/claude-sonnet # litellm format: provider/model
|
|
145
|
+
# Examples: openai/gpt-4o, gemini/gemini-2.5-flash, ollama/llama3
|
|
146
|
+
api_key_env: ANTHROPIC_API_KEY # Env var name (litellm also auto-reads OPENAI_API_KEY, etc.)
|
|
147
|
+
# base_url: # Optional: custom API endpoint (gateway/proxy)
|
|
148
|
+
|
|
149
|
+
rag:
|
|
150
|
+
embedding_model: all-MiniLM-L6-v2 # sentence-transformers model (80MB, CPU-friendly)
|
|
151
|
+
top_k: 10 # Max results from semantic search
|
|
152
|
+
rrf_k: 60 # Reciprocal Rank Fusion constant
|
|
153
|
+
|
|
154
|
+
watch: false # Auto-reindex archive/*.md on file change (off by default)
|
|
155
|
+
|
|
156
|
+
memory:
|
|
157
|
+
categories:
|
|
158
|
+
- id: user_model
|
|
159
|
+
description: "User identity, preferences, communication style, feedback"
|
|
160
|
+
- id: domain
|
|
161
|
+
description: "Transferable domain/technical knowledge — valid across projects"
|
|
162
|
+
- id: project_state
|
|
163
|
+
description: "Active tasks, in-progress decisions — decays within weeks"
|
|
164
|
+
- id: episodic
|
|
165
|
+
description: "Time-bound events and past decisions — has TTL"
|
|
166
|
+
- id: references
|
|
167
|
+
description: "URLs, doc links, dashboards, ticket trackers, Slack channels"
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## Integration
|
|
173
|
+
|
|
174
|
+
### One-command setup (recommended)
|
|
175
|
+
|
|
176
|
+
```bash
|
|
177
|
+
memxcore setup # auto-detects tools and configures everything
|
|
178
|
+
memxcore setup --dry-run # preview without making changes
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
Automatically detects and configures:
|
|
182
|
+
|
|
183
|
+
| Tool | What it does |
|
|
184
|
+
|------|-------------|
|
|
185
|
+
| **Claude Code** | Registers MCP server, installs hooks (auto-remember + auto-compact), appends agent rules to `~/.claude/CLAUDE.md` |
|
|
186
|
+
| **Cursor** | Writes MCP config to `~/.cursor/mcp.json`, copies rules to `~/.cursor/rules/memxcore.md` |
|
|
187
|
+
| **Windsurf** | Writes MCP config to `~/.codeium/windsurf/mcp_config.json` |
|
|
188
|
+
| **Codex (OpenAI)** | Writes MCP config to `~/.codex/config.toml` |
|
|
189
|
+
| **Gemini CLI** | Writes MCP config to `~/.gemini/settings.json` |
|
|
190
|
+
|
|
191
|
+
After setup, restart your tool and verify the MCP connection.
|
|
192
|
+
|
|
193
|
+
### Manual setup
|
|
194
|
+
|
|
195
|
+
If you prefer manual configuration, the MCP server config for any tool is:
|
|
196
|
+
|
|
197
|
+
```json
|
|
198
|
+
{
|
|
199
|
+
"mcpServers": {
|
|
200
|
+
"memxcore": {
|
|
201
|
+
"command": "/path/to/your/.venv/bin/python",
|
|
202
|
+
"args": ["-m", "memxcore.mcp_server"],
|
|
203
|
+
"env": { "MEMXCORE_WORKSPACE": "/path/to/your/workspace" }
|
|
204
|
+
}
|
|
205
|
+
}
|
|
206
|
+
}
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
---
|
|
210
|
+
|
|
211
|
+
## MCP Tools Reference
|
|
212
|
+
|
|
213
|
+
| Tool | Args | Description |
|
|
214
|
+
|------|------|-------------|
|
|
215
|
+
| `remember` | `text`, `category?`, `permanent?`, `tenant_id?` | Store a memory |
|
|
216
|
+
| `search` | `query`, `max_results?`, `tenant_id?` | Retrieve memories (semantic + keyword) |
|
|
217
|
+
| `compact` | `force?`, `tenant_id?` | Distill RECENT.md into categorized archives |
|
|
218
|
+
| `reindex` | `category?`, `tenant_id?` | Rebuild RAG index after manual edits |
|
|
219
|
+
|
|
220
|
+
**Categories for `remember`:**
|
|
221
|
+
`user_model` / `domain` / `project_state` / `episodic` / `references`
|
|
222
|
+
|
|
223
|
+
**Permanent memories:**
|
|
224
|
+
Use `permanent=True` to write directly to USER.md (L2). These are never compacted and always have the highest priority in search. Use for stable facts like core user identity or long-term architectural decisions.
|
|
225
|
+
|
|
226
|
+
---
|
|
227
|
+
|
|
228
|
+
## Multi-Tenant
|
|
229
|
+
|
|
230
|
+
All MCP tools and CLI commands accept an optional `tenant_id`. Each tenant gets isolated storage under `tenants/<tenant_id>/storage/`. Omit `tenant_id` for single-user mode.
|
|
231
|
+
|
|
232
|
+
```bash
|
|
233
|
+
# CLI
|
|
234
|
+
memxcore --tenant alice search "preferences"
|
|
235
|
+
|
|
236
|
+
# MCP
|
|
237
|
+
search("preferences", tenant_id="alice")
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
Tenants can also have their own `config.yaml` override at `tenants/<tenant_id>/config.yaml`.
|
|
241
|
+
|
|
242
|
+
---
|
|
243
|
+
|
|
244
|
+
## CLI Reference
|
|
245
|
+
|
|
246
|
+
```bash
|
|
247
|
+
memxcore setup # auto-detect tools, configure everything
|
|
248
|
+
memxcore setup --dry-run # preview without changes
|
|
249
|
+
memxcore doctor # check system readiness
|
|
250
|
+
memxcore reindex # rebuild RAG index
|
|
251
|
+
memxcore compact # force distillation
|
|
252
|
+
memxcore search "query" # search memories (debug)
|
|
253
|
+
memxcore benchmark # search precision benchmark
|
|
254
|
+
memxcore mine <path> # import conversations
|
|
255
|
+
memxcore --tenant alice doctor # multi-tenant
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
All commands also work via `python -m memxcore.cli <command>`.
|
|
259
|
+
|
|
260
|
+
---
|
|
261
|
+
|
|
262
|
+
## HTTP API Server (optional)
|
|
263
|
+
|
|
264
|
+
For development/testing. In production, use the MCP server.
|
|
265
|
+
|
|
266
|
+
```bash
|
|
267
|
+
pip install 'memxcore[server]'
|
|
268
|
+
python -m memxcore.server --host 127.0.0.1 --port 8000
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
Endpoints:
|
|
272
|
+
```
|
|
273
|
+
POST /remember { "text": "...", "level": null }
|
|
274
|
+
GET /search?query= returns List[MemoryResult]
|
|
275
|
+
POST /compact ?force=false
|
|
276
|
+
POST /rebuild-rag rebuild ChromaDB from archive files
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
---
|
|
280
|
+
|
|
281
|
+
## Storage Structure
|
|
282
|
+
|
|
283
|
+
```
|
|
284
|
+
memxcore/
|
|
285
|
+
+-- storage/
|
|
286
|
+
+-- RECENT.md Raw memory log. Cleared after each compaction.
|
|
287
|
+
+-- USER.md Permanent facts (permanent=True). Never compacted.
|
|
288
|
+
+-- index.json Keyword search index. Auto-rebuilt after compaction.
|
|
289
|
+
+-- chroma/ ChromaDB vector index. Rebuildable from archive/.
|
|
290
|
+
+-- archive/
|
|
291
|
+
+-- user_model.md User identity, preferences, feedback
|
|
292
|
+
+-- domain.md Domain/technical knowledge
|
|
293
|
+
+-- project_state.md Active tasks, decisions (stale after weeks)
|
|
294
|
+
+-- episodic.md Past events and decisions
|
|
295
|
+
+-- references.md URLs, doc links, external pointers
|
|
296
|
+
+-- general.md Fallback when LLM distillation unavailable
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
**Archive file format:**
|
|
300
|
+
```markdown
|
|
301
|
+
---
|
|
302
|
+
topic: project_state
|
|
303
|
+
tags: [api, testing]
|
|
304
|
+
last_distilled: 2026-04-07T14:23:01
|
|
305
|
+
confidence_level: 4
|
|
306
|
+
---
|
|
307
|
+
|
|
308
|
+
## [2026-04-07T14:23:01]
|
|
309
|
+
|
|
310
|
+
Unit test initiative complete: 234 tests passing across 9 packages.
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
Archive files are plain Markdown — edit them directly. After editing, run `memxcore reindex <category>` to update the RAG index.
|
|
314
|
+
|
|
315
|
+
---
|
|
316
|
+
|
|
317
|
+
## Security Considerations
|
|
318
|
+
|
|
319
|
+
**Storage permissions:** Memory files (`storage/`, `knowledge.db`) are created with default OS permissions (typically 0644). On shared/multi-user systems, restrict access manually:
|
|
320
|
+
```bash
|
|
321
|
+
chmod 700 memxcore/storage/
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
**HTTP server:** The optional HTTP API (`memxcore.server`) has **no authentication** and is intended for local development only. Do **not** expose it to a network. It binds to `127.0.0.1` by default — never change this to `0.0.0.0` in production.
|
|
325
|
+
|
|
326
|
+
**`memxcore mine` command:** Reads any file the user specifies. This is by design (same as `cat`), but be careful not to pipe untrusted paths into it if used in scripts.
|
|
327
|
+
|
|
328
|
+
**LLM prompt injection:** Like all LLM-based systems, stored memory content is passed to the LLM during distillation and search. Malicious content in memories could influence LLM outputs. Category names from LLM responses are sanitized to prevent path traversal, but fabricated memory content is a fundamental limitation of LLM-based extraction.
|
|
329
|
+
|
|
330
|
+
**Dependency versions:** Core dependencies use `>=` pins without upper bounds. For production deployments, use a lock file (`pip freeze > requirements.lock`) to pin exact versions.
|
|
331
|
+
|
|
332
|
+
---
|
|
333
|
+
|
|
334
|
+
## Troubleshooting
|
|
335
|
+
|
|
336
|
+
Run `memxcore doctor` first — it checks everything and tells you what to fix.
|
|
337
|
+
|
|
338
|
+
**RAG not working**
|
|
339
|
+
```bash
|
|
340
|
+
pip install 'memxcore[rag]'
|
|
341
|
+
# First run downloads the embedding model (~80MB) + torch (~500MB)
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
**LLM distillation falling back to basic**
|
|
345
|
+
```bash
|
|
346
|
+
echo $ANTHROPIC_API_KEY # must be set
|
|
347
|
+
# Without API key, memories go to archive/general.md (uncategorized)
|
|
348
|
+
```
|
|
349
|
+
|
|
350
|
+
**Rebuild RAG index**
|
|
351
|
+
```bash
|
|
352
|
+
memxcore reindex
|
|
353
|
+
```
|
|
354
|
+
|
|
355
|
+
---
|
|
356
|
+
|
|
357
|
+
## License
|
|
358
|
+
|
|
359
|
+
MIT. See [LICENSE](LICENSE).
|
memxcore-0.1.0/README.md
ADDED
|
@@ -0,0 +1,321 @@
|
|
|
1
|
+
# MemXCore
|
|
2
|
+
|
|
3
|
+
**English** | [中文](README.zh-CN.md)
|
|
4
|
+
|
|
5
|
+
Persistent memory system for AI coding assistants. Stores and retrieves memories across sessions via MCP (Model Context Protocol), with semantic search powered by ChromaDB + sentence-transformers.
|
|
6
|
+
|
|
7
|
+
Works with **Claude Code**, **Cursor**, **Gemini CLI**, and any MCP-compatible tool.
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Architecture
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
RECENT.md <- L0: raw append log (WAL). Every remember() lands here.
|
|
15
|
+
archive/<cat>.md <- L1: distilled long-term memory, per category, with YAML front matter
|
|
16
|
+
USER.md <- L2: permanent facts (remember(permanent=True)), never compacted
|
|
17
|
+
chroma/ <- RAG vector index (rebuilt from archive/ files, lossy-safe)
|
|
18
|
+
index.json <- Keyword search index (tags + summaries from archive/ front matter)
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
**Search tiers (in order):**
|
|
22
|
+
1. **RAG semantic search** (ChromaDB + BM25, RRF fusion) — most accurate, requires `chromadb` + `sentence-transformers`
|
|
23
|
+
2. **LLM relevance judgment** — Claude reads all facts and picks relevant ones, requires API key
|
|
24
|
+
3. **Keyword fallback** — always available, scans archive files directly
|
|
25
|
+
|
|
26
|
+
**Write flow:**
|
|
27
|
+
1. `remember(text)` -> append to RECENT.md
|
|
28
|
+
2. When RECENT.md exceeds token threshold -> LLM distills into structured facts
|
|
29
|
+
3. Each fact -> written to `archive/<category>.md` + upserted into ChromaDB (dual-write)
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Quick Start
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
pip install memxcore # core
|
|
37
|
+
pip install 'memxcore[rag]' # + semantic search (recommended)
|
|
38
|
+
pip install 'memxcore[all]' # everything
|
|
39
|
+
|
|
40
|
+
# Or from source:
|
|
41
|
+
pip install -r memxcore/requirements.txt
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
**Check your setup:**
|
|
45
|
+
```bash
|
|
46
|
+
memxcore doctor
|
|
47
|
+
# or: python -m memxcore.cli doctor
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
**Requirements:** Python 3.11+
|
|
51
|
+
|
|
52
|
+
**LLM API key** (any provider supported via [litellm](https://docs.litellm.ai/)):
|
|
53
|
+
```bash
|
|
54
|
+
# Pick one:
|
|
55
|
+
export ANTHROPIC_API_KEY=sk-ant-... # Anthropic (direct API)
|
|
56
|
+
export ANTHROPIC_AUTH_TOKEN=xxx # Anthropic (gateway/proxy, with ANTHROPIC_BASE_URL)
|
|
57
|
+
export OPENAI_API_KEY=sk-... # OpenAI
|
|
58
|
+
export GEMINI_API_KEY=... # Google Gemini
|
|
59
|
+
# Or use Ollama for fully local/offline: no key needed, just set model in config.yaml
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Without any API key, compaction falls back to basic mode (truncation instead of LLM distillation).
|
|
63
|
+
|
|
64
|
+
> **Important: Compaction is not automatic.**
|
|
65
|
+
> MemXCore does not detect session boundaries. If compaction is never triggered,
|
|
66
|
+
> memories accumulate in RECENT.md and are not distilled into long-term storage.
|
|
67
|
+
>
|
|
68
|
+
> **Recommended setup** (pick one):
|
|
69
|
+
> - Use the provided **Stop hook** (`memxcore.hooks.auto_remember`) — extracts facts after each response and triggers compaction automatically.
|
|
70
|
+
> - Call `compact(force=True)` via the MCP tool at the end of each session.
|
|
71
|
+
> - Run `memxcore compact` from the command line as a cron job or manual step.
|
|
72
|
+
>
|
|
73
|
+
> See the [Integration](#integration-claude-code) section for hook setup examples.
|
|
74
|
+
|
|
75
|
+
**Workspace resolution:**
|
|
76
|
+
|
|
77
|
+
By default, MemXCore resolves the workspace from the module's install location (suitable for development / editable installs). For pip-installed usage, set `MEMXCORE_WORKSPACE` to your project root:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
export MEMXCORE_WORKSPACE=/path/to/your/project
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Or in your MCP server config:
|
|
84
|
+
```json
|
|
85
|
+
{
|
|
86
|
+
"env": { "MEMXCORE_WORKSPACE": "/path/to/your/project" }
|
|
87
|
+
}
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## Configuration (`config.yaml`)
|
|
93
|
+
|
|
94
|
+
Located at `memxcore/config.yaml`. All fields are optional — sensible defaults apply.
|
|
95
|
+
|
|
96
|
+
```yaml
|
|
97
|
+
compaction:
|
|
98
|
+
strategy: llm # 'llm' = LLM distills to structured facts (via litellm)
|
|
99
|
+
# 'basic' = truncate first 50 lines to general.md
|
|
100
|
+
threshold_tokens: 2000 # Trigger compaction when RECENT.md exceeds this
|
|
101
|
+
min_entries: 3 # Don't compact if fewer than N entries in RECENT.md
|
|
102
|
+
check_interval: 5 # Check token count every N remember() calls (reduces I/O)
|
|
103
|
+
stale_minutes: 10 # Auto-compact on search() if RECENT.md older than this
|
|
104
|
+
|
|
105
|
+
llm:
|
|
106
|
+
model: anthropic/claude-sonnet # litellm format: provider/model
|
|
107
|
+
# Examples: openai/gpt-4o, gemini/gemini-2.5-flash, ollama/llama3
|
|
108
|
+
api_key_env: ANTHROPIC_API_KEY # Env var name (litellm also auto-reads OPENAI_API_KEY, etc.)
|
|
109
|
+
# base_url: # Optional: custom API endpoint (gateway/proxy)
|
|
110
|
+
|
|
111
|
+
rag:
|
|
112
|
+
embedding_model: all-MiniLM-L6-v2 # sentence-transformers model (80MB, CPU-friendly)
|
|
113
|
+
top_k: 10 # Max results from semantic search
|
|
114
|
+
rrf_k: 60 # Reciprocal Rank Fusion constant
|
|
115
|
+
|
|
116
|
+
watch: false # Auto-reindex archive/*.md on file change (off by default)
|
|
117
|
+
|
|
118
|
+
memory:
|
|
119
|
+
categories:
|
|
120
|
+
- id: user_model
|
|
121
|
+
description: "User identity, preferences, communication style, feedback"
|
|
122
|
+
- id: domain
|
|
123
|
+
description: "Transferable domain/technical knowledge — valid across projects"
|
|
124
|
+
- id: project_state
|
|
125
|
+
description: "Active tasks, in-progress decisions — decays within weeks"
|
|
126
|
+
- id: episodic
|
|
127
|
+
description: "Time-bound events and past decisions — has TTL"
|
|
128
|
+
- id: references
|
|
129
|
+
description: "URLs, doc links, dashboards, ticket trackers, Slack channels"
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## Integration
|
|
135
|
+
|
|
136
|
+
### One-command setup (recommended)
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
memxcore setup # auto-detects tools and configures everything
|
|
140
|
+
memxcore setup --dry-run # preview without making changes
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
Automatically detects and configures:
|
|
144
|
+
|
|
145
|
+
| Tool | What it does |
|
|
146
|
+
|------|-------------|
|
|
147
|
+
| **Claude Code** | Registers MCP server, installs hooks (auto-remember + auto-compact), appends agent rules to `~/.claude/CLAUDE.md` |
|
|
148
|
+
| **Cursor** | Writes MCP config to `~/.cursor/mcp.json`, copies rules to `~/.cursor/rules/memxcore.md` |
|
|
149
|
+
| **Windsurf** | Writes MCP config to `~/.codeium/windsurf/mcp_config.json` |
|
|
150
|
+
| **Codex (OpenAI)** | Writes MCP config to `~/.codex/config.toml` |
|
|
151
|
+
| **Gemini CLI** | Writes MCP config to `~/.gemini/settings.json` |
|
|
152
|
+
|
|
153
|
+
After setup, restart your tool and verify the MCP connection.
|
|
154
|
+
|
|
155
|
+
### Manual setup
|
|
156
|
+
|
|
157
|
+
If you prefer manual configuration, the MCP server config for any tool is:
|
|
158
|
+
|
|
159
|
+
```json
|
|
160
|
+
{
|
|
161
|
+
"mcpServers": {
|
|
162
|
+
"memxcore": {
|
|
163
|
+
"command": "/path/to/your/.venv/bin/python",
|
|
164
|
+
"args": ["-m", "memxcore.mcp_server"],
|
|
165
|
+
"env": { "MEMXCORE_WORKSPACE": "/path/to/your/workspace" }
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
---
|
|
172
|
+
|
|
173
|
+
## MCP Tools Reference
|
|
174
|
+
|
|
175
|
+
| Tool | Args | Description |
|
|
176
|
+
|------|------|-------------|
|
|
177
|
+
| `remember` | `text`, `category?`, `permanent?`, `tenant_id?` | Store a memory |
|
|
178
|
+
| `search` | `query`, `max_results?`, `tenant_id?` | Retrieve memories (semantic + keyword) |
|
|
179
|
+
| `compact` | `force?`, `tenant_id?` | Distill RECENT.md into categorized archives |
|
|
180
|
+
| `reindex` | `category?`, `tenant_id?` | Rebuild RAG index after manual edits |
|
|
181
|
+
|
|
182
|
+
**Categories for `remember`:**
|
|
183
|
+
`user_model` / `domain` / `project_state` / `episodic` / `references`
|
|
184
|
+
|
|
185
|
+
**Permanent memories:**
|
|
186
|
+
Use `permanent=True` to write directly to USER.md (L2). These are never compacted and always have the highest priority in search. Use for stable facts like core user identity or long-term architectural decisions.
|
|
187
|
+
|
|
188
|
+
---
|
|
189
|
+
|
|
190
|
+
## Multi-Tenant
|
|
191
|
+
|
|
192
|
+
All MCP tools and CLI commands accept an optional `tenant_id`. Each tenant gets isolated storage under `tenants/<tenant_id>/storage/`. Omit `tenant_id` for single-user mode.
|
|
193
|
+
|
|
194
|
+
```bash
|
|
195
|
+
# CLI
|
|
196
|
+
memxcore --tenant alice search "preferences"
|
|
197
|
+
|
|
198
|
+
# MCP
|
|
199
|
+
search("preferences", tenant_id="alice")
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
Tenants can also have their own `config.yaml` override at `tenants/<tenant_id>/config.yaml`.
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
206
|
+
## CLI Reference
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
memxcore setup # auto-detect tools, configure everything
|
|
210
|
+
memxcore setup --dry-run # preview without changes
|
|
211
|
+
memxcore doctor # check system readiness
|
|
212
|
+
memxcore reindex # rebuild RAG index
|
|
213
|
+
memxcore compact # force distillation
|
|
214
|
+
memxcore search "query" # search memories (debug)
|
|
215
|
+
memxcore benchmark # search precision benchmark
|
|
216
|
+
memxcore mine <path> # import conversations
|
|
217
|
+
memxcore --tenant alice doctor # multi-tenant
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
All commands also work via `python -m memxcore.cli <command>`.
|
|
221
|
+
|
|
222
|
+
---
|
|
223
|
+
|
|
224
|
+
## HTTP API Server (optional)
|
|
225
|
+
|
|
226
|
+
For development/testing. In production, use the MCP server.
|
|
227
|
+
|
|
228
|
+
```bash
|
|
229
|
+
pip install 'memxcore[server]'
|
|
230
|
+
python -m memxcore.server --host 127.0.0.1 --port 8000
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Endpoints:
|
|
234
|
+
```
|
|
235
|
+
POST /remember { "text": "...", "level": null }
|
|
236
|
+
GET /search?query= returns List[MemoryResult]
|
|
237
|
+
POST /compact ?force=false
|
|
238
|
+
POST /rebuild-rag rebuild ChromaDB from archive files
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
---
|
|
242
|
+
|
|
243
|
+
## Storage Structure
|
|
244
|
+
|
|
245
|
+
```
|
|
246
|
+
memxcore/
|
|
247
|
+
+-- storage/
|
|
248
|
+
+-- RECENT.md Raw memory log. Cleared after each compaction.
|
|
249
|
+
+-- USER.md Permanent facts (permanent=True). Never compacted.
|
|
250
|
+
+-- index.json Keyword search index. Auto-rebuilt after compaction.
|
|
251
|
+
+-- chroma/ ChromaDB vector index. Rebuildable from archive/.
|
|
252
|
+
+-- archive/
|
|
253
|
+
+-- user_model.md User identity, preferences, feedback
|
|
254
|
+
+-- domain.md Domain/technical knowledge
|
|
255
|
+
+-- project_state.md Active tasks, decisions (stale after weeks)
|
|
256
|
+
+-- episodic.md Past events and decisions
|
|
257
|
+
+-- references.md URLs, doc links, external pointers
|
|
258
|
+
+-- general.md Fallback when LLM distillation unavailable
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
**Archive file format:**
|
|
262
|
+
```markdown
|
|
263
|
+
---
|
|
264
|
+
topic: project_state
|
|
265
|
+
tags: [api, testing]
|
|
266
|
+
last_distilled: 2026-04-07T14:23:01
|
|
267
|
+
confidence_level: 4
|
|
268
|
+
---
|
|
269
|
+
|
|
270
|
+
## [2026-04-07T14:23:01]
|
|
271
|
+
|
|
272
|
+
Unit test initiative complete: 234 tests passing across 9 packages.
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
Archive files are plain Markdown — edit them directly. After editing, run `memxcore reindex <category>` to update the RAG index.
|
|
276
|
+
|
|
277
|
+
---
|
|
278
|
+
|
|
279
|
+
## Security Considerations
|
|
280
|
+
|
|
281
|
+
**Storage permissions:** Memory files (`storage/`, `knowledge.db`) are created with default OS permissions (typically 0644). On shared/multi-user systems, restrict access manually:
|
|
282
|
+
```bash
|
|
283
|
+
chmod 700 memxcore/storage/
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
**HTTP server:** The optional HTTP API (`memxcore.server`) has **no authentication** and is intended for local development only. Do **not** expose it to a network. It binds to `127.0.0.1` by default — never change this to `0.0.0.0` in production.
|
|
287
|
+
|
|
288
|
+
**`memxcore mine` command:** Reads any file the user specifies. This is by design (same as `cat`), but be careful not to pipe untrusted paths into it if used in scripts.
|
|
289
|
+
|
|
290
|
+
**LLM prompt injection:** Like all LLM-based systems, stored memory content is passed to the LLM during distillation and search. Malicious content in memories could influence LLM outputs. Category names from LLM responses are sanitized to prevent path traversal, but fabricated memory content is a fundamental limitation of LLM-based extraction.
|
|
291
|
+
|
|
292
|
+
**Dependency versions:** Core dependencies use `>=` pins without upper bounds. For production deployments, use a lock file (`pip freeze > requirements.lock`) to pin exact versions.
|
|
293
|
+
|
|
294
|
+
---
|
|
295
|
+
|
|
296
|
+
## Troubleshooting
|
|
297
|
+
|
|
298
|
+
Run `memxcore doctor` first — it checks everything and tells you what to fix.
|
|
299
|
+
|
|
300
|
+
**RAG not working**
|
|
301
|
+
```bash
|
|
302
|
+
pip install 'memxcore[rag]'
|
|
303
|
+
# First run downloads the embedding model (~80MB) + torch (~500MB)
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
**LLM distillation falling back to basic**
|
|
307
|
+
```bash
|
|
308
|
+
echo $ANTHROPIC_API_KEY # must be set
|
|
309
|
+
# Without API key, memories go to archive/general.md (uncategorized)
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
**Rebuild RAG index**
|
|
313
|
+
```bash
|
|
314
|
+
memxcore reindex
|
|
315
|
+
```
|
|
316
|
+
|
|
317
|
+
---
|
|
318
|
+
|
|
319
|
+
## License
|
|
320
|
+
|
|
321
|
+
MIT. See [LICENSE](LICENSE).
|