@199-bio/engram 0.11.1 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +35 -31
- package/dist/consolidation/plan.d.ts.map +1 -1
- package/dist/index.js +7 -3
- package/dist/retrieval/hybrid.d.ts.map +1 -1
- package/dist/retrieval/index.d.ts.map +1 -1
- package/dist/retrieval/jina.d.ts.map +1 -0
- package/dist/web/server.d.ts.map +1 -1
- package/logo.png +0 -0
- package/package.json +2 -3
- package/src/consolidation/consolidator.ts +9 -9
- package/src/consolidation/plan.ts +9 -9
- package/src/index.ts +7 -3
- package/src/retrieval/hybrid.ts +11 -11
- package/src/retrieval/index.ts +1 -1
- package/src/retrieval/jina-bridge.py +297 -0
- package/src/retrieval/{colbert.ts → jina.ts} +31 -16
- package/src/web/chat-handler.ts +4 -4
- package/src/web/server.ts +117 -6
- package/src/web/static/app.js +12 -0
- package/src/web/static/index.html +35 -0
- package/tests/retrieval/hybrid.test.ts +158 -0
- package/tests/settings.test.ts +68 -0
- package/tests/storage/database.test.ts +315 -0
- package/vitest.config.ts +7 -0
- package/LIVING_PLAN.md +0 -180
- package/PLAN.md +0 -514
- package/boba-prompt.md +0 -107
- package/src/retrieval/colbert-bridge.py +0 -222
- package/tests/test-interactive.js +0 -218
- package/tests/test-mcp.sh +0 -81
package/README.md
CHANGED
|
@@ -1,12 +1,16 @@
|
|
|
1
1
|
# Engram
|
|
2
2
|
|
|
3
|
+
<p align="center">
|
|
4
|
+
<img src="logo.png" alt="Engram" width="480" />
|
|
5
|
+
</p>
|
|
6
|
+
|
|
3
7
|
**Give your AI a perfect memory.**
|
|
4
8
|
|
|
5
|
-
Every conversation you have with your AI disappears the moment it ends. Names you've mentioned, preferences you've shared, the context of your life—
|
|
9
|
+
Every conversation you have with your AI disappears the moment it ends. Names you've mentioned, preferences you've shared, the context of your life — gone. You repeat yourself. You re-explain who people are. You remind it of things you've already said.
|
|
6
10
|
|
|
7
|
-
Engram
|
|
11
|
+
Engram fixes that.
|
|
8
12
|
|
|
9
|
-
It
|
|
13
|
+
It lets your AI remember. Not just store text — actually remember, the way you do. Important things stick. Trivial things fade. And everything connects.
|
|
10
14
|
|
|
11
15
|
> *An engram is a unit of cognitive information imprinted in a physical substance—the biological basis of memory.*
|
|
12
16
|
|
|
@@ -14,7 +18,7 @@ It gives your AI the ability to remember. Not just store text, but truly remembe
|
|
|
14
18
|
|
|
15
19
|
## How It Works
|
|
16
20
|
|
|
17
|
-
Tell your AI something once:
|
|
21
|
+
Tell your AI something once. Just once:
|
|
18
22
|
|
|
19
23
|
> "My colleague Sarah is allergic to shellfish and prefers window seats. She's leading the Q1 product launch."
|
|
20
24
|
|
|
@@ -22,7 +26,7 @@ Weeks later, ask:
|
|
|
22
26
|
|
|
23
27
|
> "I'm booking a team lunch and flights for the offsite—what should I know?"
|
|
24
28
|
|
|
25
|
-
Engram connects the dots. It remembers Sarah
|
|
29
|
+
Engram connects the dots. It remembers Sarah — the allergy, the seating preference, the workload. Your AI can now actually help. It'll suggest restaurants without shellfish and book her a window seat. It'll flag that she's probably swamped with the launch.
|
|
26
30
|
|
|
27
31
|
This isn't keyword matching. It's understanding.
|
|
28
32
|
|
|
@@ -32,11 +36,11 @@ This isn't keyword matching. It's understanding.
|
|
|
32
36
|
|
|
33
37
|
Engram models memory the way your brain does.
|
|
34
38
|
|
|
35
|
-
**Things fade.** A memory from six months ago that you've never
|
|
39
|
+
**Things fade.** A memory from six months ago that you've never revisited becomes harder to find. But something important — a name, a birthday, a preference — stays accessible even as time passes.
|
|
36
40
|
|
|
37
|
-
**Recall strengthens.** Every time a memory surfaces, it becomes more permanent. The things you think about often
|
|
41
|
+
**Recall strengthens.** Every time a memory surfaces, it becomes more permanent. The things you think about often are the things you won't forget.
|
|
38
42
|
|
|
39
|
-
**Everything connects.** People link to places
|
|
43
|
+
**Everything connects.** People link to places, places to events. Ask about Sarah, and her company, her projects, her preferences all surface together.
|
|
40
44
|
|
|
41
45
|
---
|
|
42
46
|
|
|
@@ -48,7 +52,7 @@ Install globally:
|
|
|
48
52
|
npm install -g @199-bio/engram
|
|
49
53
|
```
|
|
50
54
|
|
|
51
|
-
Add to **
|
|
55
|
+
Add to **MCP desktop client** (`~/Library/Application Support/AI/AI_desktop_config.json`):
|
|
52
56
|
|
|
53
57
|
```json
|
|
54
58
|
{
|
|
@@ -64,10 +68,10 @@ Add to **Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_
|
|
|
64
68
|
}
|
|
65
69
|
```
|
|
66
70
|
|
|
67
|
-
Or with **
|
|
71
|
+
Or with **AI coding assistant**:
|
|
68
72
|
|
|
69
73
|
```bash
|
|
70
|
-
|
|
74
|
+
AI mcp add engram -- npx -y @199-bio/engram
|
|
71
75
|
```
|
|
72
76
|
|
|
73
77
|
That's it. Your AI now remembers.
|
|
@@ -89,15 +93,15 @@ Just talk naturally. Your AI handles the rest.
|
|
|
89
93
|
- "When is my anniversary?"
|
|
90
94
|
|
|
91
95
|
**Build a knowledge graph** of your world:
|
|
92
|
-
- People, places, organizations
|
|
96
|
+
- People, places, organizations — how they connect
|
|
93
97
|
- Observations about each entity
|
|
94
|
-
- Relationships that span your
|
|
98
|
+
- Relationships that span your whole life
|
|
95
99
|
|
|
96
100
|
---
|
|
97
101
|
|
|
98
102
|
## Privacy
|
|
99
103
|
|
|
100
|
-
Your memories stay on your machine. Everything
|
|
104
|
+
Your memories stay on your machine. Everything lives in `~/.engram/`. The only external call is optional — if you provide an API key, Engram can compress old memories into summaries. But core functionality works offline.
|
|
101
105
|
|
|
102
106
|
---
|
|
103
107
|
|
|
@@ -110,14 +114,14 @@ Your AI gets these capabilities:
|
|
|
110
114
|
|
|
111
115
|
| Tool | Purpose |
|
|
112
116
|
|------|---------|
|
|
113
|
-
| `remember` | Store new information with importance
|
|
117
|
+
| `remember` | Store new information with importance and emotional weight |
|
|
114
118
|
| `recall` | Find relevant memories ranked by relevance and recency |
|
|
115
119
|
| `forget` | Remove a specific memory |
|
|
116
|
-
| `create_entity` | Add a person, place, or concept to the
|
|
120
|
+
| `create_entity` | Add a person, place, or concept to the graph |
|
|
117
121
|
| `observe` | Record a fact about an entity |
|
|
118
122
|
| `relate` | Connect two entities (e.g., "works at", "married to") |
|
|
119
123
|
| `query_entity` | Get everything known about someone or something |
|
|
120
|
-
| `list_entities` | See all tracked people
|
|
124
|
+
| `list_entities` | See all tracked people and places |
|
|
121
125
|
| `stats` | View memory statistics |
|
|
122
126
|
| `consolidate` | Compress old memories and detect contradictions |
|
|
123
127
|
| `engram_web` | Launch a visual memory browser |
|
|
@@ -127,13 +131,13 @@ Your AI gets these capabilities:
|
|
|
127
131
|
<details>
|
|
128
132
|
<summary><strong>How Search Works</strong></summary>
|
|
129
133
|
|
|
130
|
-
Engram
|
|
134
|
+
Engram runs three search methods at once:
|
|
131
135
|
|
|
132
|
-
1. **Keywords** — SQLite FTS5 finds exact matches for names
|
|
133
|
-
2. **Meaning** —
|
|
136
|
+
1. **Keywords** — SQLite FTS5 finds exact matches for names and phrases
|
|
137
|
+
2. **Meaning** — Jina v5 embeddings find conceptually related content
|
|
134
138
|
3. **Connections** — The knowledge graph expands to related entities
|
|
135
139
|
|
|
136
|
-
Results are
|
|
140
|
+
Results are merged, then ranked by recency and importance. Fresh memories surface first. Important memories resist fading.
|
|
137
141
|
|
|
138
142
|
</details>
|
|
139
143
|
|
|
@@ -147,23 +151,23 @@ Retention = e^(-time / stability)
|
|
|
147
151
|
```
|
|
148
152
|
|
|
149
153
|
- **Time** is days since the memory was last accessed
|
|
150
|
-
- **Stability** is memory strength, which
|
|
154
|
+
- **Stability** is memory strength, which grows each time you recall something
|
|
151
155
|
|
|
152
|
-
High-importance and emotionally
|
|
156
|
+
High-importance and emotionally weighted memories decay slower. Frequently accessed memories become permanent.
|
|
153
157
|
|
|
154
158
|
</details>
|
|
155
159
|
|
|
156
160
|
<details>
|
|
157
161
|
<summary><strong>How Consolidation Works</strong></summary>
|
|
158
162
|
|
|
159
|
-
With an API key, Engram
|
|
163
|
+
With an API key, Engram compresses old memories — like sleep turning experiences into long-term storage.
|
|
160
164
|
|
|
161
|
-
1. Groups related low-importance memories
|
|
165
|
+
1. Groups related low-importance memories
|
|
162
166
|
2. Creates AI-generated summaries (digests)
|
|
163
|
-
3.
|
|
167
|
+
3. Flags contradictory information
|
|
164
168
|
4. Archives the originals
|
|
165
169
|
|
|
166
|
-
|
|
170
|
+
Storage stays lean, but nothing important gets lost.
|
|
167
171
|
|
|
168
172
|
</details>
|
|
169
173
|
|
|
@@ -177,7 +181,7 @@ engram/
|
|
|
177
181
|
│ ├── storage/database.ts # SQLite with temporal fields
|
|
178
182
|
│ ├── graph/knowledge-graph.ts
|
|
179
183
|
│ ├── retrieval/
|
|
180
|
-
│ │ ├──
|
|
184
|
+
│ │ ├── jina.ts # Jina v5 semantic search
|
|
181
185
|
│ │ └── hybrid.ts # Fusion + decay + salience
|
|
182
186
|
│ ├── consolidation/consolidator.ts
|
|
183
187
|
│ └── web/server.ts # Visual browser
|
|
@@ -208,13 +212,13 @@ npm run build
|
|
|
208
212
|
npm install -g .
|
|
209
213
|
```
|
|
210
214
|
|
|
211
|
-
For semantic search, install
|
|
215
|
+
For semantic search, install the Jina embeddings package:
|
|
212
216
|
|
|
213
217
|
```bash
|
|
214
|
-
pip install
|
|
218
|
+
pip install jina-grep
|
|
215
219
|
```
|
|
216
220
|
|
|
217
|
-
If unavailable, Engram falls back to keyword-only search
|
|
221
|
+
This uses Jina v5 embeddings with MLX Metal acceleration (~9ms/query). If unavailable, Engram falls back to keyword-only search.
|
|
218
222
|
|
|
219
223
|
</details>
|
|
220
224
|
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"plan.d.ts","sourceRoot":"","sources":["../../src/consolidation/plan.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAGH,OAAO,EAAE,cAAc,EAAE,MAAM,EAAE,OAAO,EAAE,uBAAuB,EAAE,MAAM,wBAAwB,CAAC;AAiClG,MAAM,WAAW,iBAAiB;IAChC,sBAAsB,EAAE,MAAM,CAAC;IAC/B,sBAAsB,EAAE,MAAM,CAAC;IAC/B,cAAc,EAAE,OAAO,CAAC;IACxB,gBAAgB,EAAE,MAAM,CAAC;IACzB,aAAa,EAAE,MAAM,CAAC;IACtB,WAAW,EAAE,MAAM,CAAC;IACpB,UAAU,EAAE,MAAM,CAAC;IACnB,eAAe,EAAE,MAAM,CAAC;IACxB,UAAU,EAAE,OAAO,CAAC;IACpB,kBAAkB,EAAE,MAAM,CAAC;IAC3B,MAAM,EAAE,SAAS,EAAE,CAAC;CACrB;AAED,MAAM,WAAW,SAAS;IACxB,KAAK,EAAE,UAAU,GAAG,UAAU,GAAG,OAAO,GAAG,SAAS,CAAC;IACrD,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,aAAa,EAAE,MAAM,CAAC;IACtB,eAAe,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,WAAW,qBAAqB;IACpC,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,uBAAuB,CAAC,OAAO,CAAC,CAAC;IACxC,gBAAgB,EAAE,MAAM,CAAC;IACzB,YAAY,EAAE,MAAM,CAAC;IACrB,iBAAiB,EAAE,MAAM,CAAC;IAC1B,iBAAiB,EAAE,MAAM,CAAC;IAC1B,cAAc,EAAE,MAAM,CAAC;IACvB,mBAAmB,EAAE,MAAM,CAAC;IAC5B,UAAU,EAAE,MAAM,CAAC;IACnB,aAAa,EAAE,MAAM,CAAC;IACtB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,SAAS,EAAE,IAAI,CAAC;IAChB,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,YAAY,GAAG,eAAe,GAAG,oBAAoB,GAAG,iBAAiB,CAAC;IAChF,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,OAAO,CAAC;IACnB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,qBAAa,iBAAiB;IAC5B,OAAO,CAAC,EAAE,CAAiB;IAC3B,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,MAAM,CAAgB;IAC9B,OAAO,CAAC,YAAY,CAAa;IACjC,OAAO,CAAC,YAAY,CAAa;IACjC,OAAO,CAAC,QAAQ,CAAa;IAC7B,OAAO,CAAC,SAAS,CAAa;gBAElB,EAAE,EAAE,cAAc;IAK9B;;OAEG;IACH,aAAa,IAAI,iBAAiB;IA2ElC;;;OAGG;IACH,sBAAsB,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,EAAE;IAsC/C;;OAEG;IACH,sBAAsB,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,EAAE;IAShD;;OAEG;IACH,gBAAgB,CAAC,KAAK,EAAE,uBAAuB,CAAC,OAAO,CAAC,EAAE,YAAY,EAAE,MAAM,GAAG,uBAAuB;IAIxG;;OAEG;IACH,cAAc,CAAC,OAAO,EAAE,OAAO,CAAC;QAC9B,KAAK,EAAE,uBAAuB,CAAC,OAAO,CAAC,CAAC;QACxC,gBAAgB,EAAE,MAAM,CAAC;QACzB,YAAY,EAAE,MAAM,CAAC;QACrB,iBAAiB,EAAE,MAAM,CAAC;QAC1B,iBAAiB,EAAE,MAAM,CAAC;QAC1B,cAAc,EAAE,MAAM,CAAC;QACvB,mBAAmB,EAAE,MAAM,CAAC;QAC5B,UAAU,EAAE,MAAM,CAAC;QACnB,aAAa,EAAE,MAAM,CAAC;KACvB,CAAC,GAAG,IAAI;IAcT;;OAEG;IACH,QAAQ,IAAI,IAAI;IAIhB;;OAEG;IACH,IAAI,CAAC,KAAK,EAAE,MAAM,GAAG,IAAI;IAIzB;;OAEG;IACH,WAAW,IAAI,qBAAqB;IAoBpC;;OAEG;IACH,cAAc,IAAI,uBAAuB,GAAG,IAAI;IAIhD;;OAEG;IACH,UAAU,CAAC,UAAU,EAAE,uBAAuB,GAAG,IAAI;IAIrD;;OAEG;IACH,aAAa,CAAC,OAAO,EAAE,OAAO,EAAE,UAAU,CAAC,EAAE,MAAM,GAAG,IAAI;IAO1D;;OAEG;IACH,YAAY,CAAC,OAAO,EAAE,OAAO,GAAG,IAAI;IAOpC;;OAEG;IACH,WAAW,CAAC,KAAK,EAAE,MAAM,GAAG,IAAI;IAIhC;;OAEG;IACH,qBAAqB,IAAI,eAAe,EAAE;IA2C1C;;OAEG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAK5B;;OAEG;IACH,OAAO,CAAC,wBAAwB;IAKhC;;OAEG;IACH,OAAO,CAAC,uBAAuB;IAO/B;;OAEG;IACH,aAAa,CAAC,KAAK,EAAE,MAAM,GAAG,
|
|
1
|
+
{"version":3,"file":"plan.d.ts","sourceRoot":"","sources":["../../src/consolidation/plan.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAGH,OAAO,EAAE,cAAc,EAAE,MAAM,EAAE,OAAO,EAAE,uBAAuB,EAAE,MAAM,wBAAwB,CAAC;AAiClG,MAAM,WAAW,iBAAiB;IAChC,sBAAsB,EAAE,MAAM,CAAC;IAC/B,sBAAsB,EAAE,MAAM,CAAC;IAC/B,cAAc,EAAE,OAAO,CAAC;IACxB,gBAAgB,EAAE,MAAM,CAAC;IACzB,aAAa,EAAE,MAAM,CAAC;IACtB,WAAW,EAAE,MAAM,CAAC;IACpB,UAAU,EAAE,MAAM,CAAC;IACnB,eAAe,EAAE,MAAM,CAAC;IACxB,UAAU,EAAE,OAAO,CAAC;IACpB,kBAAkB,EAAE,MAAM,CAAC;IAC3B,MAAM,EAAE,SAAS,EAAE,CAAC;CACrB;AAED,MAAM,WAAW,SAAS;IACxB,KAAK,EAAE,UAAU,GAAG,UAAU,GAAG,OAAO,GAAG,SAAS,CAAC;IACrD,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,aAAa,EAAE,MAAM,CAAC;IACtB,eAAe,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,WAAW,qBAAqB;IACpC,KAAK,EAAE,MAAM,CAAC;IACd,KAAK,EAAE,uBAAuB,CAAC,OAAO,CAAC,CAAC;IACxC,gBAAgB,EAAE,MAAM,CAAC;IACzB,YAAY,EAAE,MAAM,CAAC;IACrB,iBAAiB,EAAE,MAAM,CAAC;IAC1B,iBAAiB,EAAE,MAAM,CAAC;IAC1B,cAAc,EAAE,MAAM,CAAC;IACvB,mBAAmB,EAAE,MAAM,CAAC;IAC5B,UAAU,EAAE,MAAM,CAAC;IACnB,aAAa,EAAE,MAAM,CAAC;IACtB,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,SAAS,EAAE,IAAI,CAAC;IAChB,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,YAAY,GAAG,eAAe,GAAG,oBAAoB,GAAG,iBAAiB,CAAC;IAChF,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,OAAO,CAAC;IACnB,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,qBAAa,iBAAiB;IAC5B,OAAO,CAAC,EAAE,CAAiB;IAC3B,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,MAAM,CAAgB;IAC9B,OAAO,CAAC,YAAY,CAAa;IACjC,OAAO,CAAC,YAAY,CAAa;IACjC,OAAO,CAAC,QAAQ,CAAa;IAC7B,OAAO,CAAC,SAAS,CAAa;gBAElB,EAAE,EAAE,cAAc;IAK9B;;OAEG;IACH,aAAa,IAAI,iBAAiB;IA2ElC;;;OAGG;IACH,sBAAsB,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,EAAE;IAsC/C;;OAEG;IACH,sBAAsB,CAAC,KAAK,EAAE,MAAM,GAAG,OAAO,EAAE;IAShD;;OAEG;IACH,gBAAgB,CAAC,KAAK,EAAE,uBAAuB,CAAC,OAAO,CAAC,EAAE,YAAY,EAAE,MAAM,GAAG,uBAAuB;IAIxG;;OAEG;IACH,cAAc,CAAC,OAAO,EAAE,OAAO,CAAC;QAC9B,KAAK,EAAE,uBAAuB,CAAC,OAAO,CAAC,CAAC;QACxC,gBAAgB,EAAE,MAAM,CAAC;QACzB,YAAY,EAAE,MAAM,CAAC;QACrB,iBAAiB,EAAE,MAAM,CAAC;QAC1B,iBAAiB,EAAE,MAAM,CAAC;QAC1B,cAAc,EAAE,MAAM,CAAC;QACvB,mBAAmB,EAAE,MAAM,CAAC;QAC5B,UAAU,EAAE,MAAM,CAAC;QACnB,aAAa,EAAE,MAAM,CAAC;KACvB,CAAC,GAAG,IAAI;IAcT;;OAEG;IACH,QAAQ,IAAI,IAAI;IAIhB;;OAEG;IACH,IAAI,CAAC,KAAK,EAAE,MAAM,GAAG,IAAI;IAIzB;;OAEG;IACH,WAAW,IAAI,qBAAqB;IAoBpC;;OAEG;IACH,cAAc,IAAI,uBAAuB,GAAG,IAAI;IAIhD;;OAEG;IACH,UAAU,CAAC,UAAU,EAAE,uBAAuB,GAAG,IAAI;IAIrD;;OAEG;IACH,aAAa,CAAC,OAAO,EAAE,OAAO,EAAE,UAAU,CAAC,EAAE,MAAM,GAAG,IAAI;IAO1D;;OAEG;IACH,YAAY,CAAC,OAAO,EAAE,OAAO,GAAG,IAAI;IAOpC;;OAEG;IACH,WAAW,CAAC,KAAK,EAAE,MAAM,GAAG,IAAI;IAIhC;;OAEG;IACH,qBAAqB,IAAI,eAAe,EAAE;IA2C1C;;OAEG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAK5B;;OAEG;IACH,OAAO,CAAC,wBAAwB;IAKhC;;OAEG;IACH,OAAO,CAAC,uBAAuB;IAO/B;;OAEG;IACH,aAAa,CAAC,KAAK,EAAE,MAAM,GAAG,QAAQ,EAAE,WAAW,EAAE,MAAM,EAAE,YAAY,EAAE,MAAM,EAAE,cAAc,CAAC,EAAE,MAAM,GAAG,MAAM;IAWnH;;OAEG;IACH,QAAQ,IAAI,MAAM;CAGnB"}
|
package/dist/index.js
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
/**
|
|
3
3
|
* Engram - High-quality personal memory for AI assistants
|
|
4
4
|
*
|
|
5
|
-
* Local-first MCP server with
|
|
5
|
+
* Local-first MCP server with Jina v5 + BM25 hybrid search
|
|
6
6
|
* and a lightweight knowledge graph.
|
|
7
7
|
*/
|
|
8
8
|
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
|
|
@@ -15,7 +15,7 @@ import { getTransportMode, getHttpPort } from "./transport/index.js";
|
|
|
15
15
|
import { startHttpServer } from "./transport/http.js";
|
|
16
16
|
import { EngramDatabase } from "./storage/database.js";
|
|
17
17
|
import { KnowledgeGraph } from "./graph/knowledge-graph.js";
|
|
18
|
-
import { createRetriever } from "./retrieval/
|
|
18
|
+
import { createRetriever } from "./retrieval/jina.js";
|
|
19
19
|
import { HybridSearch } from "./retrieval/hybrid.js";
|
|
20
20
|
import { EngramWebServer, getRunningServerUrl } from "./web/server.js";
|
|
21
21
|
import { Consolidator } from "./consolidation/consolidator.js";
|
|
@@ -81,6 +81,9 @@ function cleanup() {
|
|
|
81
81
|
if (webServer) {
|
|
82
82
|
webServer.stop();
|
|
83
83
|
}
|
|
84
|
+
if (retriever && "stop" in retriever) {
|
|
85
|
+
retriever.stop();
|
|
86
|
+
}
|
|
84
87
|
if (db) {
|
|
85
88
|
db.close();
|
|
86
89
|
}
|
|
@@ -119,13 +122,14 @@ if (getTransportMode() === "stdio") {
|
|
|
119
122
|
let db;
|
|
120
123
|
let graph;
|
|
121
124
|
let search;
|
|
125
|
+
let retriever;
|
|
122
126
|
let consolidator;
|
|
123
127
|
let webServer = null;
|
|
124
128
|
async function initialize() {
|
|
125
129
|
console.error(`[Engram] Initializing with database at ${DB_FILE}`);
|
|
126
130
|
db = new EngramDatabase(DB_FILE);
|
|
127
131
|
graph = new KnowledgeGraph(db);
|
|
128
|
-
|
|
132
|
+
retriever = await createRetriever(DB_PATH);
|
|
129
133
|
search = new HybridSearch(db, graph, retriever);
|
|
130
134
|
consolidator = new Consolidator(db, graph, search);
|
|
131
135
|
// Rebuild index with existing memories
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"hybrid.d.ts","sourceRoot":"","sources":["../../src/retrieval/hybrid.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,EAAE,cAAc,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,wBAAwB,CAAC;AACxE,OAAO,EAAE,cAAc,EAAE,MAAM,6BAA6B,CAAC;AAC7D,OAAO,EAAE,
|
|
1
|
+
{"version":3,"file":"hybrid.d.ts","sourceRoot":"","sources":["../../src/retrieval/hybrid.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AAEH,OAAO,EAAE,cAAc,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,wBAAwB,CAAC;AACxE,OAAO,EAAE,cAAc,EAAE,MAAM,6BAA6B,CAAC;AAC7D,OAAO,EAAE,aAAa,EAAE,eAAe,EAA0B,MAAM,WAAW,CAAC;AAEnF,MAAM,WAAW,kBAAkB;IACjC,MAAM,EAAE,MAAM,CAAC;IACf,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE;QACP,IAAI,CAAC,EAAE,MAAM,CAAC;QACd,QAAQ,CAAC,EAAE,MAAM,CAAC;QAClB,KAAK,CAAC,EAAE,MAAM,CAAC;QACf,SAAS,CAAC,EAAE,MAAM,CAAC;KACpB,CAAC;CACH;AAED,MAAM,WAAW,kBAAkB;IACjC,MAAM,EAAE,MAAM,CAAC;IACf,KAAK,EAAE,MAAM,CAAC;IACd,YAAY,EAAE,MAAM,EAAE,CAAC;CACxB;AAED,MAAM,WAAW,oBAAoB;IACnC,OAAO,EAAE,kBAAkB,EAAE,CAAC;IAC9B,OAAO,EAAE,kBAAkB,EAAE,CAAC;IAC9B,SAAS,EAAE,MAAM,CAAC;IAClB,kBAAkB,EAAE,KAAK,CAAC;QACxB,MAAM,EAAE,MAAM,CAAC;QACf,YAAY,EAAE,MAAM,CAAC;QACrB,QAAQ,EAAE,MAAM,CAAC;KAClB,CAAC,CAAC;CACJ;AAED;;;;;;GAMG;AACH,wBAAgB,kBAAkB,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,GAAG,MAAM,CAepE;AAED;;;GAGG;AACH,wBAAgB,iBAAiB,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,CAOxD;AAED;;GAEG;AACH,wBAAgB,WAAW,CAAC,MAAM,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,EAAE,GAAG,EAAE,IAAI,GAAG;IAAE,QAAQ,EAAE,MAAM,CAAC;IAAC,SAAS,EAAE,MAAM,CAAA;CAAE,CAYjH;AAED,qBAAa,YAAY;IAIrB,OAAO,CAAC,EAAE;IACV,OAAO,CAAC,KAAK;IACb,OAAO,CAAC,SAAS;IALnB,OAAO,CAAC,SAAS,CAAS;gBAGhB,EAAE,EAAE,cAAc,EAClB,KAAK,EAAE,cAAc,EACrB,SAAS,EAAE,aAAa,GAAG,eAAe;IAMpD;;OAEG;IACH,OAAO,CAAC,gBAAgB;IAIxB;;;;;OAKG;IACG,MAAM,CACV,KAAK,EAAE,MAAM,EACb,OAAO,GAAE;QACP,KAAK,CAAC,EAAE,MAAM,CAAC;QACf,YAAY,CAAC,EAAE,OAAO,CAAC;QACvB,kBAAkB,CAAC,EAAE,OAAO,CAAC;QAC7B,gBAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,qBAAqB,CAAC,EAAE,MAAM,CAAC;QAC/B,UAAU,CAAC,EAAE,MAAM,CAAC;QACpB,cAAc,CAAC,EAAE,MAAM,CAAC;QACxB,WAAW,CAAC,EAAE,MAAM,CAAC;QACrB,gBAAgB,CAAC,EAAE,MAAM,CAAC;QAC1B,YAAY,CAAC,EAAE,OAAO,CAAC;KACnB,GACL,OAAO,CAAC,oBAAoB,CAAC;IAiPhC;;;OAGG;IACH,OAAO,CAAC,aAAa;IA2BrB;;;OAGG;IACG,YAAY,CAChB,QAAQ,EAAE,MAAM,EAChB,OAAO,GAAE;QAAE,eAAe,CAAC,EAAE,MAAM,CAAA;KAAO,GACzC,OAAO,CAAC,oBAAoB,CAAC;IAyBhC;;OAEG;YACW,UAAU;IASxB;;OAEG;YACW,cAAc;IAS5B;;;OAGG;IACH,OAAO,CAAC,aAAa;IAQrB;;;OAGG;YACW,WAAW;IA8BzB;;OAEG;IACG,WAAW,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAOhD;;OAEG;IACG,YAAY,IAAI,OAAO,CAAC;QAAE,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAYhD;;OAEG;IACG,eAAe,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;CAGvD"}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/retrieval/index.ts"],"names":[],"mappings":"AAAA,cAAc,
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/retrieval/index.ts"],"names":[],"mappings":"AAAA,cAAc,WAAW,CAAC;AAC1B,cAAc,aAAa,CAAC"}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"jina.d.ts","sourceRoot":"","sources":["../../src/retrieval/jina.ts"],"names":[],"mappings":"AAAA;;GAEG;AAaH,MAAM,WAAW,QAAQ;IACvB,EAAE,EAAE,MAAM,CAAC;IACX,OAAO,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,YAAY;IAC3B,EAAE,EAAE,MAAM,CAAC;IACX,KAAK,EAAE,MAAM,CAAC;IACd,OAAO,EAAE,MAAM,CAAC;CACjB;AAUD,qBAAa,aAAa;IAaZ,OAAO,CAAC,SAAS;IAZ7B,OAAO,CAAC,OAAO,CAA6B;IAC5C,OAAO,CAAC,QAAQ,CAA0B;IAC1C,OAAO,CAAC,eAAe,CAGR;IACf,OAAO,CAAC,SAAS,CAAK;IACtB,OAAO,CAAC,KAAK,CAAS;IACtB,OAAO,CAAC,YAAY,CAAgB;IACpC,OAAO,CAAC,YAAY,CAA6B;IACjD,OAAO,CAAC,MAAM,CAAM;gBAEA,SAAS,EAAE,MAAM;IAMrC;;OAEG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IA0C5B,OAAO,CAAC,UAAU;YAuBJ,IAAI;IAuBlB;;OAEG;IACG,KAAK,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,OAAO,CAAC;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAYhF;;OAEG;IACG,GAAG,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,OAAO,CAAC;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAY9E;;OAEG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,CAAC,GAAE,MAAW,GAAG,OAAO,CAAC,YAAY,EAAE,CAAC;IAUpE;;OAEG;IACG,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,SAAS,EAAE,QAAQ,EAAE,EAAE,CAAC,GAAE,MAAW,GAAG,OAAO,CAAC,YAAY,EAAE,CAAC;IAW3F;;OAEG;IACG,MAAM,CAAC,GAAG,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAYzE;;OAEG;IACG,IAAI,IAAI,OAAO,CAAC,OAAO,CAAC;IAS9B;;OAEG;IACG,IAAI,IAAI,OAAO,CAAC,IAAI,CAAC;CAa5B;AAED;;;GAGG;AACH,qBAAa,eAAe;IAC1B,OAAO,CAAC,SAAS,CAAoC;IAE/C,KAAK,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,OAAO,CAAC;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAO1E,GAAG,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,OAAO,CAAC;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;IAIxE,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,CAAC,GAAE,MAAW,GAAG,OAAO,CAAC,YAAY,EAAE,CAAC;IAwB9D,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,SAAS,EAAE,QAAQ,EAAE,EAAE,CAAC,GAAE,MAAW,GAAG,OAAO,CAAC,YAAY,EAAE,CAAC;IAcrF,MAAM,CAAC,GAAG,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAA;KAAE,CAAC;CAS1E;AAED;;GAEG;AACH,wBAAsB,eAAe,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,aAAa,GAAG,eAAe,CAAC,CAejG"}
|
package/dist/web/server.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"server.d.ts","sourceRoot":"","sources":["../../src/web/server.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAOH,OAAO,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AACxD,OAAO,EAAE,cAAc,EAAE,MAAM,6BAA6B,CAAC;AAC7D,OAAO,EAAE,YAAY,EAAE,MAAM,wBAAwB,CAAC;AAyBtD;;;GAGG;AACH,wBAAgB,mBAAmB,IAAI,MAAM,GAAG,IAAI,CAmBnD;AAED,UAAU,gBAAgB;IACxB,EAAE,EAAE,cAAc,CAAC;IACnB,KAAK,EAAE,cAAc,CAAC;IACtB,MAAM,EAAE,YAAY,CAAC;IACrB,IAAI,CAAC,EAAE,MAAM,CAAC;CACf;AAED,qBAAa,eAAe;IAC1B,OAAO,CAAC,MAAM,CAA4B;IAC1C,OAAO,CAAC,EAAE,CAAiB;IAC3B,OAAO,CAAC,KAAK,CAAiB;IAC9B,OAAO,CAAC,MAAM,CAAe;IAC7B,OAAO,CAAC,IAAI,CAAc;IAC1B,OAAO,CAAC,YAAY,CAAe;IACnC,OAAO,CAAC,IAAI,CAAS;gBAET,OAAO,EAAE,gBAAgB;IAa/B,KAAK,IAAI,OAAO,CAAC,MAAM,CAAC;IAwD9B,OAAO,CAAC,aAAa;IAkBrB,OAAO,CAAC,cAAc;IActB,IAAI,IAAI,IAAI;YAQE,aAAa;YA+Bb,SAAS;
|
|
1
|
+
{"version":3,"file":"server.d.ts","sourceRoot":"","sources":["../../src/web/server.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAOH,OAAO,EAAE,cAAc,EAAE,MAAM,wBAAwB,CAAC;AACxD,OAAO,EAAE,cAAc,EAAE,MAAM,6BAA6B,CAAC;AAC7D,OAAO,EAAE,YAAY,EAAE,MAAM,wBAAwB,CAAC;AAyBtD;;;GAGG;AACH,wBAAgB,mBAAmB,IAAI,MAAM,GAAG,IAAI,CAmBnD;AAED,UAAU,gBAAgB;IACxB,EAAE,EAAE,cAAc,CAAC;IACnB,KAAK,EAAE,cAAc,CAAC;IACtB,MAAM,EAAE,YAAY,CAAC;IACrB,IAAI,CAAC,EAAE,MAAM,CAAC;CACf;AAED,qBAAa,eAAe;IAC1B,OAAO,CAAC,MAAM,CAA4B;IAC1C,OAAO,CAAC,EAAE,CAAiB;IAC3B,OAAO,CAAC,KAAK,CAAiB;IAC9B,OAAO,CAAC,MAAM,CAAe;IAC7B,OAAO,CAAC,IAAI,CAAc;IAC1B,OAAO,CAAC,YAAY,CAAe;IACnC,OAAO,CAAC,IAAI,CAAS;gBAET,OAAO,EAAE,gBAAgB;IAa/B,KAAK,IAAI,OAAO,CAAC,MAAM,CAAC;IAwD9B,OAAO,CAAC,aAAa;IAkBrB,OAAO,CAAC,cAAc;IActB,IAAI,IAAI,IAAI;YAQE,aAAa;YA+Bb,SAAS;YAofT,WAAW;IAgCzB,OAAO,CAAC,SAAS;CA0BlB"}
|
package/logo.png
ADDED
|
Binary file
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@199-bio/engram",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.0",
|
|
4
4
|
"description": "Give Claude a perfect memory. Local-first MCP server with hybrid search.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -15,7 +15,6 @@
|
|
|
15
15
|
"start": "node dist/index.js",
|
|
16
16
|
"test": "vitest",
|
|
17
17
|
"test:run": "vitest run",
|
|
18
|
-
"download-models": "node scripts/download-models.js",
|
|
19
18
|
"lint": "eslint src/",
|
|
20
19
|
"format": "prettier --write src/"
|
|
21
20
|
},
|
|
@@ -28,7 +27,7 @@
|
|
|
28
27
|
"memory",
|
|
29
28
|
"ai",
|
|
30
29
|
"claude",
|
|
31
|
-
"
|
|
30
|
+
"jina",
|
|
32
31
|
"rag",
|
|
33
32
|
"knowledge-graph",
|
|
34
33
|
"semantic-search",
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Memory Consolidator
|
|
3
3
|
*
|
|
4
|
-
* Uses Opus 4.
|
|
4
|
+
* Uses Opus 4.6 with extended thinking to consolidate memories into digests
|
|
5
5
|
* and detect contradictions. Inspired by how the brain consolidates
|
|
6
6
|
* short-term memories into long-term storage during sleep.
|
|
7
7
|
*
|
|
@@ -241,7 +241,7 @@ export class Consolidator {
|
|
|
241
241
|
}
|
|
242
242
|
|
|
243
243
|
/**
|
|
244
|
-
* Consolidate a batch of memories using Opus 4.
|
|
244
|
+
* Consolidate a batch of memories using Opus 4.6 with extended thinking
|
|
245
245
|
*/
|
|
246
246
|
private async consolidateBatch(
|
|
247
247
|
memories: Memory[]
|
|
@@ -273,7 +273,7 @@ Create a detailed digest that preserves all important information. Respond with
|
|
|
273
273
|
|
|
274
274
|
try {
|
|
275
275
|
const response = await client.messages.create({
|
|
276
|
-
model: "claude-opus-4-
|
|
276
|
+
model: "claude-opus-4-6-20250514",
|
|
277
277
|
max_tokens: 16000,
|
|
278
278
|
temperature: 1, // Required for extended thinking
|
|
279
279
|
thinking: {
|
|
@@ -363,7 +363,7 @@ Create a rich, detailed profile. Do not summarize away important nuances. Respon
|
|
|
363
363
|
|
|
364
364
|
try {
|
|
365
365
|
const response = await client.messages.create({
|
|
366
|
-
model: "claude-opus-4-
|
|
366
|
+
model: "claude-opus-4-6-20250514",
|
|
367
367
|
max_tokens: 16000,
|
|
368
368
|
temperature: 1, // Required for extended thinking
|
|
369
369
|
thinking: {
|
|
@@ -560,7 +560,7 @@ Create a rich, detailed profile. Do not summarize away important nuances. Respon
|
|
|
560
560
|
}
|
|
561
561
|
|
|
562
562
|
/**
|
|
563
|
-
* Extract memories from conversation episodes using
|
|
563
|
+
* Extract memories from conversation episodes using Sonnet 4.6 (fast, affordable)
|
|
564
564
|
*/
|
|
565
565
|
private async extractMemoriesFromEpisodes(
|
|
566
566
|
episodes: Episode[]
|
|
@@ -583,9 +583,9 @@ Remember: Only extract information worth remembering long-term. Skip transient t
|
|
|
583
583
|
Respond with JSON only.`;
|
|
584
584
|
|
|
585
585
|
try {
|
|
586
|
-
// Use
|
|
586
|
+
// Use Sonnet 4.6 for speed/cost (no extended thinking needed)
|
|
587
587
|
const response = await client.messages.create({
|
|
588
|
-
model: "claude-
|
|
588
|
+
model: "claude-sonnet-4-6-20250514",
|
|
589
589
|
max_tokens: 4000,
|
|
590
590
|
messages: [
|
|
591
591
|
{
|
|
@@ -781,10 +781,10 @@ Respond with JSON only.`;
|
|
|
781
781
|
episodesProcessed += sessionEpisodes.length;
|
|
782
782
|
batchIndex++;
|
|
783
783
|
|
|
784
|
-
// Estimate tokens (
|
|
784
|
+
// Estimate tokens (Sonnet 4.6)
|
|
785
785
|
const batchTokens = 3000; // Conservative estimate
|
|
786
786
|
totalTokens += batchTokens;
|
|
787
|
-
totalCost += plan.calculateCost("
|
|
787
|
+
totalCost += plan.calculateCost("sonnet", 2000, 1000);
|
|
788
788
|
|
|
789
789
|
plan.updateProgress({
|
|
790
790
|
batchesCompleted: batchIndex,
|
|
@@ -14,16 +14,16 @@
|
|
|
14
14
|
import { randomUUID } from "crypto";
|
|
15
15
|
import { EngramDatabase, Memory, Episode, ConsolidationCheckpoint } from "../storage/database.js";
|
|
16
16
|
|
|
17
|
-
// Token pricing (Opus 4.
|
|
17
|
+
// Token pricing (Opus 4.6 with extended thinking)
|
|
18
18
|
const PRICING = {
|
|
19
19
|
opus: {
|
|
20
20
|
input: 15 / 1_000_000, // $15 per 1M input tokens
|
|
21
21
|
output: 75 / 1_000_000, // $75 per 1M output tokens
|
|
22
22
|
thinking: 15 / 1_000_000, // $15 per 1M thinking tokens (same as input)
|
|
23
23
|
},
|
|
24
|
-
|
|
25
|
-
input:
|
|
26
|
-
output:
|
|
24
|
+
sonnet: {
|
|
25
|
+
input: 3.00 / 1_000_000, // $3.00 per 1M input tokens
|
|
26
|
+
output: 15.00 / 1_000_000, // $15.00 per 1M output tokens
|
|
27
27
|
},
|
|
28
28
|
};
|
|
29
29
|
|
|
@@ -130,7 +130,7 @@ export class ConsolidationPlan {
|
|
|
130
130
|
const phases: PhasePlan[] = [];
|
|
131
131
|
const delayMs = this.db.getConfigNumber("delay_between_calls_ms", 2000);
|
|
132
132
|
|
|
133
|
-
// Episode phase (
|
|
133
|
+
// Episode phase (Sonnet 4.6)
|
|
134
134
|
if (unconsolidatedEp.length >= 4) {
|
|
135
135
|
const batchCount = Math.min(episodeBatches, maxBatchesPerRun);
|
|
136
136
|
const cost = batchCount * this.estimateEpisodeBatchCost();
|
|
@@ -139,7 +139,7 @@ export class ConsolidationPlan {
|
|
|
139
139
|
itemCount: Math.min(unconsolidatedEp.length, batchCount * 20),
|
|
140
140
|
batchCount,
|
|
141
141
|
estimatedCost: cost,
|
|
142
|
-
estimatedTimeMs: batchCount * (2000 + delayMs), // ~2s per
|
|
142
|
+
estimatedTimeMs: batchCount * (2000 + delayMs), // ~2s per Sonnet call + delay
|
|
143
143
|
});
|
|
144
144
|
}
|
|
145
145
|
|
|
@@ -404,11 +404,11 @@ export class ConsolidationPlan {
|
|
|
404
404
|
}
|
|
405
405
|
|
|
406
406
|
/**
|
|
407
|
-
* Estimate cost for an episode batch (
|
|
407
|
+
* Estimate cost for an episode batch (Sonnet 4.6)
|
|
408
408
|
*/
|
|
409
409
|
private estimateEpisodeBatchCost(): number {
|
|
410
410
|
const { input, output } = TOKEN_ESTIMATES.episodeBatch;
|
|
411
|
-
return (input * PRICING.
|
|
411
|
+
return (input * PRICING.sonnet.input) + (output * PRICING.sonnet.output);
|
|
412
412
|
}
|
|
413
413
|
|
|
414
414
|
/**
|
|
@@ -424,7 +424,7 @@ export class ConsolidationPlan {
|
|
|
424
424
|
/**
|
|
425
425
|
* Calculate actual cost from token usage
|
|
426
426
|
*/
|
|
427
|
-
calculateCost(model: "opus" | "
|
|
427
|
+
calculateCost(model: "opus" | "sonnet", inputTokens: number, outputTokens: number, thinkingTokens?: number): number {
|
|
428
428
|
const pricing = PRICING[model];
|
|
429
429
|
let cost = (inputTokens * pricing.input) + (outputTokens * pricing.output);
|
|
430
430
|
|
package/src/index.ts
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
/**
|
|
3
3
|
* Engram - High-quality personal memory for AI assistants
|
|
4
4
|
*
|
|
5
|
-
* Local-first MCP server with
|
|
5
|
+
* Local-first MCP server with Jina v5 + BM25 hybrid search
|
|
6
6
|
* and a lightweight knowledge graph.
|
|
7
7
|
*/
|
|
8
8
|
|
|
@@ -21,7 +21,7 @@ import { startHttpServer } from "./transport/http.js";
|
|
|
21
21
|
|
|
22
22
|
import { EngramDatabase } from "./storage/database.js";
|
|
23
23
|
import { KnowledgeGraph } from "./graph/knowledge-graph.js";
|
|
24
|
-
import { createRetriever } from "./retrieval/
|
|
24
|
+
import { createRetriever, JinaRetriever, SimpleRetriever } from "./retrieval/jina.js";
|
|
25
25
|
import { HybridSearch } from "./retrieval/hybrid.js";
|
|
26
26
|
import { EngramWebServer, getRunningServerUrl } from "./web/server.js";
|
|
27
27
|
import { Consolidator } from "./consolidation/consolidator.js";
|
|
@@ -91,6 +91,9 @@ function cleanup(): void {
|
|
|
91
91
|
if (webServer) {
|
|
92
92
|
webServer.stop();
|
|
93
93
|
}
|
|
94
|
+
if (retriever && "stop" in retriever) {
|
|
95
|
+
(retriever as JinaRetriever).stop();
|
|
96
|
+
}
|
|
94
97
|
if (db) {
|
|
95
98
|
db.close();
|
|
96
99
|
}
|
|
@@ -135,6 +138,7 @@ if (getTransportMode() === "stdio") {
|
|
|
135
138
|
let db: EngramDatabase;
|
|
136
139
|
let graph: KnowledgeGraph;
|
|
137
140
|
let search: HybridSearch;
|
|
141
|
+
let retriever: JinaRetriever | SimpleRetriever;
|
|
138
142
|
let consolidator: Consolidator;
|
|
139
143
|
let webServer: EngramWebServer | null = null;
|
|
140
144
|
|
|
@@ -144,7 +148,7 @@ async function initialize(): Promise<void> {
|
|
|
144
148
|
db = new EngramDatabase(DB_FILE);
|
|
145
149
|
graph = new KnowledgeGraph(db);
|
|
146
150
|
|
|
147
|
-
|
|
151
|
+
retriever = await createRetriever(DB_PATH);
|
|
148
152
|
search = new HybridSearch(db, graph, retriever);
|
|
149
153
|
consolidator = new Consolidator(db, graph, search);
|
|
150
154
|
|
package/src/retrieval/hybrid.ts
CHANGED
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Hybrid Search with Reciprocal Rank Fusion (RRF)
|
|
3
|
-
* Combines BM25 (keyword) and
|
|
3
|
+
* Combines BM25 (keyword) and Jina v5 (semantic) search
|
|
4
4
|
* Enhanced with temporal decay and salience scoring
|
|
5
5
|
*/
|
|
6
6
|
|
|
7
7
|
import { EngramDatabase, Memory, Digest } from "../storage/database.js";
|
|
8
8
|
import { KnowledgeGraph } from "../graph/knowledge-graph.js";
|
|
9
|
-
import {
|
|
9
|
+
import { JinaRetriever, SimpleRetriever, SearchResult, Document } from "./jina.js";
|
|
10
10
|
|
|
11
11
|
export interface HybridSearchResult {
|
|
12
12
|
memory: Memory;
|
|
@@ -44,7 +44,7 @@ export interface HybridSearchResponse {
|
|
|
44
44
|
* Higher stability = slower forgetting
|
|
45
45
|
* Recent access = higher retention
|
|
46
46
|
*/
|
|
47
|
-
function calculateRetention(memory: Memory, now: Date): number {
|
|
47
|
+
export function calculateRetention(memory: Memory, now: Date): number {
|
|
48
48
|
// Use last_accessed if available, otherwise timestamp
|
|
49
49
|
const lastActive = memory.last_accessed || memory.timestamp;
|
|
50
50
|
const daysSinceAccess = (now.getTime() - lastActive.getTime()) / (1000 * 60 * 60 * 24);
|
|
@@ -65,7 +65,7 @@ function calculateRetention(memory: Memory, now: Date): number {
|
|
|
65
65
|
* Calculate salience score - how important/memorable is this?
|
|
66
66
|
* Combines emotional weight, importance, and access patterns
|
|
67
67
|
*/
|
|
68
|
-
function calculateSalience(memory: Memory): number {
|
|
68
|
+
export function calculateSalience(memory: Memory): number {
|
|
69
69
|
const importance = memory.importance || 0.5;
|
|
70
70
|
const emotionalWeight = memory.emotional_weight || 0.5;
|
|
71
71
|
const accessBonus = Math.min(1, Math.log(1 + (memory.access_count || 0)) / 5);
|
|
@@ -77,7 +77,7 @@ function calculateSalience(memory: Memory): number {
|
|
|
77
77
|
/**
|
|
78
78
|
* Apply temporal and salience adjustments to search results
|
|
79
79
|
*/
|
|
80
|
-
function adjustScore(memory: Memory, baseScore: number, now: Date): { adjusted: number; retention: number } {
|
|
80
|
+
export function adjustScore(memory: Memory, baseScore: number, now: Date): { adjusted: number; retention: number } {
|
|
81
81
|
const retention = calculateRetention(memory, now);
|
|
82
82
|
const salience = calculateSalience(memory);
|
|
83
83
|
|
|
@@ -97,7 +97,7 @@ export class HybridSearch {
|
|
|
97
97
|
constructor(
|
|
98
98
|
private db: EngramDatabase,
|
|
99
99
|
private graph: KnowledgeGraph,
|
|
100
|
-
private retriever:
|
|
100
|
+
private retriever: JinaRetriever | SimpleRetriever
|
|
101
101
|
) {
|
|
102
102
|
// Generate a session ID for this search instance
|
|
103
103
|
this.sessionId = `session_${Date.now()}_${Math.random().toString(36).slice(2, 8)}`;
|
|
@@ -113,8 +113,8 @@ export class HybridSearch {
|
|
|
113
113
|
/**
|
|
114
114
|
* Search using all available methods and fuse results
|
|
115
115
|
*
|
|
116
|
-
* Strategy: BM25 gets candidates fast,
|
|
117
|
-
* This is both FASTER (fewer
|
|
116
|
+
* Strategy: BM25 gets candidates fast, Jina reranks for quality
|
|
117
|
+
* This is both FASTER (fewer Jina computations) and BETTER (combines keyword + semantic)
|
|
118
118
|
*/
|
|
119
119
|
async search(
|
|
120
120
|
query: string,
|
|
@@ -128,7 +128,7 @@ export class HybridSearch {
|
|
|
128
128
|
semanticWeight?: number;
|
|
129
129
|
graphWeight?: number;
|
|
130
130
|
connectionWeight?: number; // Weight for connected memories in RRF
|
|
131
|
-
useReranking?: boolean; // Use
|
|
131
|
+
useReranking?: boolean; // Use Jina to rerank BM25 results
|
|
132
132
|
} = {}
|
|
133
133
|
): Promise<HybridSearchResponse> {
|
|
134
134
|
const {
|
|
@@ -163,7 +163,7 @@ export class HybridSearch {
|
|
|
163
163
|
// For semantic: either rerank BM25 results (faster+better) or search full index
|
|
164
164
|
let semanticResults: Array<{ id: string; score: number }>;
|
|
165
165
|
if (useReranking && bm25Results.length > 0) {
|
|
166
|
-
// Rerank BM25 candidates with
|
|
166
|
+
// Rerank BM25 candidates with Jina - faster AND better quality
|
|
167
167
|
const docs = bm25Results.map(r => ({ id: r.id, content: this.db.getMemory(r.id)?.content || '' }));
|
|
168
168
|
const reranked = await this.retriever.rerank(query, docs, candidateLimit);
|
|
169
169
|
semanticResults = reranked.map(r => ({ id: r.id, score: r.score }));
|
|
@@ -447,7 +447,7 @@ export class HybridSearch {
|
|
|
447
447
|
}
|
|
448
448
|
|
|
449
449
|
/**
|
|
450
|
-
* Semantic search via
|
|
450
|
+
* Semantic search via Jina v5
|
|
451
451
|
*/
|
|
452
452
|
private async searchSemantic(query: string, limit: number): Promise<Array<{ id: string; score: number }>> {
|
|
453
453
|
try {
|
package/src/retrieval/index.ts
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
export * from "./
|
|
1
|
+
export * from "./jina.js";
|
|
2
2
|
export * from "./hybrid.js";
|