@dex-ai/memory 0.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +164 -0
- package/package.json +43 -0
- package/src/_fakes.ts +148 -0
- package/src/db.test.ts +46 -0
- package/src/db.ts +149 -0
- package/src/embedder.ts +60 -0
- package/src/episodic.test.ts +63 -0
- package/src/episodic.ts +144 -0
- package/src/extension.test.ts +266 -0
- package/src/extension.ts +485 -0
- package/src/index.ts +32 -0
- package/src/procedural.test.ts +83 -0
- package/src/procedural.ts +183 -0
- package/src/semantic.test.ts +201 -0
- package/src/semantic.ts +210 -0
- package/src/summarize.test.ts +93 -0
- package/src/summarize.ts +154 -0
- package/src/tools.ts +283 -0
package/README.md
ADDED
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
# @dex-ai/memory-sqlite
|
|
2
|
+
|
|
3
|
+
SQLite-backed memory Extension for [`@dex-ai/sdk`](https://github.com/klxdev/dex-ai-sdk). Three memory types — **episodic**, **semantic**, **procedural** — in one extension, one SQLite file.
|
|
4
|
+
|
|
5
|
+
## Install
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
bun add @dex-ai/memory-sqlite
|
|
9
|
+
# optional: default local embedder (ONNX via Transformers.js)
|
|
10
|
+
bun add @xenova/transformers
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
`@xenova/transformers` is an optional peer dep. Skip it if you pass your own `embed()` function.
|
|
14
|
+
|
|
15
|
+
## Usage
|
|
16
|
+
|
|
17
|
+
```ts
|
|
18
|
+
import { DexAgent } from '@dex-ai/runtime';
|
|
19
|
+
import { openai } from '@dex-ai/openai';
|
|
20
|
+
import { memoryExtensionSqlite } from '@dex-ai/memory-sqlite';
|
|
21
|
+
|
|
22
|
+
const agent = new DexAgent({
|
|
23
|
+
provider: openai({ modelId: 'gpt-4.1' }),
|
|
24
|
+
extensions: [
|
|
25
|
+
memoryExtensionSqlite({
|
|
26
|
+
path: '~/.dex/memory.db',
|
|
27
|
+
userId: 'alice',
|
|
28
|
+
|
|
29
|
+
// Optional: a cheaper model for memory's summarize + extract calls.
|
|
30
|
+
llm: { model: 'gpt-4o-mini' },
|
|
31
|
+
|
|
32
|
+
// Optional: bring your own embedder (e.g. a remote endpoint).
|
|
33
|
+
// If omitted, a local Transformers.js embedder (all-MiniLM-L6-v2) is used.
|
|
34
|
+
// embed: async (texts) => await myRemoteEmbedder(texts),
|
|
35
|
+
}),
|
|
36
|
+
],
|
|
37
|
+
});
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## What lives where
|
|
41
|
+
|
|
42
|
+
### Episodic memory — past turns, auto-summarized
|
|
43
|
+
|
|
44
|
+
Every `generate()` iteration, the extension fires a background task:
|
|
45
|
+
1. Summarize the turn's new messages into 1-3 sentences via the LLM.
|
|
46
|
+
2. Embed the summary and write both to SQLite.
|
|
47
|
+
|
|
48
|
+
Background writes are tracked; `agent.dispose()` awaits them.
|
|
49
|
+
|
|
50
|
+
At recall time (`onRequest`), the extension fetches:
|
|
51
|
+
- The last **3 most-recent** episodes for the user.
|
|
52
|
+
- The **3 most-similar** episodes via sqlite-vec cosine against the user's last message.
|
|
53
|
+
- De-duplicated by id, sorted newest-first.
|
|
54
|
+
|
|
55
|
+
### Semantic memory — durable facts
|
|
56
|
+
|
|
57
|
+
Facts are `(subject, predicate, object)` tuples, unique by `(userId, subject, predicate)`. Written in two ways:
|
|
58
|
+
|
|
59
|
+
**Automatic extraction** at each iteration stop: the LLM extracts durable claims from the turn and upserts them.
|
|
60
|
+
|
|
61
|
+
**Model-driven** via tools:
|
|
62
|
+
- `memory.remember_fact({ subject, predicate, object })` — upsert by key.
|
|
63
|
+
- `memory.forget_fact({ subject, predicate })` — delete by key.
|
|
64
|
+
|
|
65
|
+
At recall time, **all of the user's facts** are injected as a synthetic system message. The set is typically small (tens to low hundreds); if it grows we'll add top-k filtering later.
|
|
66
|
+
|
|
67
|
+
### Procedural memory — runbooks
|
|
68
|
+
|
|
69
|
+
Long-form how-to content. Stored by unique `title`, tagged, with an embedding over `title + body`.
|
|
70
|
+
|
|
71
|
+
**Tools**:
|
|
72
|
+
- `memory.store_procedure({ title, body, tags? })` — upsert by title.
|
|
73
|
+
- `memory.list_procedures({ query?, tag?, limit? })` — with `query`, returns vector-ranked results; with `tag`, filters; with neither, returns most-recently-updated.
|
|
74
|
+
- `memory.get_procedure({ title })` — fetch full body.
|
|
75
|
+
|
|
76
|
+
**Auto-inject**: at recall time, the extension does a vector similarity lookup against the user's last message. If the top match scores above the threshold (default 0.5), its full body is prepended to the prompt as a synthetic system message.
|
|
77
|
+
|
|
78
|
+
## LLM configuration
|
|
79
|
+
|
|
80
|
+
```ts
|
|
81
|
+
memoryExtensionSqlite({
|
|
82
|
+
// ...
|
|
83
|
+
llm: {
|
|
84
|
+
provider: customProvider, // optional — overrides the agent's provider entirely
|
|
85
|
+
model: 'gpt-4o-mini', // optional — passes via providerOptions.model per call
|
|
86
|
+
},
|
|
87
|
+
});
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Resolution rule:
|
|
91
|
+
|
|
92
|
+
1. If `llm.provider` is set, use it.
|
|
93
|
+
2. Otherwise use the agent's provider (captured from `actx.provider` in `onAgentStart`).
|
|
94
|
+
3. If `llm.model` is set, it's passed via `providerOptions.model` on every memory-internal request. Providers that merge `providerOptions` over the default body — like `@dex-ai/openai` — honor this override per-call. Providers that don't merge `providerOptions` require passing a full `llm.provider` instance.
|
|
95
|
+
|
|
96
|
+
## Extension options
|
|
97
|
+
|
|
98
|
+
```ts
|
|
99
|
+
interface MemoryExtensionSqliteOptions {
|
|
100
|
+
path: string; // SQLite file path or ':memory:'
|
|
101
|
+
userId: string; // owner for episodic + semantic (procedural is global)
|
|
102
|
+
|
|
103
|
+
llm?: { provider?: Provider; model?: string };
|
|
104
|
+
embed?: (texts: string[]) => Promise<number[][]>; // 384-dim
|
|
105
|
+
|
|
106
|
+
episodicRecent?: number; // default 3
|
|
107
|
+
episodicSimilar?: number; // default 3
|
|
108
|
+
proceduralThreshold?: number; // default 0.5
|
|
109
|
+
autoWrite?: boolean; // default true; set false to disable auto-summarize+extract
|
|
110
|
+
}
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
## Injected prompt shape
|
|
114
|
+
|
|
115
|
+
When memory is available, the extension prepends a single synthetic system message to each provider request. Example:
|
|
116
|
+
|
|
117
|
+
```
|
|
118
|
+
Recent context:
|
|
119
|
+
- 5m ago: discussed the auth flow; chose JWT over session cookies.
|
|
120
|
+
- 1d ago: debugged a test flake in session-sqlite.
|
|
121
|
+
|
|
122
|
+
Known facts:
|
|
123
|
+
- user prefers TypeScript
|
|
124
|
+
- project uses PostgreSQL 15
|
|
125
|
+
- user is on macOS
|
|
126
|
+
|
|
127
|
+
Relevant runbook — deploy-dex (similarity 0.71):
|
|
128
|
+
1. bun run typecheck
|
|
129
|
+
2. bun run test
|
|
130
|
+
3. git tag vX.Y.Z && git push --tags
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
This message is **not** persisted to `actx.messages` — it's a per-turn rewrite via the `onRequest` reducer, which never mutates the agent's canonical history.
|
|
134
|
+
|
|
135
|
+
## Requirements + gotchas
|
|
136
|
+
|
|
137
|
+
- **sqlite-vec**: loaded via the `sqlite-vec` npm package (ships prebuilt binaries for macOS, Linux, Windows). Extension construction throws if the binary isn't loadable for your platform.
|
|
138
|
+
- **Transformers.js first-run download**: ~25 MB ONNX model, cached in `~/.cache/huggingface`. Initial `embed()` takes 10-30s; subsequent calls are fast.
|
|
139
|
+
- **Background writes are fire-and-forget.** If the process is killed mid-turn, that turn's memory may not be persisted. `agent.dispose()` awaits in-flight writes normally.
|
|
140
|
+
- **Prompt quality of auto-extraction** depends on the configured model. Cheap models (gpt-4o-mini, Haiku) work but may produce noisier facts. Review with `memory.list_facts` (or just inspect the DB) occasionally and use `memory.forget_fact` to prune.
|
|
141
|
+
- **Procedural is global**, not user-scoped, in v0.1. If you need per-user runbooks, open an issue and we'll add a user_id column.
|
|
142
|
+
|
|
143
|
+
## Schema
|
|
144
|
+
|
|
145
|
+
```
|
|
146
|
+
episodic id | user_id | summary | metadata(JSON) | created_at
|
|
147
|
+
episodic_vec id | embedding(FLOAT[384]) -- sqlite-vec
|
|
148
|
+
|
|
149
|
+
semantic id | user_id | subject | predicate | object | source | created_at | updated_at
|
|
150
|
+
UNIQUE (user_id, subject, predicate)
|
|
151
|
+
|
|
152
|
+
procedural id | title(UNIQUE) | body | tags(JSON) | created_at | updated_at
|
|
153
|
+
procedural_vec id | embedding(FLOAT[384]) -- sqlite-vec
|
|
154
|
+
|
|
155
|
+
_schema_migrations name(PK) | applied_at
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
## Testing
|
|
159
|
+
|
|
160
|
+
```bash
|
|
161
|
+
bun test
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
33 tests as of v0.1: schema/migrations, each memory type's read/write path, LLM helpers, and two end-to-end tests that exercise the full Agent + Extension flow.
|
package/package.json
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@dex-ai/memory",
|
|
3
|
+
"version": "0.3.3",
|
|
4
|
+
"description": "SQLite-backed memory Extension for @dex-ai/sdk — episodic, semantic, and procedural memory in one package.",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"exports": {
|
|
7
|
+
".": {
|
|
8
|
+
"types": "./src/index.ts",
|
|
9
|
+
"default": "./src/index.ts"
|
|
10
|
+
}
|
|
11
|
+
},
|
|
12
|
+
"files": [
|
|
13
|
+
"src"
|
|
14
|
+
],
|
|
15
|
+
"scripts": {
|
|
16
|
+
"typecheck": "tsc --noEmit",
|
|
17
|
+
"test": "bun test",
|
|
18
|
+
"changeset": "changeset",
|
|
19
|
+
"version": "changeset version",
|
|
20
|
+
"release": "changeset publish"
|
|
21
|
+
},
|
|
22
|
+
"dependencies": {
|
|
23
|
+
"@dex-ai/sdk": "^0.1.2",
|
|
24
|
+
"@xenova/transformers": "^2.17.2",
|
|
25
|
+
"sqlite-vec": "^0.1.7-alpha.2"
|
|
26
|
+
},
|
|
27
|
+
"peerDependencies": {
|
|
28
|
+
"zod": "^3.23.0"
|
|
29
|
+
},
|
|
30
|
+
"devDependencies": {
|
|
31
|
+
"typescript": "^5.6.3",
|
|
32
|
+
"@types/bun": "latest",
|
|
33
|
+
"bun-types": "latest",
|
|
34
|
+
"zod": "^3.23.8",
|
|
35
|
+
"@xenova/transformers": "^2.17.2",
|
|
36
|
+
"@changesets/cli": "^2.29.0"
|
|
37
|
+
},
|
|
38
|
+
"sideEffects": false,
|
|
39
|
+
"publishConfig": {
|
|
40
|
+
"access": "public",
|
|
41
|
+
"registry": "https://registry.npmjs.org/"
|
|
42
|
+
}
|
|
43
|
+
}
|
package/src/_fakes.ts
ADDED
|
@@ -0,0 +1,148 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Test fixtures: a deterministic fake embedder and a scriptable Model/Extension.
|
|
3
|
+
* No Transformers.js; no network; fully offline.
|
|
4
|
+
*/
|
|
5
|
+
import type {
|
|
6
|
+
Extension,
|
|
7
|
+
FinishReason,
|
|
8
|
+
Message,
|
|
9
|
+
Model,
|
|
10
|
+
ModelRequest,
|
|
11
|
+
ResponseMeta,
|
|
12
|
+
StreamPart,
|
|
13
|
+
Usage,
|
|
14
|
+
} from "@dex-ai/sdk";
|
|
15
|
+
import { EMBED_DIMS } from "./db";
|
|
16
|
+
import type { Embedder } from "./embedder";
|
|
17
|
+
|
|
18
|
+
const USAGE: Usage = { inputTokens: 0, outputTokens: 0, totalTokens: 0 };
|
|
19
|
+
const FIN: FinishReason = "stop";
|
|
20
|
+
|
|
21
|
+
/**
|
|
22
|
+
* Deterministic hashed embedder — same input -> same vector.
|
|
23
|
+
*/
|
|
24
|
+
export function fakeEmbedder(): Embedder {
|
|
25
|
+
return async (texts: string[]): Promise<number[][]> => {
|
|
26
|
+
return texts.map((t) => hashToVec(t, EMBED_DIMS));
|
|
27
|
+
};
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
function hashToVec(text: string, dims: number): number[] {
|
|
31
|
+
let seed = 0x811c9dc5;
|
|
32
|
+
for (let i = 0; i < text.length; i++) {
|
|
33
|
+
seed ^= text.charCodeAt(i);
|
|
34
|
+
seed = Math.imul(seed, 0x01000193) >>> 0;
|
|
35
|
+
}
|
|
36
|
+
const vec = new Array<number>(dims);
|
|
37
|
+
let x = seed || 1;
|
|
38
|
+
for (let i = 0; i < dims; i++) {
|
|
39
|
+
x = Math.imul(x, 48271) >>> 0;
|
|
40
|
+
vec[i] = ((x & 0xffff) / 0xffff) * 2 - 1;
|
|
41
|
+
}
|
|
42
|
+
let sq = 0;
|
|
43
|
+
for (const v of vec) sq += v * v;
|
|
44
|
+
const norm = Math.sqrt(sq) || 1;
|
|
45
|
+
return vec.map((v) => v / norm);
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
/* ------------------------------------------------------------------ */
|
|
49
|
+
/* Scripted model + extension */
|
|
50
|
+
/* ------------------------------------------------------------------ */
|
|
51
|
+
|
|
52
|
+
export type FakeStep =
|
|
53
|
+
| { kind: "tool-call"; toolName: string; input: unknown; toolCallId?: string }
|
|
54
|
+
| { kind: "text"; text: string };
|
|
55
|
+
|
|
56
|
+
export const FAKE_PROVIDER = "scripted";
|
|
57
|
+
export const FAKE_MODEL = "scripted-1";
|
|
58
|
+
|
|
59
|
+
/**
|
|
60
|
+
* A Model whose stream() plays a scripted sequence of steps.
|
|
61
|
+
* Used both for the main generate loop and for summarize/extractFacts.
|
|
62
|
+
*/
|
|
63
|
+
export function scriptedModel(opts: {
|
|
64
|
+
steps?: FakeStep[];
|
|
65
|
+
generateReplies?: string[];
|
|
66
|
+
}): Model {
|
|
67
|
+
const stepQueue: FakeStep[] = (opts.steps ?? []).slice();
|
|
68
|
+
const replyQueue: string[] = (opts.generateReplies ?? []).slice();
|
|
69
|
+
let callCount = 0;
|
|
70
|
+
|
|
71
|
+
return {
|
|
72
|
+
id: FAKE_MODEL,
|
|
73
|
+
async *stream(_r: ModelRequest): AsyncIterable<StreamPart> {
|
|
74
|
+
callCount += 1;
|
|
75
|
+
|
|
76
|
+
// stepQueue is for the main generate loop (tool calls + text responses).
|
|
77
|
+
// replyQueue is for side-calls (summarize, extractFacts) that happen outside the main loop.
|
|
78
|
+
let step: FakeStep;
|
|
79
|
+
if (stepQueue.length > 0) {
|
|
80
|
+
step = stepQueue.shift()!;
|
|
81
|
+
} else if (replyQueue.length > 0) {
|
|
82
|
+
step = { kind: "text", text: replyQueue.shift()! };
|
|
83
|
+
} else {
|
|
84
|
+
step = { kind: "text", text: "done" };
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
const meta: ResponseMeta = {
|
|
88
|
+
providerName: FAKE_PROVIDER,
|
|
89
|
+
modelId: FAKE_MODEL,
|
|
90
|
+
startedAt: Date.now(),
|
|
91
|
+
};
|
|
92
|
+
yield { type: "response-start", meta };
|
|
93
|
+
yield { type: "message-start", role: "assistant" };
|
|
94
|
+
|
|
95
|
+
let finish: FinishReason;
|
|
96
|
+
if (step.kind === "text") {
|
|
97
|
+
yield { type: "text-delta", delta: step.text };
|
|
98
|
+
yield {
|
|
99
|
+
type: "message-stop",
|
|
100
|
+
message: {
|
|
101
|
+
role: "assistant",
|
|
102
|
+
content: [{ type: "text", text: step.text }],
|
|
103
|
+
},
|
|
104
|
+
};
|
|
105
|
+
finish = "stop";
|
|
106
|
+
} else {
|
|
107
|
+
const toolCallId = step.toolCallId ?? `call-${callCount}`;
|
|
108
|
+
yield {
|
|
109
|
+
type: "tool-call",
|
|
110
|
+
toolCallId,
|
|
111
|
+
toolName: step.toolName,
|
|
112
|
+
input: step.input,
|
|
113
|
+
};
|
|
114
|
+
yield {
|
|
115
|
+
type: "message-stop",
|
|
116
|
+
message: {
|
|
117
|
+
role: "assistant",
|
|
118
|
+
content: [
|
|
119
|
+
{
|
|
120
|
+
type: "tool-call",
|
|
121
|
+
toolCallId,
|
|
122
|
+
toolName: step.toolName,
|
|
123
|
+
input: step.input,
|
|
124
|
+
},
|
|
125
|
+
],
|
|
126
|
+
},
|
|
127
|
+
};
|
|
128
|
+
finish = "tool-calls";
|
|
129
|
+
}
|
|
130
|
+
yield { type: "finish", reason: finish, usage: USAGE };
|
|
131
|
+
yield { type: "response-stop", meta, usage: USAGE, finishReason: finish };
|
|
132
|
+
},
|
|
133
|
+
};
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
/**
|
|
137
|
+
* Wrap a scriptedModel into an Extension for use with Agent.create/Agent.create.
|
|
138
|
+
*/
|
|
139
|
+
export function scriptedProviderExtension(opts: {
|
|
140
|
+
steps?: FakeStep[];
|
|
141
|
+
generateReplies?: string[];
|
|
142
|
+
}): Extension {
|
|
143
|
+
const model = scriptedModel(opts);
|
|
144
|
+
return {
|
|
145
|
+
name: FAKE_PROVIDER,
|
|
146
|
+
models: [model],
|
|
147
|
+
};
|
|
148
|
+
}
|
package/src/db.test.ts
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
import { describe, expect, test } from 'bun:test';
|
|
2
|
+
import { MemoryDb, EMBED_DIMS } from './db';
|
|
3
|
+
|
|
4
|
+
describe('MemoryDb', () => {
|
|
5
|
+
test('opens, loads sqlite-vec, runs migrations', () => {
|
|
6
|
+
const db = new MemoryDb({ path: ':memory:' });
|
|
7
|
+
// All four base tables should exist.
|
|
8
|
+
const tables = db.db
|
|
9
|
+
.prepare("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name")
|
|
10
|
+
.all()
|
|
11
|
+
.map((r) => (r as { name: string }).name);
|
|
12
|
+
expect(tables).toContain('episodic');
|
|
13
|
+
expect(tables).toContain('semantic');
|
|
14
|
+
expect(tables).toContain('procedural');
|
|
15
|
+
expect(tables).toContain('_schema_migrations');
|
|
16
|
+
// vec virtual tables exist too (they show up as 'episodic_vec'/'procedural_vec').
|
|
17
|
+
expect(tables).toContain('episodic_vec');
|
|
18
|
+
expect(tables).toContain('procedural_vec');
|
|
19
|
+
db.close();
|
|
20
|
+
});
|
|
21
|
+
|
|
22
|
+
test('migrations idempotent across constructors for the same file', async () => {
|
|
23
|
+
// :memory: isn't shareable — use a temp file.
|
|
24
|
+
const { mkdtemp, rm } = await import('node:fs/promises');
|
|
25
|
+
const { tmpdir } = await import('node:os');
|
|
26
|
+
const { join } = await import('node:path');
|
|
27
|
+
const dir = await mkdtemp(join(tmpdir(), 'dex-mem-'));
|
|
28
|
+
const path = join(dir, 'mem.db');
|
|
29
|
+
try {
|
|
30
|
+
const a = new MemoryDb({ path });
|
|
31
|
+
const appliedFirst = a.db.prepare('SELECT COUNT(*) as c FROM _schema_migrations').get() as { c: number };
|
|
32
|
+
a.close();
|
|
33
|
+
|
|
34
|
+
const b = new MemoryDb({ path });
|
|
35
|
+
const appliedAgain = b.db.prepare('SELECT COUNT(*) as c FROM _schema_migrations').get() as { c: number };
|
|
36
|
+
expect(appliedAgain.c).toBe(appliedFirst.c);
|
|
37
|
+
b.close();
|
|
38
|
+
} finally {
|
|
39
|
+
await rm(dir, { recursive: true, force: true });
|
|
40
|
+
}
|
|
41
|
+
});
|
|
42
|
+
|
|
43
|
+
test('EMBED_DIMS is 384', () => {
|
|
44
|
+
expect(EMBED_DIMS).toBe(384);
|
|
45
|
+
});
|
|
46
|
+
});
|
package/src/db.ts
ADDED
|
@@ -0,0 +1,149 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* SQLite database open + sqlite-vec load + schema migrations.
|
|
3
|
+
*
|
|
4
|
+
* One DB file, four tables plus two sqlite-vec virtual tables:
|
|
5
|
+
* episodic, episodic_vec, semantic, procedural, procedural_vec, _schema_migrations.
|
|
6
|
+
*
|
|
7
|
+
* sqlite-vec is loaded as a SQLite extension via `db.loadExtension()`. If the
|
|
8
|
+
* extension cannot be loaded (e.g. the prebuilt binary is missing for the host
|
|
9
|
+
* platform), we throw a descriptive error at construction time so apps see it
|
|
10
|
+
* immediately rather than on first recall.
|
|
11
|
+
*/
|
|
12
|
+
import { Database } from 'bun:sqlite';
|
|
13
|
+
import * as sqliteVec from 'sqlite-vec';
|
|
14
|
+
import { existsSync, mkdirSync, copyFileSync } from 'node:fs';
|
|
15
|
+
import { join, dirname } from 'node:path';
|
|
16
|
+
import { tmpdir } from 'node:os';
|
|
17
|
+
|
|
18
|
+
/** Embedding dimensions — matches Xenova/all-MiniLM-L6-v2. */
|
|
19
|
+
export const EMBED_DIMS = 384;
|
|
20
|
+
|
|
21
|
+
export interface OpenDbOptions {
|
|
22
|
+
/** Absolute path to the DB file, or ':memory:'. Created if missing. */
|
|
23
|
+
path: string;
|
|
24
|
+
/**
|
|
25
|
+
* Optional path to the sqlite-vec loadable extension (vec0.so / vec0.dylib).
|
|
26
|
+
* When running inside a compiled binary, the normal require.resolve() won't
|
|
27
|
+
* find the .so — pass the extracted path here instead.
|
|
28
|
+
*/
|
|
29
|
+
vecPath?: string;
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
const MIGRATIONS: Array<{ name: string; sql: string }> = [
|
|
33
|
+
{
|
|
34
|
+
name: '001_init',
|
|
35
|
+
sql: `
|
|
36
|
+
CREATE TABLE IF NOT EXISTS episodic (
|
|
37
|
+
id TEXT PRIMARY KEY,
|
|
38
|
+
user_id TEXT NOT NULL,
|
|
39
|
+
summary TEXT NOT NULL,
|
|
40
|
+
metadata TEXT NOT NULL DEFAULT '{}',
|
|
41
|
+
created_at INTEGER NOT NULL
|
|
42
|
+
);
|
|
43
|
+
CREATE INDEX IF NOT EXISTS idx_episodic_user_time ON episodic(user_id, created_at DESC);
|
|
44
|
+
|
|
45
|
+
CREATE TABLE IF NOT EXISTS semantic (
|
|
46
|
+
id TEXT PRIMARY KEY,
|
|
47
|
+
user_id TEXT NOT NULL,
|
|
48
|
+
subject TEXT NOT NULL,
|
|
49
|
+
predicate TEXT NOT NULL,
|
|
50
|
+
object TEXT NOT NULL,
|
|
51
|
+
source TEXT NOT NULL,
|
|
52
|
+
created_at INTEGER NOT NULL,
|
|
53
|
+
updated_at INTEGER NOT NULL,
|
|
54
|
+
UNIQUE(user_id, subject, predicate)
|
|
55
|
+
);
|
|
56
|
+
CREATE INDEX IF NOT EXISTS idx_semantic_user ON semantic(user_id);
|
|
57
|
+
|
|
58
|
+
CREATE TABLE IF NOT EXISTS procedural (
|
|
59
|
+
id TEXT PRIMARY KEY,
|
|
60
|
+
title TEXT NOT NULL UNIQUE,
|
|
61
|
+
body TEXT NOT NULL,
|
|
62
|
+
tags TEXT NOT NULL DEFAULT '[]',
|
|
63
|
+
created_at INTEGER NOT NULL,
|
|
64
|
+
updated_at INTEGER NOT NULL
|
|
65
|
+
);
|
|
66
|
+
CREATE INDEX IF NOT EXISTS idx_procedural_title ON procedural(title);
|
|
67
|
+
`,
|
|
68
|
+
},
|
|
69
|
+
{
|
|
70
|
+
name: '002_vec',
|
|
71
|
+
// vec0 virtual tables. Must run AFTER sqlite-vec is loaded.
|
|
72
|
+
sql: `
|
|
73
|
+
CREATE VIRTUAL TABLE IF NOT EXISTS episodic_vec USING vec0(
|
|
74
|
+
id TEXT PRIMARY KEY,
|
|
75
|
+
embedding FLOAT[${EMBED_DIMS}]
|
|
76
|
+
);
|
|
77
|
+
CREATE VIRTUAL TABLE IF NOT EXISTS procedural_vec USING vec0(
|
|
78
|
+
id TEXT PRIMARY KEY,
|
|
79
|
+
embedding FLOAT[${EMBED_DIMS}]
|
|
80
|
+
);
|
|
81
|
+
`,
|
|
82
|
+
},
|
|
83
|
+
{
|
|
84
|
+
name: '003_semantic_vec',
|
|
85
|
+
sql: `
|
|
86
|
+
CREATE VIRTUAL TABLE IF NOT EXISTS semantic_vec USING vec0(
|
|
87
|
+
id TEXT PRIMARY KEY,
|
|
88
|
+
embedding FLOAT[${EMBED_DIMS}]
|
|
89
|
+
);
|
|
90
|
+
`,
|
|
91
|
+
},
|
|
92
|
+
];
|
|
93
|
+
|
|
94
|
+
function applyMigrations(db: Database): void {
|
|
95
|
+
db.exec(`
|
|
96
|
+
CREATE TABLE IF NOT EXISTS _schema_migrations (
|
|
97
|
+
name TEXT PRIMARY KEY,
|
|
98
|
+
applied_at INTEGER NOT NULL
|
|
99
|
+
);
|
|
100
|
+
`);
|
|
101
|
+
const already = new Set(
|
|
102
|
+
db
|
|
103
|
+
.prepare('SELECT name FROM _schema_migrations')
|
|
104
|
+
.all()
|
|
105
|
+
.map((r) => (r as { name: string }).name),
|
|
106
|
+
);
|
|
107
|
+
const record = db.prepare('INSERT INTO _schema_migrations(name, applied_at) VALUES(?, ?)');
|
|
108
|
+
for (const m of MIGRATIONS) {
|
|
109
|
+
if (already.has(m.name)) continue;
|
|
110
|
+
db.transaction(() => {
|
|
111
|
+
db.exec(m.sql);
|
|
112
|
+
record.run(m.name, Date.now());
|
|
113
|
+
})();
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
export class MemoryDb {
|
|
118
|
+
readonly db: Database;
|
|
119
|
+
|
|
120
|
+
constructor(opts: OpenDbOptions) {
|
|
121
|
+
this.db = new Database(opts.path, { create: true });
|
|
122
|
+
this.db.exec('PRAGMA journal_mode = WAL; PRAGMA foreign_keys = ON;');
|
|
123
|
+
|
|
124
|
+
// Load sqlite-vec. Failure here is fatal — no point constructing the
|
|
125
|
+
// extension if vector tables won't work.
|
|
126
|
+
try {
|
|
127
|
+
if (opts.vecPath) {
|
|
128
|
+
// Compiled binary mode: load from explicit path
|
|
129
|
+
this.db.loadExtension(opts.vecPath);
|
|
130
|
+
} else {
|
|
131
|
+
// Normal mode: sqlite-vec resolves the platform-specific .so via require.resolve()
|
|
132
|
+
(sqliteVec as { load: (db: unknown) => void }).load(this.db);
|
|
133
|
+
}
|
|
134
|
+
} catch (err) {
|
|
135
|
+
this.db.close();
|
|
136
|
+
throw new Error(
|
|
137
|
+
`@dex-ai/memory-sqlite: failed to load sqlite-vec extension. ` +
|
|
138
|
+
`Ensure the sqlite-vec npm package's prebuilt binary is available for this platform. ` +
|
|
139
|
+
`Original error: ${err instanceof Error ? err.message : String(err)}`,
|
|
140
|
+
);
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
applyMigrations(this.db);
|
|
144
|
+
}
|
|
145
|
+
|
|
146
|
+
close(): void {
|
|
147
|
+
this.db.close();
|
|
148
|
+
}
|
|
149
|
+
}
|
package/src/embedder.ts
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Embedder interface + default local implementation.
|
|
3
|
+
*
|
|
4
|
+
* Default: Transformers.js pipeline with Xenova/all-MiniLM-L6-v2 (384-dim).
|
|
5
|
+
* - Lazy-loaded on first use (first call may take ~10-30s as it downloads the ONNX
|
|
6
|
+
* model, ~25 MB, to ~/.cache/huggingface).
|
|
7
|
+
* - Subsequent calls are fast (batched, ~50-100ms per text on modern hardware).
|
|
8
|
+
*
|
|
9
|
+
* Apps can override with their own embed() function — e.g. calling a remote
|
|
10
|
+
* embeddings endpoint — by passing it into memoryExtension options.
|
|
11
|
+
*/
|
|
12
|
+
import { EMBED_DIMS } from './db';
|
|
13
|
+
|
|
14
|
+
/** Embeds a batch of texts into fixed-dimension vectors. */
|
|
15
|
+
export type Embedder = (texts: string[]) => Promise<number[][]>;
|
|
16
|
+
|
|
17
|
+
/** Dimensionality the package expects. All embedders must produce this length. */
|
|
18
|
+
export const EXPECTED_DIMS = EMBED_DIMS;
|
|
19
|
+
|
|
20
|
+
// Guard to avoid loading Transformers.js more than once per process.
|
|
21
|
+
let sharedPipelinePromise: Promise<(texts: string[]) => Promise<number[][]>> | null = null;
|
|
22
|
+
|
|
23
|
+
async function loadTransformersPipeline(): Promise<(texts: string[]) => Promise<number[][]>> {
|
|
24
|
+
// Dynamic import so the dep is optional — apps that pass their own embed()
|
|
25
|
+
// never touch the Transformers.js package.
|
|
26
|
+
const mod = await import('@xenova/transformers').catch((err: unknown) => {
|
|
27
|
+
throw new Error(
|
|
28
|
+
`@dex-ai/memory-sqlite: @xenova/transformers is not installed. ` +
|
|
29
|
+
`Either install it (bun add @xenova/transformers) or pass a custom embed() function. ` +
|
|
30
|
+
`Original: ${err instanceof Error ? err.message : String(err)}`,
|
|
31
|
+
);
|
|
32
|
+
});
|
|
33
|
+
|
|
34
|
+
const pipelineFactory = (mod as { pipeline: (task: string, model: string) => Promise<unknown> }).pipeline;
|
|
35
|
+
const pipe = (await pipelineFactory('feature-extraction', 'Xenova/all-MiniLM-L6-v2')) as (
|
|
36
|
+
input: string[] | string,
|
|
37
|
+
options?: { pooling: 'mean' | 'none'; normalize: boolean },
|
|
38
|
+
) => Promise<{ tolist(): number[][] | number[][][]; data: Float32Array; dims: number[] }>;
|
|
39
|
+
|
|
40
|
+
return async function embed(texts: string[]): Promise<number[][]> {
|
|
41
|
+
if (texts.length === 0) return [];
|
|
42
|
+
const out = await pipe(texts, { pooling: 'mean', normalize: true });
|
|
43
|
+
const list = out.tolist();
|
|
44
|
+
return list as number[][];
|
|
45
|
+
};
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
/**
|
|
49
|
+
* Returns the default local embedder (Transformers.js / Xenova/all-MiniLM-L6-v2).
|
|
50
|
+
* First call is slow (downloads the model); subsequent calls are fast.
|
|
51
|
+
*/
|
|
52
|
+
export function localEmbedder(): Embedder {
|
|
53
|
+
return async (texts: string[]): Promise<number[][]> => {
|
|
54
|
+
if (sharedPipelinePromise === null) {
|
|
55
|
+
sharedPipelinePromise = loadTransformersPipeline();
|
|
56
|
+
}
|
|
57
|
+
const pipe = await sharedPipelinePromise;
|
|
58
|
+
return pipe(texts);
|
|
59
|
+
};
|
|
60
|
+
}
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
|
|
2
|
+
import { MemoryDb } from './db';
|
|
3
|
+
import { EpisodicStore } from './episodic';
|
|
4
|
+
import { fakeEmbedder } from './_fakes';
|
|
5
|
+
|
|
6
|
+
describe('EpisodicStore', () => {
|
|
7
|
+
let db: MemoryDb;
|
|
8
|
+
let store: EpisodicStore;
|
|
9
|
+
const embed = fakeEmbedder();
|
|
10
|
+
|
|
11
|
+
beforeEach(() => {
|
|
12
|
+
db = new MemoryDb({ path: ':memory:' });
|
|
13
|
+
store = new EpisodicStore(db.db);
|
|
14
|
+
});
|
|
15
|
+
|
|
16
|
+
afterEach(() => db.close());
|
|
17
|
+
|
|
18
|
+
test('record + recall-recent returns what was written, newest first', async () => {
|
|
19
|
+
const [v1] = await embed(['first episode']);
|
|
20
|
+
await store.record({ userId: 'u', summary: 'first', embedding: v1! });
|
|
21
|
+
await new Promise((r) => setTimeout(r, 3));
|
|
22
|
+
const [v2] = await embed(['second episode']);
|
|
23
|
+
await store.record({ userId: 'u', summary: 'second', embedding: v2! });
|
|
24
|
+
|
|
25
|
+
const rec = await store.recall('u', undefined, { recentLimit: 5, similarLimit: 0 });
|
|
26
|
+
expect(rec.length).toBe(2);
|
|
27
|
+
expect(rec[0]!.summary).toBe('second');
|
|
28
|
+
expect(rec[1]!.summary).toBe('first');
|
|
29
|
+
});
|
|
30
|
+
|
|
31
|
+
test('recall filters by userId', async () => {
|
|
32
|
+
const [v] = await embed(['x']);
|
|
33
|
+
await store.record({ userId: 'alice', summary: 'for alice', embedding: v! });
|
|
34
|
+
await store.record({ userId: 'bob', summary: 'for bob', embedding: v! });
|
|
35
|
+
|
|
36
|
+
const a = await store.recall('alice', undefined, { similarLimit: 0 });
|
|
37
|
+
expect(a.length).toBe(1);
|
|
38
|
+
expect(a[0]!.summary).toBe('for alice');
|
|
39
|
+
});
|
|
40
|
+
|
|
41
|
+
test('recall includes similar matches when query vector given', async () => {
|
|
42
|
+
// Deterministic fake embedder — same text -> same vector. Recalling with
|
|
43
|
+
// the vector of stored text should find that exact episode.
|
|
44
|
+
const [vA] = await embed(['alpha']);
|
|
45
|
+
const [vB] = await embed(['beta']);
|
|
46
|
+
const [vC] = await embed(['gamma']);
|
|
47
|
+
await store.record({ userId: 'u', summary: 'alpha', embedding: vA! });
|
|
48
|
+
await store.record({ userId: 'u', summary: 'beta', embedding: vB! });
|
|
49
|
+
await store.record({ userId: 'u', summary: 'gamma', embedding: vC! });
|
|
50
|
+
|
|
51
|
+
// Query with the exact vector of 'beta' — beta should be among results.
|
|
52
|
+
const rec = await store.recall('u', vB, { recentLimit: 0, similarLimit: 2 });
|
|
53
|
+
expect(rec.length).toBeGreaterThan(0);
|
|
54
|
+
expect(rec.some((r) => r.summary === 'beta')).toBe(true);
|
|
55
|
+
});
|
|
56
|
+
|
|
57
|
+
test('recall de-dupes across recent + similar', async () => {
|
|
58
|
+
const [v] = await embed(['one-and-only']);
|
|
59
|
+
await store.record({ userId: 'u', summary: 'one-and-only', embedding: v! });
|
|
60
|
+
const rec = await store.recall('u', v, { recentLimit: 3, similarLimit: 3 });
|
|
61
|
+
expect(rec.length).toBe(1);
|
|
62
|
+
});
|
|
63
|
+
});
|