@andespindola/brainlink 0.1.0-alpha.10 → 0.1.0-alpha.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +2 -0
- package/README.md +52 -5
- package/SECURITY.md +12 -0
- package/dist/application/frontend/client-js.js +12 -5
- package/dist/application/watch-vault.js +4 -1
- package/dist/domain/graph-layout.js +11 -2
- package/dist/domain/markdown.js +79 -2
- package/dist/infrastructure/bucket-vault.js +171 -0
- package/dist/infrastructure/file-system-vault.js +21 -3
- package/dist/infrastructure/sqlite/document-writer.js +4 -3
- package/dist/infrastructure/sqlite/graph-reader.js +22 -10
- package/dist/infrastructure/sqlite/schema.js +12 -1
- package/dist/infrastructure/sqlite-index.js +6 -1
- package/dist/mcp/server.js +2 -2
- package/dist/mcp/tools.js +1 -1
- package/docs/AGENT_USAGE.md +14 -1
- package/docs/ARCHITECTURE.md +22 -1
- package/package.json +2 -1
package/AGENTS.md
CHANGED
|
@@ -37,6 +37,8 @@ Use this loop when using Brainlink as memory:
|
|
|
37
37
|
|
|
38
38
|
When an agent adds durable memory, it should connect the new note to at least one existing concept unless the note is intentionally a root concept. Prefer exact note titles in links, for example `[[Architecture]]`, and run `broken-links`, `orphans` or `validate` when the graph looks disconnected.
|
|
39
39
|
|
|
40
|
+
Agents can mark important relationships by placing priority hints on the same line as a wiki link, for example `[[Architecture]] priority: high`, `[[Incident Runbook]] #important` or `[[Incident Runbook]] #critical`. Indexed graph edges expose `weight` and `priority` so agents can sort related notes by importance.
|
|
41
|
+
|
|
40
42
|
## Commands
|
|
41
43
|
|
|
42
44
|
```bash
|
package/README.md
CHANGED
|
@@ -62,10 +62,12 @@ Markdown is the source of truth. `.brainlink/brainlink.db` is only a rebuildable
|
|
|
62
62
|
|
|
63
63
|
- Local-first Markdown vault.
|
|
64
64
|
- Obsidian-compatible `[[wiki links]]` and `#tags`.
|
|
65
|
+
- Weighted graph edges so agents can rank relationship importance and priority.
|
|
65
66
|
- Backlinks, broken-link reports, orphan detection and validation.
|
|
66
67
|
- Full-text, semantic and hybrid retrieval modes.
|
|
67
68
|
- SQLite-backed semantic candidate buckets for larger vaults.
|
|
68
69
|
- Agent namespaces under `agents/<agent-id>/`.
|
|
70
|
+
- S3-compatible bucket vaults through `s3://bucket/prefix` URIs.
|
|
69
71
|
- CLI with machine-readable `--json` output.
|
|
70
72
|
- Short CLI alias: `blink`.
|
|
71
73
|
- Built-in MCP stdio server for agent tool integration.
|
|
@@ -256,6 +258,36 @@ http://127.0.0.1:4321
|
|
|
256
258
|
|
|
257
259
|
When `--vault` is omitted, commands use the default vault at `$HOME/.brainlink/vault`. Pass `--vault` or configure `vault` in `brainlink.config.json` when you want a custom project-local vault.
|
|
258
260
|
|
|
261
|
+
## Bucket Vaults
|
|
262
|
+
|
|
263
|
+
Brainlink can use an S3-compatible bucket as the Markdown source of truth:
|
|
264
|
+
|
|
265
|
+
```bash
|
|
266
|
+
export AWS_REGION="us-east-1"
|
|
267
|
+
export AWS_ACCESS_KEY_ID="..."
|
|
268
|
+
export AWS_SECRET_ACCESS_KEY="..."
|
|
269
|
+
|
|
270
|
+
blink add "Architecture" \
|
|
271
|
+
--vault "s3://my-memory-bucket/brainlink" \
|
|
272
|
+
--content "Bucket Markdown is the source of truth. #architecture"
|
|
273
|
+
|
|
274
|
+
blink index --vault "s3://my-memory-bucket/brainlink"
|
|
275
|
+
blink context "architecture" --vault "s3://my-memory-bucket/brainlink"
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
For Cloudflare R2, MinIO or another S3-compatible endpoint:
|
|
279
|
+
|
|
280
|
+
```bash
|
|
281
|
+
export BRAINLINK_S3_ENDPOINT="https://<account-id>.r2.cloudflarestorage.com"
|
|
282
|
+
export BRAINLINK_S3_FORCE_PATH_STYLE=1
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
Bucket vaults mirror Markdown into a local cache under
|
|
286
|
+
`$BRAINLINK_HOME/bucket-cache`. The bucket remains canonical; the local
|
|
287
|
+
`.brainlink/brainlink.db` stays a disposable index. Run `index` after remote
|
|
288
|
+
bucket changes before relying on `search`, `context`, graph or validation
|
|
289
|
+
commands. Watch mode is only supported for local filesystem vaults.
|
|
290
|
+
|
|
259
291
|
## Core Model
|
|
260
292
|
|
|
261
293
|
```txt
|
|
@@ -359,12 +391,21 @@ Available tools:
|
|
|
359
391
|
- `brainlink_add_note`: write durable Markdown memory and reindex.
|
|
360
392
|
- `brainlink_index`: rebuild the vault index.
|
|
361
393
|
- `brainlink_validate`: validate broken links and orphan notes.
|
|
362
|
-
- `brainlink_graph`: read indexed graph nodes and links.
|
|
394
|
+
- `brainlink_graph`: read indexed graph nodes and weighted links.
|
|
363
395
|
- `brainlink_broken_links`: list unresolved wiki links.
|
|
364
396
|
- `brainlink_orphans`: list disconnected notes.
|
|
365
397
|
|
|
366
398
|
The same linking rule applies through MCP: `brainlink_context` is read-only, and real graph links require Markdown notes with explicit `[[wiki links]]` followed by indexing.
|
|
367
399
|
|
|
400
|
+
Agents can raise the importance of a relationship by putting priority markers on the same line as a wiki link:
|
|
401
|
+
|
|
402
|
+
```md
|
|
403
|
+
- [ ] Review [[Architecture]] priority: high
|
|
404
|
+
Related: [[Incident Runbook]] #critical
|
|
405
|
+
```
|
|
406
|
+
|
|
407
|
+
Indexed edges expose `weight` and `priority` (`low`, `normal`, `high`, `critical`) through CLI JSON, HTTP graph APIs and `brainlink_graph`.
|
|
408
|
+
|
|
368
409
|
## Graph UI
|
|
369
410
|
|
|
370
411
|
Start the local frontend:
|
|
@@ -378,7 +419,7 @@ By default, the server uses `$HOME/.brainlink/vault`. Pass `--vault ./vault` onl
|
|
|
378
419
|
The graph UI shows:
|
|
379
420
|
|
|
380
421
|
- notes as nodes
|
|
381
|
-
- `[[wiki links]]` as edges
|
|
422
|
+
- `[[wiki links]]` as weighted edges
|
|
382
423
|
- backlinks and outgoing links
|
|
383
424
|
- full Markdown content for the selected note
|
|
384
425
|
- neutral graph nodes with segment/group metadata
|
|
@@ -492,7 +533,7 @@ blink links --vault ./vault
|
|
|
492
533
|
blink links --vault ./vault --agent coding-agent
|
|
493
534
|
```
|
|
494
535
|
|
|
495
|
-
Lists indexed wiki links.
|
|
536
|
+
Lists indexed wiki links. JSON output includes `weight` and `priority` for each relationship.
|
|
496
537
|
|
|
497
538
|
### `backlinks`
|
|
498
539
|
|
|
@@ -501,7 +542,7 @@ blink backlinks "Architecture" --vault ./vault
|
|
|
501
542
|
blink backlinks "Architecture" --vault ./vault --agent coding-agent
|
|
502
543
|
```
|
|
503
544
|
|
|
504
|
-
Lists notes pointing to a target title.
|
|
545
|
+
Lists notes pointing to a target title, ordered by strongest relationship first. JSON output includes `weight` and `priority`.
|
|
505
546
|
|
|
506
547
|
### `graph`
|
|
507
548
|
|
|
@@ -510,7 +551,7 @@ blink graph --vault ./vault --json
|
|
|
510
551
|
blink graph --vault ./vault --agent coding-agent --json
|
|
511
552
|
```
|
|
512
553
|
|
|
513
|
-
Prints indexed graph data.
|
|
554
|
+
Prints indexed graph data. Edges include `weight` and `priority` so agents can categorize importance.
|
|
514
555
|
|
|
515
556
|
### `stats`
|
|
516
557
|
|
|
@@ -612,6 +653,12 @@ Set `BRAINLINK_ALLOWED_VAULTS` for external wrappers, including MCP servers, so
|
|
|
612
653
|
export BRAINLINK_ALLOWED_VAULTS="/absolute/path/to/project-vault,/absolute/path/to/team-vault"
|
|
613
654
|
```
|
|
614
655
|
|
|
656
|
+
Bucket vaults can be allowlisted with the same variable:
|
|
657
|
+
|
|
658
|
+
```bash
|
|
659
|
+
export BRAINLINK_ALLOWED_VAULTS="s3://my-memory-bucket/brainlink"
|
|
660
|
+
```
|
|
661
|
+
|
|
615
662
|
## Note Format
|
|
616
663
|
|
|
617
664
|
Brainlink supports Markdown with optional frontmatter:
|
package/SECURITY.md
CHANGED
|
@@ -32,4 +32,16 @@ External tool wrappers, including MCP servers, should set `BRAINLINK_ALLOWED_VAU
|
|
|
32
32
|
export BRAINLINK_ALLOWED_VAULTS="/absolute/path/to/project-vault"
|
|
33
33
|
```
|
|
34
34
|
|
|
35
|
+
For bucket vaults, allowlist the S3 URI prefix:
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
export BRAINLINK_ALLOWED_VAULTS="s3://my-memory-bucket/brainlink"
|
|
39
|
+
```
|
|
40
|
+
|
|
35
41
|
When the allowlist is set, CLI commands fail if `--vault` points outside the allowed roots.
|
|
42
|
+
|
|
43
|
+
## Bucket Credentials
|
|
44
|
+
|
|
45
|
+
Bucket vaults use the standard AWS SDK credential chain. Prefer short-lived,
|
|
46
|
+
least-privilege credentials scoped to the specific bucket prefix used by
|
|
47
|
+
Brainlink. Do not store bucket credentials in Markdown notes.
|
|
@@ -83,6 +83,8 @@ const visibleEdges = () => {
|
|
|
83
83
|
return state.edges.filter(edge => ids.has(edge.source) && edge.target && ids.has(edge.target))
|
|
84
84
|
}
|
|
85
85
|
|
|
86
|
+
const edgeWeight = edge => Number.isFinite(edge.weight) ? Math.max(1, edge.weight) : 1
|
|
87
|
+
|
|
86
88
|
const resetView = () => {
|
|
87
89
|
const rect = canvas.getBoundingClientRect()
|
|
88
90
|
state.transform = { x: Math.max(rect.width, 320) / 2, y: Math.max(rect.height, 320) / 2, scale: 1 }
|
|
@@ -103,7 +105,7 @@ const createLayout = graph => {
|
|
|
103
105
|
|
|
104
106
|
const graphSignature = graph => JSON.stringify({
|
|
105
107
|
nodes: graph.nodes.map(node => [node.id, node.title, node.path, node.content, node.tags]),
|
|
106
|
-
edges: graph.edges.map(edge => [edge.source, edge.target, edge.targetTitle])
|
|
108
|
+
edges: graph.edges.map(edge => [edge.source, edge.target, edge.targetTitle, edge.weight, edge.priority])
|
|
107
109
|
})
|
|
108
110
|
|
|
109
111
|
const tick = delta => {
|
|
@@ -207,7 +209,7 @@ const render = now => {
|
|
|
207
209
|
ctx.moveTo(edge.sourceNode.x, edge.sourceNode.y)
|
|
208
210
|
ctx.lineTo(edge.targetNode.x, edge.targetNode.y)
|
|
209
211
|
ctx.strokeStyle = selectedEdge ? graphTheme.edgeActive : graphTheme.edge
|
|
210
|
-
ctx.lineWidth = selectedEdge ? 1.8 : 1
|
|
212
|
+
ctx.lineWidth = (selectedEdge ? 1.8 : 1) + Math.min(edgeWeight(edge) - 1, 8) * 0.22
|
|
211
213
|
ctx.stroke()
|
|
212
214
|
})
|
|
213
215
|
|
|
@@ -241,7 +243,7 @@ const render = now => {
|
|
|
241
243
|
}
|
|
242
244
|
|
|
243
245
|
const list = items => items.length
|
|
244
|
-
? items.map(item => '<li>' + (item.id ? '<button type="button" data-node-id="' + escapeHtml(item.id) + '">' + escapeHtml(item.title) + '</button>' : escapeHtml(item.title)) + '<small>' + escapeHtml(item.path) + '</small></li>').join('')
|
|
246
|
+
? items.map(item => '<li>' + (item.id ? '<button type="button" data-node-id="' + escapeHtml(item.id) + '">' + escapeHtml(item.title) + '</button>' : escapeHtml(item.title)) + '<small>' + escapeHtml(item.path) + (item.weight ? ' · weight ' + escapeHtml(item.weight) + ' · ' + escapeHtml(item.priority || 'normal') : '') + '</small></li>').join('')
|
|
245
247
|
: '<li><small>No links found.</small></li>'
|
|
246
248
|
|
|
247
249
|
const allNotesList = () => state.nodes.length
|
|
@@ -261,13 +263,18 @@ const selectNode = node => {
|
|
|
261
263
|
return
|
|
262
264
|
}
|
|
263
265
|
const nodeById = new Map(state.nodes.map(item => [item.id, item]))
|
|
266
|
+
const withEdgeMeta = (linkedNode, edge) => linkedNode ? {
|
|
267
|
+
...linkedNode,
|
|
268
|
+
weight: edge.weight,
|
|
269
|
+
priority: edge.priority
|
|
270
|
+
} : null
|
|
264
271
|
const outgoing = state.graph.edges
|
|
265
272
|
.filter(edge => edge.source === node.id)
|
|
266
|
-
.map(edge => edge.target ? nodeById.get(edge.target) : { title: edge.targetTitle + ' (unresolved)', path: 'Missing note' })
|
|
273
|
+
.map(edge => withEdgeMeta(edge.target ? nodeById.get(edge.target) : { title: edge.targetTitle + ' (unresolved)', path: 'Missing note' }, edge))
|
|
267
274
|
.filter(Boolean)
|
|
268
275
|
const incoming = state.graph.edges
|
|
269
276
|
.filter(edge => edge.target === node.id)
|
|
270
|
-
.map(edge => nodeById.get(edge.source))
|
|
277
|
+
.map(edge => withEdgeMeta(nodeById.get(edge.source), edge))
|
|
271
278
|
.filter(Boolean)
|
|
272
279
|
|
|
273
280
|
elements.title.textContent = node.title
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { watch } from 'node:fs';
|
|
2
2
|
import { indexVault } from './index-vault.js';
|
|
3
|
-
import { resolveVaultPath } from '../infrastructure/file-system-vault.js';
|
|
3
|
+
import { isBucketVaultPath, resolveVaultPath } from '../infrastructure/file-system-vault.js';
|
|
4
4
|
const shouldIgnore = (filename) => {
|
|
5
5
|
if (!filename) {
|
|
6
6
|
return false;
|
|
@@ -8,6 +8,9 @@ const shouldIgnore = (filename) => {
|
|
|
8
8
|
return filename.includes('.brainlink') || !filename.endsWith('.md');
|
|
9
9
|
};
|
|
10
10
|
export const startVaultWatcher = (input) => {
|
|
11
|
+
if (isBucketVaultPath(input.vaultPath)) {
|
|
12
|
+
throw new Error('Watch mode is only supported for local filesystem vaults.');
|
|
13
|
+
}
|
|
11
14
|
const absoluteVaultPath = resolveVaultPath(input.vaultPath);
|
|
12
15
|
const debounceMs = input.debounceMs ?? 350;
|
|
13
16
|
let timeout = null;
|
|
@@ -34,8 +34,17 @@ const groupKey = (node) => {
|
|
|
34
34
|
return segments[0] ?? 'root';
|
|
35
35
|
};
|
|
36
36
|
const groupLabel = (key) => groupLabels[key] ?? key;
|
|
37
|
-
const
|
|
38
|
-
|
|
37
|
+
const incrementDegreeBy = (degrees, id, amount) => {
|
|
38
|
+
degrees.set(id, (degrees.get(id) ?? 0) + amount);
|
|
39
|
+
return degrees;
|
|
40
|
+
};
|
|
41
|
+
const edgeDegreeWeight = (edge) => Math.max(1, Math.min(edge.weight, 8));
|
|
42
|
+
const countDegrees = (edges) => edges.reduce((degrees, edge) => {
|
|
43
|
+
const weight = edgeDegreeWeight(edge);
|
|
44
|
+
return edge.target
|
|
45
|
+
? incrementDegreeBy(incrementDegreeBy(degrees, edge.source, weight), edge.target, weight)
|
|
46
|
+
: incrementDegreeBy(degrees, edge.source, weight);
|
|
47
|
+
}, new Map());
|
|
39
48
|
const uniqueIds = (ids) => Array.from(new Set(ids));
|
|
40
49
|
const createAdjacency = (nodes, edges) => {
|
|
41
50
|
const nodeIds = new Set(nodes.map((node) => node.id));
|
package/dist/domain/markdown.js
CHANGED
|
@@ -6,8 +6,31 @@ const frontmatterPattern = /^---\n([\s\S]*?)\n---\n?/;
|
|
|
6
6
|
const wikiLinkPattern = /\[\[([^\]|#]+)(?:#[^\]|]+)?(?:\|[^\]]+)?\]\]/g;
|
|
7
7
|
const tagPattern = /(^|\s)#([A-Za-z0-9][A-Za-z0-9_-]*)/g;
|
|
8
8
|
const headingPattern = /^#\s+(.+)$/m;
|
|
9
|
+
const priorityRanks = {
|
|
10
|
+
low: 0,
|
|
11
|
+
normal: 1,
|
|
12
|
+
high: 2,
|
|
13
|
+
critical: 3
|
|
14
|
+
};
|
|
15
|
+
const priorityBoosts = {
|
|
16
|
+
low: 0,
|
|
17
|
+
normal: 1,
|
|
18
|
+
high: 3,
|
|
19
|
+
critical: 6
|
|
20
|
+
};
|
|
21
|
+
const priorityPatterns = [
|
|
22
|
+
['critical', /\b(?:priority|prioridade|importance|importancia|importância)\s*[:=]\s*(?:critical|critica|crítica|urgent|urgente|p0)\b/i],
|
|
23
|
+
['critical', /#(?:critical|critica|crítica|urgent|urgente|p0)\b/i],
|
|
24
|
+
['high', /\b(?:priority|prioridade|importance|importancia|importância)\s*[:=]\s*(?:high|alta|important|importante|p1)\b/i],
|
|
25
|
+
['high', /#(?:high-priority|important|importante|p1)\b/i],
|
|
26
|
+
['normal', /\b(?:priority|prioridade|importance|importancia|importância)\s*[:=]\s*(?:normal|medium|media|média|p2)\b/i],
|
|
27
|
+
['normal', /#(?:normal-priority|medium-priority|p2)\b/i],
|
|
28
|
+
['low', /\b(?:priority|prioridade|importance|importancia|importância)\s*[:=]\s*(?:low|baixa|p3)\b/i],
|
|
29
|
+
['low', /#(?:low-priority|baixa-prioridade|p3)\b/i]
|
|
30
|
+
];
|
|
9
31
|
const normalizeTitle = (title) => title.trim().replace(/\.md$/i, '');
|
|
10
32
|
const unique = (values) => Array.from(new Set(values.map((value) => value.trim()).filter(Boolean)));
|
|
33
|
+
const maxPriority = (left, right) => priorityRanks[left] >= priorityRanks[right] ? left : right;
|
|
11
34
|
const parseFrontmatter = (content) => {
|
|
12
35
|
const match = content.match(frontmatterPattern);
|
|
13
36
|
if (!match) {
|
|
@@ -24,6 +47,57 @@ const parseFrontmatter = (content) => {
|
|
|
24
47
|
};
|
|
25
48
|
const stripFrontmatter = (content) => content.replace(frontmatterPattern, '');
|
|
26
49
|
const stripFencedCodeBlocks = (content) => content.replace(/```[\s\S]*?```/g, '').replace(/~~~[\s\S]*?~~~/g, '');
|
|
50
|
+
const visibleMarkdownLines = (content) => content.split('\n').reduce((state, line) => {
|
|
51
|
+
const togglesFence = /^\s*(?:```|~~~)/.test(line);
|
|
52
|
+
const fenced = togglesFence ? !state.fenced : state.fenced;
|
|
53
|
+
state.lines.push({ content: line, fenced });
|
|
54
|
+
return {
|
|
55
|
+
lines: state.lines,
|
|
56
|
+
fenced
|
|
57
|
+
};
|
|
58
|
+
}, {
|
|
59
|
+
lines: [],
|
|
60
|
+
fenced: false
|
|
61
|
+
}).lines;
|
|
62
|
+
const linePriority = (line) => priorityPatterns.find(([, pattern]) => pattern.test(line))?.[0] ?? null;
|
|
63
|
+
const linkReferenceWeight = (line, priority) => {
|
|
64
|
+
const headingBoost = /^\s{0,3}#{1,6}\s+/.test(line) ? 2 : 0;
|
|
65
|
+
const taskBoost = /^\s*[-*]\s+\[[ x]\]/i.test(line) ? 1 : 0;
|
|
66
|
+
return 1 + (priority ? priorityBoosts[priority] : 0) + headingBoost + taskBoost;
|
|
67
|
+
};
|
|
68
|
+
export const extractWikiLinkReferences = (content) => visibleMarkdownLines(content)
|
|
69
|
+
.filter((line) => !line.fenced)
|
|
70
|
+
.flatMap((line) => {
|
|
71
|
+
const priority = linePriority(line.content);
|
|
72
|
+
const weight = linkReferenceWeight(line.content, priority);
|
|
73
|
+
return Array.from(line.content.matchAll(wikiLinkPattern), (match) => ({
|
|
74
|
+
title: normalizeTitle(match[1]),
|
|
75
|
+
weight,
|
|
76
|
+
priority
|
|
77
|
+
}));
|
|
78
|
+
});
|
|
79
|
+
const priorityFromWeight = (weight) => weight >= 8 ? 'critical' : weight >= 4 ? 'high' : 'normal';
|
|
80
|
+
export const extractWikiLinkWeights = (content) => {
|
|
81
|
+
const weights = extractWikiLinkReferences(content).reduce((state, reference) => {
|
|
82
|
+
const titleKey = reference.title.toLowerCase();
|
|
83
|
+
const current = state.get(titleKey);
|
|
84
|
+
const weight = (current?.weight ?? 0) + reference.weight;
|
|
85
|
+
const explicitPriority = reference.priority
|
|
86
|
+
? maxPriority(current?.priority ?? reference.priority, reference.priority)
|
|
87
|
+
: current?.priority;
|
|
88
|
+
const derivedPriority = priorityFromWeight(weight);
|
|
89
|
+
const priority = explicitPriority === 'low' && weight === 1
|
|
90
|
+
? 'low'
|
|
91
|
+
: maxPriority(explicitPriority ?? derivedPriority, derivedPriority);
|
|
92
|
+
state.set(titleKey, {
|
|
93
|
+
title: current?.title ?? reference.title,
|
|
94
|
+
weight,
|
|
95
|
+
priority
|
|
96
|
+
});
|
|
97
|
+
return state;
|
|
98
|
+
}, new Map());
|
|
99
|
+
return Array.from(weights.values());
|
|
100
|
+
};
|
|
27
101
|
const extractTitle = (filePath, content, frontmatter) => {
|
|
28
102
|
if (frontmatter.title) {
|
|
29
103
|
return normalizeTitle(frontmatter.title);
|
|
@@ -34,7 +108,7 @@ const extractTitle = (filePath, content, frontmatter) => {
|
|
|
34
108
|
}
|
|
35
109
|
return normalizeTitle(basename(filePath));
|
|
36
110
|
};
|
|
37
|
-
export const extractWikiLinks = (content) => unique(
|
|
111
|
+
export const extractWikiLinks = (content) => unique(extractWikiLinkReferences(content).map((reference) => reference.title));
|
|
38
112
|
export const extractTags = (content) => unique(Array.from(stripFencedCodeBlocks(content).matchAll(tagPattern), (match) => match[2]));
|
|
39
113
|
const normalizeChunkContent = (content) => content
|
|
40
114
|
.split('\n')
|
|
@@ -87,10 +161,13 @@ export const parseMarkdownDocument = (input) => {
|
|
|
87
161
|
};
|
|
88
162
|
export const createIndexedDocument = (document, titleToDocumentId, maxChunkCharacters = 1200) => {
|
|
89
163
|
const chunks = splitIntoChunks(document.id, document.content, maxChunkCharacters);
|
|
164
|
+
const linkWeights = new Map(extractWikiLinkWeights(document.content).map((link) => [link.title.toLowerCase(), link]));
|
|
90
165
|
const links = document.links.map((toTitle) => ({
|
|
91
166
|
fromDocumentId: document.id,
|
|
92
167
|
toTitle,
|
|
93
|
-
toDocumentId: titleToDocumentId.get(toTitle.toLowerCase()) ?? null
|
|
168
|
+
toDocumentId: titleToDocumentId.get(toTitle.toLowerCase()) ?? null,
|
|
169
|
+
weight: linkWeights.get(toTitle.toLowerCase())?.weight ?? 1,
|
|
170
|
+
priority: linkWeights.get(toTitle.toLowerCase())?.priority ?? 'normal'
|
|
94
171
|
}));
|
|
95
172
|
return {
|
|
96
173
|
document,
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
import { GetObjectCommand, ListObjectsV2Command, PutObjectCommand, S3Client } from '@aws-sdk/client-s3';
|
|
2
|
+
import { chmod, mkdir, readFile, rm, writeFile } from 'node:fs/promises';
|
|
3
|
+
import { createHash } from 'node:crypto';
|
|
4
|
+
import { dirname, isAbsolute, join, relative } from 'node:path';
|
|
5
|
+
import { posix } from 'node:path';
|
|
6
|
+
import { getBrainlinkHomePath } from './paths.js';
|
|
7
|
+
const directoryMode = 0o700;
|
|
8
|
+
const fileMode = 0o600;
|
|
9
|
+
const bucketScheme = 's3:';
|
|
10
|
+
const manifestPath = '.brainlink/bucket-manifest.json';
|
|
11
|
+
const excludedSegments = new Set(['.brainlink', '.git', 'node_modules', 'dist']);
|
|
12
|
+
export const isBucketVaultUri = (value) => value.trim().toLowerCase().startsWith('s3://');
|
|
13
|
+
const trimSlashes = (value) => value.replace(/^\/+|\/+$/g, '');
|
|
14
|
+
const normalizePrefix = (value) => trimSlashes(posix.normalize(trimSlashes(value))).replace(/^\.$/, '');
|
|
15
|
+
export const parseBucketVaultUri = (uri) => {
|
|
16
|
+
const parsed = new URL(uri);
|
|
17
|
+
if (parsed.protocol !== bucketScheme || !parsed.hostname) {
|
|
18
|
+
throw new Error(`Unsupported bucket vault URI: ${uri}. Use s3://bucket/prefix.`);
|
|
19
|
+
}
|
|
20
|
+
return {
|
|
21
|
+
uri: formatBucketVaultUri(parsed.hostname, normalizePrefix(decodeURIComponent(parsed.pathname))),
|
|
22
|
+
bucket: parsed.hostname,
|
|
23
|
+
prefix: normalizePrefix(decodeURIComponent(parsed.pathname))
|
|
24
|
+
};
|
|
25
|
+
};
|
|
26
|
+
export const formatBucketVaultUri = (bucket, prefix) => prefix ? `s3://${bucket}/${prefix}` : `s3://${bucket}`;
|
|
27
|
+
export const getBucketVaultCachePath = (uri) => {
|
|
28
|
+
const hash = createHash('sha256').update(parseBucketVaultUri(uri).uri).digest('hex').slice(0, 24);
|
|
29
|
+
return join(getBrainlinkHomePath(), 'bucket-cache', hash);
|
|
30
|
+
};
|
|
31
|
+
const ensureDirectory = async (path) => {
|
|
32
|
+
await mkdir(path, { recursive: true, mode: directoryMode });
|
|
33
|
+
await chmod(path, directoryMode);
|
|
34
|
+
};
|
|
35
|
+
const isPathInside = (parent, child) => {
|
|
36
|
+
const path = relative(parent, child);
|
|
37
|
+
return path === '' || (!path.startsWith('..') && !isAbsolute(path));
|
|
38
|
+
};
|
|
39
|
+
const toSafeRelativePath = (key) => {
|
|
40
|
+
const normalized = normalizePrefix(key);
|
|
41
|
+
if (!normalized || normalized.split('/').some((segment) => segment === '..' || excludedSegments.has(segment))) {
|
|
42
|
+
return null;
|
|
43
|
+
}
|
|
44
|
+
return normalized.endsWith('.md') ? normalized : null;
|
|
45
|
+
};
|
|
46
|
+
const toObjectKey = (reference, relativePath) => reference.prefix ? `${reference.prefix}/${relativePath}` : relativePath;
|
|
47
|
+
const toRelativeObjectKey = (reference, objectKey) => {
|
|
48
|
+
const relativePath = reference.prefix
|
|
49
|
+
? objectKey.startsWith(`${reference.prefix}/`)
|
|
50
|
+
? objectKey.slice(reference.prefix.length + 1)
|
|
51
|
+
: null
|
|
52
|
+
: objectKey;
|
|
53
|
+
return relativePath ? toSafeRelativePath(relativePath) : null;
|
|
54
|
+
};
|
|
55
|
+
const createBucketClient = () => new S3Client({
|
|
56
|
+
region: process.env.AWS_REGION ?? process.env.BRAINLINK_S3_REGION ?? 'us-east-1',
|
|
57
|
+
endpoint: process.env.BRAINLINK_S3_ENDPOINT ?? process.env.AWS_ENDPOINT_URL,
|
|
58
|
+
forcePathStyle: process.env.BRAINLINK_S3_FORCE_PATH_STYLE === '1'
|
|
59
|
+
});
|
|
60
|
+
const streamToString = async (body) => {
|
|
61
|
+
if (body && typeof body === 'object' && 'transformToString' in body && typeof body.transformToString === 'function') {
|
|
62
|
+
return body.transformToString();
|
|
63
|
+
}
|
|
64
|
+
throw new Error('Unsupported S3 object body.');
|
|
65
|
+
};
|
|
66
|
+
const readManifest = async (cachePath) => {
|
|
67
|
+
try {
|
|
68
|
+
return JSON.parse(await readFile(join(cachePath, manifestPath), 'utf8'));
|
|
69
|
+
}
|
|
70
|
+
catch (error) {
|
|
71
|
+
if (error instanceof Error && 'code' in error && error.code === 'ENOENT') {
|
|
72
|
+
return {
|
|
73
|
+
uri: '',
|
|
74
|
+
keys: []
|
|
75
|
+
};
|
|
76
|
+
}
|
|
77
|
+
throw error;
|
|
78
|
+
}
|
|
79
|
+
};
|
|
80
|
+
const writeManifest = async (cachePath, manifest) => {
|
|
81
|
+
const path = join(cachePath, manifestPath);
|
|
82
|
+
await ensureDirectory(dirname(path));
|
|
83
|
+
await writeFile(path, `${JSON.stringify(manifest, null, 2)}\n`, { encoding: 'utf8', mode: fileMode });
|
|
84
|
+
await chmod(path, fileMode);
|
|
85
|
+
};
|
|
86
|
+
const listBucketMarkdownKeys = async (client, reference) => {
|
|
87
|
+
const keys = [];
|
|
88
|
+
let continuationToken;
|
|
89
|
+
do {
|
|
90
|
+
const result = await client.send(new ListObjectsV2Command({
|
|
91
|
+
Bucket: reference.bucket,
|
|
92
|
+
Prefix: reference.prefix ? `${reference.prefix}/` : undefined,
|
|
93
|
+
ContinuationToken: continuationToken
|
|
94
|
+
}));
|
|
95
|
+
keys.push(...(result.Contents ?? []).flatMap((object) => (object.Key ? [object.Key] : [])));
|
|
96
|
+
continuationToken = result.NextContinuationToken;
|
|
97
|
+
} while (continuationToken);
|
|
98
|
+
return keys.flatMap((key) => {
|
|
99
|
+
const relativePath = toRelativeObjectKey(reference, key);
|
|
100
|
+
return relativePath ? [relativePath] : [];
|
|
101
|
+
});
|
|
102
|
+
};
|
|
103
|
+
const removeStaleCachedFiles = async (cachePath, previousKeys, currentKeys) => {
|
|
104
|
+
await Promise.all(previousKeys
|
|
105
|
+
.filter((key) => !currentKeys.has(key))
|
|
106
|
+
.map(async (key) => {
|
|
107
|
+
const absolutePath = join(cachePath, key);
|
|
108
|
+
if (isPathInside(cachePath, absolutePath)) {
|
|
109
|
+
await rm(absolutePath, { force: true });
|
|
110
|
+
}
|
|
111
|
+
}));
|
|
112
|
+
};
|
|
113
|
+
const downloadMarkdownFiles = async (client, reference, cachePath, keys) => {
|
|
114
|
+
await Promise.all(keys.map(async (key) => {
|
|
115
|
+
const absolutePath = join(cachePath, key);
|
|
116
|
+
if (!isPathInside(cachePath, absolutePath)) {
|
|
117
|
+
throw new Error(`Refusing to cache bucket object outside vault cache: ${key}`);
|
|
118
|
+
}
|
|
119
|
+
const result = await client.send(new GetObjectCommand({
|
|
120
|
+
Bucket: reference.bucket,
|
|
121
|
+
Key: toObjectKey(reference, key)
|
|
122
|
+
}));
|
|
123
|
+
await ensureDirectory(dirname(absolutePath));
|
|
124
|
+
await writeFile(absolutePath, await streamToString(result.Body), { encoding: 'utf8', mode: fileMode });
|
|
125
|
+
await chmod(absolutePath, fileMode);
|
|
126
|
+
}));
|
|
127
|
+
};
|
|
128
|
+
export const syncBucketVaultToCache = async (uri) => {
|
|
129
|
+
const reference = parseBucketVaultUri(uri);
|
|
130
|
+
const cachePath = getBucketVaultCachePath(reference.uri);
|
|
131
|
+
const client = createBucketClient();
|
|
132
|
+
const previousManifest = await readManifest(cachePath);
|
|
133
|
+
const keys = await listBucketMarkdownKeys(client, reference);
|
|
134
|
+
const currentKeys = new Set(keys);
|
|
135
|
+
await ensureDirectory(join(cachePath, '.brainlink'));
|
|
136
|
+
await removeStaleCachedFiles(cachePath, previousManifest.uri === reference.uri ? previousManifest.keys : [], currentKeys);
|
|
137
|
+
await downloadMarkdownFiles(client, reference, cachePath, keys);
|
|
138
|
+
await writeManifest(cachePath, {
|
|
139
|
+
uri: reference.uri,
|
|
140
|
+
keys
|
|
141
|
+
});
|
|
142
|
+
return cachePath;
|
|
143
|
+
};
|
|
144
|
+
export const writeBucketMarkdownFile = async (uri, filename, content) => {
|
|
145
|
+
const reference = parseBucketVaultUri(uri);
|
|
146
|
+
const cachePath = getBucketVaultCachePath(reference.uri);
|
|
147
|
+
const relativePath = toSafeRelativePath(filename.endsWith('.md') ? filename : `${filename}.md`);
|
|
148
|
+
if (!relativePath) {
|
|
149
|
+
throw new Error(`Invalid bucket Markdown path: ${filename}`);
|
|
150
|
+
}
|
|
151
|
+
const absolutePath = join(cachePath, relativePath);
|
|
152
|
+
if (!isPathInside(cachePath, absolutePath)) {
|
|
153
|
+
throw new Error(`Refusing to write outside bucket cache: ${absolutePath}`);
|
|
154
|
+
}
|
|
155
|
+
await ensureDirectory(join(cachePath, '.brainlink'));
|
|
156
|
+
await ensureDirectory(dirname(absolutePath));
|
|
157
|
+
await writeFile(absolutePath, content, { encoding: 'utf8', mode: fileMode });
|
|
158
|
+
await chmod(absolutePath, fileMode);
|
|
159
|
+
await createBucketClient().send(new PutObjectCommand({
|
|
160
|
+
Bucket: reference.bucket,
|
|
161
|
+
Key: toObjectKey(reference, relativePath),
|
|
162
|
+
Body: content,
|
|
163
|
+
ContentType: 'text/markdown; charset=utf-8'
|
|
164
|
+
}));
|
|
165
|
+
const manifest = await readManifest(cachePath);
|
|
166
|
+
await writeManifest(cachePath, {
|
|
167
|
+
uri: reference.uri,
|
|
168
|
+
keys: Array.from(new Set([...manifest.keys, relativePath])).sort()
|
|
169
|
+
});
|
|
170
|
+
return `${reference.uri}/${relativePath}`;
|
|
171
|
+
};
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { chmod, mkdir, readdir, readFile, stat, writeFile } from 'node:fs/promises';
|
|
2
2
|
import { dirname, extname, isAbsolute, join, relative, resolve } from 'node:path';
|
|
3
3
|
import { resolvePath } from './paths.js';
|
|
4
|
+
import { getBucketVaultCachePath, isBucketVaultUri, parseBucketVaultUri, syncBucketVaultToCache, writeBucketMarkdownFile } from './bucket-vault.js';
|
|
4
5
|
const excludedDirectories = new Set(['.brainlink', '.git', 'node_modules', 'dist']);
|
|
5
6
|
const directoryMode = 0o700;
|
|
6
7
|
const fileMode = 0o600;
|
|
@@ -15,30 +16,44 @@ const walkMarkdownFiles = async (directory) => {
|
|
|
15
16
|
}));
|
|
16
17
|
return nested.flat();
|
|
17
18
|
};
|
|
18
|
-
export const resolveVaultPath = (vaultPath) => resolvePath(vaultPath);
|
|
19
|
+
export const resolveVaultPath = (vaultPath) => isBucketVaultUri(vaultPath) ? getBucketVaultCachePath(vaultPath) : resolvePath(vaultPath);
|
|
20
|
+
export const isBucketVaultPath = (vaultPath) => isBucketVaultUri(vaultPath);
|
|
19
21
|
const isPathInside = (parent, child) => {
|
|
20
22
|
const path = relative(parent, child);
|
|
21
23
|
return path === '' || (!path.startsWith('..') && !isAbsolute(path));
|
|
22
24
|
};
|
|
25
|
+
const isBucketPrefixInside = (parent, child) => parent === '' || child === parent || child.startsWith(`${parent}/`);
|
|
23
26
|
const secureDirectory = async (path) => {
|
|
24
27
|
await mkdir(path, { recursive: true, mode: directoryMode });
|
|
25
28
|
await chmod(path, directoryMode);
|
|
26
29
|
};
|
|
27
30
|
export const assertVaultAllowed = (vaultPath, allowedVaults) => {
|
|
31
|
+
if (isBucketVaultUri(vaultPath)) {
|
|
32
|
+
const vault = parseBucketVaultUri(vaultPath);
|
|
33
|
+
const allowed = allowedVaults.filter(isBucketVaultUri).map(parseBucketVaultUri);
|
|
34
|
+
if (allowedVaults.length > 0 &&
|
|
35
|
+
!allowed.some((allowedVault) => vault.bucket === allowedVault.bucket && isBucketPrefixInside(allowedVault.prefix, vault.prefix))) {
|
|
36
|
+
throw new Error(`Vault path is not allowed: ${vault.uri}. Configure BRAINLINK_ALLOWED_VAULTS or allowedVaults.`);
|
|
37
|
+
}
|
|
38
|
+
return vault.uri;
|
|
39
|
+
}
|
|
28
40
|
const absoluteVaultPath = resolveVaultPath(vaultPath);
|
|
29
|
-
const allowed = allowedVaults.map(resolveVaultPath);
|
|
41
|
+
const allowed = allowedVaults.filter((allowedVault) => !isBucketVaultUri(allowedVault)).map(resolveVaultPath);
|
|
30
42
|
if (allowed.length > 0 && !allowed.some((allowedPath) => isPathInside(allowedPath, absoluteVaultPath))) {
|
|
31
43
|
throw new Error(`Vault path is not allowed: ${absoluteVaultPath}. Configure BRAINLINK_ALLOWED_VAULTS or allowedVaults.`);
|
|
32
44
|
}
|
|
33
45
|
return absoluteVaultPath;
|
|
34
46
|
};
|
|
35
47
|
export const ensureVault = async (vaultPath) => {
|
|
48
|
+
if (isBucketVaultUri(vaultPath)) {
|
|
49
|
+
return syncBucketVaultToCache(vaultPath);
|
|
50
|
+
}
|
|
36
51
|
const absoluteVaultPath = resolveVaultPath(vaultPath);
|
|
37
52
|
await secureDirectory(join(absoluteVaultPath, '.brainlink'));
|
|
38
53
|
return absoluteVaultPath;
|
|
39
54
|
};
|
|
40
55
|
export const readMarkdownFiles = async (vaultPath) => {
|
|
41
|
-
const absoluteVaultPath =
|
|
56
|
+
const absoluteVaultPath = await ensureVault(vaultPath);
|
|
42
57
|
const paths = await walkMarkdownFiles(absoluteVaultPath);
|
|
43
58
|
return Promise.all(paths.map(async (absolutePath) => {
|
|
44
59
|
const [content, stats] = await Promise.all([readFile(absolutePath, 'utf8'), stat(absolutePath)]);
|
|
@@ -51,6 +66,9 @@ export const readMarkdownFiles = async (vaultPath) => {
|
|
|
51
66
|
}));
|
|
52
67
|
};
|
|
53
68
|
export const writeMarkdownFile = async (vaultPath, filename, content) => {
|
|
69
|
+
if (isBucketVaultUri(vaultPath)) {
|
|
70
|
+
return writeBucketMarkdownFile(vaultPath, filename, content);
|
|
71
|
+
}
|
|
54
72
|
const absoluteVaultPath = await ensureVault(vaultPath);
|
|
55
73
|
const absolutePath = resolve(absoluteVaultPath, filename.endsWith('.md') ? filename : `${filename}.md`);
|
|
56
74
|
if (!isPathInside(absoluteVaultPath, absolutePath)) {
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { createEmbeddingBuckets } from '../../domain/embeddings.js';
|
|
2
|
+
const toTitleKey = (title) => title.toLowerCase();
|
|
2
3
|
export const createIndexWriter = (database) => ({
|
|
3
4
|
reset: () => {
|
|
4
5
|
database.exec(`
|
|
@@ -27,8 +28,8 @@ export const createIndexWriter = (database) => ({
|
|
|
27
28
|
VALUES (?, ?)
|
|
28
29
|
`);
|
|
29
30
|
const insertLink = database.prepare(`
|
|
30
|
-
INSERT INTO links (from_document_id, to_title, to_document_id)
|
|
31
|
-
VALUES (?, ?, ?)
|
|
31
|
+
INSERT INTO links (from_document_id, to_title, to_title_key, to_document_id, weight, priority)
|
|
32
|
+
VALUES (?, ?, ?, ?, ?, ?)
|
|
32
33
|
`);
|
|
33
34
|
const transaction = database.transaction(() => {
|
|
34
35
|
documents.forEach(({ document, chunks, links }) => {
|
|
@@ -41,7 +42,7 @@ export const createIndexWriter = (database) => ({
|
|
|
41
42
|
});
|
|
42
43
|
documents.forEach(({ links }) => {
|
|
43
44
|
links.forEach((link) => {
|
|
44
|
-
insertLink.run(link.fromDocumentId, link.toTitle, link.toDocumentId);
|
|
45
|
+
insertLink.run(link.fromDocumentId, link.toTitle, toTitleKey(link.toTitle), link.toDocumentId, link.weight, link.priority);
|
|
45
46
|
});
|
|
46
47
|
});
|
|
47
48
|
});
|
|
@@ -4,9 +4,12 @@ const toGraphLink = (row) => ({
|
|
|
4
4
|
fromTitle: row.from_title,
|
|
5
5
|
fromPath: row.from_path,
|
|
6
6
|
toTitle: row.to_title,
|
|
7
|
-
toPath: row.to_path
|
|
7
|
+
toPath: row.to_path,
|
|
8
|
+
weight: row.weight,
|
|
9
|
+
priority: row.priority
|
|
8
10
|
});
|
|
9
11
|
const normalizeAgentFilter = (agentId) => agentId ? sanitizeAgentId(agentId) : undefined;
|
|
12
|
+
const toTitleKey = (title) => title.toLowerCase();
|
|
10
13
|
export const createGraphReader = (database) => ({
|
|
11
14
|
listLinks: (agentId) => {
|
|
12
15
|
const normalizedAgentId = normalizeAgentFilter(agentId);
|
|
@@ -18,12 +21,14 @@ export const createGraphReader = (database) => ({
|
|
|
18
21
|
source.title AS from_title,
|
|
19
22
|
source.path AS from_path,
|
|
20
23
|
COALESCE(target.title, links.to_title) AS to_title,
|
|
21
|
-
target.path AS to_path
|
|
24
|
+
target.path AS to_path,
|
|
25
|
+
links.weight AS weight,
|
|
26
|
+
links.priority AS priority
|
|
22
27
|
FROM links
|
|
23
28
|
JOIN documents source ON source.id = links.from_document_id
|
|
24
29
|
LEFT JOIN documents target ON target.id = links.to_document_id
|
|
25
30
|
${agentFilter}
|
|
26
|
-
ORDER BY source.title, to_title
|
|
31
|
+
ORDER BY source.title, links.weight DESC, to_title
|
|
27
32
|
`)
|
|
28
33
|
.all(...(normalizedAgentId ? [normalizedAgentId] : []));
|
|
29
34
|
return rows.map(toGraphLink);
|
|
@@ -31,6 +36,7 @@ export const createGraphReader = (database) => ({
|
|
|
31
36
|
listBacklinks: (title, agentId) => {
|
|
32
37
|
const normalizedAgentId = normalizeAgentFilter(agentId);
|
|
33
38
|
const agentFilter = normalizedAgentId ? 'AND source.agent_id = ?' : '';
|
|
39
|
+
const titleKey = toTitleKey(title);
|
|
34
40
|
const rows = database
|
|
35
41
|
.prepare(`
|
|
36
42
|
SELECT
|
|
@@ -38,15 +44,17 @@ export const createGraphReader = (database) => ({
|
|
|
38
44
|
source.title AS from_title,
|
|
39
45
|
source.path AS from_path,
|
|
40
46
|
COALESCE(target.title, links.to_title) AS to_title,
|
|
41
|
-
target.path AS to_path
|
|
47
|
+
target.path AS to_path,
|
|
48
|
+
links.weight AS weight,
|
|
49
|
+
links.priority AS priority
|
|
42
50
|
FROM links
|
|
43
51
|
JOIN documents source ON source.id = links.from_document_id
|
|
44
52
|
LEFT JOIN documents target ON target.id = links.to_document_id
|
|
45
|
-
WHERE
|
|
53
|
+
WHERE links.to_title_key = ?
|
|
46
54
|
${agentFilter}
|
|
47
|
-
ORDER BY source.title
|
|
55
|
+
ORDER BY links.weight DESC, source.title
|
|
48
56
|
`)
|
|
49
|
-
.all(...(normalizedAgentId ? [
|
|
57
|
+
.all(...(normalizedAgentId ? [titleKey, normalizedAgentId] : [titleKey]));
|
|
50
58
|
return rows.map(toGraphLink);
|
|
51
59
|
},
|
|
52
60
|
getGraph: (agentId) => {
|
|
@@ -66,11 +74,13 @@ export const createGraphReader = (database) => ({
|
|
|
66
74
|
SELECT
|
|
67
75
|
links.from_document_id AS source,
|
|
68
76
|
links.to_document_id AS target,
|
|
69
|
-
links.to_title AS target_title
|
|
77
|
+
links.to_title AS target_title,
|
|
78
|
+
links.weight AS weight,
|
|
79
|
+
links.priority AS priority
|
|
70
80
|
FROM links
|
|
71
81
|
JOIN documents source ON source.id = links.from_document_id
|
|
72
82
|
${edgeAgentFilter}
|
|
73
|
-
ORDER BY links.from_document_id, links.to_title
|
|
83
|
+
ORDER BY links.from_document_id, links.weight DESC, links.to_title
|
|
74
84
|
`)
|
|
75
85
|
.all(...(normalizedAgentId ? [normalizedAgentId] : []));
|
|
76
86
|
const nodes = nodeRows.map((row) => ({
|
|
@@ -84,7 +94,9 @@ export const createGraphReader = (database) => ({
|
|
|
84
94
|
const edges = edgeRows.map((row) => ({
|
|
85
95
|
source: row.source,
|
|
86
96
|
target: row.target,
|
|
87
|
-
targetTitle: row.target_title
|
|
97
|
+
targetTitle: row.target_title,
|
|
98
|
+
weight: row.weight,
|
|
99
|
+
priority: row.priority
|
|
88
100
|
}));
|
|
89
101
|
return {
|
|
90
102
|
nodes,
|
|
@@ -1,7 +1,8 @@
|
|
|
1
|
-
const schemaVersion =
|
|
1
|
+
const schemaVersion = 5;
|
|
2
2
|
const requiredTableColumns = {
|
|
3
3
|
documents: ['id', 'agent_id', 'title', 'path', 'content', 'tags_json', 'frontmatter_json', 'created_at', 'updated_at'],
|
|
4
4
|
chunks: ['id', 'document_id', 'ordinal', 'content', 'token_count', 'embedding_provider', 'embedding_json'],
|
|
5
|
+
links: ['from_document_id', 'to_title', 'to_title_key', 'to_document_id', 'weight', 'priority'],
|
|
5
6
|
chunks_fts: ['chunk_id', 'document_id', 'agent_id', 'title', 'content']
|
|
6
7
|
};
|
|
7
8
|
const getStoredSchemaVersion = (database) => {
|
|
@@ -65,6 +66,9 @@ export const createSchema = (database) => {
|
|
|
65
66
|
FOREIGN KEY (document_id) REFERENCES documents(id) ON DELETE CASCADE
|
|
66
67
|
);
|
|
67
68
|
|
|
69
|
+
CREATE INDEX IF NOT EXISTS idx_documents_agent_title ON documents(agent_id, title);
|
|
70
|
+
CREATE INDEX IF NOT EXISTS idx_chunks_document_ordinal ON chunks(document_id, ordinal);
|
|
71
|
+
|
|
68
72
|
CREATE TABLE IF NOT EXISTS embedding_buckets (
|
|
69
73
|
bucket TEXT NOT NULL,
|
|
70
74
|
chunk_id TEXT NOT NULL,
|
|
@@ -77,11 +81,18 @@ export const createSchema = (database) => {
|
|
|
77
81
|
CREATE TABLE IF NOT EXISTS links (
|
|
78
82
|
from_document_id TEXT NOT NULL,
|
|
79
83
|
to_title TEXT NOT NULL,
|
|
84
|
+
to_title_key TEXT NOT NULL,
|
|
80
85
|
to_document_id TEXT,
|
|
86
|
+
weight INTEGER NOT NULL,
|
|
87
|
+
priority TEXT NOT NULL,
|
|
88
|
+
PRIMARY KEY (from_document_id, to_title_key),
|
|
81
89
|
FOREIGN KEY (from_document_id) REFERENCES documents(id) ON DELETE CASCADE,
|
|
82
90
|
FOREIGN KEY (to_document_id) REFERENCES documents(id) ON DELETE SET NULL
|
|
83
91
|
);
|
|
84
92
|
|
|
93
|
+
CREATE INDEX IF NOT EXISTS idx_links_to_document_id ON links(to_document_id);
|
|
94
|
+
CREATE INDEX IF NOT EXISTS idx_links_to_title_key ON links(to_title_key);
|
|
95
|
+
|
|
85
96
|
CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(
|
|
86
97
|
chunk_id UNINDEXED,
|
|
87
98
|
document_id UNINDEXED,
|
|
@@ -9,7 +9,12 @@ export const openSqliteIndex = (vaultPath) => {
|
|
|
9
9
|
const databasePath = join(vaultPath, '.brainlink', 'brainlink.db');
|
|
10
10
|
const database = new Database(databasePath);
|
|
11
11
|
chmodSync(databasePath, 0o600);
|
|
12
|
-
database.exec(
|
|
12
|
+
database.exec(`
|
|
13
|
+
PRAGMA foreign_keys = ON;
|
|
14
|
+
PRAGMA journal_mode = WAL;
|
|
15
|
+
PRAGMA synchronous = NORMAL;
|
|
16
|
+
PRAGMA temp_store = MEMORY;
|
|
17
|
+
`);
|
|
13
18
|
createSchema(database);
|
|
14
19
|
return {
|
|
15
20
|
...createIndexWriter(database),
|
package/dist/mcp/server.js
CHANGED
|
@@ -27,7 +27,7 @@ export const createBrainlinkMcpServer = () => {
|
|
|
27
27
|
}, searchTool);
|
|
28
28
|
server.registerTool('brainlink_add_note', {
|
|
29
29
|
title: 'Add Brainlink Note',
|
|
30
|
-
description: 'Write durable Markdown memory, then reindex the vault. Include explicit [[wiki links]] for connected graph memory.',
|
|
30
|
+
description: 'Write durable Markdown memory, then reindex the vault. Include explicit [[wiki links]] for connected graph memory. Add priority markers near links, such as priority: high, #important or #critical, when a relationship should be weighted higher.',
|
|
31
31
|
inputSchema: addNoteInputSchema
|
|
32
32
|
}, addNoteTool);
|
|
33
33
|
server.registerTool('brainlink_index', {
|
|
@@ -42,7 +42,7 @@ export const createBrainlinkMcpServer = () => {
|
|
|
42
42
|
}, validateTool);
|
|
43
43
|
server.registerTool('brainlink_graph', {
|
|
44
44
|
title: 'Read Brainlink Graph',
|
|
45
|
-
description: 'Read indexed graph nodes and wiki-link edges.',
|
|
45
|
+
description: 'Read indexed graph nodes and wiki-link edges. Edges include weight and priority fields so agents can rank importance and priority.',
|
|
46
46
|
inputSchema: graphInputSchema
|
|
47
47
|
}, graphTool);
|
|
48
48
|
server.registerTool('brainlink_broken_links', {
|
package/dist/mcp/tools.js
CHANGED
|
@@ -57,7 +57,7 @@ export const addNoteInputSchema = {
|
|
|
57
57
|
content: z
|
|
58
58
|
.string()
|
|
59
59
|
.min(1)
|
|
60
|
-
.describe('Durable Markdown memory. Include explicit [[wiki links]] and #tags when the memory should be connected.'),
|
|
60
|
+
.describe('Durable Markdown memory. Include explicit [[wiki links]] and #tags when the memory should be connected. Put priority markers near important links, for example priority: high, #important or #critical.'),
|
|
61
61
|
agent: z.string().min(1).optional().default('shared').describe('Agent memory namespace. Defaults to shared.'),
|
|
62
62
|
allowSensitive: z.boolean().optional().default(false).describe('Allow content that looks like a secret.')
|
|
63
63
|
};
|
package/docs/AGENT_USAGE.md
CHANGED
|
@@ -133,6 +133,7 @@ Rules:
|
|
|
133
133
|
|
|
134
134
|
- Use a clear title.
|
|
135
135
|
- Use `[[Note Title]]` for relationships.
|
|
136
|
+
- Put priority markers near links when the relationship is important.
|
|
136
137
|
- Use tags for retrieval.
|
|
137
138
|
- Keep each note focused.
|
|
138
139
|
- Prefer summaries over raw transcripts.
|
|
@@ -144,6 +145,15 @@ Brainlink only builds graph edges from Markdown `[[wiki links]]`.
|
|
|
144
145
|
|
|
145
146
|
The `context` command is read-only. It retrieves indexed notes and returns a compact package for the model, but it does not write memory, create backlinks, infer relationships or modify the graph. If an agent reads context and then learns something durable, the agent must write a note with explicit links before that knowledge becomes connected memory.
|
|
146
147
|
|
|
148
|
+
Graph edges are weighted during indexing. Repeated links increase weight. Links inside headings or task-list lines receive a small boost. Priority markers on the same line as a link raise its priority:
|
|
149
|
+
|
|
150
|
+
```md
|
|
151
|
+
- [ ] Review [[Architecture]] priority: high
|
|
152
|
+
Related: [[Incident Runbook]] #critical
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Agents should use weighted graph output to sort relationships by importance. Edges expose `weight` and `priority`, where priority is one of `low`, `normal`, `high` or `critical`.
|
|
156
|
+
|
|
147
157
|
Required write behavior:
|
|
148
158
|
|
|
149
159
|
1. Choose a clear title for the new note.
|
|
@@ -464,6 +474,8 @@ Available MCP tools:
|
|
|
464
474
|
|
|
465
475
|
MCP clients can pass `vault` and `agent` arguments per tool call. Set `BRAINLINK_ALLOWED_VAULTS` when exposing Brainlink to an external agent process so a tool cannot pass arbitrary vault paths:
|
|
466
476
|
|
|
477
|
+
`brainlink_graph` returns weighted edges. Agents should prefer higher `weight` and stronger `priority` when deciding which related notes matter most.
|
|
478
|
+
|
|
467
479
|
```bash
|
|
468
480
|
export BRAINLINK_ALLOWED_VAULTS="/absolute/path/to/project-vault"
|
|
469
481
|
```
|
|
@@ -552,4 +564,5 @@ Weak retrieval usually means:
|
|
|
552
564
|
- Local embeddings are deterministic and provider-free; remote embedding providers are not implemented yet.
|
|
553
565
|
- MCP integration is available through the `brainlink-mcp` stdio server.
|
|
554
566
|
- HTTP API is local and unauthenticated.
|
|
555
|
-
-
|
|
567
|
+
- Bucket vaults support S3-compatible `s3://bucket/prefix` URIs and use a local cache for SQLite indexes.
|
|
568
|
+
- Watch mode depends on platform filesystem watcher behavior and is only supported for local filesystem vaults.
|
package/docs/ARCHITECTURE.md
CHANGED
|
@@ -105,13 +105,15 @@ Application code depends on domain rules and infrastructure interfaces.
|
|
|
105
105
|
The infrastructure layer handles side effects:
|
|
106
106
|
|
|
107
107
|
- reading Markdown files from disk
|
|
108
|
+
- mirroring S3-compatible bucket Markdown into a local cache
|
|
108
109
|
- writing Markdown notes
|
|
109
110
|
- creating `.brainlink`
|
|
110
111
|
- writing and querying SQLite
|
|
111
112
|
- running FTS, semantic and hybrid retrieval
|
|
112
113
|
- narrowing semantic candidates through SQLite embedding buckets before cosine scoring
|
|
113
114
|
|
|
114
|
-
SQLite is an index, not the canonical storage model.
|
|
115
|
+
SQLite is an index, not the canonical storage model. For bucket vaults, Markdown
|
|
116
|
+
objects in the bucket remain canonical and SQLite is still local derived data.
|
|
115
117
|
|
|
116
118
|
## Indexing Flow
|
|
117
119
|
|
|
@@ -216,6 +218,23 @@ source note -> target note
|
|
|
216
218
|
|
|
217
219
|
The `backlinks` command queries indexed links pointing to a target title. With `--agent`, it only returns links from that namespace.
|
|
218
220
|
|
|
221
|
+
## Weighted Links
|
|
222
|
+
|
|
223
|
+
Each indexed wiki link is stored as a graph edge with:
|
|
224
|
+
|
|
225
|
+
- `weight`: numeric relationship strength.
|
|
226
|
+
- `priority`: one of `low`, `normal`, `high` or `critical`.
|
|
227
|
+
|
|
228
|
+
The parser derives weight from repeated links, task-list context, heading context and priority markers on the same line as a wiki link. Examples:
|
|
229
|
+
|
|
230
|
+
```md
|
|
231
|
+
Related: [[Architecture]]
|
|
232
|
+
- [ ] Review [[Architecture]] priority: high
|
|
233
|
+
Escalate [[Incident Runbook]] #critical
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
Backlink and graph readers return those fields to CLI JSON, HTTP API and MCP clients. Backlink queries use the normalized `to_title_key` column instead of applying `lower(...)` at read time.
|
|
237
|
+
|
|
219
238
|
## Context Building
|
|
220
239
|
|
|
221
240
|
`context` uses search results and selects one chunk per document while staying inside an estimated token budget.
|
|
@@ -240,6 +259,7 @@ Relevant content
|
|
|
240
259
|
Permanent:
|
|
241
260
|
|
|
242
261
|
- Markdown files
|
|
262
|
+
- S3-compatible Markdown objects when the vault is `s3://bucket/prefix`
|
|
243
263
|
- optional Git history around the vault
|
|
244
264
|
|
|
245
265
|
Canonical agent memory lives under:
|
|
@@ -251,6 +271,7 @@ vault/agents/<agent-id>/**/*.md
|
|
|
251
271
|
Rebuildable:
|
|
252
272
|
|
|
253
273
|
- `.brainlink/brainlink.db`
|
|
274
|
+
- `$BRAINLINK_HOME/bucket-cache`
|
|
254
275
|
- FTS records
|
|
255
276
|
- local embedding vectors
|
|
256
277
|
- local embedding bucket index
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@andespindola/brainlink",
|
|
3
|
-
"version": "0.1.0-alpha.
|
|
3
|
+
"version": "0.1.0-alpha.12",
|
|
4
4
|
"description": "Local-first knowledge memory for agents with Markdown, backlinks, indexing and context retrieval.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -54,6 +54,7 @@
|
|
|
54
54
|
"pack:smoke": "npm pack --dry-run"
|
|
55
55
|
},
|
|
56
56
|
"dependencies": {
|
|
57
|
+
"@aws-sdk/client-s3": "^3.1038.0",
|
|
57
58
|
"@modelcontextprotocol/sdk": "^1.29.0",
|
|
58
59
|
"better-sqlite3": "^12.9.0",
|
|
59
60
|
"commander": "^14.0.2",
|