claude-second-brain 0.5.1 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +48 -32
- package/bin/create.js +826 -84
- package/package.json +3 -2
- package/template/.claude/skills/brain-ingest/SKILL.md +21 -4
- package/template/.claude/skills/brain-rebuild/SKILL.md +9 -9
- package/template/.claude/skills/brain-refresh/SKILL.md +18 -5
- package/template/.claude/skills/brain-search/SKILL.md +17 -5
- package/template/CLAUDE.md +7 -7
- package/template/README.md +17 -19
- package/template/scripts/qmd/reindex.ts +31 -19
- package/template/scripts/qmd/setup.ts +45 -36
- package/template/.claude/skills/qmd-cli/SKILL.md +0 -168
- package/template/.claude/skills/setup/SKILL.md +0 -66
package/package.json
CHANGED
|
@@ -1,10 +1,11 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "claude-second-brain",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "1.0.0",
|
|
4
4
|
"description": "The fastest way to start your personal knowledge base powered by Obsidian, Claude Code, qmd, and GitHub.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
|
-
"claude-second-brain": "./bin/create.js"
|
|
7
|
+
"claude-second-brain": "./bin/create.js",
|
|
8
|
+
"csb": "./bin/create.js"
|
|
8
9
|
},
|
|
9
10
|
"scripts": {
|
|
10
11
|
"docs:dev": "vocs dev",
|
|
@@ -8,6 +8,23 @@ argument-hint: "File path (e.g. raw-sources/articles/my-article.md), URL, or lea
|
|
|
8
8
|
|
|
9
9
|
Runs the full 9-step ingest workflow defined in CLAUDE.md. Do not skip steps.
|
|
10
10
|
|
|
11
|
+
## Brain Discovery
|
|
12
|
+
|
|
13
|
+
All commands target the **default brain** registered in `~/.claude-second-brain/config.toml`.
|
|
14
|
+
Resolve paths at call time via the `claude-second-brain` CLI — do not bake paths into this file:
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
BRAIN_PATH=$(npx -y claude-second-brain path) # absolute path to the default brain's directory
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
qmd commands go through the CLI proxy, which sets `INDEX_PATH` for you:
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
npx -y claude-second-brain qmd -- query -c wiki "<terms>"
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Pass `--brain <name>` before the `--` to target a non-default brain.
|
|
27
|
+
|
|
11
28
|
## Inputs
|
|
12
29
|
|
|
13
30
|
- **File path** — a file in `raw-sources/articles/`, `raw-sources/pdfs/`, or `raw-sources/personal/`
|
|
@@ -27,7 +44,7 @@ Runs the full 9-step ingest workflow defined in CLAUDE.md. Do not skip steps.
|
|
|
27
44
|
- Let the user's response shape which topics get deep treatment before proceeding
|
|
28
45
|
|
|
29
46
|
**Step 3 — Create source summary page**
|
|
30
|
-
- File:
|
|
47
|
+
- File: `$BRAIN_PATH/wiki/sources/[slug].md` (slug = kebab-case from title)
|
|
31
48
|
- Sections: title, author/date, one-paragraph abstract, key claims (bulleted), notable quotes (max 3), synthesis note, links to wiki pages this source touches
|
|
32
49
|
- Frontmatter: `type: source-summary`, `tags`, `updated: YYYY-MM-DD`
|
|
33
50
|
|
|
@@ -35,8 +52,8 @@ Runs the full 9-step ingest workflow defined in CLAUDE.md. Do not skip steps.
|
|
|
35
52
|
- Add the new source to the Sources Ingested section: one-line description + `[[wiki/sources/slug]]` link
|
|
36
53
|
|
|
37
54
|
**Step 5 — Identify affected wiki pages**
|
|
38
|
-
- Run: `
|
|
39
|
-
- Also Glob
|
|
55
|
+
- Run: `npx -y claude-second-brain qmd -- query -c wiki "<source topic and key claims>"`
|
|
56
|
+
- Also Glob `$BRAIN_PATH/wiki/*.md` and `$BRAIN_PATH/wiki/sources/*.md` to catch anything qmd missed
|
|
40
57
|
- List all pages to create or update before proceeding
|
|
41
58
|
|
|
42
59
|
**Step 6 — Update or create wiki pages**
|
|
@@ -54,7 +71,7 @@ Runs the full 9-step ingest workflow defined in CLAUDE.md. Do not skip steps.
|
|
|
54
71
|
- Otherwise skip this step
|
|
55
72
|
|
|
56
73
|
**Step 9 — Append to log**
|
|
57
|
-
- Append to
|
|
74
|
+
- Append to `$BRAIN_PATH/wiki/log.md` (never overwrite):
|
|
58
75
|
```
|
|
59
76
|
## [YYYY-MM-DD] ingest | Source Title
|
|
60
77
|
|
|
@@ -25,9 +25,9 @@ All commands run from the vault root.
|
|
|
25
25
|
- Read `CLAUDE.md` to understand the documented schema and any references to collection names
|
|
26
26
|
- List current state from qmd:
|
|
27
27
|
```bash
|
|
28
|
-
INDEX_PATH
|
|
29
|
-
INDEX_PATH
|
|
30
|
-
INDEX_PATH
|
|
28
|
+
INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd collection list
|
|
29
|
+
INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd context list
|
|
30
|
+
INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd status
|
|
31
31
|
```
|
|
32
32
|
|
|
33
33
|
### Step 2 — Analyze the wiki
|
|
@@ -66,7 +66,7 @@ Edit `scripts/qmd/setup.ts` to reflect the new schema:
|
|
|
66
66
|
If collection names changed:
|
|
67
67
|
|
|
68
68
|
- Update every occurrence in `CLAUDE.md` (e.g. `qmd query -c wiki` → new name)
|
|
69
|
-
- Update `.claude/skills/brain-ingest/SKILL.md`, `.claude/skills/brain-search/SKILL.md`, `.claude/skills/lint/SKILL.md
|
|
69
|
+
- Update `.claude/skills/brain-ingest/SKILL.md`, `.claude/skills/brain-search/SKILL.md`, `.claude/skills/lint/SKILL.md` — anywhere collection names are hard-coded
|
|
70
70
|
- Use Grep to find any remaining stale references before moving on
|
|
71
71
|
|
|
72
72
|
### Step 7 — Tear down the old schema
|
|
@@ -74,13 +74,13 @@ If collection names changed:
|
|
|
74
74
|
For each existing collection that no longer fits the new schema:
|
|
75
75
|
|
|
76
76
|
```bash
|
|
77
|
-
INDEX_PATH
|
|
77
|
+
INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd collection remove <old-name>
|
|
78
78
|
```
|
|
79
79
|
|
|
80
80
|
For each obsolete context:
|
|
81
81
|
|
|
82
82
|
```bash
|
|
83
|
-
INDEX_PATH
|
|
83
|
+
INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd context rm <path>
|
|
84
84
|
```
|
|
85
85
|
|
|
86
86
|
### Step 8 — Register the new schema
|
|
@@ -104,9 +104,9 @@ This indexes all files under the new collections and generates fresh embeddings.
|
|
|
104
104
|
### Step 10 — Verify and report
|
|
105
105
|
|
|
106
106
|
```bash
|
|
107
|
-
INDEX_PATH
|
|
108
|
-
INDEX_PATH
|
|
109
|
-
INDEX_PATH
|
|
107
|
+
INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd collection list
|
|
108
|
+
INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd context list
|
|
109
|
+
INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd status
|
|
110
110
|
```
|
|
111
111
|
|
|
112
112
|
Confirm the new collections appear, contexts match the plan, and document/embedding counts are non-zero. Report a summary of what changed (collections added/removed/renamed, contexts updated, files re-indexed, references updated in CLAUDE.md and skills).
|
|
@@ -8,6 +8,18 @@ argument-hint: "Optional: 'force' to force re-embedding of every chunk (slow, us
|
|
|
8
8
|
|
|
9
9
|
Refreshes the qmd index so search reflects the current state of the vault. Wraps `pnpm qmd:reindex` (incremental) and the qmd CLI's force-embed flag.
|
|
10
10
|
|
|
11
|
+
## Brain Discovery
|
|
12
|
+
|
|
13
|
+
All commands target the **default brain** registered in `~/.claude-second-brain/config.toml`.
|
|
14
|
+
Resolve the brain root via the CLI and run qmd through the CLI proxy:
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
BRAIN_PATH=$(npx -y claude-second-brain path)
|
|
18
|
+
npx -y claude-second-brain qmd -- status
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Pass `--brain <name>` before the `--` to target a non-default brain.
|
|
22
|
+
|
|
11
23
|
## When to Use
|
|
12
24
|
|
|
13
25
|
- After a `/brain-ingest` session (or several) — batch the refresh, don't run after every file edit
|
|
@@ -17,13 +29,13 @@ Refreshes the qmd index so search reflects the current state of the vault. Wraps
|
|
|
17
29
|
|
|
18
30
|
## Procedure
|
|
19
31
|
|
|
20
|
-
All commands run from the
|
|
32
|
+
All `pnpm qmd:*` commands must run from `$BRAIN_PATH`; the CLI proxy (`claude-second-brain qmd`) works from anywhere.
|
|
21
33
|
|
|
22
34
|
### Default — Incremental Refresh
|
|
23
35
|
|
|
24
36
|
Run:
|
|
25
37
|
```bash
|
|
26
|
-
pnpm qmd:reindex
|
|
38
|
+
cd "$BRAIN_PATH" && pnpm qmd:reindex
|
|
27
39
|
```
|
|
28
40
|
|
|
29
41
|
This script does two things:
|
|
@@ -38,10 +50,10 @@ When the user passes `force`, re-embed **every** chunk — not just the changed
|
|
|
38
50
|
|
|
39
51
|
```bash
|
|
40
52
|
# Step 1 — update the file index first
|
|
41
|
-
pnpm qmd:reindex
|
|
53
|
+
cd "$BRAIN_PATH" && pnpm qmd:reindex
|
|
42
54
|
|
|
43
55
|
# Step 2 — force re-embed everything
|
|
44
|
-
|
|
56
|
+
npx -y claude-second-brain qmd -- embed -f
|
|
45
57
|
```
|
|
46
58
|
|
|
47
59
|
Force mode is slow — confirm with the user before proceeding if the wiki is large.
|
|
@@ -51,7 +63,7 @@ Force mode is slow — confirm with the user before proceeding if the wiki is la
|
|
|
51
63
|
After refresh, confirm with:
|
|
52
64
|
|
|
53
65
|
```bash
|
|
54
|
-
|
|
66
|
+
npx -y claude-second-brain qmd -- status
|
|
55
67
|
```
|
|
56
68
|
|
|
57
69
|
Document and embedding counts should be non-zero and reflect recent activity. If embeddings show as `0` or far below document count, re-run the refresh.
|
|
@@ -61,3 +73,4 @@ Document and embedding counts should be non-zero and reflect recent activity. If
|
|
|
61
73
|
- First run downloads ~2GB of GGUF models — expected, one-time
|
|
62
74
|
- Do not run this after every single file edit — batch it after a session
|
|
63
75
|
- This skill does **not** change the qmd schema (collections, contexts). For that, use `/brain-rebuild`
|
|
76
|
+
- If `query` or `search` commands fail with "collection not found", run `pnpm qmd:setup` from the vault root to re-register collections, then re-run this skill
|
|
@@ -8,12 +8,24 @@ argument-hint: "The question or topic to query (e.g. 'what do I know about trans
|
|
|
8
8
|
|
|
9
9
|
Runs the 4-step query workflow defined in CLAUDE.md. Answers come with inline `[[wiki/page]]` citations — not a list of files.
|
|
10
10
|
|
|
11
|
+
## Brain Discovery
|
|
12
|
+
|
|
13
|
+
All commands target the **default brain** registered in `~/.claude-second-brain/config.toml`.
|
|
14
|
+
Resolve paths and run qmd via the `claude-second-brain` CLI — do not bake paths into this file:
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
BRAIN_PATH=$(npx -y claude-second-brain path) # absolute path to the default brain's directory
|
|
18
|
+
npx -y claude-second-brain qmd -- query -c wiki "<question>"
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Pass `--brain <name>` before the `--` to target a non-default brain.
|
|
22
|
+
|
|
11
23
|
## Workflow
|
|
12
24
|
|
|
13
25
|
**Step 1 — Search the wiki**
|
|
14
|
-
- Run hybrid search: `
|
|
15
|
-
- Read
|
|
16
|
-
- Read the 2–5 most relevant pages in full before synthesizing
|
|
26
|
+
- Run hybrid search: `npx -y claude-second-brain qmd -- query -c wiki "<question>"`
|
|
27
|
+
- Read `$BRAIN_PATH/wiki/index.md` to confirm coverage and catch any pages qmd didn't surface
|
|
28
|
+
- Read the 2–5 most relevant pages in full before synthesizing (paths relative to `$BRAIN_PATH`)
|
|
17
29
|
|
|
18
30
|
**Step 2 — Synthesize an answer**
|
|
19
31
|
- Write the answer with inline `[[wiki/page]]` citations throughout
|
|
@@ -24,12 +36,12 @@ Runs the 4-step query workflow defined in CLAUDE.md. Answers come with inline `[
|
|
|
24
36
|
- If the answer draws together multiple pages in a novel way, ask:
|
|
25
37
|
> "This answer synthesizes several pages in a useful way — want me to file it as a Q&A page?"
|
|
26
38
|
- If yes:
|
|
27
|
-
- Create
|
|
39
|
+
- Create `$BRAIN_PATH/wiki/qa/[slug].md` with frontmatter `type: qa`
|
|
28
40
|
- Add to the Q&A section in `wiki/index.md`
|
|
29
41
|
- Add `[[wiki/qa/slug]]` cross-links from the relevant topic pages
|
|
30
42
|
|
|
31
43
|
**Step 4 — Log (if significant)**
|
|
32
|
-
- For queries that reveal new understanding or surface a gap worth tracking, append to
|
|
44
|
+
- For queries that reveal new understanding or surface a gap worth tracking, append to `$BRAIN_PATH/wiki/log.md`:
|
|
33
45
|
```
|
|
34
46
|
## [YYYY-MM-DD] query | Brief question summary
|
|
35
47
|
|
package/template/CLAUDE.md
CHANGED
|
@@ -111,7 +111,7 @@ Run this workflow whenever the user adds a new source. Do not skip steps.
|
|
|
111
111
|
- Add the new source to the Sources section with a one-line description and link.
|
|
112
112
|
|
|
113
113
|
**Step 5 — Identify affected wiki pages**
|
|
114
|
-
- Run `INDEX_PATH
|
|
114
|
+
- Run `INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd query -c wiki "<source topic and key claims>"` to surface related existing wiki pages.
|
|
115
115
|
- Also Glob `wiki/*.md` and `wiki/sources/*.md` to ensure completeness.
|
|
116
116
|
- List all pages to create or update.
|
|
117
117
|
|
|
@@ -142,7 +142,7 @@ Run this workflow whenever the user adds a new source. Do not skip steps.
|
|
|
142
142
|
Run this workflow when the user asks a question against the wiki.
|
|
143
143
|
|
|
144
144
|
**Step 1 — Search the wiki**
|
|
145
|
-
- Run `INDEX_PATH
|
|
145
|
+
- Run `INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd query -c wiki "<question>"` to surface semantically relevant pages.
|
|
146
146
|
- Read `wiki/index.md` to confirm coverage and catch any pages qmd didn't surface.
|
|
147
147
|
- Read the 2-5 most relevant pages fully.
|
|
148
148
|
|
|
@@ -212,15 +212,15 @@ N issues found, N fixed. [Brief summary of notable findings.]
|
|
|
212
212
|
## Search Tool
|
|
213
213
|
|
|
214
214
|
This vault uses **qmd** (`pnpm dlx @tobilu/qmd`) for local semantic and full-text search.
|
|
215
|
-
Collections and contexts are registered via `pnpm qmd:setup` and stored at `
|
|
215
|
+
Collections and contexts are registered via `pnpm qmd:setup` and stored at `.qmd/index.sqlite` (gitignored, relative to vault root).
|
|
216
216
|
|
|
217
|
-
Key commands:
|
|
217
|
+
Key commands (run from the vault root):
|
|
218
218
|
|
|
219
219
|
| Command | Use |
|
|
220
220
|
|---------|-----|
|
|
221
|
-
| `INDEX_PATH
|
|
222
|
-
| `INDEX_PATH
|
|
223
|
-
| `INDEX_PATH
|
|
221
|
+
| `INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd query -c wiki "<question>"` | Hybrid search — best for topic discovery |
|
|
222
|
+
| `INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd search -c wiki "<terms>"` | Fast keyword search (BM25, no LLM) |
|
|
223
|
+
| `INDEX_PATH=.qmd/index.sqlite pnpm dlx @tobilu/qmd vsearch -c wiki "<question>"` | Pure vector/semantic search |
|
|
224
224
|
|
|
225
225
|
Add `--json` for structured output. Omit `-c wiki` to search all collections (wiki, raw-sources, human, daily-notes).
|
|
226
226
|
|
package/template/README.md
CHANGED
|
@@ -13,23 +13,14 @@ Inspired by [Andrej Karpathy's approach to LLM-powered knowledge management](htt
|
|
|
13
13
|
## Quick start
|
|
14
14
|
|
|
15
15
|
```bash
|
|
16
|
-
# 1.
|
|
17
|
-
|
|
16
|
+
# 1. Generate vector embeddings (first run downloads ~2GB of GGUF models — once per machine)
|
|
17
|
+
pnpm qmd:reindex
|
|
18
18
|
|
|
19
|
-
# 2.
|
|
20
|
-
pnpm install
|
|
21
|
-
|
|
22
|
-
# 3. Open Claude Code
|
|
19
|
+
# 2. Open Claude Code
|
|
23
20
|
claude
|
|
24
21
|
```
|
|
25
22
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
```
|
|
29
|
-
/setup
|
|
30
|
-
```
|
|
31
|
-
|
|
32
|
-
Registers the qmd collections and generates local vector embeddings. First run downloads ~2GB of GGUF models — once.
|
|
23
|
+
qmd collections are already registered by the CLI during scaffolding — `pnpm qmd:reindex` just needs to build the index.
|
|
33
24
|
|
|
34
25
|
**Then open this folder in Obsidian.** The Git plugin is pre-configured — enable it in Obsidian settings and your wiki syncs automatically.
|
|
35
26
|
|
|
@@ -51,10 +42,6 @@ Registers the qmd collections and generates local vector embeddings. First run d
|
|
|
51
42
|
|
|
52
43
|
**`/brain-rebuild`** — **Destructive.** Redesigns the qmd schema: analyzes the wiki, proposes new collections and contexts, waits for your approval, then patches `scripts/qmd/setup.ts`, drops the old index, and rebuilds embeddings from scratch.
|
|
53
44
|
|
|
54
|
-
### Setup
|
|
55
|
-
|
|
56
|
-
**`/setup`** — First-time initialization. Registers the qmd collections and generates local vector embeddings. Run once after scaffolding.
|
|
57
|
-
|
|
58
45
|
---
|
|
59
46
|
|
|
60
47
|
## How it works
|
|
@@ -158,14 +145,14 @@ claude-second-brain/
|
|
|
158
145
|
|
|
159
146
|
## Installing and updating skills
|
|
160
147
|
|
|
161
|
-
Skills are slash commands Claude Code loads from `.claude/skills/[name]/SKILL.md` in this vault. The wiki ships with `/brain-ingest`, `/brain-search`, `/brain-refresh`, `/brain-rebuild`,
|
|
148
|
+
Skills are slash commands Claude Code loads from `.claude/skills/[name]/SKILL.md` in this vault. The wiki ships with `/brain-ingest`, `/brain-search`, `/brain-refresh`, `/brain-rebuild`, and `/lint` pre-installed.
|
|
162
149
|
|
|
163
150
|
### Update built-in wiki skills
|
|
164
151
|
|
|
165
152
|
Pull the latest skills from the upstream template:
|
|
166
153
|
|
|
167
154
|
```bash
|
|
168
|
-
# Install or update all
|
|
155
|
+
# Install or update all 5 wiki skills
|
|
169
156
|
npx skills add https://github.com/jessepinkman9900/claude-second-brain/tree/main/template/.claude/skills -a claude-code -y
|
|
170
157
|
|
|
171
158
|
# Or update a specific skill
|
|
@@ -192,6 +179,17 @@ This wraps `pnpm qmd:reindex` — you can also run that command directly if you'
|
|
|
192
179
|
|
|
193
180
|
---
|
|
194
181
|
|
|
182
|
+
## Managing brains
|
|
183
|
+
|
|
184
|
+
From anywhere:
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
npx claude-second-brain ls # list all brains
|
|
188
|
+
npx claude-second-brain rm <name> # remove a brain
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
---
|
|
192
|
+
|
|
195
193
|
## License
|
|
196
194
|
|
|
197
195
|
MIT
|
|
@@ -11,26 +11,38 @@
|
|
|
11
11
|
*/
|
|
12
12
|
|
|
13
13
|
import { createStore } from "@tobilu/qmd"
|
|
14
|
+
import { mkdir } from "node:fs/promises"
|
|
15
|
+
import { dirname, join } from "node:path"
|
|
16
|
+
import { fileURLToPath } from "node:url"
|
|
14
17
|
|
|
15
|
-
const
|
|
18
|
+
const VAULT = join(dirname(fileURLToPath(import.meta.url)), "../..")
|
|
19
|
+
const DB = join(VAULT, ".qmd", "index.sqlite")
|
|
16
20
|
|
|
17
|
-
|
|
21
|
+
try {
|
|
22
|
+
await mkdir(dirname(DB), { recursive: true })
|
|
23
|
+
const store = await createStore({ dbPath: DB })
|
|
18
24
|
|
|
19
|
-
console.log("Updating file index...")
|
|
20
|
-
const result = await store.update({
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
})
|
|
25
|
-
console.log(
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
)
|
|
25
|
+
console.log("Updating file index...")
|
|
26
|
+
const result = await store.update({
|
|
27
|
+
onProgress: ({ collection, file, current, total }: any) => {
|
|
28
|
+
process.stdout.write(`\r [${collection}] ${current}/${total} ${file}`.padEnd(80))
|
|
29
|
+
},
|
|
30
|
+
})
|
|
31
|
+
console.log(
|
|
32
|
+
`\n indexed: ${result.indexed} | updated: ${result.updated} | ` +
|
|
33
|
+
`removed: ${result.removed} | needs embedding: ${result.needsEmbedding}`
|
|
34
|
+
)
|
|
29
35
|
|
|
30
|
-
console.log("Generating embeddings...")
|
|
31
|
-
await store.embed({
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
})
|
|
36
|
-
console.log("\nDone.")
|
|
36
|
+
console.log("Generating embeddings...")
|
|
37
|
+
await store.embed({
|
|
38
|
+
onProgress: ({ current, total, collection }: any) => {
|
|
39
|
+
process.stdout.write(`\r [${collection}] ${current}/${total}`.padEnd(60))
|
|
40
|
+
},
|
|
41
|
+
})
|
|
42
|
+
console.log("\nDone.")
|
|
43
|
+
} catch (err: any) {
|
|
44
|
+
console.error(`\n[qmd:reindex] ${err?.message || err}`)
|
|
45
|
+
console.error(` vault: ${VAULT}`)
|
|
46
|
+
console.error(` db: ${DB}`)
|
|
47
|
+
process.exit(1)
|
|
48
|
+
}
|
|
@@ -10,48 +10,57 @@
|
|
|
10
10
|
*/
|
|
11
11
|
|
|
12
12
|
import { createStore } from "@tobilu/qmd"
|
|
13
|
+
import { mkdir } from "node:fs/promises"
|
|
13
14
|
import { dirname, join } from "node:path"
|
|
14
15
|
import { fileURLToPath } from "node:url"
|
|
15
16
|
|
|
16
17
|
const VAULT = join(dirname(fileURLToPath(import.meta.url)), "../..")
|
|
17
|
-
const DB = "
|
|
18
|
+
const DB = join(VAULT, ".qmd", "index.sqlite")
|
|
18
19
|
|
|
19
|
-
|
|
20
|
+
try {
|
|
21
|
+
await mkdir(dirname(DB), { recursive: true })
|
|
22
|
+
const store = await createStore({ dbPath: DB })
|
|
20
23
|
|
|
21
|
-
// --- Collections (idempotent) ---
|
|
22
|
-
const existingCols = await store.listCollections()
|
|
23
|
-
const existing = new Set(existingCols.map((c: any) => c.name))
|
|
24
|
+
// --- Collections (idempotent) ---
|
|
25
|
+
const existingCols = await store.listCollections()
|
|
26
|
+
const existing = new Set(existingCols.map((c: any) => c.name))
|
|
24
27
|
|
|
25
|
-
async function ensureCollection(name: string, relPath: string, pattern: string) {
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
28
|
+
async function ensureCollection(name: string, relPath: string, pattern: string) {
|
|
29
|
+
if (existing.has(name)) {
|
|
30
|
+
console.log(` skip (exists): ${name}`)
|
|
31
|
+
return
|
|
32
|
+
}
|
|
33
|
+
await store.addCollection(name, { path: join(VAULT, relPath), pattern })
|
|
34
|
+
console.log(` added: ${name}`)
|
|
29
35
|
}
|
|
30
|
-
await store.addCollection(name, { path: join(VAULT, relPath), pattern })
|
|
31
|
-
console.log(` added: ${name}`)
|
|
32
|
-
}
|
|
33
36
|
|
|
34
|
-
console.log("Collections:")
|
|
35
|
-
await ensureCollection("wiki", "wiki", "**/*.md")
|
|
36
|
-
await ensureCollection("raw-sources", "raw-sources", "**/*.md")
|
|
37
|
-
|
|
38
|
-
// --- Global context ---
|
|
39
|
-
console.log("\nContexts:")
|
|
40
|
-
await store.setGlobalContext(
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
)
|
|
45
|
-
|
|
46
|
-
// wiki/
|
|
47
|
-
await store.addContext("wiki", "", "LLM-maintained synthesized knowledge base — the authoritative wiki layer")
|
|
48
|
-
await store.addContext("wiki", "/sources", "Source summaries — one page per ingested source, with abstract, key claims, and synthesis notes")
|
|
49
|
-
await store.addContext("wiki", "/qa", "Filed Q&A answers that synthesize multiple wiki pages around a notable question")
|
|
50
|
-
|
|
51
|
-
// raw-sources/
|
|
52
|
-
await store.addContext("raw-sources", "", "Raw source material — immutable originals, never modified after ingestion")
|
|
53
|
-
await store.addContext("raw-sources", "/articles", "Web articles saved as markdown")
|
|
54
|
-
await store.addContext("raw-sources", "/pdfs", "PDF files or their extracted text")
|
|
55
|
-
await store.addContext("raw-sources", "/personal", "Personal notes flagged for wiki ingestion")
|
|
56
|
-
|
|
57
|
-
console.log("\nSetup complete. Run: pnpm qmd:reindex")
|
|
37
|
+
console.log("Collections:")
|
|
38
|
+
await ensureCollection("wiki", "wiki", "**/*.md")
|
|
39
|
+
await ensureCollection("raw-sources", "raw-sources", "**/*.md")
|
|
40
|
+
|
|
41
|
+
// --- Global context ---
|
|
42
|
+
console.log("\nContexts:")
|
|
43
|
+
await store.setGlobalContext(
|
|
44
|
+
"Personal knowledge vault. Uses Obsidian [[wiki/page]] wikilinks. " +
|
|
45
|
+
"Wiki pages have YAML frontmatter: type (overview|topic|entity|source-summary|qa), " +
|
|
46
|
+
"tags, sources, related, updated."
|
|
47
|
+
)
|
|
48
|
+
|
|
49
|
+
// wiki/
|
|
50
|
+
await store.addContext("wiki", "", "LLM-maintained synthesized knowledge base — the authoritative wiki layer")
|
|
51
|
+
await store.addContext("wiki", "/sources", "Source summaries — one page per ingested source, with abstract, key claims, and synthesis notes")
|
|
52
|
+
await store.addContext("wiki", "/qa", "Filed Q&A answers that synthesize multiple wiki pages around a notable question")
|
|
53
|
+
|
|
54
|
+
// raw-sources/
|
|
55
|
+
await store.addContext("raw-sources", "", "Raw source material — immutable originals, never modified after ingestion")
|
|
56
|
+
await store.addContext("raw-sources", "/articles", "Web articles saved as markdown")
|
|
57
|
+
await store.addContext("raw-sources", "/pdfs", "PDF files or their extracted text")
|
|
58
|
+
await store.addContext("raw-sources", "/personal", "Personal notes flagged for wiki ingestion")
|
|
59
|
+
|
|
60
|
+
console.log("\nSetup complete. Run: pnpm qmd:reindex")
|
|
61
|
+
} catch (err: any) {
|
|
62
|
+
console.error(`\n[qmd:setup] ${err?.message || err}`)
|
|
63
|
+
console.error(` vault: ${VAULT}`)
|
|
64
|
+
console.error(` db: ${DB}`)
|
|
65
|
+
process.exit(1)
|
|
66
|
+
}
|
|
@@ -1,168 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: qmd-cli
|
|
3
|
-
description: 'Use qmd CLI to manage collections, index documents, search, and query a local semantic knowledge base. Use when the user mentions qmd, searching a collection, indexing files, running queries, hybrid search, BM25, vector search, or managing a document index. Covers: collection add/list/remove/rename, context management, get/multi-get, update, embed, query, search, vsearch, and MCP server commands.'
|
|
4
|
-
argument-hint: 'qmd command to run or describe what you want to do (e.g., "query wiki for X", "add folder to collection")'
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
# qmd CLI
|
|
8
|
-
|
|
9
|
-
Local semantic document index with hybrid search (BM25 + vector), collection management, and MCP server.
|
|
10
|
-
|
|
11
|
-
## Vault DB
|
|
12
|
-
|
|
13
|
-
This vault keeps its index at `__QMD_PATH__`. All CLI commands **must** prefix with `INDEX_PATH=__QMD_PATH__`:
|
|
14
|
-
|
|
15
|
-
```bash
|
|
16
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd query -c wiki "..."
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
The scripts `pnpm qmd:setup` and `pnpm qmd:reindex` write to this same file. The CLI must match.
|
|
20
|
-
|
|
21
|
-
---
|
|
22
|
-
|
|
23
|
-
## Collection Management
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
# Create/index a collection
|
|
27
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd collection add <path> --name <name> --mask <glob-pattern>
|
|
28
|
-
# e.g.:
|
|
29
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd collection add wiki --name wiki --mask "**/*.md"
|
|
30
|
-
|
|
31
|
-
# List all collections with details
|
|
32
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd collection list
|
|
33
|
-
|
|
34
|
-
# Remove a collection
|
|
35
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd collection remove <name>
|
|
36
|
-
|
|
37
|
-
# Rename a collection
|
|
38
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd collection rename <old-name> <new-name>
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
## Listing Files
|
|
42
|
-
|
|
43
|
-
```bash
|
|
44
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd ls
|
|
45
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd ls [collection[/path]]
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
## Context
|
|
49
|
-
|
|
50
|
-
```bash
|
|
51
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd context add [path] "description text"
|
|
52
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd context list
|
|
53
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd context rm <path>
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
## Retrieving Documents
|
|
57
|
-
|
|
58
|
-
```bash
|
|
59
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd get <file>[:line] [-l N] [--from N]
|
|
60
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd multi-get <pattern> [-l N] [--max-bytes N]
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
**Multi-get format flags:** `--json`, `--csv`, `--md`, `--xml`, `--files`
|
|
64
|
-
|
|
65
|
-
## Indexing & Maintenance
|
|
66
|
-
|
|
67
|
-
```bash
|
|
68
|
-
# Show index status and collections
|
|
69
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd status
|
|
70
|
-
|
|
71
|
-
# Re-index all collections
|
|
72
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd update [--pull]
|
|
73
|
-
|
|
74
|
-
# Create vector embeddings (900 tokens/chunk, 15% overlap)
|
|
75
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd embed [-f]
|
|
76
|
-
|
|
77
|
-
# Remove cache and orphaned data, vacuum DB
|
|
78
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd cleanup
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
> After bulk ingest sessions, run `update` then `embed` to keep the index fresh.
|
|
82
|
-
|
|
83
|
-
## Search & Query
|
|
84
|
-
|
|
85
|
-
| Command | Description |
|
|
86
|
-
|---------|-------------|
|
|
87
|
-
| `qmd query <query>` | **Recommended** — hybrid search with query expansion + reranking |
|
|
88
|
-
| `qmd search <query>` | Full-text keyword search (BM25, no LLM) |
|
|
89
|
-
| `qmd vsearch <query>` | Pure vector/semantic similarity search |
|
|
90
|
-
|
|
91
|
-
### Search Options
|
|
92
|
-
|
|
93
|
-
```
|
|
94
|
-
-n <num> Number of results (default: 5, or 20 for --files)
|
|
95
|
-
--all Return all matches (use with --min-score to filter)
|
|
96
|
-
--min-score <num> Minimum similarity score
|
|
97
|
-
--full Output full document instead of snippet
|
|
98
|
-
--line-numbers Add line numbers to output
|
|
99
|
-
--files Output docid,score,filepath,context
|
|
100
|
-
--json JSON output with snippets
|
|
101
|
-
--csv CSV output
|
|
102
|
-
--md Markdown output
|
|
103
|
-
--xml XML output
|
|
104
|
-
-c, --collection <name> Filter results to a specific collection
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
### Examples
|
|
108
|
-
|
|
109
|
-
```bash
|
|
110
|
-
# Hybrid search across all collections
|
|
111
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd query "distributed systems consensus"
|
|
112
|
-
|
|
113
|
-
# Search only the wiki collection, JSON output
|
|
114
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd query -c wiki "transformer architecture" --json
|
|
115
|
-
|
|
116
|
-
# Keyword-only search, top 10, markdown output
|
|
117
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd search -c wiki "kafka" -n 10 --md
|
|
118
|
-
|
|
119
|
-
# Vector search with full documents
|
|
120
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd vsearch "attention mechanism" --full
|
|
121
|
-
|
|
122
|
-
# Filter by score threshold
|
|
123
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd query "machine learning" --all --min-score 0.7
|
|
124
|
-
```
|
|
125
|
-
|
|
126
|
-
## MCP Server
|
|
127
|
-
|
|
128
|
-
```bash
|
|
129
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd mcp
|
|
130
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd mcp --http [--port N]
|
|
131
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd mcp --http --daemon
|
|
132
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd mcp stop
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
## Global Options
|
|
136
|
-
|
|
137
|
-
```
|
|
138
|
-
--index <name> Use a custom index name (default: index)
|
|
139
|
-
```
|
|
140
|
-
|
|
141
|
-
Note: use `INDEX_PATH` env var rather than `--index` — `INDEX_PATH` accepts a file path while `--index` only accepts a short name mapped to `~/.cache/qmd/`.
|
|
142
|
-
|
|
143
|
-
## Typical Workflows
|
|
144
|
-
|
|
145
|
-
### First-time setup for this vault
|
|
146
|
-
```bash
|
|
147
|
-
# Handled by the /setup skill:
|
|
148
|
-
pnpm qmd:setup
|
|
149
|
-
pnpm qmd:reindex
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
### After a bulk ingest session
|
|
153
|
-
```bash
|
|
154
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd update
|
|
155
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd embed
|
|
156
|
-
```
|
|
157
|
-
|
|
158
|
-
### Research workflow
|
|
159
|
-
```bash
|
|
160
|
-
# Discover relevant pages
|
|
161
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd query -c wiki "<topic>"
|
|
162
|
-
|
|
163
|
-
# Get a specific file
|
|
164
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd get wiki/distributed-systems.md
|
|
165
|
-
|
|
166
|
-
# Get multiple related files
|
|
167
|
-
INDEX_PATH=__QMD_PATH__ pnpm dlx @tobilu/qmd multi-get "wiki/kafka.md,wiki/distributed-systems.md" --md
|
|
168
|
-
```
|