wikimem 0.2.3 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +2 -2
- package/README.md +65 -65
- package/dist/templates/config-yaml.js +1 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
All notable changes to
|
|
3
|
+
All notable changes to wikimem will be documented in this file.
|
|
4
4
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
@@ -34,4 +34,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
34
34
|
- **AGENTS.md schema:** co-evolving wiki structure document
|
|
35
35
|
- **YAML configuration:** `config.yaml` for provider, sources, schedules, processing options
|
|
36
36
|
|
|
37
|
-
[0.1.0]: https://github.com/naman10parikh/
|
|
37
|
+
[0.1.0]: https://github.com/naman10parikh/wikimem/releases/tag/v0.1.0
|
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
#
|
|
1
|
+
# wikimem
|
|
2
2
|
|
|
3
|
-
[](https://www.npmjs.com/package/wikimem)
|
|
4
4
|
[](https://opensource.org/licenses/MIT)
|
|
5
5
|
[](https://www.typescriptlang.org/)
|
|
6
6
|
[](#tests)
|
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
Drop files in. Get a structured, interlinked wiki out. It improves itself while you sleep.
|
|
11
11
|
|
|
12
12
|
```bash
|
|
13
|
-
npx
|
|
13
|
+
npx wikimem init my-wiki
|
|
14
14
|
```
|
|
15
15
|
|
|
16
16
|
```
|
|
@@ -22,7 +22,7 @@ raw/ wiki/
|
|
|
22
22
|
blog-url syntheses/ ......... cross-cutting analysis
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
|
|
25
|
+
wikimem processes any source (text, PDF, audio, video, images, URLs), compiles it into an interlinked markdown wiki with frontmatter and `[[wikilinks]]`, and opens directly in Obsidian.
|
|
26
26
|
|
|
27
27
|
Works with **Claude, OpenAI, or Ollama** (local). Your data stays on your machine.
|
|
28
28
|
|
|
@@ -31,7 +31,7 @@ Inspired by [Andrej Karpathy's LLM Wiki pattern](https://x.com/karpathy/status/1
|
|
|
31
31
|
## Install
|
|
32
32
|
|
|
33
33
|
```bash
|
|
34
|
-
npm install -g
|
|
34
|
+
npm install -g wikimem
|
|
35
35
|
```
|
|
36
36
|
|
|
37
37
|
**Requirements:** Node.js >= 18 · An LLM API key (or Ollama running locally)
|
|
@@ -40,28 +40,28 @@ npm install -g llmwiki
|
|
|
40
40
|
|
|
41
41
|
```bash
|
|
42
42
|
# 1. Create a vault
|
|
43
|
-
|
|
43
|
+
wikimem init my-wiki
|
|
44
44
|
cd my-wiki
|
|
45
45
|
|
|
46
46
|
# 2. Ingest something
|
|
47
|
-
|
|
48
|
-
|
|
47
|
+
wikimem ingest https://en.wikipedia.org/wiki/Large_language_model
|
|
48
|
+
wikimem ingest ~/Documents/research-paper.pdf
|
|
49
49
|
|
|
50
50
|
# 3. Ask questions
|
|
51
|
-
|
|
51
|
+
wikimem query "What are the key differences between RAG and compiled knowledge?"
|
|
52
52
|
```
|
|
53
53
|
|
|
54
54
|
That's it. Your wiki is now a folder of markdown files you can open in Obsidian, VS Code, or any text editor.
|
|
55
55
|
|
|
56
|
-
## Why
|
|
56
|
+
## Why wikimem?
|
|
57
57
|
|
|
58
58
|
**The problem:** You have dozens of sources — papers, podcasts, articles, screenshots, meeting recordings. They sit in folders. You forget what's in them. When you need something, you search and re-read.
|
|
59
59
|
|
|
60
60
|
**RAG approach:** Chunk documents, embed them, retrieve at query time. Lossy, opaque, and the "knowledge" lives in a vector database you can't read.
|
|
61
61
|
|
|
62
|
-
**
|
|
62
|
+
**wikimem approach:** Compile sources into structured markdown pages with summaries, cross-references, and citations. The knowledge is readable, editable, version-controlled, and improves itself over time.
|
|
63
63
|
|
|
64
|
-
| | RAG |
|
|
64
|
+
| | RAG | wikimem |
|
|
65
65
|
|---|---|---|
|
|
66
66
|
| **Storage** | Vector embeddings | Plain markdown files |
|
|
67
67
|
| **Readable?** | No (opaque vectors) | Yes (open in any editor) |
|
|
@@ -92,16 +92,16 @@ That's it. Your wiki is now a folder of markdown files you can open in Obsidian,
|
|
|
92
92
|
|
|
93
93
|
```
|
|
94
94
|
┌────────────────────────────────────────────────────┐
|
|
95
|
-
│
|
|
95
|
+
│ wikimem CLI │
|
|
96
96
|
│ │
|
|
97
|
-
│
|
|
98
|
-
│
|
|
99
|
-
│
|
|
100
|
-
│
|
|
101
|
-
│
|
|
102
|
-
│
|
|
103
|
-
│
|
|
104
|
-
│
|
|
97
|
+
│ wikimem init Create a new vault │
|
|
98
|
+
│ wikimem ingest Process source → wiki pages │
|
|
99
|
+
│ wikimem query Ask questions with citations │
|
|
100
|
+
│ wikimem lint Health-check the wiki │
|
|
101
|
+
│ wikimem watch Auto-ingest on file drop │
|
|
102
|
+
│ wikimem scrape Fetch from external sources │
|
|
103
|
+
│ wikimem improve Self-improvement cycle │
|
|
104
|
+
│ wikimem status Vault statistics │
|
|
105
105
|
├──────────────────────┬─────────────────────────────┤
|
|
106
106
|
│ Three Layers │ Three Automations │
|
|
107
107
|
│ │ │
|
|
@@ -137,57 +137,57 @@ That's it. Your wiki is now a folder of markdown files you can open in Obsidian,
|
|
|
137
137
|
|
|
138
138
|
## All Commands
|
|
139
139
|
|
|
140
|
-
### `
|
|
140
|
+
### `wikimem init [directory]`
|
|
141
141
|
|
|
142
142
|
Create a new vault with the standard directory structure.
|
|
143
143
|
|
|
144
144
|
```bash
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
145
|
+
wikimem init my-wiki # Create in my-wiki/
|
|
146
|
+
wikimem init . # Initialize current directory
|
|
147
|
+
wikimem init my-wiki --template research # Use research template
|
|
148
|
+
wikimem init my-wiki --force # Overwrite existing
|
|
149
149
|
```
|
|
150
150
|
|
|
151
151
|
Templates: `personal` (default), `research`, `business`, `codebase`
|
|
152
152
|
|
|
153
|
-
### `
|
|
153
|
+
### `wikimem ingest <source>`
|
|
154
154
|
|
|
155
155
|
Process a file or URL into wiki pages.
|
|
156
156
|
|
|
157
157
|
```bash
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
158
|
+
wikimem ingest paper.pdf # PDF → extract text → wiki pages
|
|
159
|
+
wikimem ingest podcast.mp3 # Audio → Whisper transcription → wiki
|
|
160
|
+
wikimem ingest screenshot.png # Image → Claude Vision description → wiki
|
|
161
|
+
wikimem ingest lecture.mp4 # Video → ffmpeg → Whisper → wiki
|
|
162
|
+
wikimem ingest article.md # Markdown → wiki pages
|
|
163
|
+
wikimem ingest data.json # JSON → code block in wiki
|
|
164
|
+
wikimem ingest page.html # HTML → strip tags → wiki
|
|
165
|
+
wikimem ingest report.docx # Office → basic extraction → wiki
|
|
166
|
+
wikimem ingest https://example.com/post # URL → Firecrawl/fetch → wiki
|
|
167
|
+
wikimem ingest raw/2026-04-07/file.md # Re-ingest from raw/
|
|
168
168
|
```
|
|
169
169
|
|
|
170
170
|
Each source is auto-detected by file type, copied to `raw/{date}/`, checked for duplicates, compiled into wiki pages by the LLM, and indexed. Use `-p` to pick a provider, `-m` for a specific model, `--verbose` for detailed output.
|
|
171
171
|
|
|
172
|
-
### `
|
|
172
|
+
### `wikimem query <question>`
|
|
173
173
|
|
|
174
174
|
Ask a question and get an answer synthesized from your wiki.
|
|
175
175
|
|
|
176
176
|
```bash
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
177
|
+
wikimem query "What are the main themes across my sources?"
|
|
178
|
+
wikimem query "Compare approaches to knowledge management" --file
|
|
179
|
+
wikimem query "Who is mentioned most frequently?" -p openai
|
|
180
180
|
```
|
|
181
181
|
|
|
182
182
|
Use `--file` to save the answer as a synthesis page in `wiki/syntheses/`. The query engine uses BM25 search to find relevant pages, reads the top 10, and synthesizes an answer with `[[wikilink]]` citations.
|
|
183
183
|
|
|
184
|
-
### `
|
|
184
|
+
### `wikimem lint`
|
|
185
185
|
|
|
186
186
|
Health-check the wiki for structural issues.
|
|
187
187
|
|
|
188
188
|
```bash
|
|
189
|
-
|
|
190
|
-
|
|
189
|
+
wikimem lint # Check for issues
|
|
190
|
+
wikimem lint --fix # Auto-fix where possible
|
|
191
191
|
```
|
|
192
192
|
|
|
193
193
|
Checks for:
|
|
@@ -198,24 +198,24 @@ Checks for:
|
|
|
198
198
|
|
|
199
199
|
Reports a quality score out of 100.
|
|
200
200
|
|
|
201
|
-
### `
|
|
201
|
+
### `wikimem watch`
|
|
202
202
|
|
|
203
203
|
Watch the `raw/` directory and auto-ingest new files.
|
|
204
204
|
|
|
205
205
|
```bash
|
|
206
|
-
|
|
207
|
-
|
|
206
|
+
wikimem watch # Watch current vault
|
|
207
|
+
wikimem watch -v ./my-wiki # Watch a specific vault
|
|
208
208
|
```
|
|
209
209
|
|
|
210
210
|
Uses `chokidar` for reliable cross-platform file watching. Waits for writes to stabilize before ingesting (2-second debounce). Press `Ctrl+C` to stop.
|
|
211
211
|
|
|
212
|
-
### `
|
|
212
|
+
### `wikimem scrape`
|
|
213
213
|
|
|
214
214
|
Fetch content from configured external sources and deposit in `raw/`.
|
|
215
215
|
|
|
216
216
|
```bash
|
|
217
|
-
|
|
218
|
-
|
|
217
|
+
wikimem scrape # Run all configured sources
|
|
218
|
+
wikimem scrape -s "HN Top" # Run a specific source
|
|
219
219
|
```
|
|
220
220
|
|
|
221
221
|
Sources are configured in `config.yaml`:
|
|
@@ -237,14 +237,14 @@ sources:
|
|
|
237
237
|
|
|
238
238
|
Supported source types: `rss`, `github`, `url`
|
|
239
239
|
|
|
240
|
-
### `
|
|
240
|
+
### `wikimem improve`
|
|
241
241
|
|
|
242
242
|
Run the self-improvement cycle (Automation 3).
|
|
243
243
|
|
|
244
244
|
```bash
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
245
|
+
wikimem improve # Evaluate and improve
|
|
246
|
+
wikimem improve --dry-run # Show what would change
|
|
247
|
+
wikimem improve --threshold 90 # Stricter quality bar
|
|
248
248
|
```
|
|
249
249
|
|
|
250
250
|
The improvement cycle:
|
|
@@ -254,16 +254,16 @@ The improvement cycle:
|
|
|
254
254
|
3. **Improve** — Proposes actions: add cross-links, create missing pages, expand stubs, flag contradictions
|
|
255
255
|
4. **Log** — Records what changed and why in `log.md`
|
|
256
256
|
|
|
257
|
-
### `
|
|
257
|
+
### `wikimem status`
|
|
258
258
|
|
|
259
259
|
Show vault statistics at a glance.
|
|
260
260
|
|
|
261
261
|
```bash
|
|
262
|
-
|
|
262
|
+
wikimem status
|
|
263
263
|
```
|
|
264
264
|
|
|
265
265
|
```
|
|
266
|
-
|
|
266
|
+
wikimem vault status
|
|
267
267
|
────────────────────────────────────
|
|
268
268
|
Pages: 42
|
|
269
269
|
Words: 18,340
|
|
@@ -275,7 +275,7 @@ llmwiki vault status
|
|
|
275
275
|
|
|
276
276
|
## Configuration
|
|
277
277
|
|
|
278
|
-
After `
|
|
278
|
+
After `wikimem init`, your vault contains a `config.yaml` where you set the LLM provider, external sources, self-improvement schedule, and processing options.
|
|
279
279
|
|
|
280
280
|
See [docs/configuration.md](docs/configuration.md) for the full reference.
|
|
281
281
|
|
|
@@ -291,7 +291,7 @@ See [docs/configuration.md](docs/configuration.md) for the full reference.
|
|
|
291
291
|
|
|
292
292
|
## Multi-Model Support
|
|
293
293
|
|
|
294
|
-
|
|
294
|
+
wikimem works with any major LLM provider. Choose at init time or per-command.
|
|
295
295
|
|
|
296
296
|
| Provider | Flag | Default Model | Env Variable |
|
|
297
297
|
|----------|------|---------------|-------------|
|
|
@@ -301,13 +301,13 @@ llmwiki works with any major LLM provider. Choose at init time or per-command.
|
|
|
301
301
|
|
|
302
302
|
```bash
|
|
303
303
|
# Use Claude (default)
|
|
304
|
-
|
|
304
|
+
wikimem ingest paper.pdf
|
|
305
305
|
|
|
306
306
|
# Use OpenAI
|
|
307
|
-
|
|
307
|
+
wikimem ingest paper.pdf -p openai -m gpt-4o-mini
|
|
308
308
|
|
|
309
309
|
# Use Ollama (fully local, no API keys)
|
|
310
|
-
|
|
310
|
+
wikimem ingest paper.pdf -p ollama -m llama3.2
|
|
311
311
|
```
|
|
312
312
|
|
|
313
313
|
## Multi-Format Support
|
|
@@ -324,11 +324,11 @@ llmwiki ingest paper.pdf -p ollama -m llama3.2
|
|
|
324
324
|
| **Office** | `.docx`, `.pptx`, `.xlsx` | Basic extraction | None (enhanced coming) |
|
|
325
325
|
| **URL** | `https://...` | Firecrawl / fetch | Optional `FIRECRAWL_API_KEY` |
|
|
326
326
|
|
|
327
|
-
When a processor's requirements are not met (e.g., Whisper not installed for audio),
|
|
327
|
+
When a processor's requirements are not met (e.g., Whisper not installed for audio), wikimem creates a reference page noting the source file and suggests installing the missing tool. The raw file is always preserved.
|
|
328
328
|
|
|
329
329
|
## Obsidian Integration
|
|
330
330
|
|
|
331
|
-
|
|
331
|
+
wikimem vaults are Obsidian vaults. Open any wikimem directory in Obsidian and you get:
|
|
332
332
|
|
|
333
333
|
- **Graph view** showing all pages and their `[[wikilinks]]`
|
|
334
334
|
- **YAML frontmatter** rendered as page metadata
|
|
@@ -377,7 +377,7 @@ my-wiki/
|
|
|
377
377
|
## Tests
|
|
378
378
|
|
|
379
379
|
```bash
|
|
380
|
-
cd /path/to/
|
|
380
|
+
cd /path/to/wikimem && pnpm test
|
|
381
381
|
```
|
|
382
382
|
|
|
383
383
|
## Contributing
|