wikimem 0.2.3 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (143) hide show
  1. package/CHANGELOG.md +138 -29
  2. package/README.md +173 -311
  3. package/dist/cli/commands/ask.d.ts +3 -0
  4. package/dist/cli/commands/ask.d.ts.map +1 -0
  5. package/dist/cli/commands/ask.js +63 -0
  6. package/dist/cli/commands/ask.js.map +1 -0
  7. package/dist/cli/commands/export.d.ts +3 -0
  8. package/dist/cli/commands/export.d.ts.map +1 -0
  9. package/dist/cli/commands/export.js +108 -0
  10. package/dist/cli/commands/export.js.map +1 -0
  11. package/dist/cli/commands/history.d.ts +3 -0
  12. package/dist/cli/commands/history.d.ts.map +1 -0
  13. package/dist/cli/commands/history.js +61 -0
  14. package/dist/cli/commands/history.js.map +1 -0
  15. package/dist/cli/commands/improve.d.ts.map +1 -1
  16. package/dist/cli/commands/improve.js +4 -3
  17. package/dist/cli/commands/improve.js.map +1 -1
  18. package/dist/cli/commands/ingest.d.ts.map +1 -1
  19. package/dist/cli/commands/ingest.js +5 -4
  20. package/dist/cli/commands/ingest.js.map +1 -1
  21. package/dist/cli/commands/init.d.ts.map +1 -1
  22. package/dist/cli/commands/init.js +246 -79
  23. package/dist/cli/commands/init.js.map +1 -1
  24. package/dist/cli/commands/lint.d.ts.map +1 -1
  25. package/dist/cli/commands/lint.js +4 -3
  26. package/dist/cli/commands/lint.js.map +1 -1
  27. package/dist/cli/commands/mcp.d.ts +3 -0
  28. package/dist/cli/commands/mcp.d.ts.map +1 -0
  29. package/dist/cli/commands/mcp.js +11 -0
  30. package/dist/cli/commands/mcp.js.map +1 -0
  31. package/dist/cli/commands/open.d.ts +3 -0
  32. package/dist/cli/commands/open.d.ts.map +1 -0
  33. package/dist/cli/commands/open.js +36 -0
  34. package/dist/cli/commands/open.js.map +1 -0
  35. package/dist/cli/commands/query.d.ts.map +1 -1
  36. package/dist/cli/commands/query.js +5 -4
  37. package/dist/cli/commands/query.js.map +1 -1
  38. package/dist/cli/commands/search.d.ts +3 -0
  39. package/dist/cli/commands/search.d.ts.map +1 -0
  40. package/dist/cli/commands/search.js +61 -0
  41. package/dist/cli/commands/search.js.map +1 -0
  42. package/dist/cli/commands/serve.d.ts.map +1 -1
  43. package/dist/cli/commands/serve.js +41 -2
  44. package/dist/cli/commands/serve.js.map +1 -1
  45. package/dist/cli/commands/watch.d.ts.map +1 -1
  46. package/dist/cli/commands/watch.js +4 -3
  47. package/dist/cli/commands/watch.js.map +1 -1
  48. package/dist/cli/index.d.ts.map +1 -1
  49. package/dist/cli/index.js +27 -1
  50. package/dist/cli/index.js.map +1 -1
  51. package/dist/core/audit-trail.d.ts +15 -0
  52. package/dist/core/audit-trail.d.ts.map +1 -0
  53. package/dist/core/audit-trail.js +43 -0
  54. package/dist/core/audit-trail.js.map +1 -0
  55. package/dist/core/claude-code.d.ts +10 -0
  56. package/dist/core/claude-code.d.ts.map +1 -0
  57. package/dist/core/claude-code.js +81 -0
  58. package/dist/core/claude-code.js.map +1 -0
  59. package/dist/core/config.d.ts +23 -0
  60. package/dist/core/config.d.ts.map +1 -1
  61. package/dist/core/config.js.map +1 -1
  62. package/dist/core/connectors.d.ts +58 -0
  63. package/dist/core/connectors.d.ts.map +1 -0
  64. package/dist/core/connectors.js +189 -0
  65. package/dist/core/connectors.js.map +1 -0
  66. package/dist/core/folder-scanner.d.ts +10 -0
  67. package/dist/core/folder-scanner.d.ts.map +1 -0
  68. package/dist/core/folder-scanner.js +84 -0
  69. package/dist/core/folder-scanner.js.map +1 -0
  70. package/dist/core/git.d.ts +137 -0
  71. package/dist/core/git.d.ts.map +1 -0
  72. package/dist/core/git.js +520 -0
  73. package/dist/core/git.js.map +1 -0
  74. package/dist/core/history.d.ts +21 -0
  75. package/dist/core/history.d.ts.map +1 -0
  76. package/dist/core/history.js +107 -0
  77. package/dist/core/history.js.map +1 -0
  78. package/dist/core/improve.d.ts.map +1 -1
  79. package/dist/core/improve.js +9 -0
  80. package/dist/core/improve.js.map +1 -1
  81. package/dist/core/ingest.d.ts +1 -0
  82. package/dist/core/ingest.d.ts.map +1 -1
  83. package/dist/core/ingest.js +78 -5
  84. package/dist/core/ingest.js.map +1 -1
  85. package/dist/core/observer.d.ts +71 -0
  86. package/dist/core/observer.d.ts.map +1 -0
  87. package/dist/core/observer.js +350 -0
  88. package/dist/core/observer.js.map +1 -0
  89. package/dist/core/pipeline-events.d.ts +63 -0
  90. package/dist/core/pipeline-events.d.ts.map +1 -0
  91. package/dist/core/pipeline-events.js +109 -0
  92. package/dist/core/pipeline-events.js.map +1 -0
  93. package/dist/core/query.d.ts.map +1 -1
  94. package/dist/core/query.js +16 -8
  95. package/dist/core/query.js.map +1 -1
  96. package/dist/core/scraper.d.ts +41 -0
  97. package/dist/core/scraper.d.ts.map +1 -0
  98. package/dist/core/scraper.js +277 -0
  99. package/dist/core/scraper.js.map +1 -0
  100. package/dist/index.js +3 -2
  101. package/dist/index.js.map +1 -1
  102. package/dist/mcp-entry.d.ts +10 -0
  103. package/dist/mcp-entry.d.ts.map +1 -0
  104. package/dist/mcp-entry.js +21 -0
  105. package/dist/mcp-entry.js.map +1 -0
  106. package/dist/mcp-server.d.ts +15 -0
  107. package/dist/mcp-server.d.ts.map +1 -0
  108. package/dist/mcp-server.js +390 -0
  109. package/dist/mcp-server.js.map +1 -0
  110. package/dist/processors/audio.d.ts.map +1 -1
  111. package/dist/processors/audio.js +42 -4
  112. package/dist/processors/audio.js.map +1 -1
  113. package/dist/processors/pdf.d.ts.map +1 -1
  114. package/dist/processors/pdf.js +2 -3
  115. package/dist/processors/pdf.js.map +1 -1
  116. package/dist/processors/url.js +4 -1
  117. package/dist/processors/url.js.map +1 -1
  118. package/dist/providers/claude.d.ts +1 -0
  119. package/dist/providers/claude.d.ts.map +1 -1
  120. package/dist/providers/claude.js +5 -3
  121. package/dist/providers/claude.js.map +1 -1
  122. package/dist/providers/index.d.ts +17 -1
  123. package/dist/providers/index.d.ts.map +1 -1
  124. package/dist/providers/index.js +144 -0
  125. package/dist/providers/index.js.map +1 -1
  126. package/dist/providers/openai.d.ts +1 -0
  127. package/dist/providers/openai.d.ts.map +1 -1
  128. package/dist/providers/openai.js +5 -3
  129. package/dist/providers/openai.js.map +1 -1
  130. package/dist/providers/types.d.ts +18 -0
  131. package/dist/providers/types.d.ts.map +1 -1
  132. package/dist/templates/config-yaml.d.ts.map +1 -1
  133. package/dist/templates/config-yaml.js +13 -2
  134. package/dist/templates/config-yaml.js.map +1 -1
  135. package/dist/web/public/index.html +8157 -745
  136. package/dist/web/server.d.ts.map +1 -1
  137. package/dist/web/server.js +2101 -29
  138. package/dist/web/server.js.map +1 -1
  139. package/package.json +7 -4
  140. package/scripts/install.sh +54 -0
  141. package/src/web/public/index.html +8157 -745
  142. package/templates/mcp-config.json +9 -0
  143. package/dist/web/public/public/index.html +0 -946
package/CHANGELOG.md CHANGED
@@ -1,37 +1,146 @@
1
1
  # Changelog
2
2
 
3
- All notable changes to llmwiki will be documented in this file.
3
+ All notable changes to wikimem are documented here.
4
4
 
5
- The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
- and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
5
+ ## [0.5.1] - 2026-04-08
6
+
7
+ ### Fixed (E2E Audit — Ralph Loop)
8
+
9
+ - **Duplicate settings icon**: Removed settings gear from icon rail (was redundant with sidebar bottom gear). Settings now accessed only from sidebar bottom, matching Obsidian pattern.
10
+ - **Dead "New Note" button**: Wired up with prompt for title + new `POST /api/pages` endpoint. Creates page with frontmatter, refreshes tree, and opens the page.
11
+ - **Search icon active state**: Rail search icon now highlights blue when search overlay is open, de-highlights when closed.
12
+ - **Settings gear active state**: Sidebar bottom gear icon now turns blue when settings view is active.
13
+ - **Stale tree highlight**: Navigating away from page view (to home/graph/settings) now clears the active tree item highlight.
14
+ - **Stale status bar path**: Status bar right side now clears when not viewing a page.
15
+ - **Misleading empty folders**: Empty folders (0 children) now start collapsed instead of expanded, preventing visual confusion with root-level files.
16
+ - **Drop zone text**: Changed from "Drop files to add to vault" to "Drop files or click to upload".
17
+
18
+ ### Verified (E2E Testing)
19
+
20
+ - File upload pipeline: `.md`, `.json`, `.csv`, `.txt`, `.yaml` all ingest correctly (tested via API and file watcher)
21
+ - File watcher (`wikimem watch`): Detects new files in raw/, auto-ingests with chokidar
22
+ - Search (Cmd+K): Fuzzy search → select → navigate to page works end-to-end
23
+ - Command palette (Cmd+P): All 8 commands visible and functional
24
+ - Settings page: General and Provider sections load config data correctly
25
+ - Graph view: 37-node force-directed graph renders with edges
26
+ - Tab management: Open/close/switch tabs works correctly
27
+ - Ask Your Knowledge: LLM query returns rendered markdown with wikilinks
28
+ - Keyboard shortcuts: Cmd+K, Cmd+P, Cmd+G, Cmd+comma, Escape all verified
29
+ - Wiki growth: 5 source files → 37 pages, 6,100 words, 251 cross-links, 0 orphans
30
+
31
+ ## [0.5.0] - 2026-04-08
32
+
33
+ ### Added
34
+
35
+ - **Obsidian-style icon rail**: Thin vertical strip on the extreme left with SVG icons for Files, Search, and Graph. Active icon gets blue highlight bar. Tooltips on hover.
36
+ - **Settings page**: Full two-column settings view (General, Provider, Appearance, Automations, Hotkeys, About). Accessible from sidebar gear.
37
+ - **API key configuration**: Configure Anthropic and Gemini keys from the web UI. Model selector (Claude Sonnet 4, 3.5 Sonnet, 3 Haiku). Connection test button with live feedback.
38
+ - **Settings API**: GET/PUT `/api/config` and POST `/api/config/test-provider` endpoints with API key masking.
39
+ - **Create page API**: POST `/api/pages` endpoint for creating new wiki pages from the web UI.
40
+ - **Command palette**: `Cmd+P` opens a fuzzy command palette with 8 commands (Home, Graph, Search, Settings, Sidebar, Upload, Refresh, Collapse).
41
+ - **Graph node highlighting**: Click a node to highlight it + connected neighbors. All other nodes dim to 12% opacity. Click empty space to reset. Double-click opens page.
42
+ - **Table of contents**: Auto-generated from headings (h1-h4) on pages with 3+ headings. Click to smooth-scroll.
43
+ - **URL ingestion**: "Add a URL to your vault" section on home page with validation, spinner, and success feedback.
44
+ - **Sidebar action bar**: EXPLORER header with New Note, Collapse All, and Refresh buttons.
45
+ - **Sidebar bottom bar**: Vault name/icon and settings gear button.
46
+ - **Keyboard shortcuts**: `Cmd+P` (palette), `Cmd+G` (graph), `Cmd+,` (settings), `Cmd+W` (close tab).
47
+
48
+ ### Changed
49
+
50
+ - **Complete color overhaul**: Removed all purple accents. New palette: `#1e1e1e` background, `#4f9eff` blue accent (only on active states), warm neutral text hierarchy. Professional and minimal.
51
+ - **Font overhaul**: Replaced Poppins with Inter + system font stack. Instrument Serif restricted to logo/display only. JetBrains Mono for code.
52
+ - **Tab bar**: Moved to top strip with Chrome-style rounded tabs. Active tab visually connects to content. `+` button for new tab/home.
53
+ - **File tree**: Replaced emoji chevrons with SVG arrows with smooth rotation animation. Empty folders start collapsed.
54
+ - **Topbar**: Removed centered action buttons (☰⌂⊛⌕↑). Tab bar and status indicator only.
55
+ - **Inline code**: Now uses warm `#ce9178` color instead of purple.
56
+ - **Links**: Blue (`#4f9eff`) instead of purple throughout.
57
+
58
+ ## [0.4.0] - 2026-04-08
59
+
60
+ ### Added
61
+
62
+ - `wikimem init --from-folder <path>` — Create a wiki from any existing folder with recursive file scanning and batch ingest
63
+ - `wikimem init --from-repo <url-or-path>` — Create a wiki from a GitHub repository (clone + scan)
64
+ - `wikimem history list` — View chronological audit trail of all wiki changes
65
+ - `wikimem history restore <id>` — Restore wiki to a previous snapshot
66
+ - History tab in web UI with timeline visualization, automation type icons, and color-coded entries
67
+ - Raw file viewer — click any raw source in the dashboard to view its contents in a modal
68
+ - Auto-ingest on upload — web UI file uploads now automatically trigger the ingest pipeline
69
+ - Install script (`scripts/install.sh`) for curl one-liner installation
70
+ - Git-style audit trail system (`.wikimem/history/`) tracking every wiki change with snapshots
71
+
72
+ ### Fixed
73
+
74
+ - Query tab now correctly sends POST requests (was broken with GET)
75
+ - CLI version now reads dynamically from package.json (was hardcoded to 0.1.0)
76
+ - Raw files in dashboard now clickable and browseable
77
+ - Upload results now show ingest outcome (pages created count)
78
+
79
+ ### Changed
80
+
81
+ - Raw files listing now walks subdirectories recursively
82
+ - Upload endpoint stores files in date-stamped subdirectories
83
+ - Init command refactored into modular scaffold + mode-specific handlers
84
+
85
+ ## [0.3.0] - 2026-04-07
86
+
87
+ ### Added
88
+
89
+ - Unified naming to wikimem everywhere
90
+ - PDF extraction with pdf-parse v1
91
+ - Fixed query and ingest API endpoints
92
+
93
+ ### Fixed
94
+
95
+ - PDF extraction works with real PDF files
96
+ - API endpoint consistency (query/ingest)
97
+
98
+ ## [0.2.3] - 2026-04-07
99
+
100
+ ### Fixed
101
+
102
+ - PDF text extraction using pdf-parse
103
+ - Query and ingest API endpoint routing
104
+
105
+ ## [0.2.2] - 2026-04-07
106
+
107
+ ### Fixed
108
+
109
+ - Dynamic version display from package.json
110
+ - Init command no longer adds 'cd .' for in-place init
111
+ - Removed npx prefix from help text
112
+
113
+ ## [0.2.1] - 2026-04-07
114
+
115
+ ### Fixed
116
+
117
+ - Markdown rendering works — built-in renderer with no CDN dependency
118
+
119
+ ## [0.2.0] - 2026-04-07
120
+
121
+ ### Added
122
+
123
+ - Web UI markdown rendering with clickable wikilinks
124
+ - Page detail modal with frontmatter badges
7
125
 
8
126
  ## [0.1.0] - 2026-04-07
9
127
 
10
128
  ### Added
11
129
 
12
- - **CLI commands:** `init`, `ingest`, `query`, `lint`, `watch`, `scrape`, `improve`, `status`
13
- - **Three-layer architecture:** `raw/` (immutable sources), `wiki/` (LLM-generated), `AGENTS.md` (schema)
14
- - **Three automations:** ingest pipeline (A1), external scraping (A2), self-improvement (A3)
15
- - **Multi-model support:** Claude (Anthropic), OpenAI (GPT), Ollama (local)
16
- - **Multi-format processing:**
17
- - Text: `.md`, `.txt`, `.csv`, `.json`, `.html`
18
- - PDF: built-in text extraction from PDF binary format
19
- - Audio: `.mp3`, `.wav`, `.m4a`, `.ogg`, `.flac`, `.aac` via Whisper or Deepgram
20
- - Video: `.mp4`, `.mov`, `.avi`, `.mkv`, `.webm` via ffmpeg + Whisper
21
- - Image: `.jpg`, `.png`, `.gif`, `.webp` via Claude Vision
22
- - URL: web pages via Firecrawl API or native fetch fallback
23
- - Office: `.docx`, `.pptx`, `.xlsx` basic reference extraction
24
- - **BM25 search engine:** zero-dependency full-text search with title boosting
25
- - **Semantic dedup:** Jaccard similarity check prevents near-duplicate ingestion
26
- - **Watch mode:** `chokidar`-based file watcher auto-ingests new files in `raw/`
27
- - **External scraping:** RSS feeds, GitHub trending repos, plain URL scraping
28
- - **Self-improvement cycle:** 5-dimension scoring (coverage, consistency, cross-linking, freshness, organization) with configurable threshold and dry-run mode
29
- - **Wiki health checks:** orphan detection, broken wikilink detection, missing summary detection, empty page detection
30
- - **Auto-maintained files:** `wiki/index.md` (content catalog) and `wiki/log.md` (operation log)
31
- - **Obsidian compatibility:** YAML frontmatter, `[[wikilinks]]`, section-based organization
32
- - **Domain templates:** personal, research, business, codebase
33
- - **File-back queries:** optionally save query answers as synthesis pages
34
- - **AGENTS.md schema:** co-evolving wiki structure document
35
- - **YAML configuration:** `config.yaml` for provider, sources, schedules, processing options
36
-
37
- [0.1.0]: https://github.com/naman10parikh/llmwiki/releases/tag/v0.1.0
130
+ - CLI with 10 commands: init, ingest, query, lint, status, watch, scrape, improve, duplicates, serve
131
+ - Three layers: raw/ (immutable sources), wiki/ (LLM-generated), AGENTS.md (schema)
132
+ - Three automations: ingest & process, external scrape, self-improvement via LLM Council
133
+ - 3 LLM providers: Claude (Anthropic), OpenAI, Ollama
134
+ - 9 file processors: text, URL, image (Claude Vision), audio (Whisper), video (ffmpeg), PDF, DOCX, XLSX, PPTX
135
+ - Web UI with d3 knowledge graph, dashboard, pages list, query tab, upload tab
136
+ - Obsidian-native output (.obsidian/ config, wikilinks, frontmatter, graph colors)
137
+ - BM25 full-text search with title boosting
138
+ - Semantic search with RRF (Reciprocal Rank Fusion) hybrid
139
+ - Semantic dedup with Jaccard similarity (0.7 threshold)
140
+ - Batch ingest with progress display
141
+ - Interactive tagging (--tags, --category, --interactive)
142
+ - File watcher mode (wikimem watch)
143
+ - 4 domain templates: personal, research, business, codebase
144
+ - Auto-update checker (npm registry, 24h cache)
145
+ - 58 unit tests (4 test suites)
146
+ - Published to npm as wikimem@0.1.0