@ulrichc1/sparn 1.2.1 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/PRIVACY.md +1 -1
- package/README.md +136 -642
- package/SECURITY.md +1 -1
- package/dist/cli/dashboard.cjs +3977 -0
- package/dist/cli/dashboard.cjs.map +1 -0
- package/dist/cli/dashboard.d.cts +17 -0
- package/dist/cli/dashboard.d.ts +17 -0
- package/dist/cli/dashboard.js +3932 -0
- package/dist/cli/dashboard.js.map +1 -0
- package/dist/cli/index.cjs +3855 -486
- package/dist/cli/index.cjs.map +1 -1
- package/dist/cli/index.js +3812 -459
- package/dist/cli/index.js.map +1 -1
- package/dist/daemon/index.cjs +411 -99
- package/dist/daemon/index.cjs.map +1 -1
- package/dist/daemon/index.js +423 -103
- package/dist/daemon/index.js.map +1 -1
- package/dist/hooks/post-tool-result.cjs +129 -225
- package/dist/hooks/post-tool-result.cjs.map +1 -1
- package/dist/hooks/post-tool-result.js +129 -225
- package/dist/hooks/post-tool-result.js.map +1 -1
- package/dist/hooks/pre-prompt.cjs +206 -242
- package/dist/hooks/pre-prompt.cjs.map +1 -1
- package/dist/hooks/pre-prompt.js +192 -243
- package/dist/hooks/pre-prompt.js.map +1 -1
- package/dist/hooks/stop-docs-refresh.cjs +123 -0
- package/dist/hooks/stop-docs-refresh.cjs.map +1 -0
- package/dist/hooks/stop-docs-refresh.d.cts +1 -0
- package/dist/hooks/stop-docs-refresh.d.ts +1 -0
- package/dist/hooks/stop-docs-refresh.js +126 -0
- package/dist/hooks/stop-docs-refresh.js.map +1 -0
- package/dist/index.cjs +1756 -339
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +540 -41
- package/dist/index.d.ts +540 -41
- package/dist/index.js +1739 -331
- package/dist/index.js.map +1 -1
- package/dist/mcp/index.cjs +306 -73
- package/dist/mcp/index.cjs.map +1 -1
- package/dist/mcp/index.js +310 -73
- package/dist/mcp/index.js.map +1 -1
- package/package.json +10 -3
package/README.md
CHANGED
|
@@ -1,764 +1,258 @@
|
|
|
1
1
|
# Sparn
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
[](https://www.npmjs.com/package/@ulrichc1/sparn)
|
|
5
|
-
[](https://opensource.org/licenses/MIT)
|
|
6
|
-
|
|
7
|
-
> 🧠 Neuroscience-inspired context optimization for AI coding agents
|
|
8
|
-
|
|
9
|
-
**Status**: ✅ **Production Ready** - Full Feature Set with MCP Server + Interactive Mode
|
|
10
|
-
|
|
11
|
-
**Version**: 1.2.0
|
|
12
|
-
|
|
13
|
-
---
|
|
14
|
-
|
|
15
|
-
## What is Sparn?
|
|
16
|
-
|
|
17
|
-
Sparn is an npm CLI package that applies 6 neuroscience principles to intelligently prune, score, and compress AI agent context memory. It reduces token usage by 60-90% while maintaining task completion quality.
|
|
18
|
-
|
|
19
|
-
### Neuroscience Principles
|
|
20
|
-
|
|
21
|
-
1. **Sparse Coding** - Keep only 2-5% most relevant context
|
|
22
|
-
2. **Engram Theory** - Memories decay over time
|
|
23
|
-
3. **Hippocampal KV** - Separate what to store from how to retrieve
|
|
24
|
-
4. **Multi-State Synapses** - Silent, ready, or active states
|
|
25
|
-
5. **Sleep Replay** - Periodic memory consolidation
|
|
26
|
-
6. **BTSP** - One-shot learning for critical events
|
|
27
|
-
|
|
28
|
-
---
|
|
29
|
-
|
|
30
|
-
## Features
|
|
31
|
-
|
|
32
|
-
### ✅ Core Functionality
|
|
33
|
-
- ✅ **All 6 Neuroscience Modules** - Complete brain-inspired architecture
|
|
34
|
-
- SparsePruner (Sparse Coding)
|
|
35
|
-
- EngramScorer (Engram Theory)
|
|
36
|
-
- ConfidenceStates (Multi-State Synapses)
|
|
37
|
-
- BTSPEmbedder (One-Shot Learning)
|
|
38
|
-
- GenericAdapter + ClaudeCodeAdapter (Agent-Agnostic)
|
|
39
|
-
- SleepCompressor (Sleep Replay)
|
|
40
|
-
- ✅ **Context Optimization** - 60-90% token reduction pipeline
|
|
41
|
-
- ✅ **Real-Time Optimization** - Always-on background daemon and hooks
|
|
42
|
-
- Background daemon with auto-optimization at 80K token threshold
|
|
43
|
-
- Automated consolidation scheduler (configurable intervals)
|
|
44
|
-
- Claude Code hooks (pre-prompt & post-tool-result)
|
|
45
|
-
- Incremental optimization with <50ms delta processing
|
|
46
|
-
- Budget-aware pruning targeting specific token counts
|
|
47
|
-
- Tool output compression (npm, docker, tests, git diffs)
|
|
48
|
-
- ✅ **MCP Server** (NEW!) - Model Context Protocol integration
|
|
49
|
-
- Expose Sparn as MCP tools for Claude Desktop and other MCP clients
|
|
50
|
-
- Three tools: sparn_optimize, sparn_stats, sparn_consolidate
|
|
51
|
-
- Full SDK integration with stdio transport
|
|
52
|
-
- Comprehensive configuration guide
|
|
53
|
-
- ✅ **Interactive Mode** (NEW!) - Conversational CLI interface
|
|
54
|
-
- Configuration wizard with guided prompts
|
|
55
|
-
- Optimization preview with file browsing
|
|
56
|
-
- Stats dashboard with multiple views
|
|
57
|
-
- Memory consolidation with confirmation
|
|
58
|
-
- Quick actions and shortcuts
|
|
59
|
-
- ✅ **CLI Commands** - init, optimize, stats, relay, consolidate, config, daemon, hooks, interactive
|
|
60
|
-
- ✅ **Programmatic API** - Full TypeScript support, JSDoc, standalone modules, MCP server factory
|
|
61
|
-
- ✅ **Database** - SQLite with dual index/value tables, corruption detection
|
|
62
|
-
- ✅ **Configuration** - YAML config with runtime modification
|
|
63
|
-
- ✅ **Metrics & Telemetry** - P50/P95/P99 latency tracking, cache hit rates, token savings
|
|
64
|
-
|
|
65
|
-
### ✨ Polish & UX
|
|
66
|
-
- ✨ **Progress Indicators** - Real-time ora spinners for all long operations
|
|
67
|
-
- ✨ **Visual Impact** - Before/after token savings with progress bars and celebration messages
|
|
68
|
-
- ✨ **Detailed Help** - Comprehensive --help text with examples for every command
|
|
69
|
-
- ✨ **Branded UI** - Pink brain logo colors, neural cyan, synapse violet
|
|
70
|
-
- ✨ **Error Handling** - Context-aware recovery suggestions, database backup on corruption
|
|
71
|
-
- ✨ **Lazy Loading** - Fast startup (<200ms for --help/--version)
|
|
72
|
-
|
|
73
|
-
### 📊 Quality & CI/CD
|
|
74
|
-
- 📊 **230 Tests** - Comprehensive unit + integration test coverage, 230 passing
|
|
75
|
-
- 📊 **Performance Benchmarks** - Validates <50ms incremental optimization target
|
|
76
|
-
- 📊 **CI Pipeline** - GitHub Actions with cross-platform tests (Ubuntu, macOS, Windows)
|
|
77
|
-
- 📊 **Documentation** - NEUROSCIENCE.md, CONTRIBUTING.md, CHANGELOG.md, MCP.md, comprehensive README
|
|
78
|
-
- 📊 **NPM Ready** - Package validated with publish --dry-run
|
|
79
|
-
|
|
80
|
-
---
|
|
81
|
-
|
|
82
|
-
## Screenshots
|
|
83
|
-
|
|
84
|
-
### Branded Banner & Initialization
|
|
3
|
+
Context optimization for AI coding agents. Reduces token usage by 60-90% so your Claude Code sessions last longer and cost less.
|
|
85
4
|
|
|
86
|
-
|
|
87
|
-
____ ____ ___ ____ _ __
|
|
88
|
-
/ __ \/ __ \/ | / __ \/ | / /
|
|
89
|
-
/ /_/ / /_/ / /| | / /_/ / |/ /
|
|
90
|
-
\__, / ____/ ___ |/ _, _/ /| /
|
|
91
|
-
/____/_/ /_/ |_/_/ |_/_/ |_/
|
|
92
|
-
|
|
93
|
-
🧠 Neuroscience-inspired context optimization
|
|
94
|
-
v0.1.0
|
|
95
|
-
```
|
|
5
|
+
## Why
|
|
96
6
|
|
|
97
|
-
|
|
7
|
+
Long Claude Code sessions burn through context windows fast. Tool outputs, file reads, and conversation history pile up until you hit the limit and lose your thread. Sparn watches for this and keeps your context lean:
|
|
98
8
|
|
|
99
|
-
|
|
100
|
-
|
|
9
|
+
- Compresses verbose tool outputs (test results, build logs, file reads)
|
|
10
|
+
- Tracks which context is still relevant and which has gone stale
|
|
11
|
+
- Runs as a background daemon or hooks directly into Claude Code
|
|
12
|
+
- Analyzes your codebase structure so Claude gets the right context first
|
|
101
13
|
|
|
102
|
-
|
|
103
|
-
Entries: Active 12 | Ready 34 | Pruned 189
|
|
104
|
-
Duration: 287ms
|
|
14
|
+
## Install
|
|
105
15
|
|
|
106
|
-
|
|
16
|
+
```bash
|
|
17
|
+
npm install -g @ulrichc1/sparn
|
|
107
18
|
```
|
|
108
19
|
|
|
109
|
-
|
|
20
|
+
## Setup with Claude Code
|
|
110
21
|
|
|
111
|
-
|
|
22
|
+
The fastest way to get value is to install the Claude Code hooks. These run automatically in the background - no workflow changes needed.
|
|
112
23
|
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
✔ Sparse coding complete
|
|
118
|
-
⠼ Calculating engram scores...
|
|
119
|
-
✔ Optimization complete
|
|
120
|
-
```
|
|
121
|
-
|
|
122
|
-
### Visual Impact Display
|
|
24
|
+
```bash
|
|
25
|
+
# Initialize sparn in your project
|
|
26
|
+
cd your-project
|
|
27
|
+
sparn init
|
|
123
28
|
|
|
124
|
-
|
|
29
|
+
# Install hooks into Claude Code
|
|
30
|
+
sparn hooks install
|
|
125
31
|
|
|
32
|
+
# Or install globally (all projects)
|
|
33
|
+
sparn hooks install --global
|
|
126
34
|
```
|
|
127
|
-
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
128
|
-
📊 Token Optimization Results
|
|
129
|
-
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
130
|
-
[█████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 94.9% reduced
|
|
131
35
|
|
|
132
|
-
|
|
36
|
+
That's it. Sparn now:
|
|
37
|
+
- Summarizes large tool outputs after Bash, Read, Grep, and Glob calls
|
|
38
|
+
- Warns Claude when your session transcript is getting large
|
|
133
39
|
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
40
|
+
To check status or remove:
|
|
41
|
+
```bash
|
|
42
|
+
sparn hooks status
|
|
43
|
+
sparn hooks uninstall
|
|
138
44
|
```
|
|
139
45
|
|
|
140
|
-
|
|
46
|
+
## Daily Usage
|
|
141
47
|
|
|
142
|
-
|
|
48
|
+
### Optimize context manually
|
|
143
49
|
|
|
144
|
-
|
|
50
|
+
Pipe any text through sparn to compress it:
|
|
145
51
|
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
```bash
|
|
150
|
-
# Skill is pre-installed at ~/.claude/skills/sparn/
|
|
151
|
-
# Just ensure Sparn is installed globally
|
|
152
|
-
npm install -g @ulrichc1/sparn
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
2. **Use in Claude Code**:
|
|
156
|
-
```bash
|
|
157
|
-
claude-code
|
|
158
|
-
> /sparn.go # Start real-time optimization
|
|
159
|
-
> /sparn.optimize # Manual optimization
|
|
160
|
-
> /sparn.stats # View statistics
|
|
161
|
-
```
|
|
162
|
-
|
|
163
|
-
**See** [`CLAUDE-CODE-SKILL.md`](./CLAUDE-CODE-SKILL.md) **for complete documentation.**
|
|
164
|
-
|
|
165
|
-
---
|
|
52
|
+
```bash
|
|
53
|
+
cat large-context.txt | sparn optimize
|
|
54
|
+
```
|
|
166
55
|
|
|
167
|
-
|
|
56
|
+
### Relay commands
|
|
168
57
|
|
|
169
|
-
|
|
58
|
+
Wrap any CLI command to automatically optimize its output:
|
|
170
59
|
|
|
171
60
|
```bash
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
sparn --version
|
|
61
|
+
sparn relay git log --oneline -50
|
|
62
|
+
sparn relay npm test
|
|
63
|
+
sparn relay cargo build --verbose
|
|
176
64
|
```
|
|
177
65
|
|
|
178
|
-
###
|
|
66
|
+
### Check your savings
|
|
179
67
|
|
|
180
68
|
```bash
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
# Or with other package managers
|
|
184
|
-
yarn add @ulrichc1/sparn
|
|
185
|
-
pnpm add @ulrichc1/sparn
|
|
69
|
+
sparn stats
|
|
186
70
|
```
|
|
187
71
|
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
## Quick Start
|
|
72
|
+
### Background daemon
|
|
191
73
|
|
|
192
|
-
|
|
74
|
+
For always-on optimization, start the daemon. It watches your session files and optimizes automatically when context exceeds the configured threshold.
|
|
193
75
|
|
|
194
76
|
```bash
|
|
195
|
-
|
|
196
|
-
sparn
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
**Output:**
|
|
77
|
+
sparn daemon start
|
|
78
|
+
sparn daemon status
|
|
79
|
+
sparn daemon stop
|
|
200
80
|
```
|
|
201
|
-
🧠 Sparn initialized!
|
|
202
81
|
|
|
203
|
-
|
|
204
|
-
Database: /your-project/.sparn/memory.db
|
|
82
|
+
## Codebase Intelligence (v1.4)
|
|
205
83
|
|
|
206
|
-
|
|
207
|
-
```
|
|
84
|
+
Sparn can analyze your project structure to provide smarter context to Claude Code.
|
|
208
85
|
|
|
209
|
-
###
|
|
86
|
+
### Dependency graph
|
|
210
87
|
|
|
211
|
-
|
|
212
|
-
```bash
|
|
213
|
-
cat large-context.txt | sparn optimize > optimized.txt
|
|
214
|
-
```
|
|
88
|
+
Map your project's import/export relationships:
|
|
215
89
|
|
|
216
|
-
**From file:**
|
|
217
90
|
```bash
|
|
218
|
-
|
|
219
|
-
|
|
91
|
+
# Full analysis: entry points, hot paths, orphaned files
|
|
92
|
+
sparn graph --analyze
|
|
220
93
|
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
⚡ Optimized context
|
|
94
|
+
# Focus on files related to "auth"
|
|
95
|
+
sparn graph --focus auth
|
|
224
96
|
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
Duration: 287ms
|
|
97
|
+
# Trace dependencies from an entry point
|
|
98
|
+
sparn graph --entry src/index.ts
|
|
228
99
|
```
|
|
229
100
|
|
|
230
|
-
###
|
|
101
|
+
### Search
|
|
231
102
|
|
|
232
|
-
|
|
103
|
+
Full-text search across your codebase using FTS5 (SQLite) with ripgrep fallback:
|
|
233
104
|
|
|
234
105
|
```bash
|
|
235
|
-
|
|
236
|
-
sparn
|
|
237
|
-
sparn relay cargo test --verbose
|
|
238
|
-
```
|
|
106
|
+
# First time: initialize and index
|
|
107
|
+
sparn search init
|
|
239
108
|
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
[optimized command output]
|
|
109
|
+
# Search
|
|
110
|
+
sparn search validateToken
|
|
243
111
|
|
|
244
|
-
|
|
112
|
+
# Re-index after changes
|
|
113
|
+
sparn search refresh
|
|
245
114
|
```
|
|
246
115
|
|
|
247
|
-
###
|
|
248
|
-
|
|
249
|
-
```bash
|
|
250
|
-
sparn stats
|
|
116
|
+
### Generate CLAUDE.md
|
|
251
117
|
|
|
252
|
-
|
|
253
|
-
sparn stats --graph
|
|
254
|
-
```
|
|
118
|
+
Auto-generate a `CLAUDE.md` file from your project structure, dependencies, and scripts:
|
|
255
119
|
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
📊 Sparn Statistics
|
|
120
|
+
```bash
|
|
121
|
+
sparn docs
|
|
259
122
|
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
Sessions: 23
|
|
123
|
+
# Skip dependency graph analysis
|
|
124
|
+
sparn docs --no-graph
|
|
263
125
|
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
2026-02-21 ████████████ 89K saved
|
|
126
|
+
# Custom output path
|
|
127
|
+
sparn docs -o docs/CLAUDE.md
|
|
267
128
|
```
|
|
268
129
|
|
|
269
|
-
###
|
|
130
|
+
### Workflow planner
|
|
270
131
|
|
|
271
|
-
|
|
132
|
+
Create and track implementation plans with token budgets:
|
|
272
133
|
|
|
273
134
|
```bash
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
**Output:**
|
|
278
|
-
```
|
|
279
|
-
🌙 Sleep consolidation complete
|
|
280
|
-
|
|
281
|
-
Entries: 8,234 → 2,145 (73.9% compression)
|
|
282
|
-
Duplicates merged: 412 groups
|
|
283
|
-
Duration: 1.8s
|
|
284
|
-
```
|
|
135
|
+
# Create a plan
|
|
136
|
+
sparn plan "Add user authentication" --files src/auth.ts src/routes.ts
|
|
285
137
|
|
|
286
|
-
|
|
138
|
+
# Create with search context
|
|
139
|
+
sparn plan "Fix login bug" --searches "login handler" "auth middleware"
|
|
287
140
|
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
sparn config
|
|
141
|
+
# List existing plans
|
|
142
|
+
sparn plan list
|
|
291
143
|
|
|
292
|
-
#
|
|
293
|
-
sparn
|
|
294
|
-
# Output: 5
|
|
144
|
+
# Execute a plan
|
|
145
|
+
sparn exec <plan-id>
|
|
295
146
|
|
|
296
|
-
#
|
|
297
|
-
sparn
|
|
298
|
-
# Output: Config updated: pruning.threshold = 10
|
|
147
|
+
# Verify completion
|
|
148
|
+
sparn verify <plan-id>
|
|
299
149
|
```
|
|
300
150
|
|
|
301
|
-
###
|
|
151
|
+
### Technical debt tracker
|
|
302
152
|
|
|
303
|
-
|
|
153
|
+
Track technical debt with severity levels and repayment dates:
|
|
304
154
|
|
|
305
155
|
```bash
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
sparn i
|
|
309
|
-
```
|
|
156
|
+
# Add debt
|
|
157
|
+
sparn debt add "Refactor auth middleware" --severity P1 --tokens 5000
|
|
310
158
|
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
- **Optimization Preview** - Test optimization with file preview and confirmation
|
|
314
|
-
- **Stats Dashboard** - Beautiful metrics display with multiple views
|
|
315
|
-
- **Memory Consolidation** - Interactive cleanup with confirmation
|
|
316
|
-
- **Quick Actions** - Common tasks and shortcuts
|
|
159
|
+
# Add with due date and affected files
|
|
160
|
+
sparn debt add "Fix N+1 queries" --severity P0 --due 2026-03-01 --files src/db.ts
|
|
317
161
|
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
321
|
-
🧠 Sparn Interactive Mode
|
|
322
|
-
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
323
|
-
Conversational configuration and exploration
|
|
324
|
-
|
|
325
|
-
? What would you like to do?
|
|
326
|
-
⚙️ Configure Settings
|
|
327
|
-
🔍 Optimize Preview
|
|
328
|
-
📊 Stats Dashboard
|
|
329
|
-
🧹 Memory Consolidation
|
|
330
|
-
🚀 Quick Actions
|
|
331
|
-
❯ ❌ Exit
|
|
332
|
-
```
|
|
162
|
+
# List all debt
|
|
163
|
+
sparn debt list
|
|
333
164
|
|
|
334
|
-
|
|
165
|
+
# List overdue items
|
|
166
|
+
sparn debt list --overdue
|
|
335
167
|
|
|
336
|
-
|
|
168
|
+
# Mark as resolved
|
|
169
|
+
sparn debt resolve <id>
|
|
337
170
|
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
```typescript
|
|
341
|
-
import { createSparsePruner, estimateTokens } from 'sparn';
|
|
342
|
-
|
|
343
|
-
const context = `
|
|
344
|
-
Line 1: Old file content
|
|
345
|
-
Line 2: More old content
|
|
346
|
-
Line 3: Error: Connection timeout
|
|
347
|
-
...
|
|
348
|
-
`;
|
|
349
|
-
|
|
350
|
-
const pruner = createSparsePruner();
|
|
351
|
-
const result = pruner.prune(context, 5);
|
|
352
|
-
|
|
353
|
-
console.log(`Tokens before: ${estimateTokens(context)}`);
|
|
354
|
-
console.log(`Tokens after: ${estimateTokens(result.prunedContext)}`);
|
|
355
|
-
console.log(`Kept ${result.entriesKept.length} entries`);
|
|
171
|
+
# View stats
|
|
172
|
+
sparn debt stats
|
|
356
173
|
```
|
|
357
174
|
|
|
358
|
-
### Using Memory Store
|
|
359
|
-
|
|
360
|
-
```typescript
|
|
361
|
-
import { createKVMemory, type MemoryEntry } from 'sparn';
|
|
362
|
-
|
|
363
|
-
const memory = await createKVMemory('./.sparn/memory.db');
|
|
364
|
-
|
|
365
|
-
// Store entry
|
|
366
|
-
const entry: MemoryEntry = {
|
|
367
|
-
id: crypto.randomUUID(),
|
|
368
|
-
content: 'Error: Connection timeout',
|
|
369
|
-
hash: hashContent('Error: Connection timeout'),
|
|
370
|
-
timestamp: Date.now() / 1000,
|
|
371
|
-
score: 0.85,
|
|
372
|
-
ttl: 86400,
|
|
373
|
-
state: 'active',
|
|
374
|
-
accessCount: 0,
|
|
375
|
-
tags: ['error'],
|
|
376
|
-
metadata: { severity: 'high' },
|
|
377
|
-
isBTSP: true,
|
|
378
|
-
};
|
|
379
|
-
|
|
380
|
-
await memory.put(entry);
|
|
381
|
-
|
|
382
|
-
// Query active entries
|
|
383
|
-
const activeEntries = await memory.query({
|
|
384
|
-
state: 'active',
|
|
385
|
-
minScore: 0.7,
|
|
386
|
-
limit: 10,
|
|
387
|
-
});
|
|
388
|
-
|
|
389
|
-
await memory.close();
|
|
390
|
-
```
|
|
391
|
-
|
|
392
|
-
### Full Pipeline
|
|
393
|
-
|
|
394
|
-
```typescript
|
|
395
|
-
import {
|
|
396
|
-
createSparsePruner,
|
|
397
|
-
createEngramScorer,
|
|
398
|
-
createConfidenceStates,
|
|
399
|
-
createBTSPEmbedder,
|
|
400
|
-
estimateTokens,
|
|
401
|
-
hashContent,
|
|
402
|
-
} from 'sparn';
|
|
403
|
-
|
|
404
|
-
async function optimizeWithPipeline(context: string) {
|
|
405
|
-
const pruner = createSparsePruner({ threshold: 5 });
|
|
406
|
-
const scorer = createEngramScorer({ defaultTTL: 24 });
|
|
407
|
-
const states = createConfidenceStates();
|
|
408
|
-
const btsp = createBTSPEmbedder();
|
|
409
|
-
|
|
410
|
-
// 1. Sparse pruning
|
|
411
|
-
const pruneResult = pruner.prune(context, 5);
|
|
412
|
-
|
|
413
|
-
// 2. Create entries
|
|
414
|
-
const entries = pruneResult.entriesKept.map(line => {
|
|
415
|
-
const isBTSPEntry = btsp.detectBTSP(line);
|
|
416
|
-
return {
|
|
417
|
-
id: crypto.randomUUID(),
|
|
418
|
-
content: line,
|
|
419
|
-
hash: hashContent(line),
|
|
420
|
-
timestamp: Date.now() / 1000,
|
|
421
|
-
score: isBTSPEntry ? 1.0 : 0.5,
|
|
422
|
-
ttl: isBTSPEntry ? 172800 : 86400,
|
|
423
|
-
state: isBTSPEntry ? 'active' : 'ready',
|
|
424
|
-
accessCount: 0,
|
|
425
|
-
tags: [],
|
|
426
|
-
metadata: {},
|
|
427
|
-
isBTSP: isBTSPEntry,
|
|
428
|
-
};
|
|
429
|
-
});
|
|
430
|
-
|
|
431
|
-
// 3. Score and transition
|
|
432
|
-
const scoredEntries = entries.map(entry => {
|
|
433
|
-
const score = scorer.calculateScore(entry);
|
|
434
|
-
return states.transition({ ...entry, score });
|
|
435
|
-
});
|
|
436
|
-
|
|
437
|
-
return {
|
|
438
|
-
optimizedContext: scoredEntries.map(e => e.content).join('\n'),
|
|
439
|
-
tokensBefore: estimateTokens(context),
|
|
440
|
-
tokensAfter: estimateTokens(pruneResult.prunedContext),
|
|
441
|
-
distribution: states.getDistribution(scoredEntries),
|
|
442
|
-
};
|
|
443
|
-
}
|
|
444
|
-
```
|
|
445
|
-
|
|
446
|
-
---
|
|
447
|
-
|
|
448
175
|
## Configuration
|
|
449
176
|
|
|
450
|
-
|
|
177
|
+
After `sparn init`, edit `.sparn/config.yaml`:
|
|
451
178
|
|
|
452
179
|
```yaml
|
|
453
|
-
# Pruning settings
|
|
454
180
|
pruning:
|
|
455
|
-
threshold: 5 # Keep top 5% (1-100)
|
|
456
|
-
aggressiveness: 50 #
|
|
181
|
+
threshold: 5 # Keep top 5% of context (1-100)
|
|
182
|
+
aggressiveness: 50 # How aggressively to prune (0-100)
|
|
457
183
|
|
|
458
|
-
# Decay settings
|
|
459
184
|
decay:
|
|
460
|
-
defaultTTL: 24 # Hours
|
|
461
|
-
decayThreshold: 0.95 #
|
|
185
|
+
defaultTTL: 24 # Hours before context starts fading
|
|
186
|
+
decayThreshold: 0.95 # Score threshold for pruning
|
|
462
187
|
|
|
463
|
-
# Confidence state thresholds
|
|
464
188
|
states:
|
|
465
|
-
activeThreshold: 0.7 # Score
|
|
466
|
-
readyThreshold: 0.3 # Score 0.3-0.
|
|
467
|
-
|
|
468
|
-
# Agent adapter
|
|
469
|
-
agent: generic # claude-code | generic
|
|
470
|
-
|
|
471
|
-
# UI settings
|
|
472
|
-
ui:
|
|
473
|
-
colors: true # Colored output
|
|
474
|
-
sounds: false # Sound effects
|
|
475
|
-
verbose: false # Verbose logging
|
|
476
|
-
|
|
477
|
-
# Auto-consolidation (hours, or null for manual)
|
|
478
|
-
autoConsolidate: null
|
|
479
|
-
```
|
|
480
|
-
|
|
481
|
-
---
|
|
482
|
-
|
|
483
|
-
## Common Workflows
|
|
189
|
+
activeThreshold: 0.7 # Score >= 0.7 = active (kept)
|
|
190
|
+
readyThreshold: 0.3 # Score 0.3-0.69 = ready (may be kept)
|
|
484
191
|
|
|
485
|
-
|
|
192
|
+
realtime:
|
|
193
|
+
tokenBudget: 40000 # Target token count after optimization
|
|
194
|
+
autoOptimizeThreshold: 60000 # Trigger optimization above this
|
|
195
|
+
debounceMs: 5000 # Wait time between optimizations
|
|
486
196
|
|
|
487
|
-
|
|
488
|
-
# Generate context file
|
|
489
|
-
cat src/**/*.ts > context.txt
|
|
490
|
-
|
|
491
|
-
# Optimize
|
|
492
|
-
sparn optimize --input context.txt --output optimized.txt
|
|
493
|
-
```
|
|
494
|
-
|
|
495
|
-
### Daily Cleanup
|
|
496
|
-
|
|
497
|
-
Schedule via cron:
|
|
498
|
-
|
|
499
|
-
```bash
|
|
500
|
-
# Add to crontab
|
|
501
|
-
0 2 * * * cd /your-project && sparn consolidate
|
|
197
|
+
agent: generic # or claude-code
|
|
502
198
|
```
|
|
503
199
|
|
|
504
|
-
Or
|
|
505
|
-
|
|
200
|
+
Or use the CLI:
|
|
506
201
|
```bash
|
|
507
|
-
sparn config
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
### Git Diff Optimization
|
|
511
|
-
|
|
512
|
-
```bash
|
|
513
|
-
git diff | sparn optimize | less
|
|
514
|
-
|
|
515
|
-
# Or via relay
|
|
516
|
-
sparn relay git diff --stat
|
|
202
|
+
sparn config get pruning.threshold
|
|
203
|
+
sparn config set pruning.threshold 10
|
|
517
204
|
```
|
|
518
205
|
|
|
519
|
-
|
|
520
|
-
|
|
206
|
+
Or the interactive mode:
|
|
521
207
|
```bash
|
|
522
|
-
sparn
|
|
523
|
-
sparn relay cargo build --verbose
|
|
524
|
-
sparn relay tsc --noEmit
|
|
208
|
+
sparn interactive
|
|
525
209
|
```
|
|
526
210
|
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
## Troubleshooting
|
|
211
|
+
## Programmatic API
|
|
530
212
|
|
|
531
|
-
|
|
213
|
+
```typescript
|
|
214
|
+
import { createSparsePruner, estimateTokens } from '@ulrichc1/sparn';
|
|
532
215
|
|
|
533
|
-
|
|
216
|
+
const pruner = createSparsePruner({ threshold: 5 });
|
|
217
|
+
const result = pruner.prune(largeContext, 5);
|
|
534
218
|
|
|
535
|
-
|
|
536
|
-
```bash
|
|
537
|
-
cd your-project/
|
|
538
|
-
sparn init
|
|
219
|
+
console.log(`${estimateTokens(largeContext)} -> ${estimateTokens(result.prunedContext)} tokens`);
|
|
539
220
|
```
|
|
540
221
|
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
**Error:** `Error: Context exceeds 500K tokens`
|
|
544
|
-
|
|
545
|
-
**Solution:** Increase threshold or split context:
|
|
546
|
-
```bash
|
|
547
|
-
sparn config set pruning.threshold 10 # Keep top 10%
|
|
548
|
-
```
|
|
222
|
+
The full API exports all core modules: `createDependencyGraph`, `createSearchEngine`, `createWorkflowPlanner`, `createDocsGenerator`, `createDebtTracker`, `createKVMemory`, `createBudgetPrunerFromConfig`, `createIncrementalOptimizer`, and more.
|
|
549
223
|
|
|
550
|
-
|
|
224
|
+
## MCP Server
|
|
551
225
|
|
|
552
|
-
|
|
226
|
+
Sparn can run as an MCP server for Claude Desktop or any MCP client:
|
|
553
227
|
|
|
554
|
-
**Solution:** Lower active threshold:
|
|
555
228
|
```bash
|
|
556
|
-
sparn
|
|
229
|
+
sparn mcp:server
|
|
557
230
|
```
|
|
558
231
|
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
**Error:** `Error: database is locked`
|
|
562
|
-
|
|
563
|
-
**Solution:** Close other Sparn processes or consolidate:
|
|
564
|
-
```bash
|
|
565
|
-
sparn consolidate # Unlocks and compacts database
|
|
566
|
-
```
|
|
232
|
+
Exposes three tools: `sparn_optimize`, `sparn_stats`, `sparn_consolidate`.
|
|
567
233
|
|
|
568
|
-
|
|
234
|
+
## How it works
|
|
569
235
|
|
|
570
|
-
|
|
236
|
+
Sparn uses a multi-stage pipeline to decide what context to keep:
|
|
571
237
|
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
- Log-heavy: Higher threshold (aggressive pruning)
|
|
578
|
-
5. **Use Relay for CLIs** - Wrap frequently-run commands in `sparn relay`
|
|
579
|
-
6. **BTSP for Errors** - Errors are automatically flagged as high-priority
|
|
580
|
-
7. **Tag Important Entries** - Use metadata for custom filtering (API)
|
|
581
|
-
|
|
582
|
-
---
|
|
583
|
-
|
|
584
|
-
## Performance
|
|
585
|
-
|
|
586
|
-
- **Optimization Latency**: <500ms for 100K tokens
|
|
587
|
-
- **CLI Startup Time**: <200ms (lazy loading)
|
|
588
|
-
- **Memory Usage**: <100MB for typical workloads
|
|
589
|
-
- **Token Reduction**: 60-90% average
|
|
590
|
-
- **Database Operations**: <10ms per read/write
|
|
591
|
-
|
|
592
|
-
---
|
|
593
|
-
|
|
594
|
-
## Privacy & Security
|
|
595
|
-
|
|
596
|
-
🔒 **Your data stays on your machine**
|
|
597
|
-
|
|
598
|
-
- ✅ No data transmission to external servers
|
|
599
|
-
- ✅ No telemetry or analytics
|
|
600
|
-
- ✅ No cloud storage
|
|
601
|
-
- ✅ Full user control over data
|
|
602
|
-
- ✅ GDPR compliant
|
|
603
|
-
|
|
604
|
-
All processing happens locally. See [PRIVACY.md](./PRIVACY.md) and [SECURITY.md](./SECURITY.md) for details.
|
|
605
|
-
|
|
606
|
-
---
|
|
607
|
-
|
|
608
|
-
## Architecture
|
|
609
|
-
|
|
610
|
-
### Directory Structure
|
|
611
|
-
|
|
612
|
-
```
|
|
613
|
-
sparn/
|
|
614
|
-
├── src/
|
|
615
|
-
│ ├── core/ # Neuroscience modules (library)
|
|
616
|
-
│ ├── adapters/ # Agent-specific adapters
|
|
617
|
-
│ ├── cli/ # CLI commands
|
|
618
|
-
│ ├── types/ # TypeScript interfaces
|
|
619
|
-
│ └── utils/ # Shared utilities
|
|
620
|
-
├── tests/
|
|
621
|
-
│ ├── unit/ # Unit tests
|
|
622
|
-
│ └── integration/ # Integration tests
|
|
623
|
-
└── dist/ # Build output (CJS/ESM/DTS)
|
|
624
|
-
```
|
|
625
|
-
|
|
626
|
-
### Tech Stack
|
|
627
|
-
|
|
628
|
-
- **Language**: TypeScript (strict mode)
|
|
629
|
-
- **Runtime**: Node.js 18+
|
|
630
|
-
- **Build**: tsup
|
|
631
|
-
- **Test**: vitest
|
|
632
|
-
- **Lint**: biome
|
|
633
|
-
- **Storage**: better-sqlite3
|
|
634
|
-
- **CLI**: commander + chalk + ora + boxen
|
|
635
|
-
|
|
636
|
-
---
|
|
238
|
+
- **Relevance filtering** - Only the most relevant 2-5% of context carries the signal
|
|
239
|
+
- **Time-based decay** - Older context fades over time unless reinforced by reuse
|
|
240
|
+
- **Entry classification** - Entries are active, ready, or silent based on their score
|
|
241
|
+
- **Critical event detection** - Errors and stack traces get permanently flagged as important
|
|
242
|
+
- **Periodic consolidation** - Merges duplicate entries and cleans up stale data
|
|
637
243
|
|
|
638
244
|
## Development
|
|
639
245
|
|
|
640
|
-
### Setup
|
|
641
|
-
|
|
642
246
|
```bash
|
|
643
|
-
# Clone the repository
|
|
644
247
|
git clone https://github.com/ulrichc1/sparn.git
|
|
645
248
|
cd sparn
|
|
646
|
-
|
|
647
|
-
# Install dependencies
|
|
648
249
|
npm install
|
|
649
|
-
|
|
650
|
-
# Build the project
|
|
651
250
|
npm run build
|
|
652
|
-
|
|
653
|
-
|
|
654
|
-
npm
|
|
655
|
-
```
|
|
656
|
-
|
|
657
|
-
### Available Scripts
|
|
658
|
-
|
|
659
|
-
```bash
|
|
660
|
-
npm run build # Build with tsup (CJS + ESM + DTS)
|
|
661
|
-
npm run dev # Watch mode for development
|
|
662
|
-
npm test # Run all tests with vitest
|
|
663
|
-
npm run test:watch # Run tests in watch mode
|
|
664
|
-
npm run lint # Lint code with biome
|
|
665
|
-
npm run lint:fix # Fix linting issues automatically
|
|
666
|
-
npm run typecheck # Type check without emit
|
|
667
|
-
```
|
|
668
|
-
|
|
669
|
-
### Running Locally
|
|
670
|
-
|
|
671
|
-
```bash
|
|
672
|
-
# Build first
|
|
673
|
-
npm run build
|
|
674
|
-
|
|
675
|
-
# Test CLI commands
|
|
676
|
-
node dist/cli/index.js init
|
|
677
|
-
node dist/cli/index.js --help
|
|
678
|
-
|
|
679
|
-
# Or link globally for testing
|
|
680
|
-
npm link
|
|
681
|
-
sparn --version
|
|
682
|
-
```
|
|
683
|
-
|
|
684
|
-
### Constitution Compliance
|
|
685
|
-
|
|
686
|
-
All code must follow the [Sparn Constitution](.specify/memory/constitution.md):
|
|
687
|
-
|
|
688
|
-
1. **CLI-First, Library-Second** - CLI commands wrap library modules
|
|
689
|
-
2. **Neuroscience Fidelity** - Code maps to documented brain mechanisms
|
|
690
|
-
3. **Test-First Development (TDD)** - Tests FIRST, then implementation
|
|
691
|
-
4. **Agent-Agnostic Design** - Core modules have zero agent knowledge
|
|
692
|
-
5. **Complementary to RTK** - Optimizes input context, not CLI output
|
|
693
|
-
6. **Minimal Dependencies** - Only justified runtime dependencies
|
|
694
|
-
7. **Simplicity First** - Max 3 dir levels, YAML config, no complexity
|
|
695
|
-
8. **Brand Consistency** - Pink brain (#FF6B9D), Neural Cyan, Synapse Violet
|
|
696
|
-
9. **Production-Quality TypeScript** - Strict mode, no `any`, JSDoc on all public APIs
|
|
697
|
-
|
|
698
|
-
---
|
|
699
|
-
|
|
700
|
-
## Validation
|
|
701
|
-
|
|
702
|
-
See [VALIDATION.md](./VALIDATION.md) for the complete validation guide.
|
|
703
|
-
|
|
704
|
-
### Quick Validation
|
|
705
|
-
|
|
706
|
-
```bash
|
|
707
|
-
# 1. Build
|
|
708
|
-
npm run build
|
|
709
|
-
|
|
710
|
-
# 2. Test init command
|
|
711
|
-
node dist/cli/index.js init
|
|
712
|
-
|
|
713
|
-
# 3. Check files created
|
|
714
|
-
ls -la .sparn/
|
|
715
|
-
cat .sparn/config.yaml
|
|
716
|
-
|
|
717
|
-
# 4. Run tests
|
|
718
|
-
npm test
|
|
251
|
+
npm test # 479 tests
|
|
252
|
+
npm run lint
|
|
253
|
+
npm run typecheck
|
|
719
254
|
```
|
|
720
255
|
|
|
721
|
-
---
|
|
722
|
-
|
|
723
|
-
## Roadmap
|
|
724
|
-
|
|
725
|
-
### MVP (User Stories 1-2) ✅ COMPLETE
|
|
726
|
-
- [X] US1: Initialize Sparn (`sparn init`)
|
|
727
|
-
- [X] US2: Optimize context (`sparn optimize`)
|
|
728
|
-
|
|
729
|
-
### Post-MVP
|
|
730
|
-
- [X] US3: View statistics (`sparn stats`) ✅ COMPLETE
|
|
731
|
-
- [X] US4: Relay commands (`sparn relay <cmd>`) ✅ COMPLETE
|
|
732
|
-
- [X] US5: Consolidate memory (`sparn consolidate`) ✅ COMPLETE
|
|
733
|
-
- [X] US6: Configure behavior (`sparn config`) ✅ COMPLETE
|
|
734
|
-
- [X] US7: Programmatic API (library usage) ✅ COMPLETE
|
|
735
|
-
|
|
736
|
-
---
|
|
737
|
-
|
|
738
|
-
## Contributing
|
|
739
|
-
|
|
740
|
-
This project follows strict TDD (Test-Driven Development):
|
|
741
|
-
|
|
742
|
-
1. Write tests FIRST
|
|
743
|
-
2. Confirm tests FAIL (Red phase)
|
|
744
|
-
3. Implement to make tests PASS (Green phase)
|
|
745
|
-
4. Refactor while tests stay green
|
|
746
|
-
|
|
747
|
-
See [CONTRIBUTING.md](./CONTRIBUTING.md) (coming soon) for details.
|
|
748
|
-
|
|
749
|
-
---
|
|
750
|
-
|
|
751
256
|
## License
|
|
752
257
|
|
|
753
|
-
MIT
|
|
754
|
-
|
|
755
|
-
---
|
|
756
|
-
|
|
757
|
-
## Project Links
|
|
758
|
-
|
|
759
|
-
- **Planning Docs**: `specs/001-sparn-core/`
|
|
760
|
-
- **Constitution**: `.specify/memory/constitution.md`
|
|
761
|
-
- **Tasks**: `specs/001-sparn-core/tasks.md`
|
|
762
|
-
- **Validation Guide**: `VALIDATION.md`
|
|
763
|
-
|
|
764
|
-
---
|
|
258
|
+
MIT
|