@ulrichc1/sparn 1.2.1 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. package/PRIVACY.md +1 -1
  2. package/README.md +136 -642
  3. package/SECURITY.md +1 -1
  4. package/dist/cli/dashboard.cjs +3977 -0
  5. package/dist/cli/dashboard.cjs.map +1 -0
  6. package/dist/cli/dashboard.d.cts +17 -0
  7. package/dist/cli/dashboard.d.ts +17 -0
  8. package/dist/cli/dashboard.js +3932 -0
  9. package/dist/cli/dashboard.js.map +1 -0
  10. package/dist/cli/index.cjs +3855 -486
  11. package/dist/cli/index.cjs.map +1 -1
  12. package/dist/cli/index.js +3812 -459
  13. package/dist/cli/index.js.map +1 -1
  14. package/dist/daemon/index.cjs +411 -99
  15. package/dist/daemon/index.cjs.map +1 -1
  16. package/dist/daemon/index.js +423 -103
  17. package/dist/daemon/index.js.map +1 -1
  18. package/dist/hooks/post-tool-result.cjs +129 -225
  19. package/dist/hooks/post-tool-result.cjs.map +1 -1
  20. package/dist/hooks/post-tool-result.js +129 -225
  21. package/dist/hooks/post-tool-result.js.map +1 -1
  22. package/dist/hooks/pre-prompt.cjs +206 -242
  23. package/dist/hooks/pre-prompt.cjs.map +1 -1
  24. package/dist/hooks/pre-prompt.js +192 -243
  25. package/dist/hooks/pre-prompt.js.map +1 -1
  26. package/dist/hooks/stop-docs-refresh.cjs +123 -0
  27. package/dist/hooks/stop-docs-refresh.cjs.map +1 -0
  28. package/dist/hooks/stop-docs-refresh.d.cts +1 -0
  29. package/dist/hooks/stop-docs-refresh.d.ts +1 -0
  30. package/dist/hooks/stop-docs-refresh.js +126 -0
  31. package/dist/hooks/stop-docs-refresh.js.map +1 -0
  32. package/dist/index.cjs +1756 -339
  33. package/dist/index.cjs.map +1 -1
  34. package/dist/index.d.cts +540 -41
  35. package/dist/index.d.ts +540 -41
  36. package/dist/index.js +1739 -331
  37. package/dist/index.js.map +1 -1
  38. package/dist/mcp/index.cjs +306 -73
  39. package/dist/mcp/index.cjs.map +1 -1
  40. package/dist/mcp/index.js +310 -73
  41. package/dist/mcp/index.js.map +1 -1
  42. package/package.json +10 -3
package/README.md CHANGED
@@ -1,764 +1,258 @@
1
1
  # Sparn
2
2
 
3
- [![npm version](https://img.shields.io/npm/v/@ulrichc1/sparn.svg)](https://www.npmjs.com/package/@ulrichc1/sparn)
4
- [![npm downloads](https://img.shields.io/npm/dm/@ulrichc1/sparn.svg)](https://www.npmjs.com/package/@ulrichc1/sparn)
5
- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
6
-
7
- > 🧠 Neuroscience-inspired context optimization for AI coding agents
8
-
9
- **Status**: ✅ **Production Ready** - Full Feature Set with MCP Server + Interactive Mode
10
-
11
- **Version**: 1.2.0
12
-
13
- ---
14
-
15
- ## What is Sparn?
16
-
17
- Sparn is an npm CLI package that applies 6 neuroscience principles to intelligently prune, score, and compress AI agent context memory. It reduces token usage by 60-90% while maintaining task completion quality.
18
-
19
- ### Neuroscience Principles
20
-
21
- 1. **Sparse Coding** - Keep only 2-5% most relevant context
22
- 2. **Engram Theory** - Memories decay over time
23
- 3. **Hippocampal KV** - Separate what to store from how to retrieve
24
- 4. **Multi-State Synapses** - Silent, ready, or active states
25
- 5. **Sleep Replay** - Periodic memory consolidation
26
- 6. **BTSP** - One-shot learning for critical events
27
-
28
- ---
29
-
30
- ## Features
31
-
32
- ### ✅ Core Functionality
33
- - ✅ **All 6 Neuroscience Modules** - Complete brain-inspired architecture
34
- - SparsePruner (Sparse Coding)
35
- - EngramScorer (Engram Theory)
36
- - ConfidenceStates (Multi-State Synapses)
37
- - BTSPEmbedder (One-Shot Learning)
38
- - GenericAdapter + ClaudeCodeAdapter (Agent-Agnostic)
39
- - SleepCompressor (Sleep Replay)
40
- - ✅ **Context Optimization** - 60-90% token reduction pipeline
41
- - ✅ **Real-Time Optimization** - Always-on background daemon and hooks
42
- - Background daemon with auto-optimization at 80K token threshold
43
- - Automated consolidation scheduler (configurable intervals)
44
- - Claude Code hooks (pre-prompt & post-tool-result)
45
- - Incremental optimization with <50ms delta processing
46
- - Budget-aware pruning targeting specific token counts
47
- - Tool output compression (npm, docker, tests, git diffs)
48
- - ✅ **MCP Server** (NEW!) - Model Context Protocol integration
49
- - Expose Sparn as MCP tools for Claude Desktop and other MCP clients
50
- - Three tools: sparn_optimize, sparn_stats, sparn_consolidate
51
- - Full SDK integration with stdio transport
52
- - Comprehensive configuration guide
53
- - ✅ **Interactive Mode** (NEW!) - Conversational CLI interface
54
- - Configuration wizard with guided prompts
55
- - Optimization preview with file browsing
56
- - Stats dashboard with multiple views
57
- - Memory consolidation with confirmation
58
- - Quick actions and shortcuts
59
- - ✅ **CLI Commands** - init, optimize, stats, relay, consolidate, config, daemon, hooks, interactive
60
- - ✅ **Programmatic API** - Full TypeScript support, JSDoc, standalone modules, MCP server factory
61
- - ✅ **Database** - SQLite with dual index/value tables, corruption detection
62
- - ✅ **Configuration** - YAML config with runtime modification
63
- - ✅ **Metrics & Telemetry** - P50/P95/P99 latency tracking, cache hit rates, token savings
64
-
65
- ### ✨ Polish & UX
66
- - ✨ **Progress Indicators** - Real-time ora spinners for all long operations
67
- - ✨ **Visual Impact** - Before/after token savings with progress bars and celebration messages
68
- - ✨ **Detailed Help** - Comprehensive --help text with examples for every command
69
- - ✨ **Branded UI** - Pink brain logo colors, neural cyan, synapse violet
70
- - ✨ **Error Handling** - Context-aware recovery suggestions, database backup on corruption
71
- - ✨ **Lazy Loading** - Fast startup (<200ms for --help/--version)
72
-
73
- ### 📊 Quality & CI/CD
74
- - 📊 **230 Tests** - Comprehensive unit + integration test coverage, 230 passing
75
- - 📊 **Performance Benchmarks** - Validates <50ms incremental optimization target
76
- - 📊 **CI Pipeline** - GitHub Actions with cross-platform tests (Ubuntu, macOS, Windows)
77
- - 📊 **Documentation** - NEUROSCIENCE.md, CONTRIBUTING.md, CHANGELOG.md, MCP.md, comprehensive README
78
- - 📊 **NPM Ready** - Package validated with publish --dry-run
79
-
80
- ---
81
-
82
- ## Screenshots
83
-
84
- ### Branded Banner & Initialization
3
+ Context optimization for AI coding agents. Reduces token usage by 60-90% so your Claude Code sessions last longer and cost less.
85
4
 
86
- ```
87
- ____ ____ ___ ____ _ __
88
- / __ \/ __ \/ | / __ \/ | / /
89
- / /_/ / /_/ / /| | / /_/ / |/ /
90
- \__, / ____/ ___ |/ _, _/ /| /
91
- /____/_/ /_/ |_/_/ |_/_/ |_/
92
-
93
- 🧠 Neuroscience-inspired context optimization
94
- v0.1.0
95
- ```
5
+ ## Why
96
6
 
97
- ### Optimization Summary
7
+ Long Claude Code sessions burn through context windows fast. Tool outputs, file reads, and conversation history pile up until you hit the limit and lose your thread. Sparn watches for this and keeps your context lean:
98
8
 
99
- ```
100
- Optimized context
9
+ - Compresses verbose tool outputs (test results, build logs, file reads)
10
+ - Tracks which context is still relevant and which has gone stale
11
+ - Runs as a background daemon or hooks directly into Claude Code
12
+ - Analyzes your codebase structure so Claude gets the right context first
101
13
 
102
- Tokens: 45,231 → 2,315 (94.9% reduction)
103
- Entries: Active 12 | Ready 34 | Pruned 189
104
- Duration: 287ms
14
+ ## Install
105
15
 
106
- [optimized context output follows]
16
+ ```bash
17
+ npm install -g @ulrichc1/sparn
107
18
  ```
108
19
 
109
- ### Progress Indicators
20
+ ## Setup with Claude Code
110
21
 
111
- Real-time ora spinners show optimization progress:
22
+ The fastest way to get value is to install the Claude Code hooks. These run automatically in the background - no workflow changes needed.
112
23
 
113
- ```
114
- Analyzing context...
115
- Context analyzed
116
- Applying sparse coding...
117
- ✔ Sparse coding complete
118
- ⠼ Calculating engram scores...
119
- ✔ Optimization complete
120
- ```
121
-
122
- ### Visual Impact Display
24
+ ```bash
25
+ # Initialize sparn in your project
26
+ cd your-project
27
+ sparn init
123
28
 
124
- Before/after token savings with progress bars and celebration messages:
29
+ # Install hooks into Claude Code
30
+ sparn hooks install
125
31
 
32
+ # Or install globally (all projects)
33
+ sparn hooks install --global
126
34
  ```
127
- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
128
- 📊 Token Optimization Results
129
- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
130
- [█████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 94.9% reduced
131
35
 
132
- OUTSTANDING! Mind-blowing 90%+ reduction!
36
+ That's it. Sparn now:
37
+ - Summarizes large tool outputs after Bash, Read, Grep, and Glob calls
38
+ - Warns Claude when your session transcript is getting large
133
39
 
134
- Before: 45,231 tokens
135
- After: 2,315 tokens
136
- Saved: 42,916 tokens
137
- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
40
+ To check status or remove:
41
+ ```bash
42
+ sparn hooks status
43
+ sparn hooks uninstall
138
44
  ```
139
45
 
140
- ---
46
+ ## Daily Usage
141
47
 
142
- ## 🤖 Claude Code Integration
48
+ ### Optimize context manually
143
49
 
144
- **NEW!** Sparn now integrates with Claude Code CLI for real-time context optimization!
50
+ Pipe any text through sparn to compress it:
145
51
 
146
- ### Quick Start with Claude Code
147
-
148
- 1. **Install the skill**:
149
- ```bash
150
- # Skill is pre-installed at ~/.claude/skills/sparn/
151
- # Just ensure Sparn is installed globally
152
- npm install -g @ulrichc1/sparn
153
- ```
154
-
155
- 2. **Use in Claude Code**:
156
- ```bash
157
- claude-code
158
- > /sparn.go # Start real-time optimization
159
- > /sparn.optimize # Manual optimization
160
- > /sparn.stats # View statistics
161
- ```
162
-
163
- **See** [`CLAUDE-CODE-SKILL.md`](./CLAUDE-CODE-SKILL.md) **for complete documentation.**
164
-
165
- ---
52
+ ```bash
53
+ cat large-context.txt | sparn optimize
54
+ ```
166
55
 
167
- ## Installation
56
+ ### Relay commands
168
57
 
169
- ### Global CLI Install
58
+ Wrap any CLI command to automatically optimize its output:
170
59
 
171
60
  ```bash
172
- npm install -g @ulrichc1/sparn
173
-
174
- # Verify installation
175
- sparn --version
61
+ sparn relay git log --oneline -50
62
+ sparn relay npm test
63
+ sparn relay cargo build --verbose
176
64
  ```
177
65
 
178
- ### Project Dependency
66
+ ### Check your savings
179
67
 
180
68
  ```bash
181
- npm install @ulrichc1/sparn
182
-
183
- # Or with other package managers
184
- yarn add @ulrichc1/sparn
185
- pnpm add @ulrichc1/sparn
69
+ sparn stats
186
70
  ```
187
71
 
188
- ---
189
-
190
- ## Quick Start
72
+ ### Background daemon
191
73
 
192
- ### 1. Initialize Sparn
74
+ For always-on optimization, start the daemon. It watches your session files and optimizes automatically when context exceeds the configured threshold.
193
75
 
194
76
  ```bash
195
- cd your-project/
196
- sparn init
197
- ```
198
-
199
- **Output:**
77
+ sparn daemon start
78
+ sparn daemon status
79
+ sparn daemon stop
200
80
  ```
201
- 🧠 Sparn initialized!
202
81
 
203
- Config: /your-project/.sparn/config.yaml
204
- Database: /your-project/.sparn/memory.db
82
+ ## Codebase Intelligence (v1.4)
205
83
 
206
- Run 'sparn optimize' to start optimizing context.
207
- ```
84
+ Sparn can analyze your project structure to provide smarter context to Claude Code.
208
85
 
209
- ### 2. Optimize Context
86
+ ### Dependency graph
210
87
 
211
- **From stdin:**
212
- ```bash
213
- cat large-context.txt | sparn optimize > optimized.txt
214
- ```
88
+ Map your project's import/export relationships:
215
89
 
216
- **From file:**
217
90
  ```bash
218
- sparn optimize --input context.txt --output optimized.txt
219
- ```
91
+ # Full analysis: entry points, hot paths, orphaned files
92
+ sparn graph --analyze
220
93
 
221
- **Output:**
222
- ```
223
- ⚡ Optimized context
94
+ # Focus on files related to "auth"
95
+ sparn graph --focus auth
224
96
 
225
- Tokens: 45,231 2,315 (94.9% reduction)
226
- Entries: Active 12 | Ready 34 | Pruned 189
227
- Duration: 287ms
97
+ # Trace dependencies from an entry point
98
+ sparn graph --entry src/index.ts
228
99
  ```
229
100
 
230
- ### 3. Relay Commands
101
+ ### Search
231
102
 
232
- Optimize CLI command output on-the-fly:
103
+ Full-text search across your codebase using FTS5 (SQLite) with ripgrep fallback:
233
104
 
234
105
  ```bash
235
- sparn relay git log --oneline -20
236
- sparn relay npm list
237
- sparn relay cargo test --verbose
238
- ```
106
+ # First time: initialize and index
107
+ sparn search init
239
108
 
240
- **Output:**
241
- ```
242
- [optimized command output]
109
+ # Search
110
+ sparn search validateToken
243
111
 
244
- Saved 3,456 tokens (78.2% reduction)
112
+ # Re-index after changes
113
+ sparn search refresh
245
114
  ```
246
115
 
247
- ### 4. View Statistics
248
-
249
- ```bash
250
- sparn stats
116
+ ### Generate CLAUDE.md
251
117
 
252
- # With graph
253
- sparn stats --graph
254
- ```
118
+ Auto-generate a `CLAUDE.md` file from your project structure, dependencies, and scripts:
255
119
 
256
- **Output:**
257
- ```
258
- 📊 Sparn Statistics
120
+ ```bash
121
+ sparn docs
259
122
 
260
- Total commands: 47
261
- Tokens saved: 1,234,567 (avg 82.3% reduction)
262
- Sessions: 23
123
+ # Skip dependency graph analysis
124
+ sparn docs --no-graph
263
125
 
264
- Last 7 days:
265
- 2026-02-22 ████████████████████ 156K saved
266
- 2026-02-21 ████████████ 89K saved
126
+ # Custom output path
127
+ sparn docs -o docs/CLAUDE.md
267
128
  ```
268
129
 
269
- ### 5. Consolidate Memory
130
+ ### Workflow planner
270
131
 
271
- Run periodic cleanup:
132
+ Create and track implementation plans with token budgets:
272
133
 
273
134
  ```bash
274
- sparn consolidate
275
- ```
276
-
277
- **Output:**
278
- ```
279
- 🌙 Sleep consolidation complete
280
-
281
- Entries: 8,234 → 2,145 (73.9% compression)
282
- Duplicates merged: 412 groups
283
- Duration: 1.8s
284
- ```
135
+ # Create a plan
136
+ sparn plan "Add user authentication" --files src/auth.ts src/routes.ts
285
137
 
286
- ### 6. Configure Behavior
138
+ # Create with search context
139
+ sparn plan "Fix login bug" --searches "login handler" "auth middleware"
287
140
 
288
- ```bash
289
- # View all config
290
- sparn config
141
+ # List existing plans
142
+ sparn plan list
291
143
 
292
- # Get specific value
293
- sparn config get pruning.threshold
294
- # Output: 5
144
+ # Execute a plan
145
+ sparn exec <plan-id>
295
146
 
296
- # Set value
297
- sparn config set pruning.threshold 10
298
- # Output: Config updated: pruning.threshold = 10
147
+ # Verify completion
148
+ sparn verify <plan-id>
299
149
  ```
300
150
 
301
- ### 7. Interactive Mode (NEW!)
151
+ ### Technical debt tracker
302
152
 
303
- Launch conversational interface for exploration and configuration:
153
+ Track technical debt with severity levels and repayment dates:
304
154
 
305
155
  ```bash
306
- sparn interactive
307
- # or short alias
308
- sparn i
309
- ```
156
+ # Add debt
157
+ sparn debt add "Refactor auth middleware" --severity P1 --tokens 5000
310
158
 
311
- **Features:**
312
- - **Configuration Wizard** - Guided prompts for all settings
313
- - **Optimization Preview** - Test optimization with file preview and confirmation
314
- - **Stats Dashboard** - Beautiful metrics display with multiple views
315
- - **Memory Consolidation** - Interactive cleanup with confirmation
316
- - **Quick Actions** - Common tasks and shortcuts
159
+ # Add with due date and affected files
160
+ sparn debt add "Fix N+1 queries" --severity P0 --due 2026-03-01 --files src/db.ts
317
161
 
318
- **Output:**
319
- ```
320
- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
321
- 🧠 Sparn Interactive Mode
322
- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
323
- Conversational configuration and exploration
324
-
325
- ? What would you like to do?
326
- ⚙️ Configure Settings
327
- 🔍 Optimize Preview
328
- 📊 Stats Dashboard
329
- 🧹 Memory Consolidation
330
- 🚀 Quick Actions
331
- ❯ ❌ Exit
332
- ```
162
+ # List all debt
163
+ sparn debt list
333
164
 
334
- ---
165
+ # List overdue items
166
+ sparn debt list --overdue
335
167
 
336
- ## Programmatic API
168
+ # Mark as resolved
169
+ sparn debt resolve <id>
337
170
 
338
- ### Basic Optimization
339
-
340
- ```typescript
341
- import { createSparsePruner, estimateTokens } from 'sparn';
342
-
343
- const context = `
344
- Line 1: Old file content
345
- Line 2: More old content
346
- Line 3: Error: Connection timeout
347
- ...
348
- `;
349
-
350
- const pruner = createSparsePruner();
351
- const result = pruner.prune(context, 5);
352
-
353
- console.log(`Tokens before: ${estimateTokens(context)}`);
354
- console.log(`Tokens after: ${estimateTokens(result.prunedContext)}`);
355
- console.log(`Kept ${result.entriesKept.length} entries`);
171
+ # View stats
172
+ sparn debt stats
356
173
  ```
357
174
 
358
- ### Using Memory Store
359
-
360
- ```typescript
361
- import { createKVMemory, type MemoryEntry } from 'sparn';
362
-
363
- const memory = await createKVMemory('./.sparn/memory.db');
364
-
365
- // Store entry
366
- const entry: MemoryEntry = {
367
- id: crypto.randomUUID(),
368
- content: 'Error: Connection timeout',
369
- hash: hashContent('Error: Connection timeout'),
370
- timestamp: Date.now() / 1000,
371
- score: 0.85,
372
- ttl: 86400,
373
- state: 'active',
374
- accessCount: 0,
375
- tags: ['error'],
376
- metadata: { severity: 'high' },
377
- isBTSP: true,
378
- };
379
-
380
- await memory.put(entry);
381
-
382
- // Query active entries
383
- const activeEntries = await memory.query({
384
- state: 'active',
385
- minScore: 0.7,
386
- limit: 10,
387
- });
388
-
389
- await memory.close();
390
- ```
391
-
392
- ### Full Pipeline
393
-
394
- ```typescript
395
- import {
396
- createSparsePruner,
397
- createEngramScorer,
398
- createConfidenceStates,
399
- createBTSPEmbedder,
400
- estimateTokens,
401
- hashContent,
402
- } from 'sparn';
403
-
404
- async function optimizeWithPipeline(context: string) {
405
- const pruner = createSparsePruner({ threshold: 5 });
406
- const scorer = createEngramScorer({ defaultTTL: 24 });
407
- const states = createConfidenceStates();
408
- const btsp = createBTSPEmbedder();
409
-
410
- // 1. Sparse pruning
411
- const pruneResult = pruner.prune(context, 5);
412
-
413
- // 2. Create entries
414
- const entries = pruneResult.entriesKept.map(line => {
415
- const isBTSPEntry = btsp.detectBTSP(line);
416
- return {
417
- id: crypto.randomUUID(),
418
- content: line,
419
- hash: hashContent(line),
420
- timestamp: Date.now() / 1000,
421
- score: isBTSPEntry ? 1.0 : 0.5,
422
- ttl: isBTSPEntry ? 172800 : 86400,
423
- state: isBTSPEntry ? 'active' : 'ready',
424
- accessCount: 0,
425
- tags: [],
426
- metadata: {},
427
- isBTSP: isBTSPEntry,
428
- };
429
- });
430
-
431
- // 3. Score and transition
432
- const scoredEntries = entries.map(entry => {
433
- const score = scorer.calculateScore(entry);
434
- return states.transition({ ...entry, score });
435
- });
436
-
437
- return {
438
- optimizedContext: scoredEntries.map(e => e.content).join('\n'),
439
- tokensBefore: estimateTokens(context),
440
- tokensAfter: estimateTokens(pruneResult.prunedContext),
441
- distribution: states.getDistribution(scoredEntries),
442
- };
443
- }
444
- ```
445
-
446
- ---
447
-
448
175
  ## Configuration
449
176
 
450
- Edit `.sparn/config.yaml`:
177
+ After `sparn init`, edit `.sparn/config.yaml`:
451
178
 
452
179
  ```yaml
453
- # Pruning settings
454
180
  pruning:
455
- threshold: 5 # Keep top 5% (1-100)
456
- aggressiveness: 50 # TF-IDF weight (0-100)
181
+ threshold: 5 # Keep top 5% of context (1-100)
182
+ aggressiveness: 50 # How aggressively to prune (0-100)
457
183
 
458
- # Decay settings
459
184
  decay:
460
- defaultTTL: 24 # Hours (default: 24h)
461
- decayThreshold: 0.95 # Prune threshold (0.0-1.0)
185
+ defaultTTL: 24 # Hours before context starts fading
186
+ decayThreshold: 0.95 # Score threshold for pruning
462
187
 
463
- # Confidence state thresholds
464
188
  states:
465
- activeThreshold: 0.7 # Score > 0.7 active
466
- readyThreshold: 0.3 # Score 0.3-0.7 ready
467
-
468
- # Agent adapter
469
- agent: generic # claude-code | generic
470
-
471
- # UI settings
472
- ui:
473
- colors: true # Colored output
474
- sounds: false # Sound effects
475
- verbose: false # Verbose logging
476
-
477
- # Auto-consolidation (hours, or null for manual)
478
- autoConsolidate: null
479
- ```
480
-
481
- ---
482
-
483
- ## Common Workflows
189
+ activeThreshold: 0.7 # Score >= 0.7 = active (kept)
190
+ readyThreshold: 0.3 # Score 0.3-0.69 = ready (may be kept)
484
191
 
485
- ### Claude Code Integration
192
+ realtime:
193
+ tokenBudget: 40000 # Target token count after optimization
194
+ autoOptimizeThreshold: 60000 # Trigger optimization above this
195
+ debounceMs: 5000 # Wait time between optimizations
486
196
 
487
- ```bash
488
- # Generate context file
489
- cat src/**/*.ts > context.txt
490
-
491
- # Optimize
492
- sparn optimize --input context.txt --output optimized.txt
493
- ```
494
-
495
- ### Daily Cleanup
496
-
497
- Schedule via cron:
498
-
499
- ```bash
500
- # Add to crontab
501
- 0 2 * * * cd /your-project && sparn consolidate
197
+ agent: generic # or claude-code
502
198
  ```
503
199
 
504
- Or configure auto-consolidation:
505
-
200
+ Or use the CLI:
506
201
  ```bash
507
- sparn config set autoConsolidate 24 # Every 24 hours
508
- ```
509
-
510
- ### Git Diff Optimization
511
-
512
- ```bash
513
- git diff | sparn optimize | less
514
-
515
- # Or via relay
516
- sparn relay git diff --stat
202
+ sparn config get pruning.threshold
203
+ sparn config set pruning.threshold 10
517
204
  ```
518
205
 
519
- ### Build Log Analysis
520
-
206
+ Or the interactive mode:
521
207
  ```bash
522
- sparn relay npm run build
523
- sparn relay cargo build --verbose
524
- sparn relay tsc --noEmit
208
+ sparn interactive
525
209
  ```
526
210
 
527
- ---
528
-
529
- ## Troubleshooting
211
+ ## Programmatic API
530
212
 
531
- ### "Not initialized" Error
213
+ ```typescript
214
+ import { createSparsePruner, estimateTokens } from '@ulrichc1/sparn';
532
215
 
533
- **Error:** `Error: .sparn/ not found. Run 'sparn init' first.`
216
+ const pruner = createSparsePruner({ threshold: 5 });
217
+ const result = pruner.prune(largeContext, 5);
534
218
 
535
- **Solution:**
536
- ```bash
537
- cd your-project/
538
- sparn init
219
+ console.log(`${estimateTokens(largeContext)} -> ${estimateTokens(result.prunedContext)} tokens`);
539
220
  ```
540
221
 
541
- ### Context Too Large
542
-
543
- **Error:** `Error: Context exceeds 500K tokens`
544
-
545
- **Solution:** Increase threshold or split context:
546
- ```bash
547
- sparn config set pruning.threshold 10 # Keep top 10%
548
- ```
222
+ The full API exports all core modules: `createDependencyGraph`, `createSearchEngine`, `createWorkflowPlanner`, `createDocsGenerator`, `createDebtTracker`, `createKVMemory`, `createBudgetPrunerFromConfig`, `createIncrementalOptimizer`, and more.
549
223
 
550
- ### No Entries Active
224
+ ## MCP Server
551
225
 
552
- **Problem:** All entries marked silent, no context retained
226
+ Sparn can run as an MCP server for Claude Desktop or any MCP client:
553
227
 
554
- **Solution:** Lower active threshold:
555
228
  ```bash
556
- sparn config set states.activeThreshold 0.5
229
+ sparn mcp:server
557
230
  ```
558
231
 
559
- ### Database Locked
560
-
561
- **Error:** `Error: database is locked`
562
-
563
- **Solution:** Close other Sparn processes or consolidate:
564
- ```bash
565
- sparn consolidate # Unlocks and compacts database
566
- ```
232
+ Exposes three tools: `sparn_optimize`, `sparn_stats`, `sparn_consolidate`.
567
233
 
568
- ---
234
+ ## How it works
569
235
 
570
- ## Best Practices
236
+ Sparn uses a multi-stage pipeline to decide what context to keep:
571
237
 
572
- 1. **Initialize Once** - Run `sparn init` once per project root
573
- 2. **Regular Consolidation** - Run `sparn consolidate` weekly or enable auto-consolidation
574
- 3. **Monitor Stats** - Check `sparn stats` to track savings
575
- 4. **Tune for Your Project**:
576
- - Code-heavy: Lower threshold (keep more context)
577
- - Log-heavy: Higher threshold (aggressive pruning)
578
- 5. **Use Relay for CLIs** - Wrap frequently-run commands in `sparn relay`
579
- 6. **BTSP for Errors** - Errors are automatically flagged as high-priority
580
- 7. **Tag Important Entries** - Use metadata for custom filtering (API)
581
-
582
- ---
583
-
584
- ## Performance
585
-
586
- - **Optimization Latency**: <500ms for 100K tokens
587
- - **CLI Startup Time**: <200ms (lazy loading)
588
- - **Memory Usage**: <100MB for typical workloads
589
- - **Token Reduction**: 60-90% average
590
- - **Database Operations**: <10ms per read/write
591
-
592
- ---
593
-
594
- ## Privacy & Security
595
-
596
- 🔒 **Your data stays on your machine**
597
-
598
- - ✅ No data transmission to external servers
599
- - ✅ No telemetry or analytics
600
- - ✅ No cloud storage
601
- - ✅ Full user control over data
602
- - ✅ GDPR compliant
603
-
604
- All processing happens locally. See [PRIVACY.md](./PRIVACY.md) and [SECURITY.md](./SECURITY.md) for details.
605
-
606
- ---
607
-
608
- ## Architecture
609
-
610
- ### Directory Structure
611
-
612
- ```
613
- sparn/
614
- ├── src/
615
- │ ├── core/ # Neuroscience modules (library)
616
- │ ├── adapters/ # Agent-specific adapters
617
- │ ├── cli/ # CLI commands
618
- │ ├── types/ # TypeScript interfaces
619
- │ └── utils/ # Shared utilities
620
- ├── tests/
621
- │ ├── unit/ # Unit tests
622
- │ └── integration/ # Integration tests
623
- └── dist/ # Build output (CJS/ESM/DTS)
624
- ```
625
-
626
- ### Tech Stack
627
-
628
- - **Language**: TypeScript (strict mode)
629
- - **Runtime**: Node.js 18+
630
- - **Build**: tsup
631
- - **Test**: vitest
632
- - **Lint**: biome
633
- - **Storage**: better-sqlite3
634
- - **CLI**: commander + chalk + ora + boxen
635
-
636
- ---
238
+ - **Relevance filtering** - Only the most relevant 2-5% of context carries the signal
239
+ - **Time-based decay** - Older context fades over time unless reinforced by reuse
240
+ - **Entry classification** - Entries are active, ready, or silent based on their score
241
+ - **Critical event detection** - Errors and stack traces get permanently flagged as important
242
+ - **Periodic consolidation** - Merges duplicate entries and cleans up stale data
637
243
 
638
244
  ## Development
639
245
 
640
- ### Setup
641
-
642
246
  ```bash
643
- # Clone the repository
644
247
  git clone https://github.com/ulrichc1/sparn.git
645
248
  cd sparn
646
-
647
- # Install dependencies
648
249
  npm install
649
-
650
- # Build the project
651
250
  npm run build
652
-
653
- # Run tests
654
- npm test
655
- ```
656
-
657
- ### Available Scripts
658
-
659
- ```bash
660
- npm run build # Build with tsup (CJS + ESM + DTS)
661
- npm run dev # Watch mode for development
662
- npm test # Run all tests with vitest
663
- npm run test:watch # Run tests in watch mode
664
- npm run lint # Lint code with biome
665
- npm run lint:fix # Fix linting issues automatically
666
- npm run typecheck # Type check without emit
667
- ```
668
-
669
- ### Running Locally
670
-
671
- ```bash
672
- # Build first
673
- npm run build
674
-
675
- # Test CLI commands
676
- node dist/cli/index.js init
677
- node dist/cli/index.js --help
678
-
679
- # Or link globally for testing
680
- npm link
681
- sparn --version
682
- ```
683
-
684
- ### Constitution Compliance
685
-
686
- All code must follow the [Sparn Constitution](.specify/memory/constitution.md):
687
-
688
- 1. **CLI-First, Library-Second** - CLI commands wrap library modules
689
- 2. **Neuroscience Fidelity** - Code maps to documented brain mechanisms
690
- 3. **Test-First Development (TDD)** - Tests FIRST, then implementation
691
- 4. **Agent-Agnostic Design** - Core modules have zero agent knowledge
692
- 5. **Complementary to RTK** - Optimizes input context, not CLI output
693
- 6. **Minimal Dependencies** - Only justified runtime dependencies
694
- 7. **Simplicity First** - Max 3 dir levels, YAML config, no complexity
695
- 8. **Brand Consistency** - Pink brain (#FF6B9D), Neural Cyan, Synapse Violet
696
- 9. **Production-Quality TypeScript** - Strict mode, no `any`, JSDoc on all public APIs
697
-
698
- ---
699
-
700
- ## Validation
701
-
702
- See [VALIDATION.md](./VALIDATION.md) for the complete validation guide.
703
-
704
- ### Quick Validation
705
-
706
- ```bash
707
- # 1. Build
708
- npm run build
709
-
710
- # 2. Test init command
711
- node dist/cli/index.js init
712
-
713
- # 3. Check files created
714
- ls -la .sparn/
715
- cat .sparn/config.yaml
716
-
717
- # 4. Run tests
718
- npm test
251
+ npm test # 479 tests
252
+ npm run lint
253
+ npm run typecheck
719
254
  ```
720
255
 
721
- ---
722
-
723
- ## Roadmap
724
-
725
- ### MVP (User Stories 1-2) ✅ COMPLETE
726
- - [X] US1: Initialize Sparn (`sparn init`)
727
- - [X] US2: Optimize context (`sparn optimize`)
728
-
729
- ### Post-MVP
730
- - [X] US3: View statistics (`sparn stats`) ✅ COMPLETE
731
- - [X] US4: Relay commands (`sparn relay <cmd>`) ✅ COMPLETE
732
- - [X] US5: Consolidate memory (`sparn consolidate`) ✅ COMPLETE
733
- - [X] US6: Configure behavior (`sparn config`) ✅ COMPLETE
734
- - [X] US7: Programmatic API (library usage) ✅ COMPLETE
735
-
736
- ---
737
-
738
- ## Contributing
739
-
740
- This project follows strict TDD (Test-Driven Development):
741
-
742
- 1. Write tests FIRST
743
- 2. Confirm tests FAIL (Red phase)
744
- 3. Implement to make tests PASS (Green phase)
745
- 4. Refactor while tests stay green
746
-
747
- See [CONTRIBUTING.md](./CONTRIBUTING.md) (coming soon) for details.
748
-
749
- ---
750
-
751
256
  ## License
752
257
 
753
- MIT © 2026 @ulrichc1
754
-
755
- ---
756
-
757
- ## Project Links
758
-
759
- - **Planning Docs**: `specs/001-sparn-core/`
760
- - **Constitution**: `.specify/memory/constitution.md`
761
- - **Tasks**: `specs/001-sparn-core/tasks.md`
762
- - **Validation Guide**: `VALIDATION.md`
763
-
764
- ---
258
+ MIT