sharp-context 0.1.0__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,301 @@
1
+ Metadata-Version: 2.4
2
+ Name: sharp-context
3
+ Version: 0.2.0
4
+ Summary: Information-theoretic context optimization for AI coding agents. Knapsack-optimal token budgeting, Shannon entropy scoring, SimHash dedup, predictive pre-fetch. MCP server.
5
+ Project-URL: Homepage, https://github.com/juyterman1000/sharp-context
6
+ Project-URL: Documentation, https://github.com/juyterman1000/sharp-context#readme
7
+ Project-URL: Repository, https://github.com/juyterman1000/sharp-context
8
+ Project-URL: Bug Tracker, https://github.com/juyterman1000/sharp-context/issues
9
+ Project-URL: Full Framework, https://github.com/juyterman1000/ebbiforge
10
+ Author-email: Ebbiforge Team <fastrunner10090@gmail.com>
11
+ License: MIT
12
+ Keywords: agentic-ai,checkpoint,claude,context-optimization,copilot,cursor,deduplication,entropy,knapsack,llm,mcp,token-cost
13
+ Classifier: Development Status :: 3 - Alpha
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: License :: OSI Approved :: MIT License
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
22
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
23
+ Requires-Python: >=3.10
24
+ Requires-Dist: mcp>=1.0.0
25
+ Provides-Extra: memory
26
+ Requires-Dist: hippocampus-sharp-memory>=1.0.0; extra == 'memory'
27
+ Description-Content-Type: text/markdown
28
+
29
+ # SharpContext
30
+
31
+ **Information-theoretic context optimization for AI coding agents.**
32
+
33
+ Every AI coding tool manages context with dumb FIFO truncation — stuffing tokens until the window is full, then cutting from the top. SharpContext applies mathematics to select the **optimal** context subset.
34
+
35
+ ```
36
+ pip install sharp-context
37
+ ```
38
+
39
+ ## Architecture
40
+
41
+ Hybrid Rust + Python: CPU-intensive math (knapsack DP, entropy, SimHash, LSH, dependency graph) runs in Rust via PyO3 for 50-100x speedup. MCP protocol and orchestration run in Python via FastMCP. Pure Python fallbacks activate automatically if the Rust extension isn't available.
42
+
43
+ ## What It Does
44
+
45
+ An MCP server that sits between your AI coding tool and the LLM, providing:
46
+
47
+ | Engine | What it does | How it works |
48
+ |--------|-------------|--------------|
49
+ | **Knapsack Optimizer** | Selects mathematically optimal context subset | 0/1 Knapsack DP with budget quantization (N ≤ 2000), greedy fallback (N > 2000) |
50
+ | **Entropy Scorer** | Measures information density per fragment | Shannon entropy (40%) + boilerplate detection (30%) + cross-fragment multi-scale n-gram redundancy (30%) |
51
+ | **SimHash Dedup** | Catches near-duplicate content in O(1) | 64-bit SimHash fingerprints with 4-band LSH bucketing, Hamming threshold = 3 |
52
+ | **Multi-Probe LSH Index** | Sub-linear semantic recall over 100K+ fragments | 12-table LSH with 10-bit sampling + 3-neighbor multi-probe queries |
53
+ | **Dependency Graph** | Pulls in related code fragments together | Symbol table + auto-linking (imports, type refs, function calls) + two-pass knapsack refinement |
54
+ | **Predictive Pre-fetch** | Pre-loads context before the agent asks | Static import analysis + test file inference + learned co-access patterns |
55
+ | **Checkpoint & Resume** | Crash recovery for multi-step tasks | Gzipped JSON state serialization (~100 KB per checkpoint) |
56
+ | **Feedback Loop** | Learns which context leads to good outputs | Wilson score lower-bound confidence intervals (same formula as Reddit ranking) |
57
+ | **Context Ordering** | Orders fragments for optimal LLM attention | Pinned → criticality level → dependency count → relevance score |
58
+ | **Guardrails** | Auto-pins safety-critical files, classifies tasks | Criticality levels (Safety/Critical/Important/Normal) + task-aware budget multipliers |
59
+ | **PRISM Optimizer** | Adapts scoring weights to the codebase | Anisotropic spectral optimization via Jacobi eigendecomposition on 4×4 covariance matrix |
60
+ | **Provenance Chain** | Detects hallucination risk in selected context | Tracks source verification + confidence scoring per fragment |
61
+
62
+ ## Setup
63
+
64
+ ### Cursor
65
+
66
+ Add to `.cursor/mcp.json`:
67
+
68
+ ```json
69
+ {
70
+ "mcpServers": {
71
+ "sharp-context": {
72
+ "command": "sharp-context"
73
+ }
74
+ }
75
+ }
76
+ ```
77
+
78
+ ### Claude Code
79
+
80
+ ```bash
81
+ claude mcp add sharp-context -- sharp-context
82
+ ```
83
+
84
+ ### Cline / Any MCP Client
85
+
86
+ ```json
87
+ {
88
+ "sharp-context": {
89
+ "command": "sharp-context",
90
+ "args": []
91
+ }
92
+ }
93
+ ```
94
+
95
+ ## MCP Tools
96
+
97
+ ### `remember_fragment`
98
+ Store context with auto-dedup, entropy scoring, dependency linking, and criticality detection.
99
+
100
+ ```
101
+ remember_fragment(content="def process_payment(...)...", source="payments.py", token_count=45)
102
+ → {"status": "ingested", "entropy_score": 0.82}
103
+
104
+ remember_fragment(content="def process_payment(...)...") # same content
105
+ → {"status": "duplicate", "duplicate_of": "a1b2c3", "tokens_saved": 45}
106
+ ```
107
+
108
+ ### `optimize_context`
109
+ Select the optimal context subset for a token budget. Includes dependency boosting, ε-greedy exploration, context sufficiency scoring, and provenance metadata.
110
+
111
+ ```
112
+ optimize_context(token_budget=128000, query="fix payment bug")
113
+ → {
114
+ "selected_fragments": [...],
115
+ "optimization_stats": {"method": "exact_dp", "budget_utilization": 0.73},
116
+ "tokens_saved_this_call": 42000,
117
+ "sufficiency": 0.91,
118
+ "hallucination_risk": "low"
119
+ }
120
+ ```
121
+
122
+ ### `recall_relevant`
123
+ Sub-linear semantic recall via multi-probe LSH. Falls back to brute-force scan on cold start.
124
+
125
+ ```
126
+ recall_relevant(query="database connection pooling", top_k=5)
127
+ → [{"fragment_id": "...", "relevance": 0.87, "content": "..."}]
128
+ ```
129
+
130
+ ### `record_outcome`
131
+ Feed the Wilson score feedback loop. Adjusts fragment scoring multipliers in the range [0.5, 2.0].
132
+
133
+ ```
134
+ record_outcome(fragment_ids=["a1b2c3", "d4e5f6"], success=true)
135
+ → {"status": "recorded", "fragments_updated": 2}
136
+ ```
137
+
138
+ ### `explain_context`
139
+ Per-fragment scoring breakdown with sufficiency analysis and exploration swap log.
140
+
141
+ ```
142
+ explain_context()
143
+ → {
144
+ "fragments": [{"id": "...", "recency": 0.9, "frequency": 0.3, "semantic": 0.7, "entropy": 0.8}],
145
+ "sufficiency": 0.91,
146
+ "exploration_swaps": 1
147
+ }
148
+ ```
149
+
150
+ ### `checkpoint_state` / `resume_state`
151
+ Save and restore full session state — fragments, dedup index, co-access patterns, feedback scores.
152
+
153
+ ```
154
+ checkpoint_state(task_description="Refactoring auth module", current_step="Step 5/8")
155
+ → {"status": "checkpoint_saved", "fragments_saved": 47}
156
+
157
+ resume_state()
158
+ → {"status": "resumed", "restored_fragments": 47, "metadata": {"step": "Step 5/8"}}
159
+ ```
160
+
161
+ ### `prefetch_related`
162
+ Predict and pre-load likely-needed context using import analysis, test file inference, and co-access history.
163
+
164
+ ```
165
+ prefetch_related(file_path="src/payments.py", source_content="from utils import...")
166
+ → [{"path": "src/utils.py", "reason": "import", "confidence": 0.70}]
167
+ ```
168
+
169
+ ### `get_stats`
170
+ Session statistics and cost savings.
171
+
172
+ ```
173
+ get_stats()
174
+ → {
175
+ "fragments": 142,
176
+ "total_tokens": 384000,
177
+ "savings": {
178
+ "total_tokens_saved": 284000,
179
+ "total_duplicates_caught": 12,
180
+ "estimated_cost_saved_usd": 0.85
181
+ }
182
+ }
183
+ ```
184
+
185
+ ## The Math
186
+
187
+ ### Multi-Dimensional Relevance Scoring
188
+
189
+ Each fragment is scored across four dimensions:
190
+
191
+ ```
192
+ r(f) = (w_rec · recency + w_freq · frequency + w_sem · semantic + w_ent · entropy)
193
+ / (w_rec + w_freq + w_sem + w_ent)
194
+ × feedback_multiplier
195
+ ```
196
+
197
+ Default weights: recency 0.30, frequency 0.25, semantic 0.25, entropy 0.20.
198
+
199
+ - **Recency**: Ebbinghaus forgetting curve — `exp(-ln(2) × Δt / half_life)`, half_life = 15 turns
200
+ - **Frequency**: Normalized access count (spaced repetition boost)
201
+ - **Semantic similarity**: SimHash Hamming distance to query, normalized to [0, 1]
202
+ - **Information density**: Shannon entropy + boilerplate + redundancy (see below)
203
+
204
+ ### Knapsack Context Selection
205
+
206
+ Context selection is the 0/1 Knapsack Problem:
207
+
208
+ ```
209
+ Maximize: Σ r(fᵢ) · x(fᵢ) for selected fragments
210
+ Subject to: Σ c(fᵢ) · x(fᵢ) ≤ B (token budget)
211
+ ```
212
+
213
+ **Two strategies** based on fragment count:
214
+ - **N ≤ 2000**: Exact DP with budget quantization into 1000 bins — O(N × 1000)
215
+ - **N > 2000**: Greedy density sort — O(N log N), Dantzig 0.5-optimality guarantee
216
+
217
+ Pinned fragments (safety-critical files, config files) are always included; remaining budget is allocated via DP/greedy.
218
+
219
+ ### Shannon Entropy Scoring
220
+
221
+ Three components combined:
222
+
223
+ ```
224
+ score = 0.40 × normalized_entropy + 0.30 × (1 - boilerplate_ratio) + 0.30 × (1 - redundancy)
225
+ ```
226
+
227
+ - **Shannon entropy** (40%): `H = -Σ p(char) · log₂(p(char))`, normalized by 6.0 bits/char. Stack-allocated 256-byte histogram, single O(n) pass.
228
+ - **Boilerplate detection** (30%): Pattern matching for imports, pass, dunder methods, closing delimiters.
229
+ - **Cross-fragment redundancy** (30%): Multi-scale n-gram overlap with adaptive weights by fragment length — bigram-heavy for short fragments (<20 words), 4-gram-heavy for long fragments (>100 words). Parallelized with rayon.
230
+
231
+ ### SimHash Deduplication
232
+
233
+ 64-bit fingerprints from word trigrams hashed via MD5:
234
+ - Hamming distance ≤ 3 → near-duplicate
235
+ - 4-band LSH bucketing for O(1) candidate lookup
236
+ - Separate 12-table multi-probe LSH index for semantic recall (~3 μs over 100K fragments)
237
+
238
+ ### Dependency Graph
239
+
240
+ Auto-linking via source analysis:
241
+ - **Imports** (strength 1.0): Python `from X import Y`, Rust `use`, JS `import`
242
+ - **Type references** (0.9): Type annotations, isinstance checks
243
+ - **Function calls** (0.7): General identifier usage matching against symbol table
244
+ - **Same module** (0.5): Co-located definitions
245
+
246
+ Two-pass knapsack refinement: initial selection → boost dependencies of selected fragments → re-optimize.
247
+
248
+ ### Task-Aware Budget Multipliers
249
+
250
+ ```
251
+ Bug tracing / debugging → 1.5× budget
252
+ Exploration / understanding → 1.3× budget
253
+ Refactoring / code review → 1.0× budget
254
+ Testing → 0.8× budget
255
+ Code generation → 0.7× budget
256
+ Documentation → 0.6× budget
257
+ ```
258
+
259
+ ### PRISM Spectral Optimizer
260
+
261
+ Tracks a 4×4 covariance matrix over scoring dimensions [recency, frequency, semantic, entropy] with EMA updates (β=0.95). Jacobi eigendecomposition finds principal axes. Anisotropic spectral gain dampens noisy dimensions and amplifies clean signals — automatic learning rate adaptation without hyperparameter tuning.
262
+
263
+ ## Configuration
264
+
265
+ ```python
266
+ SharpContextConfig(
267
+ default_token_budget=128_000, # GPT-4 Turbo equivalent
268
+ max_fragments=10_000, # session fragment cap
269
+ weight_recency=0.30, # scoring weights (sum to 1.0)
270
+ weight_frequency=0.25,
271
+ weight_semantic_sim=0.25,
272
+ weight_entropy=0.20,
273
+ decay_half_life_turns=15, # Ebbinghaus half-life
274
+ min_relevance_threshold=0.05, # auto-evict below this
275
+ dedup_similarity_threshold=0.92,
276
+ prefetch_depth=2,
277
+ max_prefetch_fragments=10,
278
+ auto_checkpoint_interval=5, # checkpoint every N tool calls
279
+ )
280
+ ```
281
+
282
+ ## References
283
+
284
+ - Shannon (1948) — Information Theory
285
+ - Charikar (2002) — SimHash
286
+ - Ebbinghaus (1885) — Forgetting Curve
287
+ - Dantzig (1957) — Greedy Knapsack Approximation
288
+ - Wilson (1927) — Score Confidence Intervals
289
+ - ICPC (arXiv 2025) — In-context Prompt Compression
290
+ - Proximity (arXiv 2026) — LSH-bucketed Semantic Caching
291
+ - RCC (ICLR 2025) — Recurrent Context Compression
292
+ - ILRe (ICLR 2026) — Intermediate Layer Retrieval
293
+ - Agentic Plan Caching (arXiv 2025)
294
+
295
+ ## Part of the Ebbiforge Ecosystem
296
+
297
+ SharpContext integrates with [hippocampus-sharp-memory](https://pypi.org/project/hippocampus-sharp-memory/) for persistent memory and [Ebbiforge](https://pypi.org/project/ebbiforge/) for TF embeddings and RL weight learning. Both are optional — SharpContext works standalone with pure Python fallbacks.
298
+
299
+ ## License
300
+
301
+ MIT
@@ -0,0 +1,273 @@
1
+ # SharpContext
2
+
3
+ **Information-theoretic context optimization for AI coding agents.**
4
+
5
+ Every AI coding tool manages context with dumb FIFO truncation — stuffing tokens until the window is full, then cutting from the top. SharpContext applies mathematics to select the **optimal** context subset.
6
+
7
+ ```
8
+ pip install sharp-context
9
+ ```
10
+
11
+ ## Architecture
12
+
13
+ Hybrid Rust + Python: CPU-intensive math (knapsack DP, entropy, SimHash, LSH, dependency graph) runs in Rust via PyO3 for 50-100x speedup. MCP protocol and orchestration run in Python via FastMCP. Pure Python fallbacks activate automatically if the Rust extension isn't available.
14
+
15
+ ## What It Does
16
+
17
+ An MCP server that sits between your AI coding tool and the LLM, providing:
18
+
19
+ | Engine | What it does | How it works |
20
+ |--------|-------------|--------------|
21
+ | **Knapsack Optimizer** | Selects mathematically optimal context subset | 0/1 Knapsack DP with budget quantization (N ≤ 2000), greedy fallback (N > 2000) |
22
+ | **Entropy Scorer** | Measures information density per fragment | Shannon entropy (40%) + boilerplate detection (30%) + cross-fragment multi-scale n-gram redundancy (30%) |
23
+ | **SimHash Dedup** | Catches near-duplicate content in O(1) | 64-bit SimHash fingerprints with 4-band LSH bucketing, Hamming threshold = 3 |
24
+ | **Multi-Probe LSH Index** | Sub-linear semantic recall over 100K+ fragments | 12-table LSH with 10-bit sampling + 3-neighbor multi-probe queries |
25
+ | **Dependency Graph** | Pulls in related code fragments together | Symbol table + auto-linking (imports, type refs, function calls) + two-pass knapsack refinement |
26
+ | **Predictive Pre-fetch** | Pre-loads context before the agent asks | Static import analysis + test file inference + learned co-access patterns |
27
+ | **Checkpoint & Resume** | Crash recovery for multi-step tasks | Gzipped JSON state serialization (~100 KB per checkpoint) |
28
+ | **Feedback Loop** | Learns which context leads to good outputs | Wilson score lower-bound confidence intervals (same formula as Reddit ranking) |
29
+ | **Context Ordering** | Orders fragments for optimal LLM attention | Pinned → criticality level → dependency count → relevance score |
30
+ | **Guardrails** | Auto-pins safety-critical files, classifies tasks | Criticality levels (Safety/Critical/Important/Normal) + task-aware budget multipliers |
31
+ | **PRISM Optimizer** | Adapts scoring weights to the codebase | Anisotropic spectral optimization via Jacobi eigendecomposition on 4×4 covariance matrix |
32
+ | **Provenance Chain** | Detects hallucination risk in selected context | Tracks source verification + confidence scoring per fragment |
33
+
34
+ ## Setup
35
+
36
+ ### Cursor
37
+
38
+ Add to `.cursor/mcp.json`:
39
+
40
+ ```json
41
+ {
42
+ "mcpServers": {
43
+ "sharp-context": {
44
+ "command": "sharp-context"
45
+ }
46
+ }
47
+ }
48
+ ```
49
+
50
+ ### Claude Code
51
+
52
+ ```bash
53
+ claude mcp add sharp-context -- sharp-context
54
+ ```
55
+
56
+ ### Cline / Any MCP Client
57
+
58
+ ```json
59
+ {
60
+ "sharp-context": {
61
+ "command": "sharp-context",
62
+ "args": []
63
+ }
64
+ }
65
+ ```
66
+
67
+ ## MCP Tools
68
+
69
+ ### `remember_fragment`
70
+ Store context with auto-dedup, entropy scoring, dependency linking, and criticality detection.
71
+
72
+ ```
73
+ remember_fragment(content="def process_payment(...)...", source="payments.py", token_count=45)
74
+ → {"status": "ingested", "entropy_score": 0.82}
75
+
76
+ remember_fragment(content="def process_payment(...)...") # same content
77
+ → {"status": "duplicate", "duplicate_of": "a1b2c3", "tokens_saved": 45}
78
+ ```
79
+
80
+ ### `optimize_context`
81
+ Select the optimal context subset for a token budget. Includes dependency boosting, ε-greedy exploration, context sufficiency scoring, and provenance metadata.
82
+
83
+ ```
84
+ optimize_context(token_budget=128000, query="fix payment bug")
85
+ → {
86
+ "selected_fragments": [...],
87
+ "optimization_stats": {"method": "exact_dp", "budget_utilization": 0.73},
88
+ "tokens_saved_this_call": 42000,
89
+ "sufficiency": 0.91,
90
+ "hallucination_risk": "low"
91
+ }
92
+ ```
93
+
94
+ ### `recall_relevant`
95
+ Sub-linear semantic recall via multi-probe LSH. Falls back to brute-force scan on cold start.
96
+
97
+ ```
98
+ recall_relevant(query="database connection pooling", top_k=5)
99
+ → [{"fragment_id": "...", "relevance": 0.87, "content": "..."}]
100
+ ```
101
+
102
+ ### `record_outcome`
103
+ Feed the Wilson score feedback loop. Adjusts fragment scoring multipliers in the range [0.5, 2.0].
104
+
105
+ ```
106
+ record_outcome(fragment_ids=["a1b2c3", "d4e5f6"], success=true)
107
+ → {"status": "recorded", "fragments_updated": 2}
108
+ ```
109
+
110
+ ### `explain_context`
111
+ Per-fragment scoring breakdown with sufficiency analysis and exploration swap log.
112
+
113
+ ```
114
+ explain_context()
115
+ → {
116
+ "fragments": [{"id": "...", "recency": 0.9, "frequency": 0.3, "semantic": 0.7, "entropy": 0.8}],
117
+ "sufficiency": 0.91,
118
+ "exploration_swaps": 1
119
+ }
120
+ ```
121
+
122
+ ### `checkpoint_state` / `resume_state`
123
+ Save and restore full session state — fragments, dedup index, co-access patterns, feedback scores.
124
+
125
+ ```
126
+ checkpoint_state(task_description="Refactoring auth module", current_step="Step 5/8")
127
+ → {"status": "checkpoint_saved", "fragments_saved": 47}
128
+
129
+ resume_state()
130
+ → {"status": "resumed", "restored_fragments": 47, "metadata": {"step": "Step 5/8"}}
131
+ ```
132
+
133
+ ### `prefetch_related`
134
+ Predict and pre-load likely-needed context using import analysis, test file inference, and co-access history.
135
+
136
+ ```
137
+ prefetch_related(file_path="src/payments.py", source_content="from utils import...")
138
+ → [{"path": "src/utils.py", "reason": "import", "confidence": 0.70}]
139
+ ```
140
+
141
+ ### `get_stats`
142
+ Session statistics and cost savings.
143
+
144
+ ```
145
+ get_stats()
146
+ → {
147
+ "fragments": 142,
148
+ "total_tokens": 384000,
149
+ "savings": {
150
+ "total_tokens_saved": 284000,
151
+ "total_duplicates_caught": 12,
152
+ "estimated_cost_saved_usd": 0.85
153
+ }
154
+ }
155
+ ```
156
+
157
+ ## The Math
158
+
159
+ ### Multi-Dimensional Relevance Scoring
160
+
161
+ Each fragment is scored across four dimensions:
162
+
163
+ ```
164
+ r(f) = (w_rec · recency + w_freq · frequency + w_sem · semantic + w_ent · entropy)
165
+ / (w_rec + w_freq + w_sem + w_ent)
166
+ × feedback_multiplier
167
+ ```
168
+
169
+ Default weights: recency 0.30, frequency 0.25, semantic 0.25, entropy 0.20.
170
+
171
+ - **Recency**: Ebbinghaus forgetting curve — `exp(-ln(2) × Δt / half_life)`, half_life = 15 turns
172
+ - **Frequency**: Normalized access count (spaced repetition boost)
173
+ - **Semantic similarity**: SimHash Hamming distance to query, normalized to [0, 1]
174
+ - **Information density**: Shannon entropy + boilerplate + redundancy (see below)
175
+
176
+ ### Knapsack Context Selection
177
+
178
+ Context selection is the 0/1 Knapsack Problem:
179
+
180
+ ```
181
+ Maximize: Σ r(fᵢ) · x(fᵢ) for selected fragments
182
+ Subject to: Σ c(fᵢ) · x(fᵢ) ≤ B (token budget)
183
+ ```
184
+
185
+ **Two strategies** based on fragment count:
186
+ - **N ≤ 2000**: Exact DP with budget quantization into 1000 bins — O(N × 1000)
187
+ - **N > 2000**: Greedy density sort — O(N log N), Dantzig 0.5-optimality guarantee
188
+
189
+ Pinned fragments (safety-critical files, config files) are always included; remaining budget is allocated via DP/greedy.
190
+
191
+ ### Shannon Entropy Scoring
192
+
193
+ Three components combined:
194
+
195
+ ```
196
+ score = 0.40 × normalized_entropy + 0.30 × (1 - boilerplate_ratio) + 0.30 × (1 - redundancy)
197
+ ```
198
+
199
+ - **Shannon entropy** (40%): `H = -Σ p(char) · log₂(p(char))`, normalized by 6.0 bits/char. Stack-allocated 256-byte histogram, single O(n) pass.
200
+ - **Boilerplate detection** (30%): Pattern matching for imports, pass, dunder methods, closing delimiters.
201
+ - **Cross-fragment redundancy** (30%): Multi-scale n-gram overlap with adaptive weights by fragment length — bigram-heavy for short fragments (<20 words), 4-gram-heavy for long fragments (>100 words). Parallelized with rayon.
202
+
203
+ ### SimHash Deduplication
204
+
205
+ 64-bit fingerprints from word trigrams hashed via MD5:
206
+ - Hamming distance ≤ 3 → near-duplicate
207
+ - 4-band LSH bucketing for O(1) candidate lookup
208
+ - Separate 12-table multi-probe LSH index for semantic recall (~3 μs over 100K fragments)
209
+
210
+ ### Dependency Graph
211
+
212
+ Auto-linking via source analysis:
213
+ - **Imports** (strength 1.0): Python `from X import Y`, Rust `use`, JS `import`
214
+ - **Type references** (0.9): Type annotations, isinstance checks
215
+ - **Function calls** (0.7): General identifier usage matching against symbol table
216
+ - **Same module** (0.5): Co-located definitions
217
+
218
+ Two-pass knapsack refinement: initial selection → boost dependencies of selected fragments → re-optimize.
219
+
220
+ ### Task-Aware Budget Multipliers
221
+
222
+ ```
223
+ Bug tracing / debugging → 1.5× budget
224
+ Exploration / understanding → 1.3× budget
225
+ Refactoring / code review → 1.0× budget
226
+ Testing → 0.8× budget
227
+ Code generation → 0.7× budget
228
+ Documentation → 0.6× budget
229
+ ```
230
+
231
+ ### PRISM Spectral Optimizer
232
+
233
+ Tracks a 4×4 covariance matrix over scoring dimensions [recency, frequency, semantic, entropy] with EMA updates (β=0.95). Jacobi eigendecomposition finds principal axes. Anisotropic spectral gain dampens noisy dimensions and amplifies clean signals — automatic learning rate adaptation without hyperparameter tuning.
234
+
235
+ ## Configuration
236
+
237
+ ```python
238
+ SharpContextConfig(
239
+ default_token_budget=128_000, # GPT-4 Turbo equivalent
240
+ max_fragments=10_000, # session fragment cap
241
+ weight_recency=0.30, # scoring weights (sum to 1.0)
242
+ weight_frequency=0.25,
243
+ weight_semantic_sim=0.25,
244
+ weight_entropy=0.20,
245
+ decay_half_life_turns=15, # Ebbinghaus half-life
246
+ min_relevance_threshold=0.05, # auto-evict below this
247
+ dedup_similarity_threshold=0.92,
248
+ prefetch_depth=2,
249
+ max_prefetch_fragments=10,
250
+ auto_checkpoint_interval=5, # checkpoint every N tool calls
251
+ )
252
+ ```
253
+
254
+ ## References
255
+
256
+ - Shannon (1948) — Information Theory
257
+ - Charikar (2002) — SimHash
258
+ - Ebbinghaus (1885) — Forgetting Curve
259
+ - Dantzig (1957) — Greedy Knapsack Approximation
260
+ - Wilson (1927) — Score Confidence Intervals
261
+ - ICPC (arXiv 2025) — In-context Prompt Compression
262
+ - Proximity (arXiv 2026) — LSH-bucketed Semantic Caching
263
+ - RCC (ICLR 2025) — Recurrent Context Compression
264
+ - ILRe (ICLR 2026) — Intermediate Layer Retrieval
265
+ - Agentic Plan Caching (arXiv 2025)
266
+
267
+ ## Part of the Ebbiforge Ecosystem
268
+
269
+ SharpContext integrates with [hippocampus-sharp-memory](https://pypi.org/project/hippocampus-sharp-memory/) for persistent memory and [Ebbiforge](https://pypi.org/project/ebbiforge/) for TF embeddings and RL weight learning. Both are optional — SharpContext works standalone with pure Python fallbacks.
270
+
271
+ ## License
272
+
273
+ MIT
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
4
 
5
5
  [project]
6
6
  name = "sharp-context"
7
- version = "0.1.0"
7
+ version = "0.2.0"
8
8
  description = "Information-theoretic context optimization for AI coding agents. Knapsack-optimal token budgeting, Shannon entropy scoring, SimHash dedup, predictive pre-fetch. MCP server."
9
9
  readme = "README.md"
10
10
  license = { text = "MIT" }
@@ -24,4 +24,4 @@ Quick Setup (Claude Code)::
24
24
 
25
25
  """
26
26
 
27
- __version__ = "0.1.0"
27
+ __version__ = "0.2.0"