entroly 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- entroly-0.2.0/.gitignore +75 -0
- entroly-0.2.0/PKG-INFO +302 -0
- entroly-0.2.0/README.md +273 -0
- entroly-0.2.0/entroly/__init__.py +27 -0
- entroly-0.2.0/entroly/adaptive_pruner.py +161 -0
- entroly-0.2.0/entroly/checkpoint.py +302 -0
- entroly-0.2.0/entroly/config.py +74 -0
- entroly-0.2.0/entroly/multimodal.py +880 -0
- entroly-0.2.0/entroly/prefetch.py +297 -0
- entroly-0.2.0/entroly/provenance.py +184 -0
- entroly-0.2.0/entroly/query_refiner.py +160 -0
- entroly-0.2.0/entroly/server.py +1171 -0
- entroly-0.2.0/npm/README.md +48 -0
- entroly-0.2.0/npm/index.js +33 -0
- entroly-0.2.0/npm/package.json +25 -0
- entroly-0.2.0/pyproject.toml +51 -0
entroly-0.2.0/.gitignore
ADDED
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# Node/JS
|
|
2
|
+
node_modules/
|
|
3
|
+
npm-debug.log*
|
|
4
|
+
.next/
|
|
5
|
+
web/.next/
|
|
6
|
+
|
|
7
|
+
# Editor
|
|
8
|
+
.vscode/
|
|
9
|
+
.idea/
|
|
10
|
+
|
|
11
|
+
# Python
|
|
12
|
+
__pycache__/
|
|
13
|
+
.venv/
|
|
14
|
+
*.pyc
|
|
15
|
+
.env
|
|
16
|
+
|
|
17
|
+
# Rust / Cargo
|
|
18
|
+
target/
|
|
19
|
+
Cargo.lock
|
|
20
|
+
|
|
21
|
+
# Data / Temp
|
|
22
|
+
*.log
|
|
23
|
+
.DS_Store
|
|
24
|
+
|
|
25
|
+
# Model caches / generated
|
|
26
|
+
.fastembed_cache/
|
|
27
|
+
server/.fastembed_cache/
|
|
28
|
+
*.onnx
|
|
29
|
+
*.lock
|
|
30
|
+
|
|
31
|
+
# Environment secrets (never commit)
|
|
32
|
+
web/.env.local
|
|
33
|
+
web/.env.production
|
|
34
|
+
.env.local
|
|
35
|
+
|
|
36
|
+
# =========================================================
|
|
37
|
+
# ALLOWLIST — only these directories are tracked
|
|
38
|
+
# =========================================================
|
|
39
|
+
|
|
40
|
+
# Core engine (Rust)
|
|
41
|
+
# openrustswarm-core/ — tracked
|
|
42
|
+
|
|
43
|
+
# Web dashboard (Next.js)
|
|
44
|
+
# web/ — tracked
|
|
45
|
+
|
|
46
|
+
# CogOps Skill (OpenClaw integration)
|
|
47
|
+
# cogops-skill/ — tracked
|
|
48
|
+
|
|
49
|
+
# WASM bridge
|
|
50
|
+
# cogops-wasm/ — tracked
|
|
51
|
+
|
|
52
|
+
# Python server
|
|
53
|
+
# server/ — tracked
|
|
54
|
+
|
|
55
|
+
# Public assets
|
|
56
|
+
# demo/, examples/, dist/ — tracked
|
|
57
|
+
|
|
58
|
+
# =========================================================
|
|
59
|
+
# EXCLUDED (internal / generated / one-off)
|
|
60
|
+
# =========================================================
|
|
61
|
+
openrustswarm/
|
|
62
|
+
benchmarks/
|
|
63
|
+
scripts/
|
|
64
|
+
docs/
|
|
65
|
+
hacker_news_launch.md
|
|
66
|
+
eval_*.py
|
|
67
|
+
test_*.py
|
|
68
|
+
medieval_demo.py
|
|
69
|
+
real_10k_log_swarm.py
|
|
70
|
+
CODEBASE_AUDIT.md
|
|
71
|
+
LAUNCH.md
|
|
72
|
+
*.jsonl
|
|
73
|
+
server/diagnose_api.py
|
|
74
|
+
server/verify_ui.py
|
|
75
|
+
server/run_task_help.txt
|
entroly-0.2.0/PKG-INFO
ADDED
|
@@ -0,0 +1,302 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: entroly
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Information-theoretic context optimization for AI coding agents. Knapsack-optimal token budgeting, Shannon entropy scoring, SimHash dedup, predictive pre-fetch. MCP server.
|
|
5
|
+
Project-URL: Homepage, https://github.com/juyterman1000/entroly
|
|
6
|
+
Project-URL: Documentation, https://github.com/juyterman1000/entroly#readme
|
|
7
|
+
Project-URL: Repository, https://github.com/juyterman1000/entroly
|
|
8
|
+
Project-URL: Bug Tracker, https://github.com/juyterman1000/entroly/issues
|
|
9
|
+
Project-URL: Full Framework, https://github.com/juyterman1000/ebbiforge
|
|
10
|
+
Author-email: Ebbiforge Team <fastrunner10090@gmail.com>
|
|
11
|
+
License: MIT
|
|
12
|
+
Keywords: agentic-ai,checkpoint,claude,context-optimization,copilot,cursor,deduplication,entropy,knapsack,llm,mcp,token-cost
|
|
13
|
+
Classifier: Development Status :: 4 - Beta
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
22
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
23
|
+
Requires-Python: >=3.10
|
|
24
|
+
Requires-Dist: entroly-core>=0.1.0
|
|
25
|
+
Requires-Dist: mcp>=1.0.0
|
|
26
|
+
Provides-Extra: memory
|
|
27
|
+
Requires-Dist: hippocampus-sharp-memory>=1.0.0; extra == 'memory'
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
30
|
+
# Entroly
|
|
31
|
+
|
|
32
|
+
**Information-theoretic context optimization for AI coding agents.**
|
|
33
|
+
|
|
34
|
+
Every AI coding tool manages context with dumb FIFO truncation — stuffing tokens until the window is full, then cutting from the top. Entroly applies mathematics to select the **optimal** context subset.
|
|
35
|
+
|
|
36
|
+
```
|
|
37
|
+
pip install entroly
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Architecture
|
|
41
|
+
|
|
42
|
+
Hybrid Rust + Python: CPU-intensive math (knapsack DP, entropy, SimHash, LSH, dependency graph) runs in Rust via PyO3 for 50-100x speedup. MCP protocol and orchestration run in Python via FastMCP. Pure Python fallbacks activate automatically if the Rust extension isn't available.
|
|
43
|
+
|
|
44
|
+
## What It Does
|
|
45
|
+
|
|
46
|
+
An MCP server that sits between your AI coding tool and the LLM, providing:
|
|
47
|
+
|
|
48
|
+
| Engine | What it does | How it works |
|
|
49
|
+
|--------|-------------|--------------|
|
|
50
|
+
| **Knapsack Optimizer** | Selects mathematically optimal context subset | 0/1 Knapsack DP with budget quantization (N ≤ 2000), greedy fallback (N > 2000) |
|
|
51
|
+
| **Entropy Scorer** | Measures information density per fragment | Shannon entropy (40%) + boilerplate detection (30%) + cross-fragment multi-scale n-gram redundancy (30%) |
|
|
52
|
+
| **SimHash Dedup** | Catches near-duplicate content in O(1) | 64-bit SimHash fingerprints with 4-band LSH bucketing, Hamming threshold = 3 |
|
|
53
|
+
| **Multi-Probe LSH Index** | Sub-linear semantic recall over 100K+ fragments | 12-table LSH with 10-bit sampling + 3-neighbor multi-probe queries |
|
|
54
|
+
| **Dependency Graph** | Pulls in related code fragments together | Symbol table + auto-linking (imports, type refs, function calls) + two-pass knapsack refinement |
|
|
55
|
+
| **Predictive Pre-fetch** | Pre-loads context before the agent asks | Static import analysis + test file inference + learned co-access patterns |
|
|
56
|
+
| **Checkpoint & Resume** | Crash recovery for multi-step tasks | Gzipped JSON state serialization (~100 KB per checkpoint) |
|
|
57
|
+
| **Feedback Loop** | Learns which context leads to good outputs | Wilson score lower-bound confidence intervals (same formula as Reddit ranking) |
|
|
58
|
+
| **Context Ordering** | Orders fragments for optimal LLM attention | Pinned → criticality level → dependency count → relevance score |
|
|
59
|
+
| **Guardrails** | Auto-pins safety-critical files, classifies tasks | Criticality levels (Safety/Critical/Important/Normal) + task-aware budget multipliers |
|
|
60
|
+
| **PRISM Optimizer** | Adapts scoring weights to the codebase | Anisotropic spectral optimization via Jacobi eigendecomposition on 4×4 covariance matrix |
|
|
61
|
+
| **Provenance Chain** | Detects hallucination risk in selected context | Tracks source verification + confidence scoring per fragment |
|
|
62
|
+
|
|
63
|
+
## Setup
|
|
64
|
+
|
|
65
|
+
### Cursor
|
|
66
|
+
|
|
67
|
+
Add to `.cursor/mcp.json`:
|
|
68
|
+
|
|
69
|
+
```json
|
|
70
|
+
{
|
|
71
|
+
"mcpServers": {
|
|
72
|
+
"entroly": {
|
|
73
|
+
"command": "entroly"
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
}
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
### Claude Code
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
claude mcp add entroly -- entroly
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### Cline / Any MCP Client
|
|
86
|
+
|
|
87
|
+
```json
|
|
88
|
+
{
|
|
89
|
+
"entroly": {
|
|
90
|
+
"command": "entroly",
|
|
91
|
+
"args": []
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
## MCP Tools
|
|
97
|
+
|
|
98
|
+
### `remember_fragment`
|
|
99
|
+
Store context with auto-dedup, entropy scoring, dependency linking, and criticality detection.
|
|
100
|
+
|
|
101
|
+
```
|
|
102
|
+
remember_fragment(content="def process_payment(...)...", source="payments.py", token_count=45)
|
|
103
|
+
→ {"status": "ingested", "entropy_score": 0.82}
|
|
104
|
+
|
|
105
|
+
remember_fragment(content="def process_payment(...)...") # same content
|
|
106
|
+
→ {"status": "duplicate", "duplicate_of": "a1b2c3", "tokens_saved": 45}
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
### `optimize_context`
|
|
110
|
+
Select the optimal context subset for a token budget. Includes dependency boosting, ε-greedy exploration, context sufficiency scoring, and provenance metadata.
|
|
111
|
+
|
|
112
|
+
```
|
|
113
|
+
optimize_context(token_budget=128000, query="fix payment bug")
|
|
114
|
+
→ {
|
|
115
|
+
"selected_fragments": [...],
|
|
116
|
+
"optimization_stats": {"method": "exact_dp", "budget_utilization": 0.73},
|
|
117
|
+
"tokens_saved_this_call": 42000,
|
|
118
|
+
"sufficiency": 0.91,
|
|
119
|
+
"hallucination_risk": "low"
|
|
120
|
+
}
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### `recall_relevant`
|
|
124
|
+
Sub-linear semantic recall via multi-probe LSH. Falls back to brute-force scan on cold start.
|
|
125
|
+
|
|
126
|
+
```
|
|
127
|
+
recall_relevant(query="database connection pooling", top_k=5)
|
|
128
|
+
→ [{"fragment_id": "...", "relevance": 0.87, "content": "..."}]
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
### `record_outcome`
|
|
132
|
+
Feed the Wilson score feedback loop. Adjusts fragment scoring multipliers in the range [0.5, 2.0].
|
|
133
|
+
|
|
134
|
+
```
|
|
135
|
+
record_outcome(fragment_ids=["a1b2c3", "d4e5f6"], success=true)
|
|
136
|
+
→ {"status": "recorded", "fragments_updated": 2}
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
### `explain_context`
|
|
140
|
+
Per-fragment scoring breakdown with sufficiency analysis and exploration swap log.
|
|
141
|
+
|
|
142
|
+
```
|
|
143
|
+
explain_context()
|
|
144
|
+
→ {
|
|
145
|
+
"fragments": [{"id": "...", "recency": 0.9, "frequency": 0.3, "semantic": 0.7, "entropy": 0.8}],
|
|
146
|
+
"sufficiency": 0.91,
|
|
147
|
+
"exploration_swaps": 1
|
|
148
|
+
}
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
### `checkpoint_state` / `resume_state`
|
|
152
|
+
Save and restore full session state — fragments, dedup index, co-access patterns, feedback scores.
|
|
153
|
+
|
|
154
|
+
```
|
|
155
|
+
checkpoint_state(task_description="Refactoring auth module", current_step="Step 5/8")
|
|
156
|
+
→ {"status": "checkpoint_saved", "fragments_saved": 47}
|
|
157
|
+
|
|
158
|
+
resume_state()
|
|
159
|
+
→ {"status": "resumed", "restored_fragments": 47, "metadata": {"step": "Step 5/8"}}
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
### `prefetch_related`
|
|
163
|
+
Predict and pre-load likely-needed context using import analysis, test file inference, and co-access history.
|
|
164
|
+
|
|
165
|
+
```
|
|
166
|
+
prefetch_related(file_path="src/payments.py", source_content="from utils import...")
|
|
167
|
+
→ [{"path": "src/utils.py", "reason": "import", "confidence": 0.70}]
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
### `get_stats`
|
|
171
|
+
Session statistics and cost savings.
|
|
172
|
+
|
|
173
|
+
```
|
|
174
|
+
get_stats()
|
|
175
|
+
→ {
|
|
176
|
+
"fragments": 142,
|
|
177
|
+
"total_tokens": 384000,
|
|
178
|
+
"savings": {
|
|
179
|
+
"total_tokens_saved": 284000,
|
|
180
|
+
"total_duplicates_caught": 12,
|
|
181
|
+
"estimated_cost_saved_usd": 0.85
|
|
182
|
+
}
|
|
183
|
+
}
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
## The Math
|
|
187
|
+
|
|
188
|
+
### Multi-Dimensional Relevance Scoring
|
|
189
|
+
|
|
190
|
+
Each fragment is scored across four dimensions:
|
|
191
|
+
|
|
192
|
+
```
|
|
193
|
+
r(f) = (w_rec · recency + w_freq · frequency + w_sem · semantic + w_ent · entropy)
|
|
194
|
+
/ (w_rec + w_freq + w_sem + w_ent)
|
|
195
|
+
× feedback_multiplier
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Default weights: recency 0.30, frequency 0.25, semantic 0.25, entropy 0.20.
|
|
199
|
+
|
|
200
|
+
- **Recency**: Ebbinghaus forgetting curve — `exp(-ln(2) × Δt / half_life)`, half_life = 15 turns
|
|
201
|
+
- **Frequency**: Normalized access count (spaced repetition boost)
|
|
202
|
+
- **Semantic similarity**: SimHash Hamming distance to query, normalized to [0, 1]
|
|
203
|
+
- **Information density**: Shannon entropy + boilerplate + redundancy (see below)
|
|
204
|
+
|
|
205
|
+
### Knapsack Context Selection
|
|
206
|
+
|
|
207
|
+
Context selection is the 0/1 Knapsack Problem:
|
|
208
|
+
|
|
209
|
+
```
|
|
210
|
+
Maximize: Σ r(fᵢ) · x(fᵢ) for selected fragments
|
|
211
|
+
Subject to: Σ c(fᵢ) · x(fᵢ) ≤ B (token budget)
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
**Two strategies** based on fragment count:
|
|
215
|
+
- **N ≤ 2000**: Exact DP with budget quantization into 1000 bins — O(N × 1000)
|
|
216
|
+
- **N > 2000**: Greedy density sort — O(N log N), Dantzig 0.5-optimality guarantee
|
|
217
|
+
|
|
218
|
+
Pinned fragments (safety-critical files, config files) are always included; remaining budget is allocated via DP/greedy.
|
|
219
|
+
|
|
220
|
+
### Shannon Entropy Scoring
|
|
221
|
+
|
|
222
|
+
Three components combined:
|
|
223
|
+
|
|
224
|
+
```
|
|
225
|
+
score = 0.40 × normalized_entropy + 0.30 × (1 - boilerplate_ratio) + 0.30 × (1 - redundancy)
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
- **Shannon entropy** (40%): `H = -Σ p(char) · log₂(p(char))`, normalized by 6.0 bits/char. Stack-allocated 256-byte histogram, single O(n) pass.
|
|
229
|
+
- **Boilerplate detection** (30%): Pattern matching for imports, pass, dunder methods, closing delimiters.
|
|
230
|
+
- **Cross-fragment redundancy** (30%): Multi-scale n-gram overlap with adaptive weights by fragment length — bigram-heavy for short fragments (<20 words), 4-gram-heavy for long fragments (>100 words). Parallelized with rayon.
|
|
231
|
+
|
|
232
|
+
### SimHash Deduplication
|
|
233
|
+
|
|
234
|
+
64-bit fingerprints from word trigrams hashed via MD5:
|
|
235
|
+
- Hamming distance ≤ 3 → near-duplicate
|
|
236
|
+
- 4-band LSH bucketing for O(1) candidate lookup
|
|
237
|
+
- Separate 12-table multi-probe LSH index for semantic recall (~3 μs over 100K fragments)
|
|
238
|
+
|
|
239
|
+
### Dependency Graph
|
|
240
|
+
|
|
241
|
+
Auto-linking via source analysis:
|
|
242
|
+
- **Imports** (strength 1.0): Python `from X import Y`, Rust `use`, JS `import`
|
|
243
|
+
- **Type references** (0.9): Type annotations, isinstance checks
|
|
244
|
+
- **Function calls** (0.7): General identifier usage matching against symbol table
|
|
245
|
+
- **Same module** (0.5): Co-located definitions
|
|
246
|
+
|
|
247
|
+
Two-pass knapsack refinement: initial selection → boost dependencies of selected fragments → re-optimize.
|
|
248
|
+
|
|
249
|
+
### Task-Aware Budget Multipliers
|
|
250
|
+
|
|
251
|
+
```
|
|
252
|
+
Bug tracing / debugging → 1.5× budget
|
|
253
|
+
Exploration / understanding → 1.3× budget
|
|
254
|
+
Refactoring / code review → 1.0× budget
|
|
255
|
+
Testing → 0.8× budget
|
|
256
|
+
Code generation → 0.7× budget
|
|
257
|
+
Documentation → 0.6× budget
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
### PRISM Spectral Optimizer
|
|
261
|
+
|
|
262
|
+
Tracks a 4×4 covariance matrix over scoring dimensions [recency, frequency, semantic, entropy] with EMA updates (β=0.95). Jacobi eigendecomposition finds principal axes. Anisotropic spectral gain dampens noisy dimensions and amplifies clean signals — automatic learning rate adaptation without hyperparameter tuning.
|
|
263
|
+
|
|
264
|
+
## Configuration
|
|
265
|
+
|
|
266
|
+
```python
|
|
267
|
+
EntrolyConfig(
|
|
268
|
+
default_token_budget=128_000, # GPT-4 Turbo equivalent
|
|
269
|
+
max_fragments=10_000, # session fragment cap
|
|
270
|
+
weight_recency=0.30, # scoring weights (sum to 1.0)
|
|
271
|
+
weight_frequency=0.25,
|
|
272
|
+
weight_semantic_sim=0.25,
|
|
273
|
+
weight_entropy=0.20,
|
|
274
|
+
decay_half_life_turns=15, # Ebbinghaus half-life
|
|
275
|
+
min_relevance_threshold=0.05, # auto-evict below this
|
|
276
|
+
dedup_similarity_threshold=0.92,
|
|
277
|
+
prefetch_depth=2,
|
|
278
|
+
max_prefetch_fragments=10,
|
|
279
|
+
auto_checkpoint_interval=5, # checkpoint every N tool calls
|
|
280
|
+
)
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
## References
|
|
284
|
+
|
|
285
|
+
- Shannon (1948) — Information Theory
|
|
286
|
+
- Charikar (2002) — SimHash
|
|
287
|
+
- Ebbinghaus (1885) — Forgetting Curve
|
|
288
|
+
- Dantzig (1957) — Greedy Knapsack Approximation
|
|
289
|
+
- Wilson (1927) — Score Confidence Intervals
|
|
290
|
+
- ICPC (arXiv 2025) — In-context Prompt Compression
|
|
291
|
+
- Proximity (arXiv 2026) — LSH-bucketed Semantic Caching
|
|
292
|
+
- RCC (ICLR 2025) — Recurrent Context Compression
|
|
293
|
+
- ILRe (ICLR 2026) — Intermediate Layer Retrieval
|
|
294
|
+
- Agentic Plan Caching (arXiv 2025)
|
|
295
|
+
|
|
296
|
+
## Part of the Ebbiforge Ecosystem
|
|
297
|
+
|
|
298
|
+
Entroly integrates with [hippocampus-sharp-memory](https://pypi.org/project/hippocampus-sharp-memory/) for persistent memory and [Ebbiforge](https://pypi.org/project/ebbiforge/) for TF embeddings and RL weight learning. Both are optional — Entroly works standalone with pure Python fallbacks.
|
|
299
|
+
|
|
300
|
+
## License
|
|
301
|
+
|
|
302
|
+
MIT
|
entroly-0.2.0/README.md
ADDED
|
@@ -0,0 +1,273 @@
|
|
|
1
|
+
# Entroly
|
|
2
|
+
|
|
3
|
+
**Information-theoretic context optimization for AI coding agents.**
|
|
4
|
+
|
|
5
|
+
Every AI coding tool manages context with dumb FIFO truncation — stuffing tokens until the window is full, then cutting from the top. Entroly applies mathematics to select the **optimal** context subset.
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
pip install entroly
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
## Architecture
|
|
12
|
+
|
|
13
|
+
Hybrid Rust + Python: CPU-intensive math (knapsack DP, entropy, SimHash, LSH, dependency graph) runs in Rust via PyO3 for 50-100x speedup. MCP protocol and orchestration run in Python via FastMCP. Pure Python fallbacks activate automatically if the Rust extension isn't available.
|
|
14
|
+
|
|
15
|
+
## What It Does
|
|
16
|
+
|
|
17
|
+
An MCP server that sits between your AI coding tool and the LLM, providing:
|
|
18
|
+
|
|
19
|
+
| Engine | What it does | How it works |
|
|
20
|
+
|--------|-------------|--------------|
|
|
21
|
+
| **Knapsack Optimizer** | Selects mathematically optimal context subset | 0/1 Knapsack DP with budget quantization (N ≤ 2000), greedy fallback (N > 2000) |
|
|
22
|
+
| **Entropy Scorer** | Measures information density per fragment | Shannon entropy (40%) + boilerplate detection (30%) + cross-fragment multi-scale n-gram redundancy (30%) |
|
|
23
|
+
| **SimHash Dedup** | Catches near-duplicate content in O(1) | 64-bit SimHash fingerprints with 4-band LSH bucketing, Hamming threshold = 3 |
|
|
24
|
+
| **Multi-Probe LSH Index** | Sub-linear semantic recall over 100K+ fragments | 12-table LSH with 10-bit sampling + 3-neighbor multi-probe queries |
|
|
25
|
+
| **Dependency Graph** | Pulls in related code fragments together | Symbol table + auto-linking (imports, type refs, function calls) + two-pass knapsack refinement |
|
|
26
|
+
| **Predictive Pre-fetch** | Pre-loads context before the agent asks | Static import analysis + test file inference + learned co-access patterns |
|
|
27
|
+
| **Checkpoint & Resume** | Crash recovery for multi-step tasks | Gzipped JSON state serialization (~100 KB per checkpoint) |
|
|
28
|
+
| **Feedback Loop** | Learns which context leads to good outputs | Wilson score lower-bound confidence intervals (same formula as Reddit ranking) |
|
|
29
|
+
| **Context Ordering** | Orders fragments for optimal LLM attention | Pinned → criticality level → dependency count → relevance score |
|
|
30
|
+
| **Guardrails** | Auto-pins safety-critical files, classifies tasks | Criticality levels (Safety/Critical/Important/Normal) + task-aware budget multipliers |
|
|
31
|
+
| **PRISM Optimizer** | Adapts scoring weights to the codebase | Anisotropic spectral optimization via Jacobi eigendecomposition on 4×4 covariance matrix |
|
|
32
|
+
| **Provenance Chain** | Detects hallucination risk in selected context | Tracks source verification + confidence scoring per fragment |
|
|
33
|
+
|
|
34
|
+
## Setup
|
|
35
|
+
|
|
36
|
+
### Cursor
|
|
37
|
+
|
|
38
|
+
Add to `.cursor/mcp.json`:
|
|
39
|
+
|
|
40
|
+
```json
|
|
41
|
+
{
|
|
42
|
+
"mcpServers": {
|
|
43
|
+
"entroly": {
|
|
44
|
+
"command": "entroly"
|
|
45
|
+
}
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
### Claude Code
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
claude mcp add entroly -- entroly
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
### Cline / Any MCP Client
|
|
57
|
+
|
|
58
|
+
```json
|
|
59
|
+
{
|
|
60
|
+
"entroly": {
|
|
61
|
+
"command": "entroly",
|
|
62
|
+
"args": []
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## MCP Tools
|
|
68
|
+
|
|
69
|
+
### `remember_fragment`
|
|
70
|
+
Store context with auto-dedup, entropy scoring, dependency linking, and criticality detection.
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
remember_fragment(content="def process_payment(...)...", source="payments.py", token_count=45)
|
|
74
|
+
→ {"status": "ingested", "entropy_score": 0.82}
|
|
75
|
+
|
|
76
|
+
remember_fragment(content="def process_payment(...)...") # same content
|
|
77
|
+
→ {"status": "duplicate", "duplicate_of": "a1b2c3", "tokens_saved": 45}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### `optimize_context`
|
|
81
|
+
Select the optimal context subset for a token budget. Includes dependency boosting, ε-greedy exploration, context sufficiency scoring, and provenance metadata.
|
|
82
|
+
|
|
83
|
+
```
|
|
84
|
+
optimize_context(token_budget=128000, query="fix payment bug")
|
|
85
|
+
→ {
|
|
86
|
+
"selected_fragments": [...],
|
|
87
|
+
"optimization_stats": {"method": "exact_dp", "budget_utilization": 0.73},
|
|
88
|
+
"tokens_saved_this_call": 42000,
|
|
89
|
+
"sufficiency": 0.91,
|
|
90
|
+
"hallucination_risk": "low"
|
|
91
|
+
}
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
### `recall_relevant`
|
|
95
|
+
Sub-linear semantic recall via multi-probe LSH. Falls back to brute-force scan on cold start.
|
|
96
|
+
|
|
97
|
+
```
|
|
98
|
+
recall_relevant(query="database connection pooling", top_k=5)
|
|
99
|
+
→ [{"fragment_id": "...", "relevance": 0.87, "content": "..."}]
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
### `record_outcome`
|
|
103
|
+
Feed the Wilson score feedback loop. Adjusts fragment scoring multipliers in the range [0.5, 2.0].
|
|
104
|
+
|
|
105
|
+
```
|
|
106
|
+
record_outcome(fragment_ids=["a1b2c3", "d4e5f6"], success=true)
|
|
107
|
+
→ {"status": "recorded", "fragments_updated": 2}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
### `explain_context`
|
|
111
|
+
Per-fragment scoring breakdown with sufficiency analysis and exploration swap log.
|
|
112
|
+
|
|
113
|
+
```
|
|
114
|
+
explain_context()
|
|
115
|
+
→ {
|
|
116
|
+
"fragments": [{"id": "...", "recency": 0.9, "frequency": 0.3, "semantic": 0.7, "entropy": 0.8}],
|
|
117
|
+
"sufficiency": 0.91,
|
|
118
|
+
"exploration_swaps": 1
|
|
119
|
+
}
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
### `checkpoint_state` / `resume_state`
|
|
123
|
+
Save and restore full session state — fragments, dedup index, co-access patterns, feedback scores.
|
|
124
|
+
|
|
125
|
+
```
|
|
126
|
+
checkpoint_state(task_description="Refactoring auth module", current_step="Step 5/8")
|
|
127
|
+
→ {"status": "checkpoint_saved", "fragments_saved": 47}
|
|
128
|
+
|
|
129
|
+
resume_state()
|
|
130
|
+
→ {"status": "resumed", "restored_fragments": 47, "metadata": {"step": "Step 5/8"}}
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
### `prefetch_related`
|
|
134
|
+
Predict and pre-load likely-needed context using import analysis, test file inference, and co-access history.
|
|
135
|
+
|
|
136
|
+
```
|
|
137
|
+
prefetch_related(file_path="src/payments.py", source_content="from utils import...")
|
|
138
|
+
→ [{"path": "src/utils.py", "reason": "import", "confidence": 0.70}]
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
### `get_stats`
|
|
142
|
+
Session statistics and cost savings.
|
|
143
|
+
|
|
144
|
+
```
|
|
145
|
+
get_stats()
|
|
146
|
+
→ {
|
|
147
|
+
"fragments": 142,
|
|
148
|
+
"total_tokens": 384000,
|
|
149
|
+
"savings": {
|
|
150
|
+
"total_tokens_saved": 284000,
|
|
151
|
+
"total_duplicates_caught": 12,
|
|
152
|
+
"estimated_cost_saved_usd": 0.85
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
## The Math
|
|
158
|
+
|
|
159
|
+
### Multi-Dimensional Relevance Scoring
|
|
160
|
+
|
|
161
|
+
Each fragment is scored across four dimensions:
|
|
162
|
+
|
|
163
|
+
```
|
|
164
|
+
r(f) = (w_rec · recency + w_freq · frequency + w_sem · semantic + w_ent · entropy)
|
|
165
|
+
/ (w_rec + w_freq + w_sem + w_ent)
|
|
166
|
+
× feedback_multiplier
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Default weights: recency 0.30, frequency 0.25, semantic 0.25, entropy 0.20.
|
|
170
|
+
|
|
171
|
+
- **Recency**: Ebbinghaus forgetting curve — `exp(-ln(2) × Δt / half_life)`, half_life = 15 turns
|
|
172
|
+
- **Frequency**: Normalized access count (spaced repetition boost)
|
|
173
|
+
- **Semantic similarity**: SimHash Hamming distance to query, normalized to [0, 1]
|
|
174
|
+
- **Information density**: Shannon entropy + boilerplate + redundancy (see below)
|
|
175
|
+
|
|
176
|
+
### Knapsack Context Selection
|
|
177
|
+
|
|
178
|
+
Context selection is the 0/1 Knapsack Problem:
|
|
179
|
+
|
|
180
|
+
```
|
|
181
|
+
Maximize: Σ r(fᵢ) · x(fᵢ) for selected fragments
|
|
182
|
+
Subject to: Σ c(fᵢ) · x(fᵢ) ≤ B (token budget)
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
**Two strategies** based on fragment count:
|
|
186
|
+
- **N ≤ 2000**: Exact DP with budget quantization into 1000 bins — O(N × 1000)
|
|
187
|
+
- **N > 2000**: Greedy density sort — O(N log N), Dantzig 0.5-optimality guarantee
|
|
188
|
+
|
|
189
|
+
Pinned fragments (safety-critical files, config files) are always included; remaining budget is allocated via DP/greedy.
|
|
190
|
+
|
|
191
|
+
### Shannon Entropy Scoring
|
|
192
|
+
|
|
193
|
+
Three components combined:
|
|
194
|
+
|
|
195
|
+
```
|
|
196
|
+
score = 0.40 × normalized_entropy + 0.30 × (1 - boilerplate_ratio) + 0.30 × (1 - redundancy)
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
- **Shannon entropy** (40%): `H = -Σ p(char) · log₂(p(char))`, normalized by 6.0 bits/char. Stack-allocated 256-byte histogram, single O(n) pass.
|
|
200
|
+
- **Boilerplate detection** (30%): Pattern matching for imports, pass, dunder methods, closing delimiters.
|
|
201
|
+
- **Cross-fragment redundancy** (30%): Multi-scale n-gram overlap with adaptive weights by fragment length — bigram-heavy for short fragments (<20 words), 4-gram-heavy for long fragments (>100 words). Parallelized with rayon.
|
|
202
|
+
|
|
203
|
+
### SimHash Deduplication
|
|
204
|
+
|
|
205
|
+
64-bit fingerprints from word trigrams hashed via MD5:
|
|
206
|
+
- Hamming distance ≤ 3 → near-duplicate
|
|
207
|
+
- 4-band LSH bucketing for O(1) candidate lookup
|
|
208
|
+
- Separate 12-table multi-probe LSH index for semantic recall (~3 μs over 100K fragments)
|
|
209
|
+
|
|
210
|
+
### Dependency Graph
|
|
211
|
+
|
|
212
|
+
Auto-linking via source analysis:
|
|
213
|
+
- **Imports** (strength 1.0): Python `from X import Y`, Rust `use`, JS `import`
|
|
214
|
+
- **Type references** (0.9): Type annotations, isinstance checks
|
|
215
|
+
- **Function calls** (0.7): General identifier usage matching against symbol table
|
|
216
|
+
- **Same module** (0.5): Co-located definitions
|
|
217
|
+
|
|
218
|
+
Two-pass knapsack refinement: initial selection → boost dependencies of selected fragments → re-optimize.
|
|
219
|
+
|
|
220
|
+
### Task-Aware Budget Multipliers
|
|
221
|
+
|
|
222
|
+
```
|
|
223
|
+
Bug tracing / debugging → 1.5× budget
|
|
224
|
+
Exploration / understanding → 1.3× budget
|
|
225
|
+
Refactoring / code review → 1.0× budget
|
|
226
|
+
Testing → 0.8× budget
|
|
227
|
+
Code generation → 0.7× budget
|
|
228
|
+
Documentation → 0.6× budget
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
### PRISM Spectral Optimizer
|
|
232
|
+
|
|
233
|
+
Tracks a 4×4 covariance matrix over scoring dimensions [recency, frequency, semantic, entropy] with EMA updates (β=0.95). Jacobi eigendecomposition finds principal axes. Anisotropic spectral gain dampens noisy dimensions and amplifies clean signals — automatic learning rate adaptation without hyperparameter tuning.
|
|
234
|
+
|
|
235
|
+
## Configuration
|
|
236
|
+
|
|
237
|
+
```python
|
|
238
|
+
EntrolyConfig(
|
|
239
|
+
default_token_budget=128_000, # GPT-4 Turbo equivalent
|
|
240
|
+
max_fragments=10_000, # session fragment cap
|
|
241
|
+
weight_recency=0.30, # scoring weights (sum to 1.0)
|
|
242
|
+
weight_frequency=0.25,
|
|
243
|
+
weight_semantic_sim=0.25,
|
|
244
|
+
weight_entropy=0.20,
|
|
245
|
+
decay_half_life_turns=15, # Ebbinghaus half-life
|
|
246
|
+
min_relevance_threshold=0.05, # auto-evict below this
|
|
247
|
+
dedup_similarity_threshold=0.92,
|
|
248
|
+
prefetch_depth=2,
|
|
249
|
+
max_prefetch_fragments=10,
|
|
250
|
+
auto_checkpoint_interval=5, # checkpoint every N tool calls
|
|
251
|
+
)
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
## References
|
|
255
|
+
|
|
256
|
+
- Shannon (1948) — Information Theory
|
|
257
|
+
- Charikar (2002) — SimHash
|
|
258
|
+
- Ebbinghaus (1885) — Forgetting Curve
|
|
259
|
+
- Dantzig (1957) — Greedy Knapsack Approximation
|
|
260
|
+
- Wilson (1927) — Score Confidence Intervals
|
|
261
|
+
- ICPC (arXiv 2025) — In-context Prompt Compression
|
|
262
|
+
- Proximity (arXiv 2026) — LSH-bucketed Semantic Caching
|
|
263
|
+
- RCC (ICLR 2025) — Recurrent Context Compression
|
|
264
|
+
- ILRe (ICLR 2026) — Intermediate Layer Retrieval
|
|
265
|
+
- Agentic Plan Caching (arXiv 2025)
|
|
266
|
+
|
|
267
|
+
## Part of the Ebbiforge Ecosystem
|
|
268
|
+
|
|
269
|
+
Entroly integrates with [hippocampus-sharp-memory](https://pypi.org/project/hippocampus-sharp-memory/) for persistent memory and [Ebbiforge](https://pypi.org/project/ebbiforge/) for TF embeddings and RL weight learning. Both are optional — Entroly works standalone with pure Python fallbacks.
|
|
270
|
+
|
|
271
|
+
## License
|
|
272
|
+
|
|
273
|
+
MIT
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Entroly — Information-Theoretic Context Optimization for Agentic AI
|
|
3
|
+
========================================================================
|
|
4
|
+
|
|
5
|
+
An MCP server that mathematically optimizes what goes into an LLM's
|
|
6
|
+
context window. Uses knapsack dynamic programming, Shannon entropy scoring,
|
|
7
|
+
SimHash deduplication, and predictive pre-fetching to cut token costs by
|
|
8
|
+
50–70% while improving agent accuracy.
|
|
9
|
+
|
|
10
|
+
Quick Setup (Cursor)::
|
|
11
|
+
|
|
12
|
+
Add to .cursor/mcp.json:
|
|
13
|
+
{
|
|
14
|
+
"mcpServers": {
|
|
15
|
+
"entroly": {
|
|
16
|
+
"command": "entroly"
|
|
17
|
+
}
|
|
18
|
+
}
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
Quick Setup (Claude Code)::
|
|
22
|
+
|
|
23
|
+
claude mcp add entroly -- entroly
|
|
24
|
+
|
|
25
|
+
"""
|
|
26
|
+
|
|
27
|
+
__version__ = "0.2.0"
|