entroly 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,75 @@
1
+ # Node/JS
2
+ node_modules/
3
+ npm-debug.log*
4
+ .next/
5
+ web/.next/
6
+
7
+ # Editor
8
+ .vscode/
9
+ .idea/
10
+
11
+ # Python
12
+ __pycache__/
13
+ .venv/
14
+ *.pyc
15
+ .env
16
+
17
+ # Rust / Cargo
18
+ target/
19
+ Cargo.lock
20
+
21
+ # Data / Temp
22
+ *.log
23
+ .DS_Store
24
+
25
+ # Model caches / generated
26
+ .fastembed_cache/
27
+ server/.fastembed_cache/
28
+ *.onnx
29
+ *.lock
30
+
31
+ # Environment secrets (never commit)
32
+ web/.env.local
33
+ web/.env.production
34
+ .env.local
35
+
36
+ # =========================================================
37
+ # ALLOWLIST — only these directories are tracked
38
+ # =========================================================
39
+
40
+ # Core engine (Rust)
41
+ # openrustswarm-core/ — tracked
42
+
43
+ # Web dashboard (Next.js)
44
+ # web/ — tracked
45
+
46
+ # CogOps Skill (OpenClaw integration)
47
+ # cogops-skill/ — tracked
48
+
49
+ # WASM bridge
50
+ # cogops-wasm/ — tracked
51
+
52
+ # Python server
53
+ # server/ — tracked
54
+
55
+ # Public assets
56
+ # demo/, examples/, dist/ — tracked
57
+
58
+ # =========================================================
59
+ # EXCLUDED (internal / generated / one-off)
60
+ # =========================================================
61
+ openrustswarm/
62
+ benchmarks/
63
+ scripts/
64
+ docs/
65
+ hacker_news_launch.md
66
+ eval_*.py
67
+ test_*.py
68
+ medieval_demo.py
69
+ real_10k_log_swarm.py
70
+ CODEBASE_AUDIT.md
71
+ LAUNCH.md
72
+ *.jsonl
73
+ server/diagnose_api.py
74
+ server/verify_ui.py
75
+ server/run_task_help.txt
entroly-0.2.0/PKG-INFO ADDED
@@ -0,0 +1,302 @@
1
+ Metadata-Version: 2.4
2
+ Name: entroly
3
+ Version: 0.2.0
4
+ Summary: Information-theoretic context optimization for AI coding agents. Knapsack-optimal token budgeting, Shannon entropy scoring, SimHash dedup, predictive pre-fetch. MCP server.
5
+ Project-URL: Homepage, https://github.com/juyterman1000/entroly
6
+ Project-URL: Documentation, https://github.com/juyterman1000/entroly#readme
7
+ Project-URL: Repository, https://github.com/juyterman1000/entroly
8
+ Project-URL: Bug Tracker, https://github.com/juyterman1000/entroly/issues
9
+ Project-URL: Full Framework, https://github.com/juyterman1000/ebbiforge
10
+ Author-email: Ebbiforge Team <fastrunner10090@gmail.com>
11
+ License: MIT
12
+ Keywords: agentic-ai,checkpoint,claude,context-optimization,copilot,cursor,deduplication,entropy,knapsack,llm,mcp,token-cost
13
+ Classifier: Development Status :: 4 - Beta
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: License :: OSI Approved :: MIT License
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
22
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
23
+ Requires-Python: >=3.10
24
+ Requires-Dist: entroly-core>=0.1.0
25
+ Requires-Dist: mcp>=1.0.0
26
+ Provides-Extra: memory
27
+ Requires-Dist: hippocampus-sharp-memory>=1.0.0; extra == 'memory'
28
+ Description-Content-Type: text/markdown
29
+
30
+ # Entroly
31
+
32
+ **Information-theoretic context optimization for AI coding agents.**
33
+
34
+ Every AI coding tool manages context with dumb FIFO truncation — stuffing tokens until the window is full, then cutting from the top. Entroly applies mathematics to select the **optimal** context subset.
35
+
36
+ ```
37
+ pip install entroly
38
+ ```
39
+
40
+ ## Architecture
41
+
42
+ Hybrid Rust + Python: CPU-intensive math (knapsack DP, entropy, SimHash, LSH, dependency graph) runs in Rust via PyO3 for 50-100x speedup. MCP protocol and orchestration run in Python via FastMCP. Pure Python fallbacks activate automatically if the Rust extension isn't available.
43
+
44
+ ## What It Does
45
+
46
+ An MCP server that sits between your AI coding tool and the LLM, providing:
47
+
48
+ | Engine | What it does | How it works |
49
+ |--------|-------------|--------------|
50
+ | **Knapsack Optimizer** | Selects mathematically optimal context subset | 0/1 Knapsack DP with budget quantization (N ≤ 2000), greedy fallback (N > 2000) |
51
+ | **Entropy Scorer** | Measures information density per fragment | Shannon entropy (40%) + boilerplate detection (30%) + cross-fragment multi-scale n-gram redundancy (30%) |
52
+ | **SimHash Dedup** | Catches near-duplicate content in O(1) | 64-bit SimHash fingerprints with 4-band LSH bucketing, Hamming threshold = 3 |
53
+ | **Multi-Probe LSH Index** | Sub-linear semantic recall over 100K+ fragments | 12-table LSH with 10-bit sampling + 3-neighbor multi-probe queries |
54
+ | **Dependency Graph** | Pulls in related code fragments together | Symbol table + auto-linking (imports, type refs, function calls) + two-pass knapsack refinement |
55
+ | **Predictive Pre-fetch** | Pre-loads context before the agent asks | Static import analysis + test file inference + learned co-access patterns |
56
+ | **Checkpoint & Resume** | Crash recovery for multi-step tasks | Gzipped JSON state serialization (~100 KB per checkpoint) |
57
+ | **Feedback Loop** | Learns which context leads to good outputs | Wilson score lower-bound confidence intervals (same formula as Reddit ranking) |
58
+ | **Context Ordering** | Orders fragments for optimal LLM attention | Pinned → criticality level → dependency count → relevance score |
59
+ | **Guardrails** | Auto-pins safety-critical files, classifies tasks | Criticality levels (Safety/Critical/Important/Normal) + task-aware budget multipliers |
60
+ | **PRISM Optimizer** | Adapts scoring weights to the codebase | Anisotropic spectral optimization via Jacobi eigendecomposition on 4×4 covariance matrix |
61
+ | **Provenance Chain** | Detects hallucination risk in selected context | Tracks source verification + confidence scoring per fragment |
62
+
63
+ ## Setup
64
+
65
+ ### Cursor
66
+
67
+ Add to `.cursor/mcp.json`:
68
+
69
+ ```json
70
+ {
71
+ "mcpServers": {
72
+ "entroly": {
73
+ "command": "entroly"
74
+ }
75
+ }
76
+ }
77
+ ```
78
+
79
+ ### Claude Code
80
+
81
+ ```bash
82
+ claude mcp add entroly -- entroly
83
+ ```
84
+
85
+ ### Cline / Any MCP Client
86
+
87
+ ```json
88
+ {
89
+ "entroly": {
90
+ "command": "entroly",
91
+ "args": []
92
+ }
93
+ }
94
+ ```
95
+
96
+ ## MCP Tools
97
+
98
+ ### `remember_fragment`
99
+ Store context with auto-dedup, entropy scoring, dependency linking, and criticality detection.
100
+
101
+ ```
102
+ remember_fragment(content="def process_payment(...)...", source="payments.py", token_count=45)
103
+ → {"status": "ingested", "entropy_score": 0.82}
104
+
105
+ remember_fragment(content="def process_payment(...)...") # same content
106
+ → {"status": "duplicate", "duplicate_of": "a1b2c3", "tokens_saved": 45}
107
+ ```
108
+
109
+ ### `optimize_context`
110
+ Select the optimal context subset for a token budget. Includes dependency boosting, ε-greedy exploration, context sufficiency scoring, and provenance metadata.
111
+
112
+ ```
113
+ optimize_context(token_budget=128000, query="fix payment bug")
114
+ → {
115
+ "selected_fragments": [...],
116
+ "optimization_stats": {"method": "exact_dp", "budget_utilization": 0.73},
117
+ "tokens_saved_this_call": 42000,
118
+ "sufficiency": 0.91,
119
+ "hallucination_risk": "low"
120
+ }
121
+ ```
122
+
123
+ ### `recall_relevant`
124
+ Sub-linear semantic recall via multi-probe LSH. Falls back to brute-force scan on cold start.
125
+
126
+ ```
127
+ recall_relevant(query="database connection pooling", top_k=5)
128
+ → [{"fragment_id": "...", "relevance": 0.87, "content": "..."}]
129
+ ```
130
+
131
+ ### `record_outcome`
132
+ Feed the Wilson score feedback loop. Adjusts fragment scoring multipliers in the range [0.5, 2.0].
133
+
134
+ ```
135
+ record_outcome(fragment_ids=["a1b2c3", "d4e5f6"], success=true)
136
+ → {"status": "recorded", "fragments_updated": 2}
137
+ ```
138
+
139
+ ### `explain_context`
140
+ Per-fragment scoring breakdown with sufficiency analysis and exploration swap log.
141
+
142
+ ```
143
+ explain_context()
144
+ → {
145
+ "fragments": [{"id": "...", "recency": 0.9, "frequency": 0.3, "semantic": 0.7, "entropy": 0.8}],
146
+ "sufficiency": 0.91,
147
+ "exploration_swaps": 1
148
+ }
149
+ ```
150
+
151
+ ### `checkpoint_state` / `resume_state`
152
+ Save and restore full session state — fragments, dedup index, co-access patterns, feedback scores.
153
+
154
+ ```
155
+ checkpoint_state(task_description="Refactoring auth module", current_step="Step 5/8")
156
+ → {"status": "checkpoint_saved", "fragments_saved": 47}
157
+
158
+ resume_state()
159
+ → {"status": "resumed", "restored_fragments": 47, "metadata": {"step": "Step 5/8"}}
160
+ ```
161
+
162
+ ### `prefetch_related`
163
+ Predict and pre-load likely-needed context using import analysis, test file inference, and co-access history.
164
+
165
+ ```
166
+ prefetch_related(file_path="src/payments.py", source_content="from utils import...")
167
+ → [{"path": "src/utils.py", "reason": "import", "confidence": 0.70}]
168
+ ```
169
+
170
+ ### `get_stats`
171
+ Session statistics and cost savings.
172
+
173
+ ```
174
+ get_stats()
175
+ → {
176
+ "fragments": 142,
177
+ "total_tokens": 384000,
178
+ "savings": {
179
+ "total_tokens_saved": 284000,
180
+ "total_duplicates_caught": 12,
181
+ "estimated_cost_saved_usd": 0.85
182
+ }
183
+ }
184
+ ```
185
+
186
+ ## The Math
187
+
188
+ ### Multi-Dimensional Relevance Scoring
189
+
190
+ Each fragment is scored across four dimensions:
191
+
192
+ ```
193
+ r(f) = (w_rec · recency + w_freq · frequency + w_sem · semantic + w_ent · entropy)
194
+ / (w_rec + w_freq + w_sem + w_ent)
195
+ × feedback_multiplier
196
+ ```
197
+
198
+ Default weights: recency 0.30, frequency 0.25, semantic 0.25, entropy 0.20.
199
+
200
+ - **Recency**: Ebbinghaus forgetting curve — `exp(-ln(2) × Δt / half_life)`, half_life = 15 turns
201
+ - **Frequency**: Normalized access count (spaced repetition boost)
202
+ - **Semantic similarity**: SimHash Hamming distance to query, normalized to [0, 1]
203
+ - **Information density**: Shannon entropy + boilerplate + redundancy (see below)
204
+
205
+ ### Knapsack Context Selection
206
+
207
+ Context selection is the 0/1 Knapsack Problem:
208
+
209
+ ```
210
+ Maximize: Σ r(fᵢ) · x(fᵢ) for selected fragments
211
+ Subject to: Σ c(fᵢ) · x(fᵢ) ≤ B (token budget)
212
+ ```
213
+
214
+ **Two strategies** based on fragment count:
215
+ - **N ≤ 2000**: Exact DP with budget quantization into 1000 bins — O(N × 1000)
216
+ - **N > 2000**: Greedy density sort — O(N log N), Dantzig 0.5-optimality guarantee
217
+
218
+ Pinned fragments (safety-critical files, config files) are always included; remaining budget is allocated via DP/greedy.
219
+
220
+ ### Shannon Entropy Scoring
221
+
222
+ Three components combined:
223
+
224
+ ```
225
+ score = 0.40 × normalized_entropy + 0.30 × (1 - boilerplate_ratio) + 0.30 × (1 - redundancy)
226
+ ```
227
+
228
+ - **Shannon entropy** (40%): `H = -Σ p(char) · log₂(p(char))`, normalized by 6.0 bits/char. Stack-allocated 256-byte histogram, single O(n) pass.
229
+ - **Boilerplate detection** (30%): Pattern matching for imports, pass, dunder methods, closing delimiters.
230
+ - **Cross-fragment redundancy** (30%): Multi-scale n-gram overlap with adaptive weights by fragment length — bigram-heavy for short fragments (<20 words), 4-gram-heavy for long fragments (>100 words). Parallelized with rayon.
231
+
232
+ ### SimHash Deduplication
233
+
234
+ 64-bit fingerprints from word trigrams hashed via MD5:
235
+ - Hamming distance ≤ 3 → near-duplicate
236
+ - 4-band LSH bucketing for O(1) candidate lookup
237
+ - Separate 12-table multi-probe LSH index for semantic recall (~3 μs over 100K fragments)
238
+
239
+ ### Dependency Graph
240
+
241
+ Auto-linking via source analysis:
242
+ - **Imports** (strength 1.0): Python `from X import Y`, Rust `use`, JS `import`
243
+ - **Type references** (0.9): Type annotations, isinstance checks
244
+ - **Function calls** (0.7): General identifier usage matching against symbol table
245
+ - **Same module** (0.5): Co-located definitions
246
+
247
+ Two-pass knapsack refinement: initial selection → boost dependencies of selected fragments → re-optimize.
248
+
249
+ ### Task-Aware Budget Multipliers
250
+
251
+ ```
252
+ Bug tracing / debugging → 1.5× budget
253
+ Exploration / understanding → 1.3× budget
254
+ Refactoring / code review → 1.0× budget
255
+ Testing → 0.8× budget
256
+ Code generation → 0.7× budget
257
+ Documentation → 0.6× budget
258
+ ```
259
+
260
+ ### PRISM Spectral Optimizer
261
+
262
+ Tracks a 4×4 covariance matrix over scoring dimensions [recency, frequency, semantic, entropy] with EMA updates (β=0.95). Jacobi eigendecomposition finds principal axes. Anisotropic spectral gain dampens noisy dimensions and amplifies clean signals — automatic learning rate adaptation without hyperparameter tuning.
263
+
264
+ ## Configuration
265
+
266
+ ```python
267
+ EntrolyConfig(
268
+ default_token_budget=128_000, # GPT-4 Turbo equivalent
269
+ max_fragments=10_000, # session fragment cap
270
+ weight_recency=0.30, # scoring weights (sum to 1.0)
271
+ weight_frequency=0.25,
272
+ weight_semantic_sim=0.25,
273
+ weight_entropy=0.20,
274
+ decay_half_life_turns=15, # Ebbinghaus half-life
275
+ min_relevance_threshold=0.05, # auto-evict below this
276
+ dedup_similarity_threshold=0.92,
277
+ prefetch_depth=2,
278
+ max_prefetch_fragments=10,
279
+ auto_checkpoint_interval=5, # checkpoint every N tool calls
280
+ )
281
+ ```
282
+
283
+ ## References
284
+
285
+ - Shannon (1948) — Information Theory
286
+ - Charikar (2002) — SimHash
287
+ - Ebbinghaus (1885) — Forgetting Curve
288
+ - Dantzig (1957) — Greedy Knapsack Approximation
289
+ - Wilson (1927) — Score Confidence Intervals
290
+ - ICPC (arXiv 2025) — In-context Prompt Compression
291
+ - Proximity (arXiv 2026) — LSH-bucketed Semantic Caching
292
+ - RCC (ICLR 2025) — Recurrent Context Compression
293
+ - ILRe (ICLR 2026) — Intermediate Layer Retrieval
294
+ - Agentic Plan Caching (arXiv 2025)
295
+
296
+ ## Part of the Ebbiforge Ecosystem
297
+
298
+ Entroly integrates with [hippocampus-sharp-memory](https://pypi.org/project/hippocampus-sharp-memory/) for persistent memory and [Ebbiforge](https://pypi.org/project/ebbiforge/) for TF embeddings and RL weight learning. Both are optional — Entroly works standalone with pure Python fallbacks.
299
+
300
+ ## License
301
+
302
+ MIT
@@ -0,0 +1,273 @@
1
+ # Entroly
2
+
3
+ **Information-theoretic context optimization for AI coding agents.**
4
+
5
+ Every AI coding tool manages context with dumb FIFO truncation — stuffing tokens until the window is full, then cutting from the top. Entroly applies mathematics to select the **optimal** context subset.
6
+
7
+ ```
8
+ pip install entroly
9
+ ```
10
+
11
+ ## Architecture
12
+
13
+ Hybrid Rust + Python: CPU-intensive math (knapsack DP, entropy, SimHash, LSH, dependency graph) runs in Rust via PyO3 for 50-100x speedup. MCP protocol and orchestration run in Python via FastMCP. Pure Python fallbacks activate automatically if the Rust extension isn't available.
14
+
15
+ ## What It Does
16
+
17
+ An MCP server that sits between your AI coding tool and the LLM, providing:
18
+
19
+ | Engine | What it does | How it works |
20
+ |--------|-------------|--------------|
21
+ | **Knapsack Optimizer** | Selects mathematically optimal context subset | 0/1 Knapsack DP with budget quantization (N ≤ 2000), greedy fallback (N > 2000) |
22
+ | **Entropy Scorer** | Measures information density per fragment | Shannon entropy (40%) + boilerplate detection (30%) + cross-fragment multi-scale n-gram redundancy (30%) |
23
+ | **SimHash Dedup** | Catches near-duplicate content in O(1) | 64-bit SimHash fingerprints with 4-band LSH bucketing, Hamming threshold = 3 |
24
+ | **Multi-Probe LSH Index** | Sub-linear semantic recall over 100K+ fragments | 12-table LSH with 10-bit sampling + 3-neighbor multi-probe queries |
25
+ | **Dependency Graph** | Pulls in related code fragments together | Symbol table + auto-linking (imports, type refs, function calls) + two-pass knapsack refinement |
26
+ | **Predictive Pre-fetch** | Pre-loads context before the agent asks | Static import analysis + test file inference + learned co-access patterns |
27
+ | **Checkpoint & Resume** | Crash recovery for multi-step tasks | Gzipped JSON state serialization (~100 KB per checkpoint) |
28
+ | **Feedback Loop** | Learns which context leads to good outputs | Wilson score lower-bound confidence intervals (same formula as Reddit ranking) |
29
+ | **Context Ordering** | Orders fragments for optimal LLM attention | Pinned → criticality level → dependency count → relevance score |
30
+ | **Guardrails** | Auto-pins safety-critical files, classifies tasks | Criticality levels (Safety/Critical/Important/Normal) + task-aware budget multipliers |
31
+ | **PRISM Optimizer** | Adapts scoring weights to the codebase | Anisotropic spectral optimization via Jacobi eigendecomposition on 4×4 covariance matrix |
32
+ | **Provenance Chain** | Detects hallucination risk in selected context | Tracks source verification + confidence scoring per fragment |
33
+
34
+ ## Setup
35
+
36
+ ### Cursor
37
+
38
+ Add to `.cursor/mcp.json`:
39
+
40
+ ```json
41
+ {
42
+ "mcpServers": {
43
+ "entroly": {
44
+ "command": "entroly"
45
+ }
46
+ }
47
+ }
48
+ ```
49
+
50
+ ### Claude Code
51
+
52
+ ```bash
53
+ claude mcp add entroly -- entroly
54
+ ```
55
+
56
+ ### Cline / Any MCP Client
57
+
58
+ ```json
59
+ {
60
+ "entroly": {
61
+ "command": "entroly",
62
+ "args": []
63
+ }
64
+ }
65
+ ```
66
+
67
+ ## MCP Tools
68
+
69
+ ### `remember_fragment`
70
+ Store context with auto-dedup, entropy scoring, dependency linking, and criticality detection.
71
+
72
+ ```
73
+ remember_fragment(content="def process_payment(...)...", source="payments.py", token_count=45)
74
+ → {"status": "ingested", "entropy_score": 0.82}
75
+
76
+ remember_fragment(content="def process_payment(...)...") # same content
77
+ → {"status": "duplicate", "duplicate_of": "a1b2c3", "tokens_saved": 45}
78
+ ```
79
+
80
+ ### `optimize_context`
81
+ Select the optimal context subset for a token budget. Includes dependency boosting, ε-greedy exploration, context sufficiency scoring, and provenance metadata.
82
+
83
+ ```
84
+ optimize_context(token_budget=128000, query="fix payment bug")
85
+ → {
86
+ "selected_fragments": [...],
87
+ "optimization_stats": {"method": "exact_dp", "budget_utilization": 0.73},
88
+ "tokens_saved_this_call": 42000,
89
+ "sufficiency": 0.91,
90
+ "hallucination_risk": "low"
91
+ }
92
+ ```
93
+
94
+ ### `recall_relevant`
95
+ Sub-linear semantic recall via multi-probe LSH. Falls back to brute-force scan on cold start.
96
+
97
+ ```
98
+ recall_relevant(query="database connection pooling", top_k=5)
99
+ → [{"fragment_id": "...", "relevance": 0.87, "content": "..."}]
100
+ ```
101
+
102
+ ### `record_outcome`
103
+ Feed the Wilson score feedback loop. Adjusts fragment scoring multipliers in the range [0.5, 2.0].
104
+
105
+ ```
106
+ record_outcome(fragment_ids=["a1b2c3", "d4e5f6"], success=true)
107
+ → {"status": "recorded", "fragments_updated": 2}
108
+ ```
109
+
110
+ ### `explain_context`
111
+ Per-fragment scoring breakdown with sufficiency analysis and exploration swap log.
112
+
113
+ ```
114
+ explain_context()
115
+ → {
116
+ "fragments": [{"id": "...", "recency": 0.9, "frequency": 0.3, "semantic": 0.7, "entropy": 0.8}],
117
+ "sufficiency": 0.91,
118
+ "exploration_swaps": 1
119
+ }
120
+ ```
121
+
122
+ ### `checkpoint_state` / `resume_state`
123
+ Save and restore full session state — fragments, dedup index, co-access patterns, feedback scores.
124
+
125
+ ```
126
+ checkpoint_state(task_description="Refactoring auth module", current_step="Step 5/8")
127
+ → {"status": "checkpoint_saved", "fragments_saved": 47}
128
+
129
+ resume_state()
130
+ → {"status": "resumed", "restored_fragments": 47, "metadata": {"step": "Step 5/8"}}
131
+ ```
132
+
133
+ ### `prefetch_related`
134
+ Predict and pre-load likely-needed context using import analysis, test file inference, and co-access history.
135
+
136
+ ```
137
+ prefetch_related(file_path="src/payments.py", source_content="from utils import...")
138
+ → [{"path": "src/utils.py", "reason": "import", "confidence": 0.70}]
139
+ ```
140
+
141
+ ### `get_stats`
142
+ Session statistics and cost savings.
143
+
144
+ ```
145
+ get_stats()
146
+ → {
147
+ "fragments": 142,
148
+ "total_tokens": 384000,
149
+ "savings": {
150
+ "total_tokens_saved": 284000,
151
+ "total_duplicates_caught": 12,
152
+ "estimated_cost_saved_usd": 0.85
153
+ }
154
+ }
155
+ ```
156
+
157
+ ## The Math
158
+
159
+ ### Multi-Dimensional Relevance Scoring
160
+
161
+ Each fragment is scored across four dimensions:
162
+
163
+ ```
164
+ r(f) = (w_rec · recency + w_freq · frequency + w_sem · semantic + w_ent · entropy)
165
+ / (w_rec + w_freq + w_sem + w_ent)
166
+ × feedback_multiplier
167
+ ```
168
+
169
+ Default weights: recency 0.30, frequency 0.25, semantic 0.25, entropy 0.20.
170
+
171
+ - **Recency**: Ebbinghaus forgetting curve — `exp(-ln(2) × Δt / half_life)`, half_life = 15 turns
172
+ - **Frequency**: Normalized access count (spaced repetition boost)
173
+ - **Semantic similarity**: SimHash Hamming distance to query, normalized to [0, 1]
174
+ - **Information density**: Shannon entropy + boilerplate + redundancy (see below)
175
+
176
+ ### Knapsack Context Selection
177
+
178
+ Context selection is the 0/1 Knapsack Problem:
179
+
180
+ ```
181
+ Maximize: Σ r(fᵢ) · x(fᵢ) for selected fragments
182
+ Subject to: Σ c(fᵢ) · x(fᵢ) ≤ B (token budget)
183
+ ```
184
+
185
+ **Two strategies** based on fragment count:
186
+ - **N ≤ 2000**: Exact DP with budget quantization into 1000 bins — O(N × 1000)
187
+ - **N > 2000**: Greedy density sort — O(N log N), Dantzig 0.5-optimality guarantee
188
+
189
+ Pinned fragments (safety-critical files, config files) are always included; remaining budget is allocated via DP/greedy.
190
+
191
+ ### Shannon Entropy Scoring
192
+
193
+ Three components combined:
194
+
195
+ ```
196
+ score = 0.40 × normalized_entropy + 0.30 × (1 - boilerplate_ratio) + 0.30 × (1 - redundancy)
197
+ ```
198
+
199
+ - **Shannon entropy** (40%): `H = -Σ p(char) · log₂(p(char))`, normalized by 6.0 bits/char. Stack-allocated 256-byte histogram, single O(n) pass.
200
+ - **Boilerplate detection** (30%): Pattern matching for imports, pass, dunder methods, closing delimiters.
201
+ - **Cross-fragment redundancy** (30%): Multi-scale n-gram overlap with adaptive weights by fragment length — bigram-heavy for short fragments (<20 words), 4-gram-heavy for long fragments (>100 words). Parallelized with rayon.
202
+
203
+ ### SimHash Deduplication
204
+
205
+ 64-bit fingerprints from word trigrams hashed via MD5:
206
+ - Hamming distance ≤ 3 → near-duplicate
207
+ - 4-band LSH bucketing for O(1) candidate lookup
208
+ - Separate 12-table multi-probe LSH index for semantic recall (~3 μs over 100K fragments)
209
+
210
+ ### Dependency Graph
211
+
212
+ Auto-linking via source analysis:
213
+ - **Imports** (strength 1.0): Python `from X import Y`, Rust `use`, JS `import`
214
+ - **Type references** (0.9): Type annotations, isinstance checks
215
+ - **Function calls** (0.7): General identifier usage matching against symbol table
216
+ - **Same module** (0.5): Co-located definitions
217
+
218
+ Two-pass knapsack refinement: initial selection → boost dependencies of selected fragments → re-optimize.
219
+
220
+ ### Task-Aware Budget Multipliers
221
+
222
+ ```
223
+ Bug tracing / debugging → 1.5× budget
224
+ Exploration / understanding → 1.3× budget
225
+ Refactoring / code review → 1.0× budget
226
+ Testing → 0.8× budget
227
+ Code generation → 0.7× budget
228
+ Documentation → 0.6× budget
229
+ ```
230
+
231
+ ### PRISM Spectral Optimizer
232
+
233
+ Tracks a 4×4 covariance matrix over scoring dimensions [recency, frequency, semantic, entropy] with EMA updates (β=0.95). Jacobi eigendecomposition finds principal axes. Anisotropic spectral gain dampens noisy dimensions and amplifies clean signals — automatic learning rate adaptation without hyperparameter tuning.
234
+
235
+ ## Configuration
236
+
237
+ ```python
238
+ EntrolyConfig(
239
+ default_token_budget=128_000, # GPT-4 Turbo equivalent
240
+ max_fragments=10_000, # session fragment cap
241
+ weight_recency=0.30, # scoring weights (sum to 1.0)
242
+ weight_frequency=0.25,
243
+ weight_semantic_sim=0.25,
244
+ weight_entropy=0.20,
245
+ decay_half_life_turns=15, # Ebbinghaus half-life
246
+ min_relevance_threshold=0.05, # auto-evict below this
247
+ dedup_similarity_threshold=0.92,
248
+ prefetch_depth=2,
249
+ max_prefetch_fragments=10,
250
+ auto_checkpoint_interval=5, # checkpoint every N tool calls
251
+ )
252
+ ```
253
+
254
+ ## References
255
+
256
+ - Shannon (1948) — Information Theory
257
+ - Charikar (2002) — SimHash
258
+ - Ebbinghaus (1885) — Forgetting Curve
259
+ - Dantzig (1957) — Greedy Knapsack Approximation
260
+ - Wilson (1927) — Score Confidence Intervals
261
+ - ICPC (arXiv 2025) — In-context Prompt Compression
262
+ - Proximity (arXiv 2026) — LSH-bucketed Semantic Caching
263
+ - RCC (ICLR 2025) — Recurrent Context Compression
264
+ - ILRe (ICLR 2026) — Intermediate Layer Retrieval
265
+ - Agentic Plan Caching (arXiv 2025)
266
+
267
+ ## Part of the Ebbiforge Ecosystem
268
+
269
+ Entroly integrates with [hippocampus-sharp-memory](https://pypi.org/project/hippocampus-sharp-memory/) for persistent memory and [Ebbiforge](https://pypi.org/project/ebbiforge/) for TF embeddings and RL weight learning. Both are optional — Entroly works standalone with pure Python fallbacks.
270
+
271
+ ## License
272
+
273
+ MIT
@@ -0,0 +1,27 @@
1
+ """
2
+ Entroly — Information-Theoretic Context Optimization for Agentic AI
3
+ ========================================================================
4
+
5
+ An MCP server that mathematically optimizes what goes into an LLM's
6
+ context window. Uses knapsack dynamic programming, Shannon entropy scoring,
7
+ SimHash deduplication, and predictive pre-fetching to cut token costs by
8
+ 50–70% while improving agent accuracy.
9
+
10
+ Quick Setup (Cursor)::
11
+
12
+ Add to .cursor/mcp.json:
13
+ {
14
+ "mcpServers": {
15
+ "entroly": {
16
+ "command": "entroly"
17
+ }
18
+ }
19
+ }
20
+
21
+ Quick Setup (Claude Code)::
22
+
23
+ claude mcp add entroly -- entroly
24
+
25
+ """
26
+
27
+ __version__ = "0.2.0"