pymrsf 0.4.0__tar.gz

pymrsf-0.4.0/.gitignore

@@ -0,0 +1,46 @@
+# Environment variables
+.env
+
+# Python cache
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+
+# Virtual environment
+venv/
+env/
+ENV/
+
+# MRSF database and indexes
+mrsf.db
+mrsf.faiss
+mrsf.faiss.meta
+
+# Results and outputs
+*.csv
+mrsf_results_full.csv
+
+# Large model files
+models/
+*.gguf
+*.tar.gz
+*.bin
+*.safetensors
+
+# Distribution files (can be rebuilt)
+dist/
+build/
+*.egg-info/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db

pymrsf-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Eric Monthe
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pymrsf-0.4.0/PKG-INFO

@@ -0,0 +1,204 @@
+Metadata-Version: 2.4
+Name: pymrsf
+Version: 0.4.0
+Summary: Novelty-Aware RAG scoring — Filter chunks by information gain, not just relevance
+Project-URL: Homepage, https://github.com/riiseup08/mrsf
+Author: Eric Monthe
+License: MIT
+License-File: LICENSE
+Keywords: embeddings,llm,novelty,rag,retrieval,semantic
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Requires-Dist: faiss-cpu>=1.7.4
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: llama-cpp-python>=0.2.0
+Requires-Dist: msgpack>=1.0.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: tiktoken>=0.5.0
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# pymrsf — Novelty-Aware RAG Chunk Scoring
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-33%20passing-brightgreen)]()
+
+**Stop wasting context window on information your LLM already knows.**
+
+`pymrsf` scores RAG chunks by measuring **information gain** — not just relevance. It uses the model's own predictive surprise to detect which chunks contain genuinely new information.
+
+## The Problem
+
+Standard RAG retrieves chunks by *relevance* (cosine similarity). But a chunk can be highly relevant while containing *only facts the model already memorized during training*. You waste precious context window on redundant information.
+
+Also, if the LLM already *knows the answer* to the query, even novel chunks are less useful. And if two chunks say the same thing, you don't need both.
+
+## The Solution
+
+`pymrsf` introduces **multi-factor novelty-aware scoring**:
+
+| Factor | What It Measures | Weight |
+|--------|-----------------|--------|
+| **Novelty** | How much *new* information does this chunk contain? | 40% |
+| **Relevance** | How related is this chunk to the query? | 40% |
+| **Query Ignorance** | Does the model *not* know the answer to your question? | 20% |
+| **Diversity** | Does a better chunk already cover this content? | Dedup |
+
+## Quick Start
+
+```python
+from pymrsf.rag import filter_chunks
+
+# Your retrieved chunks
+chunks = [
+    "Backpropagation computes gradients using the chain rule.",
+    "Neural networks are inspired by the human brain.",
+    "The sky is blue because of Rayleigh scattering.",
+]
+
+# Filter to only useful chunks
+query = "How does backpropagation work?"
+useful = filter_chunks(chunks, query, min_rag_score=50, verbose=True)
+
+# → Pass only useful chunks to your LLM
+answer = llm.complete(query, context=useful)
+```
+
+## Installation
+
+```bash
+pip install llama-cpp-python faiss-cpu msgpack tiktoken
+git clone https://github.com/riiseup08/mrsf.git
+cd mrsf
+pip install -e .
+```
+
+## Features
+
+### 🎯 RAG Chunk Scoring (Core Feature)
+
+```python
+from pymrsf.rag import score_chunk, score_chunks, score_chunks_batch
+
+# Single chunk scoring
+result = score_chunk(
+    "Backpropagation computes gradients using the chain rule.",
+    query="How does backpropagation work?",
+    verbose=True
+)
+print(result["rag_score"])    # 72/100
+print(result["verdict"])      # "good"
+print(result["query_knowledge"])  # how much model knows the query
+
+# Batch scoring (3-5x faster for many chunks)
+results = score_chunks_batch(chunks, query)
+
+# Custom weights (adjust the formula)
+weights = {"novelty": 0.5, "relevance": 0.3, "query_ignorance": 0.2}
+result = score_chunk(chunk, query, weights=weights)
+```
+
+### 🔍 Knowledge Probing
+
+```python
+from pymrsf import probe
+
+result = probe("To be or not to be, that is the question.")
+print(f"Knowledge: {result['knowledge_score']}/100 ({result['label']})")
+# → Knowledge: 92/100 (memorized) — Shakespeare is well-known
+
+result = probe("My proprietary algorithm uses a novel attention mechanism.")
+print(f"Knowledge: {result['knowledge_score']}/100 ({result['label']})")
+# → Knowledge: 15/100 (unknown) — novel content!
+```
+
+### 🔧 RAG Pipeline Filter
+
+```python
+from pymrsf.rag import filter_chunks
+
+chunks = retriever.get(query, top_k=20)   # your retriever
+
+# Only keep chunks worth sending to the LLM
+good = filter_chunks(
+    chunks,
+    query,
+    min_rag_score=50,      # skip low-value chunks
+    top_k=5,                # limit context window usage
+    diversity_threshold=0.85,  # dedup similar chunks
+    verbose=True,
+)
+
+answer = llm.complete(query, context=good)
+```
+
+### 📦 Delta Compression (Experimental)
+
+Store text efficiently using LLM surprises:
+
+```python
+from pymrsf import mrsf_write, mrsf_read, save_index
+
+# Write (stores only surprise tokens = ~40% compression)
+mrsf_write("The Eiffel Tower is in Paris.")
+save_index()
+
+# Read (reconstructs from delta + model)
+results = mrsf_read("famous landmark in France")
+```
+
+## Configuration
+
+Create a `.env` file:
+
+```bash
+PYMRSF_PROVIDER=local
+PYMRSF_MODEL_PATH=./models/mistral-7b-v0.1.Q4_K_M.gguf
+```
+
+## Scoring Concepts
+
+### RAG Score Formula
+```
+rag_score = novelty × 0.40 + relevance × 0.40 + query_ignorance × 0.20
+```
+
+### What the Scores Mean
+
+| Score | Verdict | Action |
+|-------|---------|--------|
+| 80-100 | Excellent | Prioritize this chunk |
+| 60-79 | Good | Include in context |
+| 40-59 | Moderate | Include if space allows |
+| 20-39 | Weak | Skip if better chunks exist |
+| 0-19 | Skip | Model already knows this |
+
+## Project Structure
+
+```
+pymrsf/
+├── __init__.py     # Public API exports
+├── core.py         # Provider routing (local + openai), lazy model loading
+├── embeddings.py   # Ollama embedding API client
+├── probe.py        # Knowledge probing (how well does model know a text?)
+├── rag.py          # RAG chunk scoring with novelty + relevance + diversity
+├── storage.py      # Delta compression storage (experimental)
+├── inspect.py      # Token-level visualization tools
+└── benchmark.py    # Compression/latency benchmarks
+```
+
+## Project Status
+
+**Alpha** — The RAG novelty scoring works and solves a real problem. The delta compression/storage system is experimental.
+
+## License
+
+MIT

pymrsf-0.4.0/README.md

@@ -0,0 +1,177 @@
+# pymrsf — Novelty-Aware RAG Chunk Scoring
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-33%20passing-brightgreen)]()
+
+**Stop wasting context window on information your LLM already knows.**
+
+`pymrsf` scores RAG chunks by measuring **information gain** — not just relevance. It uses the model's own predictive surprise to detect which chunks contain genuinely new information.
+
+## The Problem
+
+Standard RAG retrieves chunks by *relevance* (cosine similarity). But a chunk can be highly relevant while containing *only facts the model already memorized during training*. You waste precious context window on redundant information.
+
+Also, if the LLM already *knows the answer* to the query, even novel chunks are less useful. And if two chunks say the same thing, you don't need both.
+
+## The Solution
+
+`pymrsf` introduces **multi-factor novelty-aware scoring**:
+
+| Factor | What It Measures | Weight |
+|--------|-----------------|--------|
+| **Novelty** | How much *new* information does this chunk contain? | 40% |
+| **Relevance** | How related is this chunk to the query? | 40% |
+| **Query Ignorance** | Does the model *not* know the answer to your question? | 20% |
+| **Diversity** | Does a better chunk already cover this content? | Dedup |
+
+## Quick Start
+
+```python
+from pymrsf.rag import filter_chunks
+
+# Your retrieved chunks
+chunks = [
+    "Backpropagation computes gradients using the chain rule.",
+    "Neural networks are inspired by the human brain.",
+    "The sky is blue because of Rayleigh scattering.",
+]
+
+# Filter to only useful chunks
+query = "How does backpropagation work?"
+useful = filter_chunks(chunks, query, min_rag_score=50, verbose=True)
+
+# → Pass only useful chunks to your LLM
+answer = llm.complete(query, context=useful)
+```
+
+## Installation
+
+```bash
+pip install llama-cpp-python faiss-cpu msgpack tiktoken
+git clone https://github.com/riiseup08/mrsf.git
+cd mrsf
+pip install -e .
+```
+
+## Features
+
+### 🎯 RAG Chunk Scoring (Core Feature)
+
+```python
+from pymrsf.rag import score_chunk, score_chunks, score_chunks_batch
+
+# Single chunk scoring
+result = score_chunk(
+    "Backpropagation computes gradients using the chain rule.",
+    query="How does backpropagation work?",
+    verbose=True
+)
+print(result["rag_score"])    # 72/100
+print(result["verdict"])      # "good"
+print(result["query_knowledge"])  # how much model knows the query
+
+# Batch scoring (3-5x faster for many chunks)
+results = score_chunks_batch(chunks, query)
+
+# Custom weights (adjust the formula)
+weights = {"novelty": 0.5, "relevance": 0.3, "query_ignorance": 0.2}
+result = score_chunk(chunk, query, weights=weights)
+```
+
+### 🔍 Knowledge Probing
+
+```python
+from pymrsf import probe
+
+result = probe("To be or not to be, that is the question.")
+print(f"Knowledge: {result['knowledge_score']}/100 ({result['label']})")
+# → Knowledge: 92/100 (memorized) — Shakespeare is well-known
+
+result = probe("My proprietary algorithm uses a novel attention mechanism.")
+print(f"Knowledge: {result['knowledge_score']}/100 ({result['label']})")
+# → Knowledge: 15/100 (unknown) — novel content!
+```
+
+### 🔧 RAG Pipeline Filter
+
+```python
+from pymrsf.rag import filter_chunks
+
+chunks = retriever.get(query, top_k=20)   # your retriever
+
+# Only keep chunks worth sending to the LLM
+good = filter_chunks(
+    chunks,
+    query,
+    min_rag_score=50,      # skip low-value chunks
+    top_k=5,                # limit context window usage
+    diversity_threshold=0.85,  # dedup similar chunks
+    verbose=True,
+)
+
+answer = llm.complete(query, context=good)
+```
+
+### 📦 Delta Compression (Experimental)
+
+Store text efficiently using LLM surprises:
+
+```python
+from pymrsf import mrsf_write, mrsf_read, save_index
+
+# Write (stores only surprise tokens = ~40% compression)
+mrsf_write("The Eiffel Tower is in Paris.")
+save_index()
+
+# Read (reconstructs from delta + model)
+results = mrsf_read("famous landmark in France")
+```
+
+## Configuration
+
+Create a `.env` file:
+
+```bash
+PYMRSF_PROVIDER=local
+PYMRSF_MODEL_PATH=./models/mistral-7b-v0.1.Q4_K_M.gguf
+```
+
+## Scoring Concepts
+
+### RAG Score Formula
+```
+rag_score = novelty × 0.40 + relevance × 0.40 + query_ignorance × 0.20
+```
+
+### What the Scores Mean
+
+| Score | Verdict | Action |
+|-------|---------|--------|
+| 80-100 | Excellent | Prioritize this chunk |
+| 60-79 | Good | Include in context |
+| 40-59 | Moderate | Include if space allows |
+| 20-39 | Weak | Skip if better chunks exist |
+| 0-19 | Skip | Model already knows this |
+
+## Project Structure
+
+```
+pymrsf/
+├── __init__.py     # Public API exports
+├── core.py         # Provider routing (local + openai), lazy model loading
+├── embeddings.py   # Ollama embedding API client
+├── probe.py        # Knowledge probing (how well does model know a text?)
+├── rag.py          # RAG chunk scoring with novelty + relevance + diversity
+├── storage.py      # Delta compression storage (experimental)
+├── inspect.py      # Token-level visualization tools
+└── benchmark.py    # Compression/latency benchmarks
+```
+
+## Project Status
+
+**Alpha** — The RAG novelty scoring works and solves a real problem. The delta compression/storage system is experimental.
+
+## License
+
+MIT

pymrsf-0.4.0/demo_novelty.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""
+pymrsf demo — see the novelty-aware RAG scoring in action.
+
+This script tests 3 things:
+  1. Knowledge probing : Does the model know Shakespeare better than novel text?
+  2. RAG scoring       : Do novel+relevant chunks score higher?
+  3. Diversity dedup   : Are duplicate chunks filtered out?
+
+Run:  python demo_novelty.py
+"""
+
+import sys
+
+# ── Test 1: Knowledge Probe ──────────────────────────────────────────────────
+
+print("=" * 65)
+print("TEST 1: Knowledge Probing")
+print("Does the model know famous text better than novel text?")
+print("=" * 65)
+
+from pymrsf import probe
+
+famous = [
+    "To be or not to be, that is the question. Whether tis nobler in the mind to suffer.",
+    "The quick brown fox jumps over the lazy dog.",
+]
+
+novel = [
+    "My proprietary attention mechanism uses a novel sparse gating function.",
+    "The Zeta-7 protocol encrypts data using quantum-resistant lattice cryptography.",
+]
+
+print("\n📚 FAMOUS TEXT (should score HIGH — model knows this):")
+for text in famous:
+    result = probe(text)
+    if "error" in result:
+        print(f"   ⚠️  {result['error']}")
+        print(f"   (This is expected if no local model is loaded)")
+        break
+    bar = "█" * (result['knowledge_score'] // 4) + "░" * (25 - result['knowledge_score'] // 4)
+    print(f"   [{bar}] {result['knowledge_score']:>2}/100  {result['label'].upper():<12}  {text[:40]}...")
+
+else:
+    print("\n🔬 NOVEL TEXT (should score LOW — model hasn't seen this):")
+    for text in novel:
+        result = probe(text)
+        bar = "█" * (result['knowledge_score'] // 4) + "░" * (25 - result['knowledge_score'] // 4)
+        print(f"   [{bar}] {result['knowledge_score']:>2}/100  {result['label'].upper():<12}  {text[:40]}...")
+
+    # Verify the claim
+    print(f"\n{'─'*65}")
+    famous_avg = sum(probe(t)['knowledge_score'] for t in famous) / len(famous)
+    novel_avg  = sum(probe(t)['knowledge_score'] for t in novel) / len(novel)
+    print(f"  Famous avg: {famous_avg:.0f}/100  |  Novel avg: {novel_avg:.0f}/100")
+    if famous_avg > novel_avg:
+        print(f"  ✅ CONFIRMED: Model knows famous text {famous_avg - novel_avg:.0f}% better!")
+    else:
+        print(f"  ⚠️  Unexpected result — but that's interesting data too!")
+
+
+# ── Test 2: RAG Chunk Scoring ─────────────────────────────────────────────────
+
+print("\n\n" + "=" * 65)
+print("TEST 2: RAG Chunk Scoring")
+print("Do novel+relevant chunks score higher than known+irrelevant ones?")
+print("=" * 65)
+
+from pymrsf.rag import score_chunks_batch, filter_chunks
+
+# Simulated RAG chunks for the query "How does backpropagation work?"
+query = "How does backpropagation work?"
+rag_chunks = [
+    "Backpropagation computes gradients using the chain rule. It propagates error backwards.",
+    "Neural networks are inspired by the human brain. They consist of layers of neurons.",
+    "The sky is blue because of Rayleigh scattering. This is well-known physics.",
+    "A novel optimization technique uses second-order gradients for faster convergence.",
+]
+
+print(f"\nQuery: '{query}'")
+print(f"Chunks to score: {len(rag_chunks)}")
+print()
+
+results = score_chunks_batch(rag_chunks, query, diversity_threshold=0.90)
+
+for r in results:
+    bar = "█" * (r['rag_score'] // 4) + "░" * (25 - r['rag_score'] // 4)
+    status = "✅" if r['rag_score'] >= 50 else "❌"
+    print(f"  {status} [{bar}] RAG={r['rag_score']:>2}  N={r['novelty_score']:>2}  R={r['relevance_score']:>2}  "
+          f"Q={r['query_knowledge']:>2}  {r['verdict'].upper():<10}  {r['chunk'][:50]}...")
+
+best_score = results[0]['rag_score']
+worst_score = results[-1]['rag_score']
+print(f"\n  Best chunk: {results[0]['chunk'][:50]}... ({best_score}/100)")
+print(f"  Worst chunk: {results[-1]['chunk'][:50]}... ({worst_score}/100)")
+if best_score > worst_score:
+    print(f"  ✅ RAG scoring successfully ranked chunks by usefulness!")
+else:
+    print(f"  ⚠️  Scores are similar — fine-tuning threshold may help")
+
+
+# ── Test 3: Diversity Dedup ───────────────────────────────────────────────────
+
+print("\n\n" + "=" * 65)
+print("TEST 3: Diversity Dedup")
+print("Does the filter remove duplicate chunks?")
+print("=" * 65)
+
+# Create chunks where two are very similar
+dup_chunks = [
+    "Backpropagation computes gradients using the chain rule error propagation.",
+    "Backpropagation uses chain rule to compute gradients by propagating error.",
+    "The sky is blue because of Rayleigh scattering.",
+    "The sky appears blue due to Rayleigh scattering of sunlight.",
+]
+
+print(f"\nQuery: 'How does backpropagation work?'")
+print(f"Input: {len(dup_chunks)} chunks (2 pairs of near-duplicates)")
+print()
+
+# First: score without dedup
+print("--- WITHOUT dedup ---")
+raw = score_chunks_batch(dup_chunks, query, diversity_threshold=1.0)
+for r in raw:
+    print(f"  [{r['rag_score']:>2}/100] {r['chunk'][:55]}...")
+
+# Then: with dedup (default threshold 0.85)
+print("\n--- WITH dedup (threshold=0.85) ---")
+filtered = score_chunks_batch(dup_chunks, query, diversity_threshold=0.85)
+dup_count = sum(1 for r in filtered if r['rag_score'] == 0)
+for r in filtered:
+    if r['rag_score'] > 0:
+        print(f"  ✅ [{r['rag_score']:>2}/100] {r['chunk'][:55]}...")
+    else:
+        print(f"  ❌ [{r['rag_score']:>2}/100] {r['chunk'][:55]}... (DUPLICATE)")
+
+if dup_count > 0:
+    print(f"\n  ✅ Dedup removed {dup_count} duplicate chunks!")
+else:
+    print(f"\n  ℹ️  No duplicates detected (threshold may need adjustment)")
+
+
+# ── Summary ───────────────────────────────────────────────────────────────────
+
+print("\n\n" + "=" * 65)
+print("SUMMARY")
+print("=" * 65)
+print(f"\n  pymrsf gives you:")
+print(f"  1. Which chunks contain NEW information (not just relevant)")
+print(f"  2. Whether the model already KNOWS THE ANSWER to your query")
+print(f"  3. Automatic removal of DUPLICATE chunks")
+print(f"  4. A tunable RAG score to optimize context window usage")
+print(f"\n  Next step: Drop filter_chunks() into your RAG pipeline:")
+print(f"    from pymrsf.rag import filter_chunks")
+print(f"    good_chunks = filter_chunks(retriever.get(query), query)")
+print(f"    answer = llm.complete(query, context=good_chunks)")
+print("=" * 65)