ctxintel 0.1.0__tar.gz

ctxintel-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Usman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ctxintel-0.1.0/PKG-INFO

@@ -0,0 +1,210 @@
+Metadata-Version: 2.4
+Name: ctxintel
+Version: 0.1.0
+Summary: Context Operating System for LLM Applications — local, zero-cost context intelligence
+License: MIT
+Project-URL: Homepage, https://github.com/nccsuzzi/ctxintel
+Project-URL: Issues, https://github.com/nccsuzzi/ctxintel/issues
+Keywords: llm,context,compression,nlp,ai,memory
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: nltk>=3.8
+Requires-Dist: sumy>=0.11
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: tiktoken>=0.5
+Provides-Extra: spacy
+Requires-Dist: spacy>=3.7; extra == "spacy"
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers>=2.2; extra == "embeddings"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Dynamic: license-file
+
+# ctxintel — Context Operating System for LLM Applications
+
+**ctxintel** is a fully local, zero-cost context intelligence engine for LLM applications. Instead of blindly summarizing or truncating conversations, it runs a structured 6-stage deterministic pipeline that scores, extracts, remembers, compresses, and optimizes your context window — all without a single external AI API call.
+
+> 🔒 **Fully offline** · 🎯 **Deterministic & debuggable** · 📊 **Data-driven rules** · ⚡ **Drop-in for any LLM framework**
+
+---
+
+## Install
+
+```bash
+pip install ctxintel
+
+# Optional: enable NER and dependency parsing
+pip install spacy
+python -m spacy download en_core_web_sm
+```
+
+## Quick Start
+
+```python
+from ctxintel import ContextIntel
+
+sdk = ContextIntel(preset="coding_assistant")
+
+messages = [
+    {"role": "user",      "content": "Hi, I'm John. Building a REST API."},
+    {"role": "assistant", "content": "What stack are you using?"},
+    {"role": "user",      "content": "FastAPI and Python. Deploying on AWS."},
+    {"role": "user",      "content": "Don't use synchronous code."},
+    {"role": "user",      "content": "ok"},
+    {"role": "user",      "content": "cool"},
+    {"role": "user",      "content": "Now add JWT authentication."},
+]
+
+result = sdk.process(messages)
+print(f"Tokens: {result.original_token_count} → {result.token_count}")
+print(f"Memories: {result.extracted_memories}")
+print(sdk.memory_summary())
+```
+
+## How It Works
+
+```
+Raw Messages
+     │
+     ▼
+┌─────────────────┐
+│  1. RANKING     │  TF-IDF + recency + signal patterns + semantic uniqueness
+└────────┬────────┘
+         ▼
+┌─────────────────┐
+│  2. EXTRACTION  │  patterns.yaml rules + spaCy NER + dependency parsing
+└────────┬────────┘
+         ▼
+┌─────────────────┐
+│  3. MEMORY      │  Per-memory scoring + frequency tracking + JSON persistence
+└────────┬────────┘
+         ▼
+┌─────────────────┐
+│  4. COMPRESSION │  Extractive summarization via sumy LSA (zero AI calls)
+└────────┬────────┘
+         ▼
+┌─────────────────┐
+│  5. OPTIMIZATION│  Fit within token budget via tiktoken
+└────────┬────────┘
+         ▼
+   Final Context
+   (minimal tokens, maximum relevance)
+```
+
+## Presets
+
+| Preset | Use Case | Token Budget | Threshold |
+|---|---|---|---|
+| `coding_assistant` | Code generation & programming | 8,000 | 0.35 |
+| `customer_support` | Support bots & helpdesk | 6,000 | 0.40 |
+| `ai_tutor` | Educational assistants | 10,000 | 0.30 |
+| `agent_system` | Autonomous agents | 12,000 | 0.45 |
+| `general` | General chat apps | 8,000 | 0.40 |
+
+```python
+from ctxintel import list_presets
+print(list_presets())
+```
+
+## Extending Patterns
+
+The core innovation of ctxintel is its data-driven rule engine at `ctxintel/data/patterns.yaml`. You can add custom categories without changing any code:
+
+```yaml
+# Add to the categories section in patterns.yaml
+categories:
+  database:
+    patterns:
+      - "using {X} database"
+      - "store data in {X}"
+      - "migrate to {X}"
+    keywords:
+      - postgresql
+      - mysql
+      - mongodb
+      - redis
+    priority: high
+```
+
+No code changes needed — ctxintel loads the YAML at runtime.
+
+## Why Not Just Summarize?
+
+| Approach | What You Lose |
+|---|---|
+| **Truncation** | Early context, user preferences, constraints |
+| **Blind summarization** | Structured facts, decisions, priorities |
+| **ctxintel** | Nothing important — it scores, extracts, and remembers |
+
+ctxintel doesn't just shorten your context. It **understands** what matters: user preferences are preserved, tasks are tracked, decisions are remembered, and filler is compressed — all deterministically.
+
+## Zero API Calls
+
+> ⚠️ **ctxintel makes zero external API calls. Ever.**
+>
+> No OpenAI. No Anthropic. No Cohere. No network requests.
+> Everything runs locally using scikit-learn, sumy, spaCy, and tiktoken.
+> Your conversations never leave your machine.
+
+## API Reference
+
+### `ContextIntel`
+
+```python
+sdk = ContextIntel(
+    preset="coding_assistant",  # or None for manual config
+    preserve=["task", "constraint"],
+    threshold=0.4,
+    token_budget=8000,
+    memory_path=".ctxintel_memory.json",
+)
+
+# Full pipeline
+result = sdk.process(messages)
+
+# Incremental workflow
+sdk.add_message("user", "Hello!")
+sdk.add_message("assistant", "Hi there!")
+result = sdk.flush()
+
+# Utilities
+sdk.preview_compression(messages)
+sdk.memory_summary()
+sdk.reset_memory()
+sdk.supported_categories()
+```
+
+### `ContextResult`
+
+```python
+result.messages             # Final optimized messages
+result.memories             # Extracted memories
+result.token_count          # Final token count
+result.original_token_count # Original token count
+result.compression_ratio    # Reduction ratio (0.0–1.0)
+result.extracted_memories   # Number of memories found
+```
+
+## Roadmap
+
+- [ ] Sentence-transformers reranker (opt-in)
+- [ ] CLI: `ctxintel compress chat.json --budget 4000`
+- [ ] Streaming message support
+- [ ] Custom patterns.yaml path support
+- [ ] LangChain / LlamaIndex integration wrappers
+- [ ] Multi-conversation memory aggregation
+
+## License
+
+MIT

ctxintel-0.1.0/README.md

@@ -0,0 +1,177 @@
+# ctxintel — Context Operating System for LLM Applications
+
+**ctxintel** is a fully local, zero-cost context intelligence engine for LLM applications. Instead of blindly summarizing or truncating conversations, it runs a structured 6-stage deterministic pipeline that scores, extracts, remembers, compresses, and optimizes your context window — all without a single external AI API call.
+
+> 🔒 **Fully offline** · 🎯 **Deterministic & debuggable** · 📊 **Data-driven rules** · ⚡ **Drop-in for any LLM framework**
+
+---
+
+## Install
+
+```bash
+pip install ctxintel
+
+# Optional: enable NER and dependency parsing
+pip install spacy
+python -m spacy download en_core_web_sm
+```
+
+## Quick Start
+
+```python
+from ctxintel import ContextIntel
+
+sdk = ContextIntel(preset="coding_assistant")
+
+messages = [
+    {"role": "user",      "content": "Hi, I'm John. Building a REST API."},
+    {"role": "assistant", "content": "What stack are you using?"},
+    {"role": "user",      "content": "FastAPI and Python. Deploying on AWS."},
+    {"role": "user",      "content": "Don't use synchronous code."},
+    {"role": "user",      "content": "ok"},
+    {"role": "user",      "content": "cool"},
+    {"role": "user",      "content": "Now add JWT authentication."},
+]
+
+result = sdk.process(messages)
+print(f"Tokens: {result.original_token_count} → {result.token_count}")
+print(f"Memories: {result.extracted_memories}")
+print(sdk.memory_summary())
+```
+
+## How It Works
+
+```
+Raw Messages
+     │
+     ▼
+┌─────────────────┐
+│  1. RANKING     │  TF-IDF + recency + signal patterns + semantic uniqueness
+└────────┬────────┘
+         ▼
+┌─────────────────┐
+│  2. EXTRACTION  │  patterns.yaml rules + spaCy NER + dependency parsing
+└────────┬────────┘
+         ▼
+┌─────────────────┐
+│  3. MEMORY      │  Per-memory scoring + frequency tracking + JSON persistence
+└────────┬────────┘
+         ▼
+┌─────────────────┐
+│  4. COMPRESSION │  Extractive summarization via sumy LSA (zero AI calls)
+└────────┬────────┘
+         ▼
+┌─────────────────┐
+│  5. OPTIMIZATION│  Fit within token budget via tiktoken
+└────────┬────────┘
+         ▼
+   Final Context
+   (minimal tokens, maximum relevance)
+```
+
+## Presets
+
+| Preset | Use Case | Token Budget | Threshold |
+|---|---|---|---|
+| `coding_assistant` | Code generation & programming | 8,000 | 0.35 |
+| `customer_support` | Support bots & helpdesk | 6,000 | 0.40 |
+| `ai_tutor` | Educational assistants | 10,000 | 0.30 |
+| `agent_system` | Autonomous agents | 12,000 | 0.45 |
+| `general` | General chat apps | 8,000 | 0.40 |
+
+```python
+from ctxintel import list_presets
+print(list_presets())
+```
+
+## Extending Patterns
+
+The core innovation of ctxintel is its data-driven rule engine at `ctxintel/data/patterns.yaml`. You can add custom categories without changing any code:
+
+```yaml
+# Add to the categories section in patterns.yaml
+categories:
+  database:
+    patterns:
+      - "using {X} database"
+      - "store data in {X}"
+      - "migrate to {X}"
+    keywords:
+      - postgresql
+      - mysql
+      - mongodb
+      - redis
+    priority: high
+```
+
+No code changes needed — ctxintel loads the YAML at runtime.
+
+## Why Not Just Summarize?
+
+| Approach | What You Lose |
+|---|---|
+| **Truncation** | Early context, user preferences, constraints |
+| **Blind summarization** | Structured facts, decisions, priorities |
+| **ctxintel** | Nothing important — it scores, extracts, and remembers |
+
+ctxintel doesn't just shorten your context. It **understands** what matters: user preferences are preserved, tasks are tracked, decisions are remembered, and filler is compressed — all deterministically.
+
+## Zero API Calls
+
+> ⚠️ **ctxintel makes zero external API calls. Ever.**
+>
+> No OpenAI. No Anthropic. No Cohere. No network requests.
+> Everything runs locally using scikit-learn, sumy, spaCy, and tiktoken.
+> Your conversations never leave your machine.
+
+## API Reference
+
+### `ContextIntel`
+
+```python
+sdk = ContextIntel(
+    preset="coding_assistant",  # or None for manual config
+    preserve=["task", "constraint"],
+    threshold=0.4,
+    token_budget=8000,
+    memory_path=".ctxintel_memory.json",
+)
+
+# Full pipeline
+result = sdk.process(messages)
+
+# Incremental workflow
+sdk.add_message("user", "Hello!")
+sdk.add_message("assistant", "Hi there!")
+result = sdk.flush()
+
+# Utilities
+sdk.preview_compression(messages)
+sdk.memory_summary()
+sdk.reset_memory()
+sdk.supported_categories()
+```
+
+### `ContextResult`
+
+```python
+result.messages             # Final optimized messages
+result.memories             # Extracted memories
+result.token_count          # Final token count
+result.original_token_count # Original token count
+result.compression_ratio    # Reduction ratio (0.0–1.0)
+result.extracted_memories   # Number of memories found
+```
+
+## Roadmap
+
+- [ ] Sentence-transformers reranker (opt-in)
+- [ ] CLI: `ctxintel compress chat.json --budget 4000`
+- [ ] Streaming message support
+- [ ] Custom patterns.yaml path support
+- [ ] LangChain / LlamaIndex integration wrappers
+- [ ] Multi-conversation memory aggregation
+
+## License
+
+MIT

ctxintel-0.1.0/ctxintel/__init__.py

@@ -0,0 +1,16 @@
+"""ctxintel — Context Operating System for LLM Applications."""
+
+from ctxintel.pipeline import ContextIntel
+from ctxintel.models import Message, Memory, ContextResult
+from ctxintel.presets import PRESETS, load_preset, list_presets
+
+__version__ = "0.1.0"
+__all__ = [
+    "ContextIntel",
+    "Message",
+    "Memory",
+    "ContextResult",
+    "PRESETS",
+    "load_preset",
+    "list_presets",
+]

ctxintel-0.1.0/ctxintel/compressor.py

@@ -0,0 +1,122 @@
+"""Message compressor using extractive summarization (sumy LSA) with fallback."""
+
+import warnings
+from typing import Dict, List, Optional
+
+from ctxintel.models import Message
+
+try:
+    from sumy.parsers.plaintext import PlaintextParser
+    from sumy.nlp.tokenizers import Tokenizer
+    from sumy.summarizers.lsa import LsaSummarizer
+    from sumy.nlp.stemmers import Stemmer
+    _HAS_SUMY = True
+except ImportError:
+    _HAS_SUMY = False
+    warnings.warn(
+        "sumy not found. Compression will use a simpler sentence-based fallback. "
+        "Install with: pip install sumy",
+        stacklevel=2,
+    )
+
+
+class Compressor:
+    """Compresses low-importance messages into a summary while preserving critical ones.
+
+    Uses sumy's LSA summarizer for extractive summarization (zero AI API calls).
+    Falls back to a simple first-N-sentences approach if sumy is unavailable.
+    """
+
+    def compress(
+        self,
+        messages: List[Message],
+        preserve: Optional[List[str]] = None,
+        threshold: float = 0.4,
+    ) -> List[Message]:
+        """Compress messages below the importance threshold into a summary.
+
+        Args:
+            messages: Ranked list of Message objects.
+            preserve: Category names to preserve (unused — preserved flag is on Message).
+            threshold: Messages below this importance score get compressed.
+
+        Returns:
+            List of messages with low-importance ones replaced by a summary.
+        """
+        keep_msgs = [m for m in messages if m.importance >= threshold or m.preserved]
+        compress_msgs = [m for m in messages if m.importance < threshold and not m.preserved]
+
+        if not compress_msgs:
+            return keep_msgs
+
+        combined_text = " ".join(m.content for m in compress_msgs)
+
+        # If the compressed content is very short, just drop it —
+        # a summary message would be larger than the original filler
+        if len(combined_text) < 50:
+            return keep_msgs
+
+        summary = self._summarize(combined_text, len(compress_msgs))
+
+        # Only inject summary if it's actually shorter than the combined original
+        if len(summary) >= len(combined_text):
+            return keep_msgs
+
+        summary_msg = Message(
+            role="system",
+            content=f"[Earlier context summary]: {summary}",
+            importance=0.85,
+            preserved=True,
+        )
+        return [summary_msg] + keep_msgs
+
+    def estimate_reduction(self, messages: List[Message], threshold: float = 0.4) -> Dict:
+        """Preview compression stats without actually compressing.
+
+        Args:
+            messages: Ranked list of Message objects.
+            threshold: Importance threshold for compression.
+
+        Returns:
+            Dict with total_messages, will_keep, will_compress, estimated_reduction_pct.
+        """
+        keep = sum(1 for m in messages if m.importance >= threshold or m.preserved)
+        compress = sum(1 for m in messages if m.importance < threshold and not m.preserved)
+        total = len(messages)
+        return {
+            "total_messages": total,
+            "will_keep": keep,
+            "will_compress": compress,
+            "estimated_reduction_pct": round(compress / max(total, 1) * 100, 1),
+        }
+
+    def _summarize(self, text: str, msg_count: int) -> str:
+        """Summarize text using sumy LSA or a sentence-based fallback."""
+        if _HAS_SUMY:
+            return self._summarize_lsa(text, msg_count)
+        return self._summarize_fallback(text)
+
+    def _summarize_lsa(self, text: str, msg_count: int) -> str:
+        """Use sumy's LSA summarizer for extractive summarization."""
+        try:
+            sentence_count = max(3, min(7, msg_count // 3))
+            parser = PlaintextParser.from_string(text, Tokenizer("english"))
+            stemmer = Stemmer("english")
+            summarizer = LsaSummarizer(stemmer)
+            sentences = summarizer(parser.document, sentence_count)
+            summary = " ".join(str(s) for s in sentences)
+            if not summary.strip():
+                return self._summarize_fallback(text)
+            return summary
+        except Exception as exc:
+            warnings.warn(f"sumy LSA failed ({exc}). Using fallback.", stacklevel=2)
+            return self._summarize_fallback(text)
+
+    def _summarize_fallback(self, text: str) -> str:
+        """Simple fallback: take the first 5 sentences."""
+        sentences = text.split(". ")
+        selected = sentences[:5]
+        result = ". ".join(selected)
+        if not result.endswith("."):
+            result += "."
+        return result