contextpress 0.5.4__tar.gz → 0.5.6__tar.gz

contextpress-0.5.6/AGENTS.md

@@ -0,0 +1,45 @@
+# Notes for agents and contributors
+
+## Branches
+
+- **`dev`** — default working branch; all feature work and fixes land here first.
+- **`main`** — release branch; merge from `dev` when cutting a PyPI release.
+
+Always `git checkout dev` before making changes in this repo.
+
+## Where things live
+
+| Area | Path |
+|------|------|
+| Public API | `contextpress/core.py` → `ContextManager` |
+| I/O formats | `contextpress/normalizer.py` |
+| Stage presets | `contextpress/compression.py` (`STAGE_ORDER`, `VALID_STAGES`) |
+| Tier 1 stages | `contextpress/strategies/*.py` |
+| Tier 2 | `contextpress/llm/` (`LLMBackend`, `adapters.py`) |
+| Custom stages | `contextpress/registry.py` + `ContextManager.register_stage()` |
+| Orchestration | `contextpress/pipeline.py` (behavior contract at top of file) |
+| Profiles | `contextpress/profiles.py` |
+
+`contextpress/__init__.py` lazy-loads `ContextManager` so `from contextpress.models import Turn` avoids pulling optional Tier-2 / sumy import chains when possible.
+
+## Commands
+
+```bash
+pip install -e ".[dev]"
+pytest tests -q
+```
+
+## Invariants (non-negotiable)
+
+See the numbered block at the top of `contextpress/pipeline.py`. Tests assume Tier 1 is deterministic (no live LLM calls in `tests/`).
+
+## Stabilization backlog
+
+See [`AUDIT.md`](AUDIT.md) for pre-0.6.0 findings (contract gaps, refactors, fixture gaps). Prefer low-risk items from that backlog before adding stages or new dependencies.
+
+## Adding a stage
+
+1. Subclass `BaseStrategy`, implement `process`.
+2. Register it in `Pipeline._build_strategy`.
+3. Add the name to `compression.STAGE_ORDER` and presets if it should be user-selectable.
+4. Extend `profiles.Profile` / `PROFILES` if it needs per-type defaults.

contextpress-0.5.6/AUDIT.md

@@ -0,0 +1,89 @@
+# Pre-0.6.0 stabilization audit (0.5.5)
+
+Recorded after the 0.5.x ergonomics arc. This is the Phase 1 deliverable:
+findings + backlog for Phases 2–4. Behavior changes are deferred unless marked
+**fix now**.
+
+## Architecture (summary)
+
+```
+ContextManager (core.py)
+  → normalize_messages (normalizer.py)
+  → apply_stage_selection (compression.py)
+  → Pipeline.run (pipeline.py)
+      → filler → repetition → resolution → recency → budget
+      → optional Tier 2 LLM
+  → denormalize_output
+```
+
+Profiles: `chat`, `rag_doc`, `agent`. Tier 1 deps: nltk, scikit-learn, sumy, tiktoken.
+
+## Findings
+
+### Contract / correctness
+
+| ID | Severity | Finding | Action |
+|----|----------|---------|--------|
+| C1 | Medium | Invariant 1 says system turns are untouched; `BudgetStrategy` truncates system as last resort (`budget.py`) | Document in contract (Phase 2) or change behavior later |
+| C2 | Medium | LLM dedup failures in `Pipeline._run_llm_tier` fall back silently (no warning); summarize path warns | Add warning (Phase 2) |
+| C3 | Medium | Filler drops ack phrases like `"sounds good"` before resolution can use them (`filler.py` / `resolution.py`) | Document + fixture test (Phases 3–4) |
+| C4 | Low | `Turn.importance` / `Turn.resolved` set by resolution but unused by pipeline | Leave; optional cleanup later |
+
+### Duplication / maintainability (Phase 2)
+
+| ID | Finding | Action |
+|----|---------|--------|
+| R1 | TF-IDF + cosine similarity copied in `repetition.py`, `recency.py`, `resolution.py` | Extract shared helper |
+| R2 | Token counting duplicated in `stats.py` and `budget.py` | Budget uses `stats` helpers |
+| R3 | `_bootstrap.py` downloads unused `stopwords`; warning mentions recency (doesn't use NLTK) | Trim packages; fix message |
+| R4 | Strategies use ad hoc `deepcopy` vs `clone_turn` | Prefer `clone_turn` where safe |
+
+### Test / fixture gaps (Phases 3–4)
+
+| ID | Gap |
+|----|-----|
+| T1 | No assert that input `messages` are immutable after `compress()` |
+| T2 | No end-to-end `rag_doc` / richer `agent` pipeline tests |
+| T3 | Normalizer edge cases: `None`, bad types, tuple skip, LangChain turn-count mismatch |
+| T4 | No offline real-chat fixtures under `tests/fixtures/` |
+| T5 | Cross-stage filler → resolution interaction untested |
+
+### Normalizer fragility (document; fix only if fixtures break)
+
+| ID | Area |
+|----|------|
+| N1 | LangChain detection heuristic on first list element |
+| N2 | Dict/tuple skips change turn count vs input |
+| N3 | New turns (RESOLVED / LLM summary) lose `_original_dict` keys |
+| N4 | `apply_text_to_turn` only replaces the first text block |
+
+### Deferred (not in 0.5.x stabilization)
+
+- Resolution / normalizer rewrites
+- New pipeline stages
+- New LLM backends
+- New hard dependencies
+- Changing stage order without a proven bug
+
+## Phase plan
+
+| Phase | Version | Scope |
+|-------|---------|--------|
+| 1 Analysis | **0.5.5** | This document + backlog — shipped |
+| 2 Refactor | **0.5.6** | R1–R3, C1 doc, C2 warning (no semantic stage changes) — shipped |
+| 3 Fixtures | **0.5.7** | Offline chat samples + smoke/invariant tests |
+| 4 Fixes | **0.5.8** | Bugs proven by fixtures + T1–T5 regression tests |
+
+## Invariants checklist (from `pipeline.py`)
+
+1. System turns pass through stages untouched — **except** budget last-resort truncation (C1).
+2. Input never mutated — enforced by clones; **needs T1**.
+3. Output format mirrors input — denormalize; weaker after turn collapse.
+4. Last 3 non-system not compressed by recency — unit tested.
+5. Last 2 non-system not removed by budget — unit tested.
+6. Resolution (chat) needs both sides — unit tested; C3 may weaken signals.
+7. Repetition: more recent wins — unit tested.
+8. Tier 1 deterministic — assumed; sumy/NLTK have fallbacks.
+9. LLM failure → Tier 1 + warning — summarize yes; dedup **C2**.
+10. `token_budget=None` skips budget enforcement — OK.
+11–12. Presets + Tier 2 modes — covered by existing tests.

{contextpress-0.5.4 → contextpress-0.5.6}/CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to `contextpress` are recorded here.
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.6] - 2026-07-26
+
+- **Stabilization Phase 2** — low-risk refactors (see [`AUDIT.md`](AUDIT.md)):
+  - Shared ``tfidf_cosine`` / ``tfidf_similarity_matrix`` in ``contextpress.text_sim``.
+  - ``BudgetStrategy`` uses ``stats.get_encoding`` / ``count_turn_tokens``.
+  - NLTK bootstrap drops unused ``stopwords``; warning text fixed.
+  - Behavior contract clarifies budget may truncate system as last resort.
+  - LLM deduplicate failures now emit a warning (invariant 9).
+
+## [0.5.5] - 2026-07-26
+
+- **Stabilization Phase 1** — codebase audit and prioritized backlog in [`AUDIT.md`](AUDIT.md).
+- No runtime API changes; prep for low-risk refactors (0.5.6) and fixture-driven fixes (0.5.7–0.5.8).
+
 ## [0.5.4] - 2026-07-25
 
 - **`compress_many()`** — batch ``compress()`` over a list of conversations.

{contextpress-0.5.4 → contextpress-0.5.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: contextpress
-Version: 0.5.4
+Version: 0.5.6
 Summary: Deterministic context compression for LLM chat, RAG, and agent pipelines
 Project-URL: Homepage, https://github.com/Taha-azizi/contextpress
 Project-URL: Documentation, https://github.com/Taha-azizi/contextpress#readme
@@ -561,6 +561,7 @@ For academic use, cite this package in your paper’s software or methods sectio
 
 ## Extension and growth
 
+- **Stabilization audit** — See [`AUDIT.md`](AUDIT.md) for known contract gaps and the 0.5.5–0.5.8 backlog (refactors, fixtures, fixes).
 - **Custom stages** — Subclass `contextpress.strategies.base.BaseStrategy` and plug in via a custom `Pipeline` subclass or future registry hooks.
 - **Tier 2** — Implement `LLMBackend` (`summarize`, `deduplicate`) for provider-specific semantic compression; failures fall back to Tier 1.
 - **Presets API** — `from contextpress.compression import VALID_STAGES, STAGE_ORDER` for tooling and experiments.

{contextpress-0.5.4 → contextpress-0.5.6}/README.md

@@ -320,6 +320,7 @@ For academic use, cite this package in your paper’s software or methods sectio
 
 ## Extension and growth
 
+- **Stabilization audit** — See [`AUDIT.md`](AUDIT.md) for known contract gaps and the 0.5.5–0.5.8 backlog (refactors, fixtures, fixes).
 - **Custom stages** — Subclass `contextpress.strategies.base.BaseStrategy` and plug in via a custom `Pipeline` subclass or future registry hooks.
 - **Tier 2** — Implement `LLMBackend` (`summarize`, `deduplicate`) for provider-specific semantic compression; failures fall back to Tier 1.
 - **Presets API** — `from contextpress.compression import VALID_STAGES, STAGE_ORDER` for tooling and experiments.

{contextpress-0.5.4 → contextpress-0.5.6}/contextpress/__init__.py

@@ -16,7 +16,7 @@ __all__ = [
     "CompressionResult",
     "CompressionStats",
 ]
-__version__ = "0.5.4"
+__version__ = "0.5.6"
 
 
 def __getattr__(name: str) -> Any:

{contextpress-0.5.4 → contextpress-0.5.6}/contextpress/_bootstrap.py

@@ -17,7 +17,8 @@ def bootstrap_nltk() -> None:
     flag = Path.home() / ".contextpress" / "nltk_ready"
     if flag.exists():
         return
-    packages = ["punkt", "stopwords", "averaged_perceptron_tagger"]
+    # Used by ResolutionStrategy._noun_subject (word_tokenize + pos_tag).
+    packages = ["punkt", "averaged_perceptron_tagger"]
     failed: list[str] = []
     for pkg in packages:
         try:
@@ -27,7 +28,7 @@ def bootstrap_nltk() -> None:
     if failed:
         warnings.warn(
             f"contextpress: could not download NLTK data {failed}; "
-            "resolution/recency may degrade. Check network or set NLTK_DATA.",
+            "resolution subject extraction may degrade. Check network or set NLTK_DATA.",
             stacklevel=2,
         )
     flag.parent.mkdir(parents=True, exist_ok=True)

{contextpress-0.5.4 → contextpress-0.5.6}/contextpress/pipeline.py

@@ -1,7 +1,9 @@
 """
 CONTEXTPRESS BEHAVIOR CONTRACT
 ===============================
-1. system turns are ALWAYS passed through all stages untouched.
+1. system turns are ALWAYS passed through NLP stages untouched. Budget may
+   truncate system content as a last resort when the token cap cannot otherwise
+   be met (emits a warning).
 2. Input is NEVER mutated. Always return new objects.
 3. Output format ALWAYS mirrors input format.
 4. Most recent 3 non-system turns are NEVER compressed by Stage 4.
@@ -203,6 +205,10 @@ class Pipeline:
             try:
                 keep_idx = self.llm_backend.deduplicate(texts)
             except Exception:
+                warnings.warn(
+                    "contextpress: LLM deduplicate failed; keeping all non-system turns",
+                    stacklevel=2,
+                )
                 keep_idx = list(range(len(ns_turns)))
             valid = sorted(
                 {

{contextpress-0.5.4 → contextpress-0.5.6}/contextpress/strategies/budget.py

@@ -6,20 +6,10 @@ import warnings
 import tiktoken
 
 from contextpress.models import Conversation, Turn
-from contextpress.normalizer import extract_text_for_processing
+from contextpress.stats import count_turn_tokens, get_encoding
 from contextpress.strategies.base import BaseStrategy
 
 
-def _turn_tokens(turn: Turn, encoding: tiktoken.Encoding) -> int:
-    role = turn.role
-    if isinstance(turn.content, str):
-        body = turn.content
-    else:
-        body = extract_text_for_processing(turn)
-    text = f"{role}\n{body}"
-    return len(encoding.encode(text))
-
-
 def _truncate_system_turn(turn: Turn, encoding: tiktoken.Encoding, max_tokens: int) -> Turn:
     if isinstance(turn.content, str):
         tokens = encoding.encode(turn.content)
@@ -62,20 +52,12 @@ class BudgetStrategy(BaseStrategy):
         self.token_budget = int(token_budget)
         self.model = model
 
-    def _encoding(self) -> tiktoken.Encoding:
-        if self.model:
-            try:
-                return tiktoken.encoding_for_model(self.model)
-            except KeyError:
-                pass
-        return tiktoken.get_encoding("cl100k_base")
-
     def process(self, conversation: Conversation) -> Conversation:
-        enc = self._encoding()
+        enc = get_encoding(self.model)
         turns: list[Turn] = [copy.deepcopy(t) for t in conversation.turns]
 
         def total_toks(ts: list[Turn]) -> int:
-            return sum(_turn_tokens(t, enc) for t in ts)
+            return sum(count_turn_tokens(t, enc) for t in ts)
 
         if total_toks(turns) <= self.token_budget:
             return Conversation(
@@ -96,7 +78,7 @@ class BudgetStrategy(BaseStrategy):
                 n_removed += 1
                 continue
 
-            # No non-system left to remove (or only protected pair) — truncate system from end
+            # Last resort: truncate system (see behavior contract note on invariant 1)
             warnings.warn(
                 "contextpress: truncating system prompt to satisfy token budget",
                 stacklevel=2,
@@ -104,7 +86,7 @@ class BudgetStrategy(BaseStrategy):
             for si, t in enumerate(turns):
                 if t.role != "system":
                     continue
-                others = total_toks(turns) - _turn_tokens(t, enc)
+                others = total_toks(turns) - count_turn_tokens(t, enc)
                 room = max(1, self.token_budget - others)
                 turns[si] = _truncate_system_turn(t, enc, room)
             break

{contextpress-0.5.4 → contextpress-0.5.6}/contextpress/strategies/recency.py

@@ -3,8 +3,6 @@ from __future__ import annotations
 import copy
 import re
 
-from sklearn.feature_extraction.text import TfidfVectorizer
-from sklearn.metrics.pairwise import cosine_similarity
 from sumy.nlp.tokenizers import Tokenizer
 from sumy.parsers.plaintext import PlaintextParser
 from sumy.summarizers.lsa import LsaSummarizer
@@ -12,6 +10,7 @@ from sumy.summarizers.lsa import LsaSummarizer
 from contextpress.models import Conversation, Turn
 from contextpress.normalizer import apply_text_to_turn, extract_text_for_processing
 from contextpress.strategies.base import BaseStrategy
+from contextpress.text_sim import tfidf_cosine
 
 _SENT_SPLIT = re.compile(r"(?<=[.!?])\s+")
 
@@ -126,11 +125,4 @@ class RecencyStrategy(BaseStrategy):
         )
 
     def _relevance_score(self, query: str, chunk: str) -> float:
-        if not query.strip() or not chunk.strip():
-            return 0.0
-        try:
-            vec = TfidfVectorizer(min_df=1, max_df=1.0)
-            mat = vec.fit_transform([query, chunk])
-            return float(cosine_similarity(mat[0:1], mat[1:2])[0, 0])
-        except ValueError:
-            return 0.0
+        return tfidf_cosine(query, chunk)

{contextpress-0.5.4 → contextpress-0.5.6}/contextpress/strategies/repetition.py

@@ -2,12 +2,10 @@ from __future__ import annotations
 
 import copy
 
-from sklearn.feature_extraction.text import TfidfVectorizer
-from sklearn.metrics.pairwise import cosine_similarity
-
 from contextpress.models import Conversation, Turn
 from contextpress.normalizer import extract_text_for_processing
 from contextpress.strategies.base import BaseStrategy
+from contextpress.text_sim import tfidf_similarity_matrix
 
 
 def _threshold_for_aggr(aggressiveness: float) -> float:
@@ -55,12 +53,9 @@ class RepetitionStrategy(BaseStrategy):
                 g_idx.append(gi)
             if len(texts) < 2:
                 return
-            vec = TfidfVectorizer(min_df=1, max_df=1.0)
-            try:
-                mat = vec.fit_transform(texts)
-            except ValueError:
+            sim = tfidf_similarity_matrix(texts)
+            if sim is None:
                 return
-            sim = cosine_similarity(mat)
             n = len(g_idx)
             for a in range(n):
                 for b in range(a + 1, n):

{contextpress-0.5.4 → contextpress-0.5.6}/contextpress/strategies/resolution.py

@@ -4,12 +4,11 @@ import copy
 import re
 
 import nltk
-from sklearn.feature_extraction.text import TfidfVectorizer
-from sklearn.metrics.pairwise import cosine_similarity
 
 from contextpress.models import Conversation, Turn
 from contextpress.normalizer import extract_text_for_processing
 from contextpress.strategies.base import BaseStrategy
+from contextpress.text_sim import tfidf_cosine
 
 LAYER_A_PHRASES = [
     "let's go with",
@@ -78,12 +77,7 @@ def _find_thread_start(conv: Conversation, res_idx: int, max_lookback: int = 20)
         if turns[i].role == "system":
             break
         ttext = extract_text_for_processing(turns[i])
-        try:
-            vec = TfidfVectorizer(min_df=1, max_df=1.0)
-            mat = vec.fit_transform([ttext, res_text])
-            sim = cosine_similarity(mat[0:1], mat[1:2])[0, 0]
-        except ValueError:
-            sim = 0.0
+        sim = tfidf_cosine(ttext, res_text)
         if sim >= 0.3:
             start = i
         else:

contextpress-0.5.6/contextpress/text_sim.py

@@ -0,0 +1,32 @@
+"""Shared TF-IDF cosine helpers for Tier 1 strategies."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.metrics.pairwise import cosine_similarity
+
+
+def tfidf_cosine(a: str, b: str) -> float:
+    """Cosine similarity of two texts; ``0.0`` if vectors cannot be built."""
+    if not a.strip() or not b.strip():
+        return 0.0
+    try:
+        vec = TfidfVectorizer(min_df=1, max_df=1.0)
+        mat = vec.fit_transform([a, b])
+        return float(cosine_similarity(mat[0:1], mat[1:2])[0, 0])
+    except ValueError:
+        return 0.0
+
+
+def tfidf_similarity_matrix(texts: list[str]) -> Any | None:
+    """Pairwise cosine matrix for ``texts``, or ``None`` if unfit."""
+    if len(texts) < 2:
+        return None
+    try:
+        vec = TfidfVectorizer(min_df=1, max_df=1.0)
+        mat = vec.fit_transform(texts)
+        return cosine_similarity(mat)
+    except ValueError:
+        return None

{contextpress-0.5.4 → contextpress-0.5.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "contextpress"
-version = "0.5.4"
+version = "0.5.6"
 description = "Deterministic context compression for LLM chat, RAG, and agent pipelines"
 readme = "README.md"
 license = { file = "LICENSE" }
@@ -59,7 +59,7 @@ packages = ["contextpress"]
 "contextpress/py.typed" = "contextpress/py.typed"
 
 [tool.hatch.build.targets.sdist]
-include = ["/contextpress", "/tests", "/examples", "/README.md", "/LICENSE", "/NOTICE", "/CHANGELOG.md", "/CITATION.cff", "/CONTRIBUTING.md"]
+include = ["/contextpress", "/tests", "/examples", "/README.md", "/LICENSE", "/NOTICE", "/CHANGELOG.md", "/CITATION.cff", "/CONTRIBUTING.md", "/AUDIT.md", "/AGENTS.md"]
 
 [tool.black]
 line-length = 100

contextpress-0.5.6/tests/test_v056.py

@@ -0,0 +1,39 @@
+from unittest.mock import MagicMock
+
+import pytest
+
+from contextpress import ContextManager
+from contextpress.text_sim import tfidf_cosine, tfidf_similarity_matrix
+
+
+def test_tfidf_cosine_identical():
+    assert tfidf_cosine("hello world", "hello world") == pytest.approx(1.0)
+
+
+def test_tfidf_cosine_empty():
+    assert tfidf_cosine("", "hello") == 0.0
+
+
+def test_tfidf_similarity_matrix_shape():
+    mat = tfidf_similarity_matrix(["alpha beta", "alpha gamma", "unrelated words here"])
+    assert mat is not None
+    assert mat.shape == (3, 3)
+
+
+def test_llm_dedup_failure_emits_warning():
+    backend = MagicMock()
+    backend.deduplicate.side_effect = RuntimeError("boom")
+
+    cm = ContextManager(
+        type="chat",
+        llm_backend=backend,
+        llm_min_input_chars=1,
+        llm_mode="dedupe_only",
+    )
+    messages = [
+        {"role": "user", "content": "first turn about databases"},
+        {"role": "assistant", "content": "second turn about databases"},
+    ]
+    result = cm.compress(messages, token_budget=None, return_stats=True)
+    assert any("LLM deduplicate failed" in w for w in result.stats.warnings_emitted)
+    assert len(result.messages) == 2