chatfit 0.4.0__tar.gz

chatfit-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Anandita Singh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

chatfit-0.4.0/PKG-INFO

@@ -0,0 +1,156 @@
+Metadata-Version: 2.4
+Name: chatfit
+Version: 0.4.0
+Summary: Trim conversation history to fit an LLM token budget.
+Author: Anandita Singh
+License: MIT
+Project-URL: Homepage, https://github.com/ananditasinghh/chatfit
+Project-URL: Issues, https://github.com/ananditasinghh/chatfit/issues
+Keywords: llm,chat,tokens,context-window,rag,openai,anthropic
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: tiktoken
+Requires-Dist: tiktoken>=0.5; extra == "tiktoken"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: tiktoken>=0.5; extra == "dev"
+Dynamic: license-file
+
+# chatfit
+
+**Trim conversation history to fit an LLM token budget — without forgetting.**
+
+When a chat with an LLM gets long, you eventually blow past the model's context
+window and the API errors out. `chatfit` trims the conversation down to a token
+budget you choose. It keeps the system prompt and the most recent turns, and
+**condenses the older turns into a single summary** so the model retains the
+gist of earlier context instead of forgetting it.
+
+> `contextfit` packs your RAG chunks. **`chatfit` packs your chat history.**
+
+- 🧠 **Remembers, doesn't just delete** — old turns become a summary
+- 🪶 **Tiny & dependency-free** — pure Python, `tiktoken` optional
+- 📌 **Pins your system prompt** so it's never dropped
+- ✅ **Always fits** — even an oversized summary is truncated to the budget
+- 📊 **Tells you what happened** — tokens before/after, messages dropped
+
+## Install
+
+```bash
+pip install chatfit               # pure-Python word-count estimate
+pip install "chatfit[tiktoken]"   # accurate token counts
+```
+
+## Quick start
+
+```python
+from chatfit import fit
+
+messages = [
+    {"role": "system",    "content": "You are a helpful assistant."},
+    {"role": "user",      "content": "Hi!"},
+    {"role": "assistant", "content": "Hello! How can I help?"},
+    # ... 50 more turns ...
+]
+
+result = fit(messages, max_tokens=4000)
+
+send_to_llm(result.messages)     # guaranteed to fit in 4000 tokens
+print(result)                    # what got trimmed and why
+```
+
+## How it works
+
+1. If the conversation already fits the budget → returned unchanged.
+2. Otherwise: keep the system prompt + the newest turns that fit.
+3. The older turns are condensed into one `[Summary of earlier conversation]`
+   message so their gist is preserved.
+4. The result is **guaranteed** to fit `max_tokens`.
+
+## Bring your own summarizer
+
+`chatfit` never calls an LLM itself. By default it uses a no-LLM summarizer that
+lists the topics the user raised. For real AI summaries, pass your own:
+
+```python
+def my_summarizer(dropped_messages):
+    text = "\n".join(m["content"] for m in dropped_messages)
+    return openai.chat.completions.create(
+        model="gpt-4o-mini",
+        messages=[{"role": "user", "content": f"Summarize:\n{text}"}],
+    ).choices[0].message.content
+
+result = fit(messages, max_tokens=4000, summarizer=my_summarizer)
+```
+
+## `ChatMemory` — rolling memory for ongoing chats
+
+`fit()` is one-shot. For a live conversation, use `ChatMemory`: you `add()`
+turns as they happen and it keeps recent turns verbatim while *incrementally*
+folding older ones into a single rolling summary — far cheaper than
+re-summarizing from scratch every turn, and always within budget.
+
+```python
+from chatfit import ChatMemory
+
+mem = ChatMemory(max_tokens=2000, summarizer=my_llm_summarizer)
+mem.set_system("You are a helpful assistant.")
+
+mem.add_user("Hi!")
+mem.add_assistant("Hello! How can I help?")
+# ... many turns later ...
+
+messages = mem.render()   # always fits 2000 tokens; oldest turns summarized
+response = openai.chat.completions.create(model="gpt-4", messages=messages)
+```
+
+The summary stays bounded (hierarchical): each fold re-summarizes the previous
+summary together with the newly dropped turn, so it never grows without limit.
+
+## The `fit()` function
+
+```python
+fit(
+    messages,            # list of {"role": ..., "content": ...} dicts
+    max_tokens,          # the budget the result must fit within
+    pin_system=True,     # never drop system messages
+    model="gpt-4",       # used for token counting
+    summarizer=None,     # your callable; defaults to a built-in no-LLM one
+)
+```
+
+Returns a `TrimResult`:
+
+| Attribute | Meaning |
+|---|---|
+| `.messages` | the trimmed conversation |
+| `.tokens_before` / `.tokens_after` | token counts before/after |
+| `.tokens_saved` | tokens removed |
+| `.dropped_count` / `.kept_count` | original messages dropped / messages kept |
+| `.fits` | is it within budget? |
+| `.was_trimmed` | did anything get dropped? |
+
+## Run the demo & tests
+
+```bash
+pip install -e ".[dev]"
+python examples/demo.py
+python examples/try_it.py
+pytest
+```
+
+## Roadmap
+
+- `keep_relevant` — keep the most *relevant* old turns, not just the newest
+  (powered by the relevance engine from its sister library, `contextfit`)
+- semantic de-duplication of repeated turns
+- auto-detect a model's context window
+
+## License
+
+MIT

chatfit-0.4.0/README.md

@@ -0,0 +1,133 @@
+# chatfit
+
+**Trim conversation history to fit an LLM token budget — without forgetting.**
+
+When a chat with an LLM gets long, you eventually blow past the model's context
+window and the API errors out. `chatfit` trims the conversation down to a token
+budget you choose. It keeps the system prompt and the most recent turns, and
+**condenses the older turns into a single summary** so the model retains the
+gist of earlier context instead of forgetting it.
+
+> `contextfit` packs your RAG chunks. **`chatfit` packs your chat history.**
+
+- 🧠 **Remembers, doesn't just delete** — old turns become a summary
+- 🪶 **Tiny & dependency-free** — pure Python, `tiktoken` optional
+- 📌 **Pins your system prompt** so it's never dropped
+- ✅ **Always fits** — even an oversized summary is truncated to the budget
+- 📊 **Tells you what happened** — tokens before/after, messages dropped
+
+## Install
+
+```bash
+pip install chatfit               # pure-Python word-count estimate
+pip install "chatfit[tiktoken]"   # accurate token counts
+```
+
+## Quick start
+
+```python
+from chatfit import fit
+
+messages = [
+    {"role": "system",    "content": "You are a helpful assistant."},
+    {"role": "user",      "content": "Hi!"},
+    {"role": "assistant", "content": "Hello! How can I help?"},
+    # ... 50 more turns ...
+]
+
+result = fit(messages, max_tokens=4000)
+
+send_to_llm(result.messages)     # guaranteed to fit in 4000 tokens
+print(result)                    # what got trimmed and why
+```
+
+## How it works
+
+1. If the conversation already fits the budget → returned unchanged.
+2. Otherwise: keep the system prompt + the newest turns that fit.
+3. The older turns are condensed into one `[Summary of earlier conversation]`
+   message so their gist is preserved.
+4. The result is **guaranteed** to fit `max_tokens`.
+
+## Bring your own summarizer
+
+`chatfit` never calls an LLM itself. By default it uses a no-LLM summarizer that
+lists the topics the user raised. For real AI summaries, pass your own:
+
+```python
+def my_summarizer(dropped_messages):
+    text = "\n".join(m["content"] for m in dropped_messages)
+    return openai.chat.completions.create(
+        model="gpt-4o-mini",
+        messages=[{"role": "user", "content": f"Summarize:\n{text}"}],
+    ).choices[0].message.content
+
+result = fit(messages, max_tokens=4000, summarizer=my_summarizer)
+```
+
+## `ChatMemory` — rolling memory for ongoing chats
+
+`fit()` is one-shot. For a live conversation, use `ChatMemory`: you `add()`
+turns as they happen and it keeps recent turns verbatim while *incrementally*
+folding older ones into a single rolling summary — far cheaper than
+re-summarizing from scratch every turn, and always within budget.
+
+```python
+from chatfit import ChatMemory
+
+mem = ChatMemory(max_tokens=2000, summarizer=my_llm_summarizer)
+mem.set_system("You are a helpful assistant.")
+
+mem.add_user("Hi!")
+mem.add_assistant("Hello! How can I help?")
+# ... many turns later ...
+
+messages = mem.render()   # always fits 2000 tokens; oldest turns summarized
+response = openai.chat.completions.create(model="gpt-4", messages=messages)
+```
+
+The summary stays bounded (hierarchical): each fold re-summarizes the previous
+summary together with the newly dropped turn, so it never grows without limit.
+
+## The `fit()` function
+
+```python
+fit(
+    messages,            # list of {"role": ..., "content": ...} dicts
+    max_tokens,          # the budget the result must fit within
+    pin_system=True,     # never drop system messages
+    model="gpt-4",       # used for token counting
+    summarizer=None,     # your callable; defaults to a built-in no-LLM one
+)
+```
+
+Returns a `TrimResult`:
+
+| Attribute | Meaning |
+|---|---|
+| `.messages` | the trimmed conversation |
+| `.tokens_before` / `.tokens_after` | token counts before/after |
+| `.tokens_saved` | tokens removed |
+| `.dropped_count` / `.kept_count` | original messages dropped / messages kept |
+| `.fits` | is it within budget? |
+| `.was_trimmed` | did anything get dropped? |
+
+## Run the demo & tests
+
+```bash
+pip install -e ".[dev]"
+python examples/demo.py
+python examples/try_it.py
+pytest
+```
+
+## Roadmap
+
+- `keep_relevant` — keep the most *relevant* old turns, not just the newest
+  (powered by the relevance engine from its sister library, `contextfit`)
+- semantic de-duplication of repeated turns
+- auto-detect a model's context window
+
+## License
+
+MIT

chatfit-0.4.0/chatfit/__init__.py

@@ -0,0 +1,36 @@
+"""chatfit — trim conversation history to fit an LLM token budget.
+
+contextfit packs your RAG chunks; chatfit packs your chat history.
+
+chatfit keeps the newest turns that fit your token budget and condenses the
+older ones into a single summary message, so the model keeps the gist of
+earlier context instead of forgetting it.
+
+Basic usage:
+
+    from chatfit import fit
+
+    result = fit(messages, max_tokens=4000)
+    print(result.messages)        # the trimmed conversation
+    print(result.tokens_after)    # how many tokens it now uses
+
+    # Pass your own LLM-backed summarizer for richer summaries:
+    result = fit(messages, max_tokens=4000, summarizer=my_llm_summarizer)
+"""
+
+from .core import default_summarizer, fit
+from .memory import ChatMemory
+from .result import TrimResult
+from .tokens import count_message_tokens, count_tokens
+
+__version__ = "0.4.0"
+
+__all__ = [
+    "fit",
+    "ChatMemory",
+    "default_summarizer",
+    "TrimResult",
+    "count_tokens",
+    "count_message_tokens",
+    "__version__",
+]

chatfit-0.4.0/chatfit/core.py

@@ -0,0 +1,268 @@
+"""The core :func:`fit` function: trim a conversation to a token budget.
+
+chatfit keeps the newest turns that fit and replaces the older ones with a
+single summary message, so the model retains the gist of earlier context
+instead of forgetting it.
+
+Phase A design:
+- Budget allocation: the budget left for droppable content is split into a
+  reserved share for the summary and a share for recent verbatim turns, so the
+  summary can never starve the recent turns (and vice versa).
+- Target-length summarization: the summarizer is told its token budget so it
+  can generate-to-fit instead of being blindly truncated afterwards.
+"""
+
+from __future__ import annotations
+
+import inspect
+from typing import Any, Callable, Dict, List, Optional, Tuple, Union
+
+from .result import TrimResult
+from .tokens import (
+    TOKENS_PER_MESSAGE,
+    TOKENS_PER_REPLY,
+    count_conversation_tokens,
+    count_message_tokens,
+    count_tokens,
+    truncate_to_tokens,
+)
+
+# A summarizer takes the dropped messages (and, optionally, a target token
+# budget) and returns a summary string.
+Summarizer = Union[
+    Callable[[List[Dict[str, Any]]], str],
+    Callable[[List[Dict[str, Any]], int], str],
+]
+
+SUMMARY_PREFIX = "[Summary of earlier conversation]"
+
+# Default fraction of the droppable budget reserved for the summary.
+DEFAULT_SUMMARY_RATIO = 0.35
+
+
+def default_summarizer(
+    dropped_messages: List[Dict[str, Any]],
+    max_tokens: Optional[int] = None,
+    model: str = "gpt-4",
+) -> str:
+    """A no-LLM fallback summarizer: lists the topics the user raised.
+
+    Honors ``max_tokens`` (target-length) by trimming its own output. chatfit
+    never calls an LLM itself; pass your own ``summarizer`` to :func:`fit` for
+    real AI summaries.
+    """
+    user_lines = [
+        str(m.get("content", "")).strip()
+        for m in dropped_messages
+        if m.get("role") == "user"
+    ]
+    if user_lines:
+        text = "Earlier, the user asked about: " + "; ".join(user_lines)
+    else:
+        text = f"{len(dropped_messages)} earlier message(s) were omitted."
+
+    if max_tokens is not None:
+        text = truncate_to_tokens(text, max_tokens, model)
+    return text
+
+
+def _call_summarizer(
+    summarizer: Summarizer,
+    messages: List[Dict[str, Any]],
+    target_tokens: int,
+) -> str:
+    """Call ``summarizer``, passing ``target_tokens`` if it accepts a second arg.
+
+    This keeps backward compatibility with one-argument summarizers.
+    """
+    try:
+        params = list(inspect.signature(summarizer).parameters.values())
+        positional = [
+            p for p in params
+            if p.kind in (p.POSITIONAL_ONLY, p.POSITIONAL_OR_KEYWORD)
+        ]
+        accepts_target = len(positional) >= 2 or any(
+            p.kind == p.VAR_POSITIONAL for p in params
+        )
+    except (ValueError, TypeError):
+        accepts_target = False
+
+    if accepts_target:
+        return summarizer(messages, target_tokens)
+    return summarizer(messages)
+
+
+def _is_system(message: Dict[str, Any]) -> bool:
+    return message.get("role") == "system"
+
+
+def _split_pinned(
+    messages: List[Dict[str, Any]],
+    pin_system: bool,
+) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+    """Split messages into (pinned, droppable)."""
+    if pin_system:
+        pinned = [m for m in messages if _is_system(m)]
+        droppable = [m for m in messages if not _is_system(m)]
+    else:
+        pinned = []
+        droppable = list(messages)
+    return pinned, droppable
+
+
+def _keep_recent_turns(
+    droppable: List[Dict[str, Any]],
+    budget: int,
+    model: str,
+) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+    """Greedily keep the newest turns that fit in ``budget``.
+
+    Returns ``(kept, dropped_older)`` where ``kept`` are the most recent
+    messages that fit (in original order) and ``dropped_older`` are the older
+    messages that did not fit (also in original order).
+    """
+    running = 0
+    split = len(droppable)  # index where kept turns begin
+    for idx in range(len(droppable) - 1, -1, -1):
+        cost = count_message_tokens(droppable[idx], model)
+        if running + cost <= budget:
+            running += cost
+            split = idx
+        else:
+            # Once one message doesn't fit, stop so we never leave a gap in the
+            # middle of the conversation.
+            break
+
+    return droppable[split:], droppable[:split]
+
+
+def _drop_leading_assistant(turns: List[Dict[str, Any]]) -> None:
+    """Drop leading assistant replies whose user turn was removed (in place)."""
+    while turns and turns[0].get("role") == "assistant":
+        turns.pop(0)
+
+
+def _summary_fixed_overhead(model: str) -> int:
+    """Tokens used by a summary message *before* the summarizer's own text.
+
+    That is: per-message overhead + role + the ``SUMMARY_PREFIX`` line.
+    """
+    stub = {"role": "system", "content": f"{SUMMARY_PREFIX}\n"}
+    return count_message_tokens(stub, model)
+
+
+def fit(
+    messages: List[Dict[str, Any]],
+    max_tokens: int,
+    *,
+    pin_system: bool = True,
+    model: str = "gpt-4",
+    summarizer: Optional[Summarizer] = None,
+    summary_ratio: float = DEFAULT_SUMMARY_RATIO,
+) -> TrimResult:
+    """Trim a conversation so it fits within ``max_tokens``.
+
+    The newest turns that fit are kept; the older turns are condensed into a
+    single summary message so the model retains the gist of earlier context.
+
+    Budget allocation: after reserving room for pinned messages, the remaining
+    budget is split between a reserved share for the summary
+    (``summary_ratio``) and the recent verbatim turns. The summary may also
+    expand into any budget the recent turns leave unused.
+
+    Args:
+        messages: A list of chat messages, each a dict with at least ``role``
+            and ``content`` keys (OpenAI-style).
+        max_tokens: The token budget the result must fit within.
+        pin_system: If True, system messages are always kept and never counted
+            as droppable, even if they alone exceed the budget.
+        model: Model name used for token counting (passed to tiktoken).
+        summarizer: A callable taking the dropped messages (and optionally a
+            target token budget as a second argument) and returning a summary
+            string. Defaults to :func:`default_summarizer` (no LLM).
+        summary_ratio: Fraction (0-1) of the droppable budget reserved for the
+            summary. Defaults to 0.35.
+
+    Returns:
+        A :class:`TrimResult` with the trimmed messages and stats.
+
+    Raises:
+        ValueError: If ``max_tokens`` is not positive or ``summary_ratio`` is
+            not strictly between 0 and 1.
+    """
+    if max_tokens <= 0:
+        raise ValueError(f"max_tokens must be positive, got {max_tokens}")
+    if not 0.0 < summary_ratio < 1.0:
+        raise ValueError(
+            f"summary_ratio must be between 0 and 1, got {summary_ratio}"
+        )
+    if summarizer is None:
+        summarizer = default_summarizer
+
+    tokens_before = count_conversation_tokens(messages, model)
+    original_count = len(messages)
+
+    # Nothing to do if it already fits.
+    if tokens_before <= max_tokens:
+        return TrimResult(
+            messages=list(messages),
+            tokens_before=tokens_before,
+            tokens_after=tokens_before,
+            dropped_count=0,
+            max_tokens=max_tokens,
+        )
+
+    pinned, droppable = _split_pinned(messages, pin_system)
+    pinned_tokens = sum(count_message_tokens(m, model) for m in pinned)
+    budget = max_tokens - pinned_tokens - TOKENS_PER_REPLY
+
+    # #1 Budget allocation: reserve a share for the summary so the recent turns
+    # cannot consume the whole budget and leave no room to remember the past.
+    summary_budget = max(TOKENS_PER_MESSAGE + 1, round(budget * summary_ratio))
+    recent_budget = max(0, budget - summary_budget)
+
+    kept_turns, dropped_older = _keep_recent_turns(droppable, recent_budget, model)
+
+    if dropped_older:
+        recent_used = sum(count_message_tokens(m, model) for m in kept_turns)
+
+        # The summary may use its reserved share plus anything the recent turns
+        # left unused — so we never waste budget.
+        summary_allowance = budget - recent_used
+
+        # #2 Target-length summarization: tell the summarizer how many tokens it
+        # has for its own text, so it generates to fit rather than overflowing.
+        fixed_overhead = _summary_fixed_overhead(model)
+        target_text_tokens = max(1, summary_allowance - fixed_overhead)
+
+        summary_text = _call_summarizer(summarizer, dropped_older, target_text_tokens)
+        summary_body = f"{SUMMARY_PREFIX}\n{summary_text}"
+        summary_msg = {"role": "system", "content": summary_body}
+
+        # Safety net: if the summarizer ignored the target, truncate to fit.
+        if count_message_tokens(summary_msg, model) > summary_allowance:
+            overhead = count_message_tokens(summary_msg, model) - count_tokens(
+                summary_body, model
+            )
+            summary_msg["content"] = truncate_to_tokens(
+                summary_body, max(1, summary_allowance - overhead), model
+            )
+
+        _drop_leading_assistant(kept_turns)
+        trimmed = pinned + [summary_msg] + kept_turns
+    else:
+        _drop_leading_assistant(kept_turns)
+        trimmed = pinned + kept_turns
+
+    tokens_after = count_conversation_tokens(trimmed, model)
+    # How many of the ORIGINAL messages are no longer present individually.
+    # (A synthetic summary message is not counted as an original survivor.)
+    dropped_count = original_count - (len(pinned) + len(kept_turns))
+
+    return TrimResult(
+        messages=trimmed,
+        tokens_before=tokens_before,
+        tokens_after=tokens_after,
+        dropped_count=dropped_count,
+        max_tokens=max_tokens,
+    )