leancontext 2.0.5__tar.gz → 2.0.6__tar.gz

{leancontext-2.0.5 → leancontext-2.0.6}/CHANGELOG.md

@@ -5,6 +5,26 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [2.0.6] - 2026-06-21
+
+### Fixed
+- JSON reducer is now lossless on every value: rows are emitted as JSON arrays with
+  the field names factored into the header once, so values containing the column
+  delimiter, quotes, or newlines no longer corrupt the columnar layout. The JSON
+  fidelity check matches values in their encoded form, so it sees such corruption.
+- Gateway paths (LiteLLM proxy + SDK patch) now reduce OpenAI Responses requests
+  (`input=`), not just chat (`messages=`).
+- `reduce_messages` dispatches per item, so a list mixing message formats reduces
+  every tool output instead of only those matching the first format detected.
+- OpenAI Responses tool outputs shaped as a list of content parts are now reduced.
+- `__version__` is read from the installed package metadata (was a stale `0.0.1`).
+- `CostTracker` running totals are guarded by a lock for multi-threaded agents.
+
+### Docs
+- README install commands use the published package (`pip install leancontext`),
+  document the `mcp` extra, note which tokenizer the benchmark uses, and state
+  which integrations are CI-verified vs best-effort.
+
 ## [2.0.5] - 2026-06-21
 
 ### Security
@@ -30,6 +50,11 @@ All notable changes to this project are documented here. The format is based on
 - Lower the minimum Python from 3.14 to 3.10 so the package installs on current
   interpreters (the code already supports 3.10+; CI runs 3.10 through 3.14).
 
+## [2.0.1] - 2026-06-21
+
+Intermediate release during the initial PyPI rollout (Python version metadata),
+superseded by 2.0.2. Version 2.0.3 was never published.
+
 ## [2.0.0] - 2026-06-21
 
 ### Added
@@ -53,8 +78,10 @@ All notable changes to this project are documented here. The format is based on
 - Targets Python 3.14; ruff, mypy, and coverage run in CI; examples, contributor, and
   security docs included.
 
-[Unreleased]: https://github.com/pankajniet/LeanContext/compare/v2.0.5...HEAD
+[Unreleased]: https://github.com/pankajniet/LeanContext/compare/v2.0.6...HEAD
+[2.0.6]: https://github.com/pankajniet/LeanContext/releases/tag/v2.0.6
 [2.0.5]: https://github.com/pankajniet/LeanContext/releases/tag/v2.0.5
 [2.0.4]: https://github.com/pankajniet/LeanContext/releases/tag/v2.0.4
 [2.0.2]: https://github.com/pankajniet/LeanContext/releases/tag/v2.0.2
+[2.0.1]: https://github.com/pankajniet/LeanContext/releases/tag/v2.0.1
 [2.0.0]: https://github.com/pankajniet/LeanContext/releases/tag/v2.0.0

{leancontext-2.0.5 → leancontext-2.0.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: leancontext
-Version: 2.0.5
+Version: 2.0.6
 Summary: Deterministic, type-aware reduction of agent tool outputs at the source. Cut LLM token cost without making the agent do less.
 Project-URL: Homepage, https://github.com/pankajniet/LeanContext
 Project-URL: Repository, https://github.com/pankajniet/LeanContext
@@ -84,14 +84,18 @@ $ python bench.py
 sample              kind          before   after  saved  fidelity
 -----------------------------------------------------------------
 log (incident)      log            52642     100   100%      100%
-json (RAG chunks)   json            1862    1390    25%      100%
+json (RAG chunks)   json            1862    1391    25%      100%
 html (web fetch)    html            1672    1093    35%      100%
 diff (patch)        diff             639      81    87%      100%
 stacktrace          stacktrace       896      94    90%      100%
 -----------------------------------------------------------------
-TOTAL                              57711    2758    95%
+TOTAL                              57711    2759    95%
 ```
 
+Counts above use the built-in heuristic tokenizer (≈4 chars/token). Install the
+`tiktoken` extra for exact model token counts — the ratios are similar (~92% on
+this sample). The reduced text is identical either way.
+
 A real incident log, before and after:
 
 ```text
@@ -128,10 +132,11 @@ errors, anomalies, and identifiers, and collapses the rest.
 ## Install
 
 ```bash
-pip install -e .                  # core, standard library only
-pip install -e ".[integrations]"  # openai, anthropic, litellm, fastapi adapters
-pip install -e ".[otel]"          # OpenTelemetry metrics
-pip install -e ".[tiktoken]"      # exact token counts (used automatically when present)
+pip install leancontext                  # core, standard library only
+pip install "leancontext[integrations]"  # openai, anthropic, litellm, fastapi adapters
+pip install "leancontext[otel]"          # OpenTelemetry metrics
+pip install "leancontext[mcp]"           # MCP server
+pip install "leancontext[tiktoken]"      # exact token counts (used automatically when present)
 ```
 
 ## Use it
@@ -176,6 +181,11 @@ r.fidelity                        # 0..1 signal preserved
 | Frameworks | LangChain, LangGraph, Agno via `wrap(tools)`; any framework via `@reduce` on tool functions (sync or async) |
 | MCP server | `python -m leancontext.integrations.mcp_server` — reduce / expand / stats over stdio |
 
+CI exercises OpenAI (chat + Responses), Anthropic, LiteLLM, the standalone proxy, OpenTelemetry,
+and the MCP server against the real packages. Message reduction for all formats (including Gemini)
+is unit-tested directly. The framework adapters (LangChain / LangGraph / Agno) and the SDK-level
+Gemini client wrapper are provided best-effort and are not yet covered in CI against the live SDKs.
+
 ## Reducers
 
 | Kind | What it does |
@@ -185,6 +195,7 @@ r.fidelity                        # 0..1 signal preserved
 | `diff` | Keep all change, hunk, and header lines, collapse unchanged context |
 | `stacktrace` | Keep the exception and boundary frames, collapse the deep middle |
 | `html` | Strip tags, scripts, and styles, keep visible text and links |
+| `table` | Collapse whitespace-aligned command-line tables, keep header and data |
 
 Anything else, or any payload below the size, saving, or fidelity thresholds, passes through unchanged.
 
@@ -216,8 +227,8 @@ leancontext.use_tiktoken("gpt-4o")            # force a specific model's tokeniz
 
 ## Roadmap
 
-Accurate provider tokenizers by default, an MCP server, tested LangChain / LlamaIndex / CrewAI
-adapters, broader Anthropic native interop, and a PyPI release.
+CI-verified LangChain / LlamaIndex / CrewAI / Agno adapters, accurate provider tokenizers by
+default, and broader Anthropic native interop.
 
 ## Contributing
 

{leancontext-2.0.5 → leancontext-2.0.6}/README.md

@@ -38,14 +38,18 @@ $ python bench.py
 sample              kind          before   after  saved  fidelity
 -----------------------------------------------------------------
 log (incident)      log            52642     100   100%      100%
-json (RAG chunks)   json            1862    1390    25%      100%
+json (RAG chunks)   json            1862    1391    25%      100%
 html (web fetch)    html            1672    1093    35%      100%
 diff (patch)        diff             639      81    87%      100%
 stacktrace          stacktrace       896      94    90%      100%
 -----------------------------------------------------------------
-TOTAL                              57711    2758    95%
+TOTAL                              57711    2759    95%
 ```
 
+Counts above use the built-in heuristic tokenizer (≈4 chars/token). Install the
+`tiktoken` extra for exact model token counts — the ratios are similar (~92% on
+this sample). The reduced text is identical either way.
+
 A real incident log, before and after:
 
 ```text
@@ -82,10 +86,11 @@ errors, anomalies, and identifiers, and collapses the rest.
 ## Install
 
 ```bash
-pip install -e .                  # core, standard library only
-pip install -e ".[integrations]"  # openai, anthropic, litellm, fastapi adapters
-pip install -e ".[otel]"          # OpenTelemetry metrics
-pip install -e ".[tiktoken]"      # exact token counts (used automatically when present)
+pip install leancontext                  # core, standard library only
+pip install "leancontext[integrations]"  # openai, anthropic, litellm, fastapi adapters
+pip install "leancontext[otel]"          # OpenTelemetry metrics
+pip install "leancontext[mcp]"           # MCP server
+pip install "leancontext[tiktoken]"      # exact token counts (used automatically when present)
 ```
 
 ## Use it
@@ -130,6 +135,11 @@ r.fidelity                        # 0..1 signal preserved
 | Frameworks | LangChain, LangGraph, Agno via `wrap(tools)`; any framework via `@reduce` on tool functions (sync or async) |
 | MCP server | `python -m leancontext.integrations.mcp_server` — reduce / expand / stats over stdio |
 
+CI exercises OpenAI (chat + Responses), Anthropic, LiteLLM, the standalone proxy, OpenTelemetry,
+and the MCP server against the real packages. Message reduction for all formats (including Gemini)
+is unit-tested directly. The framework adapters (LangChain / LangGraph / Agno) and the SDK-level
+Gemini client wrapper are provided best-effort and are not yet covered in CI against the live SDKs.
+
 ## Reducers
 
 | Kind | What it does |
@@ -139,6 +149,7 @@ r.fidelity                        # 0..1 signal preserved
 | `diff` | Keep all change, hunk, and header lines, collapse unchanged context |
 | `stacktrace` | Keep the exception and boundary frames, collapse the deep middle |
 | `html` | Strip tags, scripts, and styles, keep visible text and links |
+| `table` | Collapse whitespace-aligned command-line tables, keep header and data |
 
 Anything else, or any payload below the size, saving, or fidelity thresholds, passes through unchanged.
 
@@ -170,8 +181,8 @@ leancontext.use_tiktoken("gpt-4o")            # force a specific model's tokeniz
 
 ## Roadmap
 
-Accurate provider tokenizers by default, an MCP server, tested LangChain / LlamaIndex / CrewAI
-adapters, broader Anthropic native interop, and a PyPI release.
+CI-verified LangChain / LlamaIndex / CrewAI / Agno adapters, accurate provider tokenizers by
+default, and broader Anthropic native interop.
 
 ## Contributing
 

{leancontext-2.0.5 → leancontext-2.0.6}/leancontext/__init__.py

@@ -49,7 +49,18 @@ from .integrations import (
 from .messages import detect_format, reduce_messages
 from .tokens import active_tokenizer, count_tokens, set_token_counter, use_tiktoken
 
-__version__ = "0.0.1"
+# Single source of truth is the installed package metadata (pyproject version);
+# the literal is only a fallback for running straight from a source tree.
+try:
+    from importlib.metadata import PackageNotFoundError
+    from importlib.metadata import version as _pkg_version
+
+    try:
+        __version__ = _pkg_version("leancontext")
+    except PackageNotFoundError:
+        __version__ = "2.0.6"
+except ImportError:  # pragma: no cover - importlib.metadata is stdlib on 3.10+
+    __version__ = "2.0.6"
 
 __all__ = [
     "reduce",

{leancontext-2.0.5 → leancontext-2.0.6}/leancontext/cost.py

@@ -13,6 +13,7 @@ no known price, token savings are still reported and ``usd_saved`` is ``None``.
 
 from __future__ import annotations
 
+import threading
 from collections.abc import Callable
 
 #: USD per 1M tokens (input, output). Indicative — override via set_price().
@@ -71,14 +72,17 @@ class CostTracker:
         self.usd_saved = 0.0
         self.has_price = _input_price(model, input_price_per_mtok) is not None
         self._hook: Callable | None = None
+        self._lock = threading.Lock()
 
     def _on(self, r) -> None:
-        self.reductions += 1
-        self.tokens_before += r.tokens_before
-        self.tokens_after += r.tokens_after
-        self.tokens_saved += r.tokens_saved
-        if self.has_price:
-            self.usd_saved += estimate_savings(r, self.model, self.price)["usd_saved"]
+        # The hook fires from every reducing thread, so guard the running totals.
+        usd = estimate_savings(r, self.model, self.price)["usd_saved"] if self.has_price else 0.0
+        with self._lock:
+            self.reductions += 1
+            self.tokens_before += r.tokens_before
+            self.tokens_after += r.tokens_after
+            self.tokens_saved += r.tokens_saved
+            self.usd_saved += usd
 
     def install(self) -> CostTracker:
         from .core import on_reduction

{leancontext-2.0.5 → leancontext-2.0.6}/leancontext/fidelity.py

@@ -60,12 +60,22 @@ def _iter_scalars(data: Any):
 
 
 def _json_fidelity(original: str, reduced: str) -> float:
-    """Fraction of JSON scalar values (strings and numbers) preserved in the output."""
+    """Fraction of JSON scalar values (strings and numbers) preserved in the output.
+
+    Values are matched in their JSON-encoded form (the reducer emits them that way),
+    so a value containing a delimiter, quote, or newline only counts as preserved if
+    its exact escaped bytes survive — the check sees structural corruption, not just
+    whether the characters appear somewhere.
+    """
     try:
         data = json.loads(original)
     except Exception:
         return 1.0
-    values = [str(v) for v in _iter_scalars(data) if str(v)]
+    values = [
+        json.dumps(v, ensure_ascii=False).strip('"')
+        for v in _iter_scalars(data)
+    ]
+    values = [v for v in values if v]
     if not values:
         return 1.0
     kept = sum(1 for v in values if v in reduced)

{leancontext-2.0.5 → leancontext-2.0.6}/leancontext/integrations/_common.py

@@ -25,19 +25,30 @@ def mark(fn: Callable) -> Callable:
     return fn
 
 
-def reduce_messages_in(mapping: Any, fmt: str, opts: dict, key: str = "messages") -> None:
-    """Fail-open, in-place reduction of ``mapping[key]`` (dict-like).
+#: Request keys that can carry a message/tool-output list across providers:
+#: ``messages`` (OpenAI chat / Anthropic), ``input`` (OpenAI Responses API).
+_LIST_KEYS = ("messages", "input")
 
-    ``key`` is ``messages`` for OpenAI/Anthropic, ``contents`` for Gemini.
+
+def reduce_messages_in(mapping: Any, fmt: str, opts: dict, key: str | None = "messages") -> None:
+    """Fail-open, in-place reduction of the message list(s) in ``mapping`` (dict-like).
+
+    ``key`` names the field to reduce (``messages`` for OpenAI/Anthropic). Pass
+    ``key=None`` to reduce whichever known list keys are present — used on gateway
+    paths (LiteLLM) where a request may be chat (``messages``) or Responses (``input``).
     """
-    if isinstance(mapping, dict) and isinstance(mapping.get(key), list):
-        try:
-            mapping[key] = reduce_messages(mapping[key], fmt=fmt, **opts)
-        except Exception:
-            pass  # fail open
+    if not isinstance(mapping, dict):
+        return
+    keys = _LIST_KEYS if key is None else (key,)
+    for k in keys:
+        if isinstance(mapping.get(k), list):
+            try:
+                mapping[k] = reduce_messages(mapping[k], fmt=fmt, **opts)
+            except Exception:
+                pass  # fail open
 
 
-def wrap_messages_create(create: Callable, *, fmt: str, opts: dict, key: str = "messages",
+def wrap_messages_create(create: Callable, *, fmt: str, opts: dict, key: str | None = "messages",
                          reduce: bool = True,
                          before: Callable[[dict], None] | None = None) -> Callable:
     """Wrap a ``create(**kwargs)`` callable to reduce its messages before calling through.

{leancontext-2.0.5 → leancontext-2.0.6}/leancontext/integrations/litellm.py

@@ -35,7 +35,8 @@ def make_handler(**opts):
     class LeanContextHandler(CustomLogger):
         async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
             if call_type in _REDUCIBLE_CALLS:
-                reduce_messages_in(data, "auto", opts)  # fail-open in-place
+                # key=None: reduce chat (messages) or Responses (input) payloads alike
+                reduce_messages_in(data, "auto", opts, key=None)  # fail-open in-place
             return data
 
     return LeanContextHandler()
@@ -48,14 +49,14 @@ def patch(**opts) -> None:
     if getattr(litellm, "_leancontext_patched", False):
         return
 
-    litellm.completion = wrap_messages_create(litellm.completion, fmt="auto", opts=opts)
+    litellm.completion = wrap_messages_create(litellm.completion, fmt="auto", opts=opts, key=None)
 
     if hasattr(litellm, "acompletion"):
         _orig_acompletion = litellm.acompletion
 
         @functools.wraps(_orig_acompletion)
         async def acompletion(*args, **kwargs):
-            reduce_messages_in(kwargs, "auto", opts)
+            reduce_messages_in(kwargs, "auto", opts, key=None)
             return await _orig_acompletion(*args, **kwargs)
 
         litellm.acompletion = mark(acompletion)

{leancontext-2.0.5 → leancontext-2.0.6}/leancontext/messages.py

@@ -125,13 +125,18 @@ def _reduce_gemini_message(content: Any, opts: dict) -> Any:
 # type "function_call_output" whose `output` is a string.
 
 def _reduce_responses_message(item: Any, opts: dict) -> Any:
-    if (
-        isinstance(item, dict)
-        and item.get("type") == "function_call_output"
-        and isinstance(item.get("output"), str)
-    ):
+    if not isinstance(item, dict) or item.get("type") != "function_call_output":
+        return item
+    output = item.get("output")
+    if isinstance(output, str):
+        new_item = dict(item)
+        new_item["output"] = _reduce_str(output, opts)
+        return new_item
+    # The Responses API also allows a list of content parts (e.g. output_text);
+    # reduce those the same way as chat parts. Anything else passes through.
+    if isinstance(output, list):
         new_item = dict(item)
-        new_item["output"] = _reduce_str(item["output"], opts)
+        new_item["output"] = [_reduce_openai_part(p, opts) for p in output]
         return new_item
     return item
 
@@ -180,6 +185,15 @@ _REDUCE_BY_NAME = {f.name: f.reduce for f in _FORMATS}
 
 # --- public ------------------------------------------------------------------
 
+def _format_for(m: Any) -> str:
+    """The format a single message belongs to (priority order); defaults to ``openai``."""
+    if isinstance(m, dict):
+        for fmt in _FORMATS:
+            if fmt.detect(m):
+                return fmt.name
+    return "openai"
+
+
 def detect_format(messages: list) -> str:
     """Best-effort detection of the message protocol; defaults to ``openai``."""
     for m in messages:
@@ -197,9 +211,14 @@ def reduce_messages(messages: Any, *, fmt: str = "auto", **opts) -> Any:
     Handles OpenAI (chat + Responses), Anthropic, and Gemini formats. Only tool-result
     content is touched; instructions are never altered. Anything unrecognised passes
     through unchanged (fail open).
+
+    With ``fmt="auto"`` each message is dispatched by its own format, so a list mixing
+    shapes (e.g. a chat tool message alongside a Responses ``function_call_output``)
+    reduces every item — not just the ones matching the first format seen.
     """
     if not isinstance(messages, list):
         return messages
-    resolved = detect_format(messages) if fmt == "auto" else fmt
-    reducer = _REDUCE_BY_NAME.get(resolved, _reduce_openai_message)
+    if fmt == "auto":
+        return [_REDUCE_BY_NAME.get(_format_for(m), _reduce_openai_message)(m, opts) for m in messages]
+    reducer = _REDUCE_BY_NAME.get(fmt, _reduce_openai_message)
     return [reducer(m, opts) for m in messages]

{leancontext-2.0.5 → leancontext-2.0.6}/leancontext/reducers/json_data.py

@@ -27,20 +27,22 @@ def _find_records(data: Any) -> list[dict] | None:
     return None
 
 
-def _fmt(value: Any) -> str:
-    if isinstance(value, str):
-        return value
-    return json.dumps(value, separators=(",", ":"), ensure_ascii=False)
-
-
 def reduce_json(text: str) -> tuple[str, list[str]]:
     data = json.loads(text)
     records = _find_records(data)
 
     if records is not None and len(records) >= 3:
         keys = list(dict.fromkeys(k for row in records for k in row.keys()))
-        header = "fields: " + " | ".join(keys)
-        rows = [" | ".join(_fmt(row.get(k, "")) for k in keys) for row in records]
+        # Each row is a JSON array of values in `keys` order, with the field names
+        # factored out into the header once (a missing field becomes null, keeping
+        # every row positional). JSON-encoding every cell keeps values that contain
+        # the delimiter, quotes, or newlines unambiguous and lossless — a plain
+        # " | " join would corrupt those, and the fidelity check wouldn't catch it.
+        header = "fields: " + json.dumps(keys, separators=(",", ":"), ensure_ascii=False)
+        rows = [
+            json.dumps([row.get(k) for k in keys], separators=(",", ":"), ensure_ascii=False)
+            for row in records
+        ]
         notes = [f"columnar: {len(records)} records × {len(keys)} fields, keys factored out once"]
         return header + "\n" + "\n".join(rows), notes
 

{leancontext-2.0.5 → leancontext-2.0.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "leancontext"
-version = "2.0.5"
+version = "2.0.6"
 description = "Deterministic, type-aware reduction of agent tool outputs at the source. Cut LLM token cost without making the agent do less."
 readme = "README.md"
 requires-python = ">=3.10"

{leancontext-2.0.5 → leancontext-2.0.6}/tests/test_core.py

@@ -41,6 +41,16 @@ def test_json_columnar_is_lossless_on_values():
     for i in range(20):
         assert f"n{i}" in r.text           # every value preserved
 
+
+def test_json_columnar_handles_delimiter_and_newline_values():
+    # Values containing the column delimiter or a newline must not corrupt rows:
+    # each row must parse back to exactly its original fields (regression test).
+    records = [{"id": i, "text": f"row {i} | part A\nrow {i} part B", "n": i} for i in range(10)]
+    r = leancontext.reduce(json.dumps(records))
+    assert r.kind == "json"                 # reduction applied, not reverted
+    rows = [json.loads(line) for line in r.text.splitlines()[1:]]  # skip the fields header
+    assert rows == [[i, f"row {i} | part A\nrow {i} part B", i] for i in range(10)]
+
 def test_decorator_preserves_contract():
     @leancontext.reduce
     def tool(_: str) -> str:

{leancontext-2.0.5 → leancontext-2.0.6}/tests/test_gateway.py

@@ -127,6 +127,20 @@ def test_proxy_reduces_before_forwarding():
     assert len(sent) < len(_big_log()) and "root cause" in sent
 
 
+# --- gateway helper: chat (messages) vs Responses (input) --------------------
+
+def test_reduce_messages_in_handles_responses_input_key():
+    # Gateway paths use key=None so a Responses request (input=) reduces too, not
+    # just chat (messages=). No third-party dependency needed for this logic.
+    from leancontext.integrations._common import reduce_messages_in
+
+    data = {"model": "gpt-4o",
+            "input": [{"type": "function_call_output", "call_id": "c", "output": _big_log()}]}
+    reduce_messages_in(data, "auto", {}, key=None)
+    sent = data["input"][0]["output"]
+    assert len(sent) < len(_big_log()) and "root cause" in sent
+
+
 # --- LiteLLM (real CustomLogger) ---------------------------------------------
 
 def test_litellm_pre_call_hook_reduces():

{leancontext-2.0.5 → leancontext-2.0.6}/tests/test_messages.py

@@ -63,6 +63,28 @@ def test_responses_format_reduced():
     assert len(reduced) < len(_log()) and "root cause" in reduced
 
 
+def test_mixed_format_list_reduces_every_item():
+    # A chat tool message AND a Responses function_call_output in one list: auto
+    # dispatch must reduce both, not just the format of the first message seen.
+    items = [
+        {"role": "tool", "tool_call_id": "c1", "content": _log()},
+        {"type": "function_call_output", "call_id": "c2", "output": _log()},
+    ]
+    out = reduce_messages(items)
+    assert len(out[0]["content"]) < len(_log()) and "root cause" in out[0]["content"]
+    assert len(out[1]["output"]) < len(_log()) and "root cause" in out[1]["output"]
+
+
+def test_responses_list_shaped_output_reduced():
+    items = [
+        {"type": "function_call_output", "call_id": "c1",
+         "output": [{"type": "output_text", "text": _log()}]},
+    ]
+    out = reduce_messages(items)
+    reduced = out[0]["output"][0]["text"]
+    assert len(reduced) < len(_log()) and "root cause" in reduced
+
+
 def test_non_list_passthrough():
     assert reduce_messages("not a list") == "not a list"