leancontext 2.0.0__tar.gz

leancontext-2.0.0/.editorconfig

@@ -0,0 +1,17 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+indent_style = space
+
+[*.py]
+indent_size = 4
+
+[*.{yml,yaml,toml,json}]
+indent_size = 2
+
+[*.md]
+trim_trailing_whitespace = false

leancontext-2.0.0/.github/CODEOWNERS

@@ -0,0 +1,2 @@
+# Default owner for everything in this repo.
+* @pankajniet

leancontext-2.0.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,25 @@
+---
+name: Bug report
+about: Something isn't working as expected
+title: ""
+labels: bug
+---
+
+**What happened**
+A clear description of the bug.
+
+**To reproduce**
+A minimal snippet, ideally the payload and the call:
+
+```python
+import leancontext
+r = leancontext.reduce(...)
+```
+
+**Expected**
+What you expected instead.
+
+**Environment**
+- LeanContext version:
+- Python version:
+- OS:

leancontext-2.0.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: false

leancontext-2.0.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,16 @@
+---
+name: Feature request
+about: Suggest an improvement (a new reducer, integration, etc.)
+title: ""
+labels: enhancement
+---
+
+**The problem**
+What are you trying to do, and what's missing today?
+
+**Proposed solution**
+What you'd like to see. For a new reducer, include a sample payload and what should
+be kept vs. collapsed.
+
+**Alternatives**
+Anything you've considered or worked around.

leancontext-2.0.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,12 @@
+## What this changes
+
+A short description of the change and why.
+
+## Checklist
+
+- [ ] `ruff check leancontext` passes
+- [ ] `mypy leancontext` passes
+- [ ] `pytest -q` passes
+- [ ] New behavior has tests
+- [ ] Reducers stay deterministic and value-preserving (see `AGENTS.md`)
+- [ ] `CHANGELOG.md` updated if user-facing

leancontext-2.0.0/.github/dependabot.yml

@@ -0,0 +1,10 @@
+version: 2
+updates:
+  - package-ecosystem: pip
+    directory: "/"
+    schedule:
+      interval: weekly
+  - package-ecosystem: github-actions
+    directory: "/"
+    schedule:
+      interval: weekly

leancontext-2.0.0/.github/workflows/ci.yml

@@ -0,0 +1,30 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.14"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev,otel,integrations,mcp]"
+      - name: Lint (ruff)
+        run: ruff check leancontext bench.py demo.py
+      - name: Type check (mypy)
+        run: mypy leancontext
+      - name: Tests
+        run: pytest -q --cov=leancontext --cov-report=term-missing

leancontext-2.0.0/.github/workflows/publish.yml

@@ -0,0 +1,28 @@
+name: Publish to PyPI
+
+# Publishes automatically when you create a GitHub Release.
+# Uses PyPI Trusted Publishing (OIDC) — no API tokens needed.
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write        # required for Trusted Publishing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.14"
+      - name: Build sdist and wheel
+        run: |
+          python -m pip install --upgrade build
+          python -m build
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

leancontext-2.0.0/.gitignore

@@ -0,0 +1,29 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+*.so
+
+# Virtual envs
+.venv/
+venv/
+env/
+
+# Tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# OS / editor
+.DS_Store
+.idea/
+.vscode/
+*.swp
+
+# Local content store (paging)
+.leancontext/

leancontext-2.0.0/.pre-commit-config.yaml

@@ -0,0 +1,12 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.8
+    hooks:
+      - id: ruff
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-toml

leancontext-2.0.0/AGENTS.md

@@ -0,0 +1,174 @@
+# AGENTS.md — LeanContext
+
+> Project name: **LeanContext** — keep agent context lean (drop redundant boilerplate, keep the signal). Easy to rename.
+> This file is the contract for any human or AI agent working in this repo. Read it before writing code.
+
+## 1. Mission
+
+Cut the cost of agentic LLM workflows by shrinking **what we send to the model**, without
+making the agent do less. The cost driver is *input tokens* — tool outputs, log dumps, RAG
+chunks, and JSON responses that arrive raw (repeated keys, boilerplate, verbose formatting)
+and are then **re-sent on every turn** (the quadratic tax).
+
+We reduce that payload **at the source** — the moment a tool result is produced — where its
+structure is still known. Deterministic, type-aware, cache-stable, and with **measurable
+fidelity** so teams can trust it in production.
+
+**Three pillars (our verified edge vs the incumbent Headroom — see §11):**
+1. **Cache-safe by construction** — deterministic + content-addressed, so we never break the
+   provider prompt-cache discount (Headroom's cache benefit is conditional).
+2. **Measurable fidelity** — a safety score on *every* reduction, not just dataset-level aggregates.
+3. **Provider-native interop** — composes with Anthropic context-editing / compaction / memory tool
+   and LiteLLM, instead of replacing them. Zero ML in the path (no overhead, no language bias).
+
+## 2. The problem (settled)
+
+- The cost is the **input**, not the model output (input is typically 10–50× output in coding agents).
+- A tool output added on turn N is paid for again on every later turn → **O(n²)**.
+- Raw payloads are mostly redundancy: repeated timestamps, JSON keys, boilerplate, near-identical lines.
+
+## 3. Why existing "compress-on-the-wire" middleware falls short
+
+These are the gaps LeanContext is designed to beat. Do not reintroduce them.
+
+| Gap | Their failure | Our fix |
+|-----|---------------|---------|
+| Structure-blind | Compresses a flattened request blob; can't tell a log from RAG | Reduce at the **tool-result boundary**, type-aware |
+| Still O(n²) | Compresses but re-sends every turn | Optional **paging**: aged results → expandable refs (O(1)) |
+| Breaks prompt caching | Non-deterministic / cross-context compression busts the prefix cache | **Deterministic + content-addressed** → cache-stable |
+| Reversibility helps humans, not the agent | Model only sees compressed text, can't ask for detail | Paging gives the agent an `expand` handle |
+| No measurable safety | "usually keeps the signal" is a probability | **Fidelity score** per reduction |
+| Re-compress churn | Re-runs every turn | Reduce **once**, cache by content hash |
+| Value loss | Drops an exact ID/number the agent needed | **Value-preserving** reducers |
+
+## 4. Design principles (non-negotiable)
+
+1. **Reduce at the source, not the wire.** Structure = safety + ratio.
+2. **No LLM in the reduction path.** Deterministic, microsecond-scale, debuggable, free.
+3. **Deterministic & content-addressed.** Same input → same output → stays cacheable.
+4. **Anomaly-preserving.** The rare line *is* the signal. Frequency-1 patterns and
+   error/warn/exception lines are kept verbatim.
+5. **Value-preserving.** Never silently drop identifiers, numbers, paths, or quoted values
+   that could be load-bearing for the agent's reasoning.
+6. **Measurable safety.** Every reduction reports a fidelity score and what it preserved/dropped.
+7. **Non-invasive by default.** `@reduce` decorator / tool wrapper. Paging is opt-in.
+8. **Honest accounting.** Report tokens before/after *and* cache-aware cost estimate.
+
+## 5. Integration & adoption (the make-or-break)
+
+Adoption dies on two things: **friction** and **fear**. These rules kill both.
+
+**A. Tiered integration — pick your friction level. All routes share one core.**
+1. Manual: `reduce(text).text` — zero coupling, works literally anywhere.
+2. Decorator: `@reduce` on a tool function — one line per tool.
+3. Bulk wrap: `tools = leancontext.wrap(tools)` — one line for all tools; detects LangChain/OpenAI/CrewAI tool objects and plain callables.
+4. Client fallback: `client = leancontext.wrap(openai_client)` — when you can't touch the tools (structure-blind, last resort).
+
+**B. Target the tool *callable*, not the framework.** Every framework ultimately invokes a Python
+callable for a tool. Operating on the callable's return value is universal — no per-framework code.
+
+**C. Fail open, ALWAYS.** Never break the agent. If the type is unknown, the reducer errors, the
+saving is below threshold, or fidelity is below threshold → return the **original, unchanged**.
+LeanContext can only ever help or no-op. This is the single most important rule.
+
+**D. Preserve the tool contract.** Same signature, same return type (`str` in → `str` out).
+The agent never knows LeanContext exists. No new required params, no behavior change.
+
+**E. Zero config, sane defaults.** `pip install leancontext`, add one line. No setup, no keys, no service.
+
+**F. Global kill switch.** `LEANCONTEXT_DISABLED=1` (env) or `leancontext.disable()` instantly no-ops
+everything — the trust lever teams need during an incident.
+
+**G. Silent by default, observable on demand.** No logging unless a hook is attached
+(`leancontext.on_reduction(cb)` for savings telemetry). Never spams stdout.
+
+**H. No forced dependencies.** Core is stdlib-only. Real tokenizers (tiktoken) and framework
+adapters are optional extras.
+
+## 6. Non-goals
+
+- Not a hosted service or dashboard. Local-first, library-first.
+- Not a model router or gateway (compose with LiteLLM etc., don't replace them).
+- Not lossy "summarization by an LLM" (that costs tokens to save tokens, and is non-deterministic).
+- Not a memory framework (compose with the agent's own history management).
+
+## 7. Architecture
+
+```
+leancontext/
+  __init__.py        # public API: reduce(), Reduction, reduce decorator
+  core.py            # dispatch, type detection, Reduction dataclass, token counting
+  fidelity.py        # salience extraction + fidelity scoring
+  tokens.py          # pluggable token counter (heuristic default, optional tiktoken)
+  reducers/
+    __init__.py
+    logs.py          # collapse near-identical lines, keep anomalies/errors verbatim
+    json_data.py     # factor schema once, send rows as values (columnar)
+    diff.py          # keep changed hunks, reference unchanged   (later)
+    stacktrace.py    # keep frames at the boundary + the cause   (later)
+  integrations/
+    decorator.py     # @reduce for tool functions
+    client.py        # provider client wrapper (fallback surface) (later)
+```
+
+Public API (stable target):
+
+```python
+from leancontext import reduce
+
+r = reduce(tool_output)            # kind auto-detected
+r.text                             # reduced string to send to the model
+r.tokens_before, r.tokens_after    # honest accounting
+r.ratio                            # 0.0–1.0 saved
+r.fidelity                         # 0.0–1.0 signal preserved
+r.kind, r.notes, r.ref             # type, human notes, content hash (for paging)
+```
+
+```python
+from leancontext import reduce as reduce_tool
+
+@reduce_tool                       # non-invasive integration
+def search_logs(query: str) -> str:
+    ...
+```
+
+## 8. Conventions
+
+- Python 3.14+. Standard library only in the core path; optional extras (tiktoken) behind a flag.
+- Type hints everywhere. Dataclasses for results.
+- Pure functions for reducers: `reduce(text, **opts) -> Reduction`. No global state, no I/O.
+- Determinism is a test requirement: same input must yield byte-identical output.
+- Every reducer ships with: a fidelity guarantee, a docstring stating what it preserves, and tests.
+
+## 9. Testing
+
+- `pytest`. Each reducer: ratio test (achieves target on representative input), determinism test,
+  and a fidelity test (known salient tokens survive).
+- A golden "incident log" fixture is the canonical demo: a ~10k-token log with one FATAL line
+  must reduce hard while preserving the FATAL line and all distinct error templates.
+
+## 10. Roadmap
+
+- **v0 (now):** core dispatch + `logs` + `json_data` reducers, fidelity, token accounting, `@reduce`, demo.
+- **v0.1:** `diff`, `stacktrace`, `table`, `html` reducers; tiktoken integration; CLI (`leancontext reduce <file>`).
+- **v0.2:** opt-in paging tier (`expand` tool + content store); client-wrapper fallback surface.
+- **v0.3:** framework adapters (LangChain/LlamaIndex/CrewAI tool wrappers); cost report.
+
+## 11. Differentiation — evidence-based (vs Headroom, verified 2026-06-21)
+
+Headroom (github.com/chopratejas/headroom, Apache-2.0, v0.26.0) is the reference incumbent and is
+genuinely strong: library + proxy + MCP + CCR reversible retrieval, hybrid statistical+ML compression.
+
+**Non-clone rule (crucial):** we do NOT re-implement Headroom. Where it is strong (Rust "SmartCrusher"
+statistical JSON, AST-aware code compaction, ML "Kompress"), we either match with a *standard*
+(e.g. TOON for structured data, later) or defer. We compete ONLY on its **verified gaps** below.
+
+| Verified gap in Headroom | LeanContext's edge |
+|---|---|
+| No interop with provider-native context mgmt (Anthropic `clear_tool_uses` / compaction / memory tool) | Compose with them: compress on ingest, provider clears on age. Cross-provider. |
+| No per-reduction fidelity score (only dataset-level aggregates) | Fidelity score on every reduction; fail-open below threshold. |
+| Prompt-cache benefit is conditional (changing prefixes erase KV-cache savings) | Cache-safe by construction: deterministic, per-block, content-addressed → reduce once, stable bytes. |
+| Hybrid uses ML (Kompress): overhead, English-biased, net-negative on fast models | Zero-ML, deterministic → no model load, no language bias, never net-negative. |
+| RAG document text + small/under-threshold payloads pass through | (Roadmap) optionally reduce RAG document blocks safely. |
+| Reversibility bounded by TTL + local LRU (RAM cost) | Determinism lets us re-derive instead of store; lighter retrieval. |
+

leancontext-2.0.0/CHANGELOG.md

@@ -0,0 +1,32 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/), and the project aims to follow semantic versioning.
+
+## [Unreleased]
+
+## [2.0.0] - 2026-06-21
+
+### Added
+- Core fail-open reduction pipeline with deterministic output and per-content-type
+  fidelity scoring (json, diff, and stack-trace reductions are verified, not assumed).
+- Reducers: `log`, `json`, `diff`, `stacktrace`, `html`, and `table` (whitespace-aligned
+  command-line output). Each reducer declares its own detector via a small registry.
+- Content-addressed cross-turn cache (each unique payload is reduced once).
+- Integrations: `@reduce` decorator and `wrap()` (sync and async tools); OpenAI / Anthropic /
+  Gemini client wrappers; `reduce_messages` (OpenAI, Anthropic, Gemini formats); LiteLLM
+  proxy + SDK; a hardened standalone FastAPI proxy (auth passthrough, streaming, 502 on
+  upstream errors); OpenTelemetry telemetry; Anthropic native context-editing interop;
+  an MCP server (`reduce`/`expand`/`stats`); and framework adapters for LangChain,
+  LangGraph, and Agno.
+- Cost accounting (`CostTracker`, `estimate_savings`) and paging (`lc://` references + `expand`).
+- Token counting auto-uses `tiktoken` when installed, with `active_tokenizer()` to report it.
+- A caching-on validation harness (`examples/validate_caching.py`) for Anthropic and OpenAI.
+- Input-size guard (`CONFIG.max_input_chars`) and a global kill switch.
+
+### Project
+- Targets Python 3.14; ruff, mypy, and coverage run in CI; examples, contributor, and
+  security docs included.
+
+[Unreleased]: https://github.com/pankajniet/LeanContext/compare/v2.0.0...HEAD
+[2.0.0]: https://github.com/pankajniet/LeanContext/releases/tag/v2.0.0

leancontext-2.0.0/CITATION.cff

@@ -0,0 +1,19 @@
+cff-version: 1.2.0
+message: "If you use LeanContext, please cite it."
+title: LeanContext
+abstract: >-
+  Deterministic, measurable context reduction for LLM agents. Trims the tool
+  output an agent re-sends to the model every turn, keeping the signal.
+authors:
+  - family-names: Pandey
+    given-names: Pankaj
+repository-code: "https://github.com/pankajniet/LeanContext"
+license: Apache-2.0
+version: 2.0.0
+date-released: "2026-06-21"
+keywords:
+  - llm
+  - ai-agents
+  - context-engineering
+  - tokens
+  - cost-optimization

leancontext-2.0.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,54 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a
+harassment-free experience for everyone, regardless of age, body size, visible or invisible
+disability, ethnicity, sex characteristics, gender identity and expression, level of
+experience, education, socio-economic status, nationality, personal appearance, race, religion,
+or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse,
+inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment:
+
+- Showing empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes
+- Focusing on what is best for the overall community
+
+Examples of unacceptable behavior:
+
+- The use of sexualized language or imagery, and sexual attention or advances of any kind
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information without their explicit permission
+- Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Enforcement Responsibilities
+
+Project maintainers are responsible for clarifying and enforcing our standards of acceptable
+behavior and will take appropriate and fair corrective action in response to any behavior that
+they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when an individual
+is officially representing the community in public spaces.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported privately
+to the maintainer via a GitHub security advisory or by contacting **@pankajniet**. All
+complaints will be reviewed and investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1,
+available at <https://www.contributor-covenant.org/version/2/1/code_of_conduct.html>.
+
+[homepage]: https://www.contributor-covenant.org

leancontext-2.0.0/CONTRIBUTING.md

@@ -0,0 +1,48 @@
+# Contributing to LeanContext
+
+Thanks for your interest. Issues and pull requests are welcome.
+
+## Setup
+
+```bash
+pip install -e ".[dev,otel,integrations]"
+```
+
+## Before you push
+
+```bash
+ruff check leancontext bench.py demo.py    # lint
+mypy leancontext                           # types
+pytest -q                                  # tests
+```
+
+CI runs all three on Python 3.14.
+
+## Writing a reducer
+
+Reducers live in `leancontext/reducers/` and are pure functions:
+
+```python
+def reduce_x(text: str) -> tuple[str, list[str]]:
+    ...
+    return reduced_text, notes
+```
+
+They must be:
+
+- **Deterministic** — the same input always produces the same output (the cache and prompt
+  caching depend on this; there is a determinism test for every reducer).
+- **Value-preserving** — never silently drop identifiers, numbers, paths, or error lines.
+- **Anomaly-preserving** — rare and error/severity lines are the signal; keep them.
+
+Register the reducer and its detector in `leancontext/core.py`, and add tests covering ratio,
+determinism, and signal preservation.
+
+## Design rules
+
+See [AGENTS.md](AGENTS.md) for the full contract (fail-open behaviour, the no-LLM rule,
+integration principles, and what LeanContext deliberately does not do).
+
+## Commit messages
+
+Plain, descriptive summaries. No AI co-author trailers.