python-token-killer 0.1.0__tar.gz

python_token_killer-0.1.0/.gitignore

@@ -0,0 +1,38 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+dist/
+build/
+*.whl
+
+# uv
+.venv/
+
+# Legacy virtual environments (not used with uv)
+venv/
+env/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+.mypy_cache/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Build
+*.so
+*.dylib
+dist/

python_token_killer-0.1.0/CHANGELOG.md

@@ -0,0 +1,74 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+
+### Changed
+
+### Fixed
+
+---
+
+## [0.1.0] - 2026-04-09
+
+Initial public release.
+
+### API
+
+- `ptk.minimize(obj)` — auto-detects content type, applies the right compression strategy, returns a minimized string. Accepts `aggressive`, `content_type`, and minimizer-specific kwargs.
+- `ptk.stats(obj)` — same compression, returns a dict with `output`, `original_tokens`, `minimized_tokens`, `savings_pct`, `content_type`.
+- `ptk.detect_type(obj)` — returns the auto-detected content type as a string.
+- `ptk(obj)` — callable module shorthand for `ptk.minimize(obj)`.
+
+### Minimizers
+
+- **DictMinimizer** — recursive null/empty stripping (preserves `0` and `False`), key shortening (`description` → `desc`, `configuration` → `cfg`, 30+ mappings), single-child flattening, kv/tabular output formats.
+- **ListMinimizer** — schema-once tabular encoding for uniform list-of-dicts, primitive dedup with `(xN)` counts, deterministic even-spaced sampling with first/last preservation.
+- **CodeMinimizer** — comment stripping with pragma preservation (`# noqa`, `# type: ignore`, `# TODO`, `# FIXME`, `// eslint-disable`), multi-line docstring collapse to first line, multi-language signature extraction (Python, JS, Rust, Go).
+- **LogMinimizer** — consecutive duplicate line collapse, timestamp stripping, error-only filtering with stack trace preservation (`Traceback`, `File`, `*Error:`, `*Exception:`), `"failed"` keyword preservation, FATAL/CRITICAL treated as errors.
+- **DiffMinimizer** — context line folding to `... N lines ...`, noise stripping (`index`, `old mode`, `new mode`, `similarity`, `Binary files`), `` preservation.
+- **TextMinimizer** — 20+ word abbreviations (`implementation` → `impl`, `configuration` → `config`, `production` → `prod`, case-preserving), 16 phrase abbreviations (`in order to` → `to`, `due to the fact that` → `because`), 13 filler phrase removals (`Furthermore,`, `Moreover,`, `Additionally,`), stopword removal (aggressive mode).
+
+### Benchmarks
+
+Real token counts via tiktoken (`cl100k_base`):
+
+| Benchmark | Original | Default | Saved | Aggressive | Saved |
+|---|---|---|---|---|---|
+| API response (JSON) | 1,450 | 792 | 45.4% | 782 | 46.1% |
+| Python module (code) | 2,734 | 2,113 | 22.7% | 309 | 88.7% |
+| Server log (58 lines) | 1,389 | 1,388 | 0.1% | 231 | 83.4% |
+| 50 user records (list) | 2,774 | 922 | 66.8% | 922 | 66.8% |
+| Verbose paragraph (text) | 101 | 96 | 5.0% | 74 | 26.7% |
+| **Total** | **11,182** | **7,424** | **33.6%** | **2,627** | **76.5%** |
+
+Bundled sample data and runner: `python benchmarks/bench.py`
+
+### Tests
+
+322 tests across two suites:
+
+- **test_ptk.py** (153 tests) — feature coverage for all 6 minimizers, type detection, base helpers, API contracts, and real-world payloads.
+- **test_adversarial.py** (169 tests) — type chaos (None, bytes, sets, circular refs, broken `__str__`, dataclasses, generators, inf/nan), deep nesting (100-level dicts, 10k-wide structures), unicode (emoji, CJK, RTL, null bytes, surrogates, BOM), regex safety (pathological backtracking, unclosed constructs, 100k newlines), API contracts (parametrized across 9 input types), input mutation verification (deepcopy before/after), thread safety (10 concurrent threads), performance (all benchmarks under 5s), idempotency, and content type mismatch degradation.
+
+### Examples
+
+- `examples/clean_api_response.py` — standalone script + stdin pipe for JSON cleanup.
+- `examples/langchain_middleware.py` — LangGraph node, callable wrapper, batch document minimizer.
+- `examples/claude_code_skill.py` — CLI tool with `--stdin`, `--type`, `--aggressive`, `--stats` flags.
+
+### Infrastructure
+
+- Zero required dependencies — stdlib only. tiktoken optional (`pip install python-token-killer[tiktoken]`).
+- `mypy --strict` clean across all 10 source files.
+- `ruff check` clean across all source, tests, benchmarks, and examples.
+- `py.typed` marker for PEP 561 type checker support.
+- GitHub Actions CI workflow (Python 3.10–3.13 matrix).
+- `AGENTS.md` + `CLAUDE.md` for coding agent context.
+- MIT license.

python_token_killer-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,141 @@
+# Contributing to ptk
+
+## Quick Start
+
+```bash
+git clone https://github.com/amahi2001/python-token-killer.git
+cd python-token-killer
+
+# Install uv (if you don't have it)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Install all dev dependencies — uv manages the venv automatically
+uv sync
+
+# Run everything CI runs — must pass before opening a PR
+make check
+```
+
+That's it. `uv sync` reads `uv.lock`, creates `.venv`, and installs every dev tool pinned to exact versions. No manual venv activation needed — `uv run` handles it.
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `make check` | Lint + typecheck + tests (the one command before every PR) |
+| `make test` | Tests only (361 tests, ~0.6s) |
+| `make lint` | `ruff check` + `ruff format --check` |
+| `make typecheck` | `mypy --strict` |
+| `make bench` | Benchmarks with tiktoken |
+| `make fix` | Auto-fix lint and formatting issues |
+| `make build` | Build wheel + sdist (`dist/`) |
+| `make clean` | Remove caches and build artifacts |
+
+All commands use `uv run` — they work whether or not you've activated the venv.
+
+## Architecture in 30 Seconds
+
+```
+ptk.minimize(obj)
+  → _types.detect(obj)          # what is this? dict, list, code, log, diff, text
+  → _ROUTER[type]               # pick the singleton minimizer
+  → minimizer.run(obj)          # _serialize for measurement, _minimize for output
+  → MinResult(output, lengths)  # frozen dataclass
+```
+
+Every file has one job:
+
+```
+src/ptk/
+  __init__.py         Public API + callable module trick + router
+  _types.py           ContentType enum + detect() heuristics
+  _base.py            Minimizer ABC + MinResult + shared helpers
+  minimizers/
+    _dict.py          DictMinimizer    (null strip, key shorten, flatten)
+    _list.py          ListMinimizer    (tabular, dedup, sampling)
+    _code.py          CodeMinimizer    (comments, docstrings, signatures)
+    _log.py           LogMinimizer     (dedup lines, error filter, stack traces)
+    _diff.py          DiffMinimizer    (context folding, noise strip)
+    _text.py          TextMinimizer    (abbreviation, filler removal, stopwords)
+```
+
+## The Three Rules
+
+These are non-negotiable. PRs that break them will be rejected.
+
+### 1. `minimize()` must never raise
+
+Any Python object passed to `ptk.minimize()` must produce a string — never an exception. `Minimizer.run()` wraps every `_minimize()` call in a try/except that catches `RecursionError`, `ValueError`, `TypeError`, and `OverflowError`, falling back to `str(obj)`.
+
+### 2. Never mutate the input
+
+All minimizers must create new objects. The original `obj` passed to `minimize()` must be identical after the call. `test_adversarial.py::TestInputMutation` verifies this with `deepcopy` comparisons.
+
+### 3. Zero required dependencies
+
+The library must work with `pip install python-token-killer` and nothing else. No numpy, no tiktoken in the core. Optional extras are fine — import them inside try/except.
+
+## Gotchas You'll Hit
+
+### The callable module trick
+
+`__init__.py` swaps its own `__class__` to `_CallableModule` so `ptk(obj)` works. Imports must be structured carefully — `sys` and `types` are imported after the public API definitions with `# noqa: E402`. Don't reorganize imports without testing `ptk({"a": 1})` interactively.
+
+### `_serialize` vs `_minimize`
+
+`_serialize(obj)` is called before `_minimize()` — it only measures the original length for stats. It must never raise (it has its own try/except). The actual output comes from `_minimize()`.
+
+### `from __future__ import annotations`
+
+Every source file uses this for PEP 563 deferred annotation evaluation on Python 3.10. Don't remove it.
+
+### Regexes are precompiled
+
+All regex patterns are compiled at module import time as module-level constants. Never call `re.compile()` inside a function.
+
+### Pragma preservation in CodeMinimizer
+
+When stripping comments, `_strip_comment_if_safe()` checks each comment against `_PRAGMA_KEYWORDS` before removing it. Comments containing `noqa`, `type: ignore`, `TODO`, `FIXME`, `eslint-disable`, etc. survive.
+
+### Thread safety
+
+Minimizers are stateless singletons stored in `_ROUTER`. Don't add instance attributes that change between calls.
+
+## Adding a New Minimizer
+
+1. Create `src/ptk/minimizers/_yourtype.py`, subclass `Minimizer`, implement `_minimize()`
+2. Add `ContentType.YOURTYPE = auto()` to `_types.py` + detection heuristic in `detect()`
+3. Register in `_ROUTER` in `__init__.py`
+4. Export from `minimizers/__init__.py`
+5. Add tests in `test_ptk.py` (feature) and `test_adversarial.py` (edge cases)
+
+## Dependency Groups
+
+ptk uses [PEP 735](https://peps.python.org/pep-0735/) dependency groups:
+
+| Group | Contents | Install |
+|---|---|---|
+| `test` | pytest | `uv sync --only-group test` |
+| `lint` | ruff | `uv sync --only-group lint` |
+| `typecheck` | mypy | `uv sync --only-group typecheck` |
+| `bench` | tiktoken | `uv sync --only-group bench` |
+| `hooks` | pre-commit | `uv sync --only-group hooks` |
+| `dev` | all of the above | `uv sync` (default) |
+
+CI installs only what each job needs. `uv sync` with no flags installs `dev` (everything).
+
+## Pre-commit Hooks
+
+```bash
+uv run pre-commit install
+```
+
+After that, `ruff` and `mypy` run automatically on every `git commit`. `make check` is the equivalent without needing hooks installed.
+
+## PR Checklist
+
+- `make check` passes
+- New code has tests in `test_ptk.py` (feature) and/or `test_adversarial.py` (edge cases)
+- No new required dependencies added
+- Docstrings on new public classes/methods
+- CHANGELOG.md updated under `[Unreleased]` if user-facing

python_token_killer-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ptk contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

python_token_killer-0.1.0/PKG-INFO

@@ -0,0 +1,269 @@
+Metadata-Version: 2.4
+Name: python-token-killer
+Version: 0.1.0
+Summary: Minimize LLM tokens from Python objects — dicts, code, logs, diffs, and more.
+Project-URL: Homepage, https://github.com/amahi2001/python-token-killer
+Project-URL: Repository, https://github.com/amahi2001/python-token-killer
+Project-URL: Issues, https://github.com/amahi2001/python-token-killer/issues
+Project-URL: Changelog, https://github.com/amahi2001/python-token-killer/blob/main/CHANGELOG.md
+Author-email: amahi2001 <amahi2001@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 ptk contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: agents,claude,compression,context-window,langchain,langgraph,llm,nlp,openai,rag,tokens
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: tiktoken
+Requires-Dist: tiktoken>=0.7; extra == 'tiktoken'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="assets/mascot.png" alt="ptk" width="200"/>
+</p>
+
+<p align="center">
+  <strong>ptk — Python Token Killer</strong><br/>
+  <strong>Minimize LLM tokens from Python objects in one call</strong><br/>
+  Zero dependencies • Auto type detection • 322 tests
+</p>
+
+<table align="center">
+  <tr>
+    <td align="left" valign="middle">
+      <a href="https://github.com/amahi2001/python-token-killer/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/amahi2001/python-token-killer/ci.yml?branch=main&style=flat-square&label=CI" alt="CI"/></a><br/>
+      <img src="https://img.shields.io/badge/python-3.10+-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.10+"/><br/>
+      <img src="https://img.shields.io/badge/mypy-strict-blue?style=flat-square" alt="mypy strict"/><br/>
+      <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-yellow?style=flat-square" alt="License"/></a>
+    </td>
+  </tr>
+</table>
+
+---
+
+## What is ptk?
+
+ptk is a **Python library** that minimizes tokens before they reach an LLM. Pass in any Python object — dict, list, code, logs, diffs, text — and get back a compressed string representation.
+
+Inspired by [RTK (Rust Token Killer)](https://github.com/rtk-ai/rtk), but designed as a library for programmatic use, not a CLI proxy.
+
+```python
+import ptk
+
+ptk.minimize({"users": [{"name": "Alice", "bio": None, "age": 30}]})
+# → '{"users":[{"name":"Alice","age":30}]}'
+
+ptk(my_dict)                   # callable shorthand
+ptk(my_dict, aggressive=True)  # max compression
+```
+
+```bash
+pip install python-token-killer
+# or
+uv add python-token-killer
+```
+
+Optional: `pip install python-token-killer[tiktoken]` or `uv add python-token-killer[tiktoken]` for exact token counting.
+
+## Benchmarks
+
+Real token counts via tiktoken (`cl100k_base`, same tokenizer as GPT-4 / Claude):
+
+```
+Benchmark                      Original  Default   Saved    Aggressive  Saved
+API response (JSON)                1450      792   45.4%         782   46.1%
+Python module (code)               2734     2113   22.7%         309   88.7%
+Server log (58 lines)              1389     1388    0.1%         231   83.4%
+50 user records (list)             2774      922   66.8%         922   66.8%
+Verbose paragraph (text)            101       96    5.0%          74   26.7%
+                                 ─────────────────────────────────────────────
+TOTAL                             11182     7424   33.6%        2627   76.5%
+```
+
+Run yourself: `python benchmarks/bench.py`
+
+## What It Does
+
+ptk auto-detects your input type and routes to the right minimizer:
+
+| Input Type | Strategy | Typical Savings |
+|---|---|---|
+| `dict` | Null stripping, key shortening, flattening, compact JSON | 30–60% |
+| `list` | Dedup, schema-once tabular, sampling | 40–70% |
+| Code `str` | Comment stripping (pragma-preserving), docstring collapse, signature extraction | 25–80% |
+| Logs `str` | Line dedup with counts, error-only filtering, stack trace preservation | 60–90% |
+| Diffs `str` | Context folding, noise stripping | 50–75% |
+| Text `str` | Word/phrase abbreviation, filler removal, stopword removal | 10–30% |
+
+## API
+
+### `ptk.minimize(obj, *, aggressive=False, content_type=None, **kw) → str`
+
+Main entry point. Auto-detects type, applies the right strategy, returns a minimized string.
+
+```python
+# auto-detect
+ptk.minimize({"key": "value"})
+
+# force content type
+ptk.minimize(some_string, content_type="code")
+ptk.minimize(some_string, content_type="log")
+
+# dict output formats
+ptk.minimize(data, format="kv")       # key:value lines
+ptk.minimize(data, format="tabular")  # header-once tabular
+
+# code: signatures only (huge savings)
+ptk.minimize(code, content_type="code", mode="signatures")
+
+# logs: errors only
+ptk.minimize(logs, content_type="log", errors_only=True)
+```
+
+### `ptk.stats(obj, **kw) → dict`
+
+Same compression, but returns statistics:
+
+```python
+ptk.stats(big_api_response)
+# {
+#   "output": "...",
+#   "original_len": 4200,
+#   "minimized_len": 1800,
+#   "savings_pct": 57.1,
+#   "content_type": "dict",
+#   "original_tokens": 1050,
+#   "minimized_tokens": 450,
+# }
+```
+
+### `ptk(obj)` — callable module
+
+```python
+import ptk
+ptk(some_dict)  # equivalent to ptk.minimize(some_dict)
+```
+
+## Features by Minimizer
+
+### DictMinimizer
+- Strips `None`, `""`, `[]`, `{}` recursively (preserves `0` and `False`)
+- Key shortening: `description` → `desc`, `timestamp` → `ts`, `configuration` → `cfg`, etc.
+- Single-child flattening: `{"a": {"b": val}}` → `{"a.b": val}` (aggressive)
+- Output formats: compact JSON (default), key-value lines, header-once tabular
+
+### ListMinimizer
+- Uniform list-of-dicts → schema-once tabular: declare fields once, one row per item
+- Primitive dedup with counts: `["a", "a", "a", "b"]` → `a (x3)\nb`
+- Large array sampling with first/last preservation (aggressive, threshold: 50)
+
+### CodeMinimizer
+- Strips comments while **preserving pragmas**: `# noqa`, `# type: ignore`, `# TODO`, `# FIXME`, `// eslint-disable`
+- Collapses multi-line docstrings to first line only
+- Signature extraction mode: pulls `def`, `class`, `fn`, `func` across Python, JS, Rust, Go
+- Normalizes blank lines and trailing whitespace
+
+### LogMinimizer
+- Consecutive duplicate line collapse with `(xN)` counts
+- Error-only filtering preserving: ERROR, WARN, FATAL, CRITICAL, stack traces, "failed" keyword
+- Timestamp stripping (aggressive)
+
+### DiffMinimizer
+- Folds unchanged context lines to `... N lines ...`
+- Strips noise: `index`, `old mode`, `new mode`, `similarity`, `Binary files` (aggressive)
+- Preserves: `+`/`-` lines, `@@` hunks, `---`/`+++` headers, ``
+
+### TextMinimizer
+- Word abbreviation: `implementation` → `impl`, `configuration` → `config`, `production` → `prod`, etc.
+- Phrase abbreviation: `in order to` → `to`, `due to the fact that` → `because`, etc.
+- Filler removal: strips `Furthermore,`, `Moreover,`, `In addition,`, `Additionally,`
+- Stopword removal (aggressive): strips `the`, `a`, `is`, `very`, etc.
+
+## Use Cases
+
+### Agent Frameworks (LangGraph / LangChain)
+
+```python
+import ptk
+
+def compress_context(state):
+    state["context"] = ptk.minimize(state["context"], aggressive=True)
+    return state
+```
+
+### Claude Code Skills
+
+```python
+#!/usr/bin/env python3
+import ptk, json, sys
+data = json.load(open(sys.argv[1]))
+print(ptk(data))
+```
+
+### API Response Cleanup
+
+```python
+response = requests.get("https://api.example.com/users").json()
+clean = ptk.minimize(response)  # strip nulls, compact JSON
+```
+
+## Comparison with Alternatives
+
+| Tool | Approach | Best For |
+|---|---|---|
+| **ptk** | Type-detecting Python library, one-liner API | Programmatic use in scripts, agents, frameworks |
+| [RTK](https://github.com/rtk-ai/rtk) | Rust CLI proxy for shell commands | Coding agents (Claude Code, OpenCode) |
+| [claw-compactor](https://github.com/open-compress/claw-compactor) | 14-stage pipeline, AST-aware | Heavy-duty workspace compression |
+| [toons](https://pypi.org/project/toons/) | TOON serialization format | Tabular data in LLM prompts |
+| [LLMLingua](https://github.com/microsoft/LLMLingua) | Neural prompt compression | Natural language, requires GPU |
+
+## Design Principles
+
+- **Zero deps** — stdlib only. tiktoken is optional for exact counts.
+- **Builtins-first** — `frozenset` for O(1) lookups, precompiled regexes, `slots=True` frozen dataclasses.
+- **DRY** — shared `strip_nullish()`, `dedup_lines()` reused across minimizers.
+- **Type-routed** — O(1) detection for dicts/lists, first-2KB heuristic for strings.
+- **Safe by default** — aggressive mode is opt-in. Default never destroys meaning.
+
+## Development
+
+```bash
+git clone https://github.com/amahi2001/python-token-killer.git
+cd python-token-killer
+uv sync          # installs all dev dependencies, creates .venv automatically
+make check       # lint + typecheck + 361 tests
+```
+
+## License
+
+MIT