tokenseive 1.1.0__tar.gz

tokenseive-1.1.0/.gitignore

@@ -0,0 +1,82 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Type checkers
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+.pytype/
+.ruff_cache/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db

tokenseive-1.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 TokenSeive Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tokenseive-1.1.0/PKG-INFO

@@ -0,0 +1,455 @@
+Metadata-Version: 2.4
+Name: tokenseive
+Version: 1.1.0
+Summary: Multi-layer token optimization for LLM applications — compress prompts, map codebases, reduce output
+Project-URL: Homepage, https://github.com/your-username/tokenseive
+Project-URL: Repository, https://github.com/your-username/tokenseive
+Project-URL: Issues, https://github.com/your-username/tokenseive/issues
+Author: TokenSeive Contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: code-graph,codebase,compression,context-window,llm,llmlingua,optimization,prompt,tokens
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: General
+Requires-Python: >=3.9
+Provides-Extra: all
+Requires-Dist: graphifyy>=0.8; extra == 'all'
+Requires-Dist: headroom-ai>=0.20; extra == 'all'
+Requires-Dist: llmlingua>=0.2; extra == 'all'
+Requires-Dist: selective-context>=0.1; extra == 'all'
+Requires-Dist: tiktoken>=0.5; extra == 'all'
+Requires-Dist: tree-sitter-language-pack>=0.1; extra == 'all'
+Requires-Dist: tree-sitter>=0.21; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Provides-Extra: headroom
+Requires-Dist: headroom-ai>=0.20; extra == 'headroom'
+Provides-Extra: mapper
+Requires-Dist: graphifyy>=0.8; extra == 'mapper'
+Requires-Dist: tree-sitter-language-pack>=0.1; extra == 'mapper'
+Requires-Dist: tree-sitter>=0.21; extra == 'mapper'
+Provides-Extra: ml
+Requires-Dist: llmlingua>=0.2; extra == 'ml'
+Requires-Dist: selective-context>=0.1; extra == 'ml'
+Provides-Extra: tokens
+Requires-Dist: tiktoken>=0.5; extra == 'tokens'
+Description-Content-Type: text/markdown
+
+# TokenSeive
+
+> **Multi-layer token optimization for LLM applications — compress prompts, map codebases, reduce output.**
+
+[![PyPI version](https://img.shields.io/pypi/v/tokenseive.svg)](https://pypi.org/project/tokenseive/)
+[![Python versions](https://img.shields.io/pypi/pyversions/tokenseive.svg)](https://pypi.org/project/tokenseive/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-82%20passing-brightgreen.svg)](#testing)
+
+TokenSeive is a standalone, **framework-agnostic** library that shrinks the
+four things that eat your context window:
+
+| Layer | What it does | Dependency cost |
+|-------|--------------|-----------------|
+| **Compress** | Shrinks *input* prompts with deterministic rules (and optional ML). | **Zero** required deps |
+| **Tool Compression** | Compresses structured tool outputs (JSON, search results, command output) by 85-93% via Headroom SmartCrusher. | Zero (fallback); headroom-ai optional |
+| **Map** | Turns a codebase into a token-budgeted ranked map / code graph. | Zero (regex fallback); tree-sitter optional |
+| **Behavioral** | Cuts *output* tokens by injecting a "lazy dev" ruleset. | Zero required deps |
+
+It works with **any** agentic framework — LangChain, AutoGen, CrewAI, raw
+OpenAI, or plain Python — because it imports nothing from them.
+
+---
+
+## Installation
+
+```bash
+# Core: works with plain Python 3.9+. Nothing else required.
+pip install tokenseive                    # zero deps
+
+# Optional extras
+pip install tokenseive[tokens]            # accurate token counts via tiktoken
+pip install tokenseive[ml]                # LLMLingua-2 + Selective Context backends
+pip install tokenseive[mapper]            # tree-sitter parsing + graphify code graphs
+pip install tokenseive[headroom]          # tool output compression (Headroom SmartCrusher)
+pip install tokenseive[all]               # everything
+```
+
+| Extra | Adds | When you want it |
+|-------|------|------------------|
+| `tokens` | `tiktoken` | Real GPT-4o token counts (otherwise a fast heuristic) |
+| `ml` | `llmlingua`, `selective-context` | Higher compression ratios on long docs |
+| `mapper` | `tree-sitter`, `tree-sitter-language-pack`, `graphifyy` | Precise multi-language parsing & visual code graphs |
+| `headroom` | `headroom-ai` | Compress structured tool outputs (JSON, search results, command stdout) by 85-93% |
+| `all` | all of the above + `headroom` | The full experience |
+
+---
+
+## Quick start
+
+### 1. Compress a prompt (zero deps)
+
+```python
+from tokenseive import RuleBasedCompressor
+
+rc = RuleBasedCompressor()
+result = rc.compress("It is important to note that, in order to proceed, "
+                     "we really must be careful.")
+
+print(result.compressed_text)
+# -> 'to proceed, we must be careful.'   (critical keyword preserved)
+
+print(f"{result.tokens_saved} tokens saved ({result.compression_ratio:.0%})")
+```
+
+`compress()` returns a [`CompressionResult`](tokenseive/compressors/rule_based.py)
+dataclass with `compressed_text`, `original_tokens`, `compressed_tokens`,
+`tokens_saved`, `compression_ratio`, and `techniques_applied`. It is
+**idempotent** and **never** mangles code blocks, XML/HTML, identity lines, or
+critical-keyword instructions (`NEVER`, `MUST`, `ALWAYS`, …).
+
+### 2. Map a codebase (zero deps)
+
+```python
+from tokenseive import CodebaseMapper
+
+mapper = CodebaseMapper("/path/to/repo", verbose=False)
+
+print(mapper.get_repo_map(max_tokens=1024))   # ranked symbol overview
+mapper.find_function("build_prompt")          # -> [{file, line, signature, ...}]
+print(mapper.get_symbol_context("build_prompt"))  # def + callers/callees, ready for the LLM
+```
+
+### 3. Cut output tokens with a behavioral ruleset
+
+```python
+from tokenseive import BehavioralRuleset
+
+ruleset = BehavioralRuleset(mode="full")   # off | lite | full | ultra
+system_prompt = base_prompt + "\n\n" + ruleset.get_instructions()
+# Injecting the "lazy dev" ladder steers the model toward the shortest
+# working diff — typically 22–54% fewer output tokens.
+```
+
+### 4. Compress tool output (zero deps)
+
+```python
+from tokenseive import HeadroomCompressor
+
+compressor = HeadroomCompressor()
+result = compressor.compress_tool_output("search_files", json_output)
+print(f"{result.tokens_saved} tokens saved ({result.compression_ratio:.1%})")
+# -> '1295 tokens saved (89.1%)'
+```
+
+`HeadroomCompressor` wraps Headroom's **SmartCrusher** to shrink structured tool
+outputs (JSON, search results, file listings) by **85-93%**. When `headroom-ai`
+isn't installed it falls back to deterministic array-truncation / structure
+collapsing, so it always returns a usable result. Install the real engine with
+`pip install tokenseive[headroom]`.
+
+---
+
+## Tool Output Compression
+
+Compress structured tool outputs (JSON, search results, file listings) by **85-93%**
+using Headroom's **SmartCrusher**.
+
+```python
+from tokenseive import HeadroomCompressor
+
+compressor = HeadroomCompressor()
+
+# Compress a single tool result
+result = compressor.compress_tool_output("search_files", json_output)
+print(f"{result.tokens_saved} tokens saved ({result.compression_ratio:.1%})")
+
+# Check whether a tool result should be compressed
+if compressor.should_compress("grep", large_output):
+    compressed = compressor.compress_tool_output("grep", large_output)
+
+# Batch-compress conversation history (protects the most recent turns)
+compressed_messages = compressor.compress_messages(messages, protect_recent=4)
+```
+
+### How it works
+- **SmartCrusher** — truncates large arrays with `[N omitted]` summaries, collapses repetitive structures, and aggregates result lists.
+- **Tool-specific policy** — always compress `search_files` / `grep` / `list_files`; never compress `bash` / `write_file` / `edit_file`.
+- **Graceful fallback** — when `headroom-ai` isn't installed, applies deterministic compression (array truncation, structure collapsing).
+
+| Tool output type | Before | After | Savings |
+|---|---|---|---|
+| File search (200 files) | 14,481 chars | 759 chars | **94.8%** |
+| JSON API response | 27,607 tokens | 2,719 tokens | **90.1%** |
+| Multi-tool turn (3 tools) | 26,890 tokens | 1,911 tokens | **92.9%** |
+
+---
+
+## Architecture
+
+```
+                           ┌──────────────────────────────────────┐
+                           │              Your Agent              │
+                           │ (LangChain / AutoGen / CrewAI / raw) │
+                           └──────────────────┬───────────────────┘
+                                              │  system prompt + context
+          ┌──────────────────────┬───────────┴───────────┬──────────────────────┐
+          ▼                      ▼                       ▼                      ▼
+ ┌─────────────────┐  ┌────────────────────┐  ┌────────────────────┐  ┌──────────────────┐
+ │     COMPRESS    │  │  TOOL COMPRESSION  │  │        MAP         │  │    BEHAVIORAL    │
+ │   (input side)  │  │   (tool results)   │  │   (context side)   │  │  (output side)   │
+ ├─────────────────┤  ├────────────────────┤  ├────────────────────┤  ├──────────────────┤
+ │ RuleBased       │  │ HeadroomCompressor │  │ CodebaseMapper     │  │ BehavioralRuleset│
+ │  Compressor     │  │ + SmartCrusher     │  │  get_repo_map()    │  │  off/lite/full/  │
+ │ LLMLingua-2     │  │ compress_messages()│  │  get_code_graph()  │  │  ultra modes     │
+ │ SelectiveContext│  │ should_compress()  │  │  find / trace /    │  │  apply_to()      │
+ │ CompressionPipe │  │                    │  │  context queries   │  │                  │
+ │  line (cascade) │  │                    │  │                    │  │                  │
+ └─────────────────┘  └────────────────────┘  └────────────────────┘  └──────────────────┘
+   rules: 0 deps        fallback: 0 deps        regex: 0 deps           0 deps
+   ml:    tokenseive[ml]   headroom: [headroom]    treesitter: tokenseive[mapper]
+```
+
+Each layer is **independent** — use one, two, three, or all four.
+
+---
+
+## API reference
+
+### Compressors ([`tokenseive/compressors/`](tokenseive/compressors/__init__.py))
+
+#### `RuleBasedCompressor(encoding="o200k_base", identity_names=())`
+Deterministic, dependency-free compression. The workhorse.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `compress(text, **kw)` | `CompressionResult` | Full rule pipeline (idempotent). |
+| `count_tokens(text)` | `int` | tiktoken if available, else heuristic. |
+
+Techniques applied, in order: redundant-phrase removal → abbreviation
+expansion → contractions → filler/verbosity removal → punctuation cleanup →
+whitespace normalization → duplicate-line removal. Each runs **only on
+non-critical lines**; protected regions are masked first and restored verbatim.
+
+#### `CompressionPipeline(backend="rules", rate=0.5)`
+Unified entry point with graceful degradation.
+
+| `backend` | Behavior |
+|-----------|----------|
+| `"rules"` *(default)* | Deterministic, always available. |
+| `"selective"` | GPT-2 phrase filtering (`tokenseive[ml]`). Falls back to rules if unavailable. |
+| `"llmlingua2"` | Microsoft LLMLingua-2 (`tokenseive[ml]`). Falls back to rules if unavailable. |
+| `"multi"` | Cascade: rules → selective → llmlingua2, stopping at the target keep-rate. |
+
+```python
+CompressionPipeline.available_backends()   # -> ['rules'] or ['rules','selective','llmlingua2']
+```
+
+#### `CompressionResult`
+Dataclass with `.original_text`, `.compressed_text`, `.original_tokens`,
+`.compressed_tokens`, `.tokens_saved`, `.compression_ratio`,
+`.techniques_applied`, plus `.as_dict()`, `["key"]`, and `.get(k, default)`
+for dict-style access.
+
+#### `LLMLingua2Compressor` / `SelectiveContextCompressor`
+Direct ML backends (lazy-loaded, raise `ImportError` with a helpful message if
+the extra isn't installed).
+
+### Mapper ([`tokenseive/mapper/`](tokenseive/mapper/__init__.py))
+
+#### `CodebaseMapper(repo_path, *, extensions=None, max_files=None, ...)`
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `get_repo_map(max_tokens=1024)` | `str` | Ranked, token-budgeted symbol tree. |
+| `get_code_graph()` | `dict` | `{nodes, edges, stats}` (graphify or tree-sitter fallback). |
+| `export_graph(format="json")` | `str` | JSON / HTML / SVG export. |
+| `find_function(name)` / `find_class(name)` | `list[dict]` | Locations of a symbol. |
+| `trace_call_chain(name, max_depth=3)` | `dict` | Outbound + inbound call tree. |
+| `get_symbol_context(name)` | `str` | Definition + callers/callees block. |
+| `get_dependencies(file)` | `dict` | Imports + dependents of a file. |
+| `get_stats()` | `dict` | File/symbol/token-reduction statistics. |
+
+### Behavioral ([`tokenseive/behavioral/`](tokenseive/behavioral/__init__.py))
+
+#### `BehavioralRuleset(mode="full")`
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `get_instructions()` | `str` | Ruleset text to inject (empty when `mode="off"`). |
+| `get_token_count()` | `int` | Estimated tokens of the ruleset. |
+| `apply_to(prompt, separator="\n\n")` | `str` | Append ruleset to a prompt. |
+
+Modes: `off` (inject nothing), `lite`, `full` *(default)*, `ultra` (YAGNI extremist).
+
+### Tool Output Compression ([`tokenseive/tool_compression/`](tokenseive/tool_compression/__init__.py))
+
+#### `HeadroomCompressor(policy=None, strict=False, ...)`
+Compress structured tool outputs via Headroom's **SmartCrusher**, with a
+deterministic zero-dependency fallback when `headroom-ai` is absent.
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `compress_tool_output(tool_name, content)` | `ToolCompressionResult` | Compress a single tool result (always returns a usable result). |
+| `should_compress(tool_name, content)` | `bool` | Whether a result is worth compressing (policy + size gate). |
+| `compress_messages(messages, protect_recent=None)` | `list[dict]` | Batch-compress older `tool` messages in a conversation. |
+| `available()` | `bool` | Whether the real `headroom-ai` engine is importable. |
+
+`ToolCompressionResult` mirrors `CompressionResult` (`.compressed_text`,
+`.tokens_saved`, `.compression_ratio`, `.transforms_applied`, `.as_dict()`).
+Tool policy: always compress `search_files` / `grep` / `list_files`; never
+compress `bash` / `write_file` / `edit_file`. Pass `strict=True` to require the
+real engine instead of the built-in fallback. *(requires `[headroom]` for the
+real SmartCrusher engine; the fallback needs no extra deps.)*
+
+---
+
+## CLI
+
+```bash
+# Compress a prompt file
+tokenseive compress prompt.txt
+tokenseive compress prompt.txt --backend multi --rate 0.5
+tokenseive compress prompt.txt --json --write
+
+# Map a codebase
+tokenseive map /path/to/repo --max-tokens 1024
+tokenseive map /path/to/repo --find-function "my_func"
+tokenseive map /path/to/repo --trace "my_func" --depth 2
+tokenseive map /path/to/repo --context "my_func"
+tokenseive map /path/to/repo --stats
+
+# Output-optimization ruleset
+tokenseive ruleset --mode full
+tokenseive ruleset --mode ultra --tokens
+
+tokenseive version
+```
+
+---
+
+## Benchmarks
+
+Rule-based compression is deterministic and free; ML backends push further on
+long, prose-heavy documents. Representative savings on typical inputs:
+
+| Input type | `rules` | `selective` | `llmlingua2` | `multi` (0.5) |
+|------------|--------:|------------:|-------------:|--------------:|
+| System prompt (verbose) | **~12%** | ~35% | ~45% | ~48% |
+| Meeting transcript | ~10% | ~40% | ~55% | ~58% |
+| API docs (long) | ~8% | ~38% | ~50% | ~52% |
+| Output tokens (behavioral `full`) | — | — | — | **22–54%** |
+
+> Rule-based ratios are stable across runs (idempotent). ML ratios vary with
+> content and the chosen keep-rate. The behavioral ruleset cuts *response*
+> tokens by steering the model toward shorter diffs.
+
+Mapper token reduction depends on repo size; for a typical mid-size Python
+project the ranked repo map is **~95–99% smaller** than reading every file.
+
+---
+
+## Framework integration
+
+TokenSeive imports nothing framework-specific, so integration is just
+"build the prompt, then call the model":
+
+```python
+from tokenseive import RuleBasedCompressor, BehavioralRuleset
+
+def system_prompt():
+    base = RuleBasedCompressor().compress(YOUR_BASE_PROMPT).compressed_text
+    return BehavioralRuleset(mode="full").apply_to(base)
+```
+
+**OpenAI (raw):**
+```python
+openai.chat.completions.create(
+    model="gpt-4o",
+    messages=[{"role": "system", "content": system_prompt()},
+              {"role": "user", "content": question}],
+)
+```
+
+**LangChain:**
+```python
+from langchain_core.prompts import ChatPromptTemplate
+prompt = ChatPromptTemplate.from_messages(
+    [("system", system_prompt()), ("human", "{question}")]
+)
+```
+
+**AutoGen / CrewAI:** pass `system_prompt()` as the agent's `system_message`
+/ `backstory`.
+
+See [`examples/agent_integration.py`](examples/agent_integration.py) for a
+complete, runnable pattern (with a codebase map appended via
+[`CodebaseMapper`](tokenseive/mapper/__init__.py)).
+
+---
+
+## Comparison
+
+| Feature | TokenSeive | LLMLingua | Selective Context | LangChain ` compres` |
+|---------|:----------:|:---------:|:-----------------:|:--------------------:|
+| Zero required deps | ✅ | ❌ (torch) | ❌ (torch/spacy) | ❌ (langchain) |
+| Deterministic / idempotent rules | ✅ | ❌ | ❌ | partial |
+| Protected regions (code, XML, identity) | ✅ | ❌ | ❌ | ❌ |
+| Codebase repo mapping | ✅ | ❌ | ❌ | ❌ |
+| Output-token ruleset | ✅ | ❌ | ❌ | ❌ |
+| Tool output compression (85-93%) | ✅ | ❌ | ❌ | ❌ |
+| Multi-backend cascade | ✅ | single | single | n/a |
+| Framework-agnostic | ✅ | ✅ | ✅ | ❌ |
+
+---
+
+## Project layout
+
+```
+tokenseive/
+├── pyproject.toml
+├── README.md
+├── LICENSE
+├── tokenseive/
+│   ├── __init__.py            # Main API + version
+│   ├── cli.py                 # `tokenseive` CLI
+│   ├── utils.py               # Token counting (tiktoken-or-heuristic) + sentinels
+│   ├── compressors/           # rule_based, llmlingua2, selective, pipeline
+│   ├── mapper/                # repo_map, code_graph, queries
+│   ├── behavioral/            # output-optimization ruleset
+│   └── tool_compression/      # headroom SmartCrusher (tool-output compression)
+├── tests/                     # 82 tests, run with zero deps
+└── examples/                  # basic, ml, repo_mapping, agent_integration
+```
+
+## Testing
+
+```bash
+pip install tokenseive[dev]   # pytest + pytest-cov
+pytest                        # 82 tests, all pass with zero optional deps
+```
+
+The full suite runs with **no extras installed** — the rule-based compressor,
+regex mapper, and behavioral ruleset are all exercised by default.
+
+## Design principles
+
+1. **Zero required dependencies** — `pip install tokenseive` just works on Python 3.9+.
+2. **Optional ML/mapping backends** — every heavy import is lazy and degrades gracefully.
+3. **Framework-agnostic** — no imports from any specific agent framework.
+4. **Deterministic by default** — rule-based compression is idempotent and reproducible.
+5. **Never destroy meaning** — code, XML/HTML, identity, and critical instructions are protected.
+
+## License
+
+[MIT](LICENSE)