token-distiller 0.1.0__tar.gz

token_distiller-0.1.0/.claude-plugin/marketplace.json

@@ -0,0 +1,18 @@
+{
+  "name": "token-distiller",
+  "owner": {
+    "name": "Sri PriYa N. Chen",
+    "email": "p.chen@NeoclassicalPopArt.com",
+    "url": "https://github.com/priyanchen/token-distiller"
+  },
+  "plugins": [
+    {
+      "name": "token-distiller",
+      "source": "./plugin",
+      "description": "Distills PDFs, photos, and repos into token-efficient context for LLM agents. Local OCR reads embedded diagrams, not just text layers; content-hash caching collapses re-reads; BM25 retrieval bounds huge documents. Nothing is discarded — anything shortened expands back in full. 33x on a 25-page PDF, 97% on pytest output.",
+      "version": "0.1.0",
+      "license": "PolyForm-Noncommercial-1.0.0",
+      "keywords": ["token", "context", "pdf", "ocr", "retrieval", "rag"]
+    }
+  ]
+}

token_distiller-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,61 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      # poppler backs pdf2image's page rasterization; tesseract is the OCR engine; the
+      # DejaVu fonts are needed by the fixtures that render real text for OCR to read.
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y poppler-utils tesseract-ocr fonts-dejavu-core
+
+      - name: Install package
+        run: pip install ".[dev]"
+
+      - name: Verify OCR engine is reachable
+        run: python -c "import pytesseract; print(pytesseract.get_tesseract_version())"
+
+      - name: Default suite
+        run: pytest -q
+
+      # Deselected by default because they drive real Tesseract; run explicitly here so the
+      # compression claims in the README and the figure-reading path stay covered.
+      - name: OCR-marked suite
+        run: pytest -q -m ocr
+
+      - name: Smoke-test the CLI end to end
+        run: |
+          distill --help
+          python - <<'PY'
+          import subprocess, sys
+          sys.path.insert(0, "tests")
+          from conftest import make_pdf
+          make_pdf("/tmp/ci_smoke.pdf", [["A page of native text long enough to skip OCR entirely."]])
+          out = subprocess.run(
+              ["distill", "file", "/tmp/ci_smoke.pdf", "--json"],
+              capture_output=True, text=True, check=True,
+          ).stdout
+          import json
+          d = json.loads(out)
+          assert d["method_counts"] == {"native_text": 1}, d["method_counts"]
+          assert d["compression_ratio"] > 1.0, d["compression_ratio"]
+          print("CLI smoke test OK:", d["raw_tokens_est"], "->", d["distilled_tokens_est"])
+          PY

token_distiller-0.1.0/.gitignore

@@ -0,0 +1,16 @@
+.venv/
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+data/
+.env
+.env.*
+.DS_Store
+**/.DS_Store
+
+# local working artifacts, not part of the project
+*.patch
+PR-BRIEF-*.md

token_distiller-0.1.0/LICENSE

@@ -0,0 +1,133 @@
+Copyright (C) 2026 Sri PriYa N. Chen
+
+Required Notice: Copyright Sri PriYa N. Chen
+
+# PolyForm Noncommercial License 1.0.0
+
+<https://polyformproject.org/licenses/noncommercial/1.0.0>
+
+## Acceptance
+
+In order to get any license under these terms, you must agree to them
+as both strict obligations and conditions to all your licenses.
+
+## Copyright License
+
+The licensor grants you a copyright license for the software to do
+everything you might do with the software that would otherwise infringe
+the licensor's copyright in it for any permitted purpose. However, you
+may only distribute the software according to Distribution License and
+make changes or new works based on the software according to Changes and
+New Works License.
+
+## Distribution License
+
+The licensor grants you an additional copyright license to distribute
+copies of the software. Your license to distribute covers distributing
+the software with changes and new works permitted by Changes and New
+Works License.
+
+## Notices
+
+You must ensure that anyone who gets a copy of any part of the software
+from you also gets a copy of these terms or the URL for them above, as
+well as copies of any plain-text lines beginning with `Required Notice:`
+that the licensor provided with the software.
+
+## Changes and New Works License
+
+The licensor grants you an additional copyright license to make changes
+and new works based on the software for any permitted purpose.
+
+## Patent License
+
+The licensor grants you a patent license for the software that covers
+patent claims the licensor can license, or becomes able to license, that
+you would infringe by using the software.
+
+## Noncommercial Purposes
+
+Any noncommercial purpose is a permitted purpose.
+
+## Personal Uses
+
+Personal use for research, experiment, and testing for the benefit of
+public knowledge, personal study, private entertainment, hobby projects,
+amateur pursuits, or religious observance, without any anticipated
+commercial application, is use for a permitted purpose.
+
+## Noncommercial Organizations
+
+Use by any charitable organization, educational institution, public
+research organization, public safety or health organization,
+environmental protection organization, or government institution is use
+for a permitted purpose regardless of the source of funding or
+obligations resulting from the funding.
+
+## Fair Use
+
+You may have "fair use" rights for the software under the law. These
+terms do not limit them.
+
+## No Other Rights
+
+These terms do not allow you to sublicense or transfer any of your
+licenses to anyone else, or prevent the licensor from granting licenses
+to anyone else. These terms do not imply any other licenses.
+
+## Patent Defense
+
+If you make any written claim that the software infringes or contributes
+to infringement of any patent, your patent license for the software
+granted under these terms ends immediately. If your company makes such a
+claim, your patent license ends immediately for work on behalf of your
+company.
+
+## Violations
+
+The first time you are notified in writing that you have violated any of
+these terms, or done anything with the software not covered by your
+licenses, your licenses can nonetheless continue if you come into full
+compliance with these terms, and take practical steps to correct past
+violations, within 32 days of receiving notice. Otherwise, all your
+licenses end immediately.
+
+## No Liability
+
+As far as the law allows, the software comes as is, without any warranty
+or condition, and the licensor will not be liable to you for any damages
+arising out of these terms or the use or nature of the software, under
+any kind of legal claim.
+
+## Definitions
+
+The **licensor** is the individual or entity offering these terms, and
+the **software** is the software the licensor makes available under
+these terms.
+
+**You** refers to the individual or entity agreeing to these terms.
+
+**Your company** is any legal entity, sole proprietorship, or other kind
+of organization that you work for, plus all organizations that have
+control over, are under the control of, or are under common control with
+that organization. **Control** means ownership of substantially all the
+assets of an entity, or the power to direct its management and policies
+by vote, contract, or otherwise. Control can be direct or indirect.
+
+**Your licenses** are all the licenses granted to you for the software
+under these terms.
+
+**Use** means anything you do with the software requiring one of your
+licenses.
+
+---
+
+## Commercial Licensing
+
+Personal, private, and noncommercial use (including by individuals,
+students, hobbyists, and noncommercial/nonprofit/educational
+organizations) is free under the terms above — no license purchase
+needed.
+
+For commercial use, please contact Sri PriYa N. Chen:
+p.chen@NeoclassicalPopArt.com

token_distiller-0.1.0/PKG-INFO

@@ -0,0 +1,257 @@
+Metadata-Version: 2.4
+Name: token-distiller
+Version: 0.1.0
+Summary: Distills PDFs, photos, and repos into token-efficient context for LLM agents. Local OCR reads embedded diagrams, not just text layers; content-hash caching collapses re-reads; BM25 retrieval bounds huge documents. Nothing is discarded — anything shortened expands back in full. 33x on a 25-page PDF, 97% on pytest output.
+Project-URL: Homepage, https://github.com/priyanchen/token-distiller
+Project-URL: Repository, https://github.com/priyanchen/token-distiller
+Project-URL: Issues, https://github.com/priyanchen/token-distiller/issues
+Author: Sri PriYa N. Chen
+License-Expression: PolyForm-Noncommercial-1.0.0
+License-File: LICENSE
+Keywords: anthropic,claude,context,llm,ocr,pdf,rag,retrieval,tokens
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: General
+Requires-Python: >=3.11
+Requires-Dist: anthropic>=0.40
+Requires-Dist: pathspec>=0.12
+Requires-Dist: pdf2image>=1.17
+Requires-Dist: pdfplumber>=0.11
+Requires-Dist: pillow-heif>=0.15
+Requires-Dist: pillow>=10.0
+Requires-Dist: pypdf>=4.0
+Requires-Dist: pytesseract>=0.3.10
+Requires-Dist: rank-bm25>=0.2
+Provides-Extra: dev
+Requires-Dist: pytest-mock; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Provides-Extra: rag-semantic
+Requires-Dist: voyageai>=0.3; extra == 'rag-semantic'
+Description-Content-Type: text/markdown
+
+# Token Distiller
+
+Distills PDFs and photos into token-efficient text before they enter an LLM's context, packs repos Repomix-style, indexes ingested content for retrieval instead of raw dumping, and tracks session activity mode to bias what a context audit flags.
+
+Pure Python. No shell scripts, no Node/TypeScript.
+
+## Setup
+
+### From PyPI
+
+```bash
+brew install poppler   # required by pdf2image for PDF page rasterization
+pip install token-distiller
+```
+
+### As a Claude Code plugin
+
+```
+/plugin marketplace add priyanchen/token-distiller
+/plugin install token-distiller@token-distiller
+```
+
+This wires the hook configuration only. Claude Code has no mechanism to run setup
+commands after installing a plugin, so it can't provision a Python environment for you —
+separately run `pip install token-distiller` so the `distill` binary the hook invokes is
+on `PATH`. Skipping that step doesn't break anything: the hook fails open, so `Read` on a
+PDF passes through unchanged rather than erroring. It just means nothing gets distilled
+until the binary is actually installed.
+
+### From source
+
+```bash
+brew install poppler   # required by pdf2image for PDF page rasterization
+python3 -m venv .venv
+source .venv/bin/activate
+pip install .
+```
+
+Use a regular install, not `pip install -e .`. On this machine the editable install's
+`_editable_impl_token_distiller.pth` was silently ignored by `site.py` (the file was
+readable and its contents correct, and a byte-identical copy under a different filename
+*was* honored — root cause unresolved), leaving `token_distiller` unimportable. A regular
+install copies the package into `site-packages` and avoids the `.pth` indirection
+entirely. Re-run `pip install .` after editing source.
+
+Optional: set **`TOKEN_DISTILLER_ANTHROPIC_API_KEY`** to enable vision-model fallback for
+figures and pages OCR can't read. Prefer that name over plain `ANTHROPIC_API_KEY` — a host
+agent (Claude Code included) may also read `ANTHROPIC_API_KEY` and switch from subscription
+authentication to per-token API billing if it finds one. The scoped name is checked first,
+so this tool gets a key without changing anyone else's auth.
+
+Set `VOYAGE_API_KEY` and `pip install ".[rag-semantic]"` to enable semantic (embedding)
+retrieval on top of the default BM25 keyword index.
+
+## CLI
+
+Placeholders are uppercase; substitute your own path, directory, or question.
+
+```bash
+distill file PATH             # distill one PDF/photo
+distill scan DIR              # batch distill a directory
+distill repo DIR              # Repomix-style repo pack
+distill index DIR             # build a retrieval index
+distill query "QUESTION"      # query the index
+distill mode                  # current session activity mode
+distill audit PATH            # CLAUDE.md/MEMORY.md structural audit (defaults to .)
+distill report                # cumulative token/savings report
+distill expand HANDLE         # full distilled text for a handle (--list to browse)
+distill compress              # compress verbose command output read from stdin
+distill install-hook          # wire the PreToolUse Read-interception hook into a project
+```
+
+Every command that distills a PDF accepts `--no-figures` to skip reading embedded figures,
+and `--no-vision` to stay on local OCR only. `distill file` additionally accepts
+`--accurate-tokens`, which calls Anthropic's `count_tokens` endpoint to report the real
+tokenizer count for the distilled output next to the chars/4 estimate. It's opt-in because
+it needs an API key (`TOKEN_DISTILLER_ANTHROPIC_API_KEY` or `ANTHROPIC_API_KEY`) and a
+network round trip — the call itself is free and doesn't count against message-creation
+rate limits, but the estimate is what every other command uses by default.
+
+## Compressing command output
+
+Verbose CLI output is the one context cost the document pipeline doesn't touch. Pipe it in:
+
+```bash
+pytest -q | distill compress --stats
+git status | distill compress
+npm install | distill compress
+```
+
+Measured on this repo's own output: a 169-test `pytest` run goes **289 → 8 tokens (97%)**,
+keeping the failure list, the first assertion detail, and the summary line while dropping
+the wall of dots. Plain `git status` goes **153 → 58 tokens (62%)**, grouped by state.
+
+Two properties worth knowing:
+
+- **It never inflates.** `git status --porcelain` is already denser than any per-state
+  summary of it, so when compression would produce more text than it consumed, the original
+  is returned unchanged.
+- **It never executes anything.** `distill compress` reads stdin and writes stdout. The
+  alternative — a hook that rewrites your Bash command to route it through a wrapper —
+  means building shell strings out of model-supplied input, which is exactly where command
+  injection lives. Piping output that you already ran has no such surface. Automatic
+  interception is deliberately not implemented for that reason.
+
+## Verified in a live session
+
+The `Read` interception was checked against a real Claude Code session, not only against
+synthetic hook payloads. A one-page PDF holding a canary string was read through the hook:
+the session reported `2,572 → 38 tokens` and the model quoted the canary correctly,
+confirming the substituted text is what actually reaches it. A second run asking for one
+specific page of a four-page PDF passed straight through to the native ranged read with no
+hook note, confirming a page range is never answered with whole-document content.
+
+## How it avoids losing anything
+
+Every distillation is stored whole, keyed by a SHA-256 of the file's bytes, before any
+shortening happens. Anything the hook shortens carries a handle, and `distill expand
+<handle>` returns the complete text. Concretely:
+
+- **Re-reads collapse.** Reading the same unchanged file twice in a session returns a
+  one-line pointer the second time instead of the whole document again (7,634 → 287
+  chars measured). Edit the file and the hash changes, so it is re-distilled in full —
+  a stale cache can never be served.
+- **Repeated page boilerplate is restated once.** A line must appear on ≥80% of pages to
+  qualify, so a running copyright footer (25/25 pages) collapses while a structural
+  marker like `Example:` (15/25) is left alone. Collapsed lines are listed at the top of
+  the output.
+- **Large documents defer rather than truncate.** Past `TOKEN_DISTILLER_LARGE_DOC_TOKENS`
+  (default 8000) the hook returns a head plus retrieval instructions. Nothing is
+  discarded — `distill expand` or `distill index` + `distill query` reach the rest.
+- **Embedded figures are read, not skipped.** Native-text extraction sees a page's text
+  layer only, so a diagram sitting beside that text would otherwise go unread. Each
+  embedded figure is cropped out by its bounding box and put through the same OCR →
+  vision chain used for scanned pages, then written into the output labelled
+  `[figure N on page M]`. Cropping to the figure matters: the surrounding prose is
+  already captured losslessly, so including it would pay vision tokens to re-read text
+  we already have. Hairline rules and background strips are skipped via
+  `TOKEN_DISTILLER_FIGURE_MIN_SIDE_PT` (default 48pt). `--no-figures` turns it off.
+- **A weak OCR pass is retried on a preprocessed copy.** Figures cropped from a PDF are
+  often below the ~300 DPI Tesseract expects and sit on a tinted panel, which is exactly
+  when raw OCR returns nothing. A second attempt greyscales, stretches contrast, upscales
+  small crops, and binarizes with an Otsu threshold. It is a retry rather than the default
+  because binarizing can destroy anti-aliased text that read fine raw, so the preprocessed
+  pass has to win on word count and confidence to be used. Measured on a real 765-page
+  book: of 13 figures that raw OCR could not read at all, **10 were recovered** — one at
+  confidence 96, transcribing `Market Research / Competitive Analysis / SWOT Analysis /
+  Goal Setting / Resource Allocation`. That book ends at 60 of 63 figures read, with no
+  API key.
+- **Figures that still can't be read are flagged, never dropped silently.** A purely
+  graphical diagram with no legible labels yields nothing from OCR, and without
+  `ANTHROPIC_API_KEY` there's no vision fallback to describe it. Those pages stay in
+  `pages_with_uncaptured_images()` and surface as one compact note — never one line per
+  page — in `distill file`, `--json`, and the hook-read response. Set `ANTHROPIC_API_KEY`
+  to close the remainder.
+
+## What leaves your machine
+
+With no API keys set, nothing does. OCR runs locally through Tesseract and retrieval runs
+locally through BM25, so the default configuration makes **zero network calls**.
+
+Two features are opt-in, and they send different things to different companies:
+
+| Enabled by | Goes to | What is sent |
+|---|---|---|
+| `TOKEN_DISTILLER_ANTHROPIC_API_KEY` (or `ANTHROPIC_API_KEY`) | Anthropic | A PNG of a **single cropped figure**, plus a fixed prompt. No source code, no file paths, no surrounding page text. |
+| `VOYAGE_API_KEY` + `pip install ".[rag-semantic]"` | Voyage AI | **Chunk text** from whatever you indexed. If you indexed a repo pack, that includes your source code. |
+
+The Voyage path is the one to think hardest about — it is a separate company under separate
+terms, and it is the only path that can transmit code. It is off unless you both install the
+extra and set the key; BM25 retrieval works without it.
+
+On the Anthropic path, [their commercial terms](https://www.anthropic.com/legal/commercial-terms)
+state that the customer "retains all rights to its Inputs", "owns its Outputs", that
+Anthropic "disclaims any rights it receives to the Customer Content", and that Anthropic
+"may not train models on Customer Content from Services". Read them yourself rather than
+relying on this summary; this is not legal advice.
+
+Turn figure reading off entirely with `--no-figures` or
+`TOKEN_DISTILLER_DESCRIBE_FIGURES=0`, and it will never reach for the vision model.
+
+Toggle any of it off: `TOKEN_DISTILLER_CACHE=0`, `TOKEN_DISTILLER_REREAD_COLLAPSE=0`,
+`TOKEN_DISTILLER_BOILERPLATE=0`.
+
+## A note on the numbers
+
+`raw_tokens_est` models what the **host** pays to ingest the file, not what its text
+alone would cost. Reading a PDF natively renders each page to an image and bills those
+pixels on top of the text, so a 25-page text PDF costs ~60,000 tokens to read raw but
+~1,800 distilled (33x). Scoring it as text-only would have reported a meaningless 1.0x.
+
+Both figures are estimates (chars/4 for text, Anthropic's published pixel formula for
+images) — good enough to compare methods, not exact. `distill file --accurate-tokens`
+swaps the distilled-side estimate for a real count from Anthropic's tokenizer, to check
+the estimate rather than trust it.
+
+That 33x describes a *sparse* page, where a fixed per-page rendering cost dominates a
+small amount of actual text — it is not a document-size-independent multiplier. A
+densely-written page compresses by far less through this mechanism alone, because the
+distilled side scales with real content: measured on a 765-page, prose-dense book, the
+whole-document ratio was 4.77x (1,567,889 raw → 328,589 distilled), not 33x. That is
+expected, not a regression — a page's text cannot be compressed below its own token
+count by an extraction step that isn't lossy.
+
+For a document that size, the number that actually matters is not the whole-document
+ratio anyway. 328,589 distilled tokens is well past `TOKEN_DISTILLER_LARGE_DOC_TOKENS`
+(default 8000), so the hook's large-document deferral fires: a live session reading that
+765-page file through the hook receives a head plus a `distill index` / `distill query`
+pointer, measured at ~1,675 tokens — roughly **940x** against the 1,567,889 raw cost.
+(The payload embeds the file's absolute path, so the exact token count shifts a little
+with where the file lives.)
+For large documents, the deferral-and-retrieval path is where the real savings come
+from; the raw-vs-native-text mechanism the 33x figure describes matters most for
+documents small enough to be read in full.
+
+## License
+
+**PolyForm Noncommercial 1.0.0.** Free for personal, private, and noncommercial use —
+no license purchase needed. Commercial use requires contacting Sri PriYa N. Chen
+(p.chen@NeoclassicalPopArt.com) for a commercial license.