token0 0.3.2__tar.gz → 0.4.0__tar.gz

token0-0.4.0/.github/workflows/publish.yml

@@ -0,0 +1,26 @@
+name: Publish
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  pypi:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Build package
+        run: |
+          python -m pip install --upgrade pip build
+          python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          password: ${{ secrets.PYPI_API_TOKEN }}

{token0-0.3.2 → token0-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: token0
-Version: 0.3.2
+Version: 0.4.0
 Summary: Open-source API proxy that makes vision LLM calls 5-10x cheaper
 License-Expression: Apache-2.0
 License-File: LICENSE
@@ -60,6 +60,21 @@ Token0 fixes this. It sits between your app and the LLM, analyzes every image an
 
 ---
 
+## When Token0 Helps (Measured)
+
+Real requests through the LiteLLM Proxy integration, GPT-4.1 token formulas:
+
+| Request | What the provider receives | Image tokens |
+|---|---|---|
+| "Classify this image", 4000×3000 photo | 512×384, low detail (40 KB instead of 1.4 MB) | 765 → 85 |
+| "Extract all text", 200dpi document page | OCR text (verified cheaper than the image) | 1,105 → 815 |
+| "Read the text on this page", dense 1080p screenshot | image — OCR text would cost 1,513 | 1,105 → 1,105 |
+| "Click the add to cart button", 1080p screenshot | original dimensions kept, PNG → JPEG | 1,105 → 1,105 |
+
+- **Biggest wins:** simple tasks on large photos (classify, yes/no, count) and document pages.
+- **No token savings on agent screenshots, by design:** UI/computer-use prompts keep the original dimensions so the coordinates the model returns still map onto the screen; only the upload gets smaller.
+- **OCR is verified, not assumed:** Token0 extracts the text, and if it would cost more than the (optimized) image, it sends the image instead. OCR runs on CPU via EasyOCR and adds seconds per image, so it suits batch/document pipelines more than latency-sensitive calls.
+
 ## How It Works
 
 ```
@@ -68,7 +83,7 @@ Your App → Token0 Proxy → [Analyze → Classify → Route → Transform →
          Database (logs every optimization decision + savings)
 ```
 
-Token0 applies **11 optimizations** automatically:
+Token0 applies **12 optimizations** automatically:
 
 ### Core Optimizations (Free Tier)
 
@@ -96,6 +111,8 @@ Token0 applies **11 optimizations** automatically:
 
 **11. Saliency-Based ROI Cropping** — Detects which region of an image the prompt is asking about and crops to that region before sending to the LLM. "What's the total on this invoice?" → crops to the bottom 40% of the image. "Read the header" → crops to the top 25%. Rule-based spatial keyword matching (zero ML deps). Delivers ~60% additional pixel reduction on document and form images before any other optimization runs.
 
+**12. Accessibility Tree Routing** — UI automation agents often have both a screenshot and an accessibility tree (AXUIElement, Playwright, Chrome DevTools). Token0 accepts both and routes to the cheaper representation automatically. If the tree is complete (no canvas/iframe/opaque elements), the screenshot is dropped and the tree is serialized as compact text — **93-97% token savings** vs a 1080p screenshot. If the tree has opaque nodes (a Figma canvas, a video element), Token0 keeps the screenshot. Supports Playwright/CDP format, macOS AXUIElement format, and pre-serialized strings.
+
 ---
 
 ## Benchmarks
@@ -240,10 +257,58 @@ Using OpenAI's published token formulas on real images and GPT-4.1 pricing ($2.0
 9. **On cloud APIs, total image savings reach 98.9%** when all optimizations are combined with model cascading.
 10. **Video deduplication collapses 60-frame clips to ~10 keyframes** — 13-45% savings on local models, ~83% projected on GPT-4.1.
 11. **Model-aware OCR skip is critical** — ultra-efficient encoders like llama3.2-vision use <50 tokens/image; OCR text output would cost more, not less.
+12. **Accessibility tree routing eliminates screenshot cost entirely** for UI agents — 93-97% savings when the tree is complete; screenshot fallback is automatic when canvas/iframe nodes are detected.
+
+### Accessibility Tree Benchmark (GPT-4o pricing)
+
+UI agents that send both a screenshot and an accessibility tree can route to the cheaper representation automatically.
+
+**Real browser results — Playwright, 1280×720, live pages** (actual reported prompt_tokens):
+
+| Page | Screenshot Tokens | Tree Tokens | Savings | Model |
+|---|---|---|---|---|
+| Hacker News | 750 | 192 | **74.4%** | moondream |
+| Hacker News | 602 | 164 | **72.8%** | llava:7b |
+| GitHub Home | 751 | 560 | **25.4%** | moondream |
+| GitHub Home | 601 | 560 | **6.8%** | llava:7b |
+| Wikipedia | 747 | 747 | **0%** | moondream |
+| Wikipedia | 599 | 1,165 | **-94.5%** | llava:7b — tree too large |
+
+> Wikipedia's rich navigation tree exceeded the screenshot token count on llava:7b — token0 would correctly fall back to the screenshot in this case. Hacker News (minimal DOM) shows the best real-world savings.
+
+**Ollama model results — 7 vision models, synthetic 800×600 screenshots** (actual reported prompt_tokens):
+
+> Synthetic screenshots: PIL-generated images with drawn UI elements (login form, todo list). Not real browser screenshots.
+
+| Model | Screenshot Tokens | Tree Tokens | Savings | Note |
+|---|---|---|---|---|
+| granite3.2-vision | 10,328 | 218 | **97.9%** | High-res encoder |
+| moondream | 1,500 | 168 | **88.8%** | |
+| llava:7b | 1,202 | 160 | **86.7%** | |
+| llava-llama3 | 1,201 | 164 | **86.3%** | |
+| minicpm-v | 704 | 128 | **81.8%** | |
+| gemma3:4b | 566 | 145 | **74.4%** | |
+| llama3.2-vision | 46 | 130 | n/a | Ultra-efficient encoder — tree costs more; screenshot wins |
+
+**Cloud API extrapolation** (tree tokens from Ollama measurements, screenshot tokens from published formulas, 800×600 image):
+
+| Provider | Screenshot Tokens | Tree Tokens (avg) | Savings | At 100K calls/day, saved/mo |
+|---|---|---|---|---|
+| OpenAI GPT-4o | 1,530 | ~80 | **89.6%** | **~$10,282** |
+| Anthropic Claude | 1,280 | ~80 | **87.6%** | **~$10,089** |
+
+> Tree token counts are text-based and provider-agnostic (~4 chars/token). Screenshot tokens use OpenAI tile formula (85 + 170×tiles) and Anthropic pixel formula (w×h/750). Canvas/iframe nodes trigger automatic screenshot fallback — no configuration needed.
+
+Run benchmarks:
+```bash
+python -m benchmarks.bench_ax_tree                        # formula-based projections
+python -m benchmarks.bench_ax_tree_models                 # all 7 Ollama vision models (synthetic)
+python -m benchmarks.bench_ax_tree_real                   # real browser pages via Playwright
+```
 
 ### Additional Test Coverage
 
-Token0 includes **171 unit tests** and benchmarks across multiple suites:
+Token0 includes **216 unit tests** and benchmarks across multiple suites:
 
 | Suite | Tests | What It Validates |
 |---|---|---|
@@ -260,6 +325,7 @@ Token0 includes **171 unit tests** and benchmarks across multiple suites:
 | `pdf` | 8 | PDF detection, decode, text extraction, token estimation |
 | `estimate` | 11 | /v1/estimate endpoint: single image, multiple images, remote URL skip, cost calc |
 | `langchain` | 8 | LangChain callback: import, text passthrough, image optimization, role mapping |
+| `ax_tree` | 22 | AX tree serialize, opaque detection, AXUIElement format, combo routing |
 
 ---
 
@@ -407,6 +473,40 @@ response = client.chat.completions.create(
 # ~83% savings on GPT-4.1
 ```
 
+### Accessibility Tree Support (UI Agents)
+
+If your agent captures both a screenshot and an accessibility tree, send both — Token0 picks the cheaper path automatically:
+
+```python
+import json
+
+# Playwright example
+page = await browser.new_page()
+snapshot = await page.accessibility.snapshot()  # returns a dict
+
+response = client.chat.completions.create(
+    model="gpt-4.1",
+    messages=[{
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "What button should I click to submit the form?"},
+            # Screenshot fallback — only used if tree has canvas/iframe nodes
+            {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
+            # Accessibility tree — token0 routes to this when complete
+            {"type": "accessibility_tree", "accessibility_tree": {
+                "data": snapshot,
+                "source": "playwright"
+            }},
+        ]
+    }],
+    extra_headers={"X-Provider-Key": "sk-..."}
+)
+# GitHub PR page: 2,125 tokens (screenshot) → 132 tokens (tree) — 93.8% savings
+# response.token0.optimizations_applied = ["ax tree → text (1,993 tokens saved vs screenshot)"]
+```
+
+Works with **Playwright**, **macOS AXUIElement**, **Chrome DevTools Protocol**, and pre-serialized strings. Canvas, iframe, and video elements trigger automatic screenshot fallback — no configuration needed.
+
 ### Streaming Support
 
 Token0 supports `stream=true` — images are optimized before streaming begins, then tokens flow word-by-word via SSE:
@@ -433,16 +533,14 @@ for chunk in stream:
 
 ### Use With LiteLLM
 
-Already using [LiteLLM](https://github.com/BerriAI/litellm)? Add Token0 as a callback hook — no proxy needed:
+Already using [LiteLLM](https://github.com/BerriAI/litellm)?
 
-```python
-import litellm
-from token0.litellm_hook import Token0Hook
+**SDK** — swap the import; same arguments as `litellm.completion` / `litellm.acompletion`:
 
-litellm.callbacks = [Token0Hook()]
+```python
+from token0.litellm_hook import completion  # or acompletion
 
-# All your existing litellm calls now get image optimization for free
-response = litellm.completion(
+response = completion(
     model="gpt-4.1",
     messages=[{
         "role": "user",
@@ -453,15 +551,16 @@ response = litellm.completion(
     }]
 )
 
-# Stats available in response metadata
-# response._hidden_params["metadata"]["token0"]["tokens_saved"]
+# Stats: response._hidden_params["token0"]["tokens_saved"]
 ```
 
-Or in LiteLLM proxy `config.yaml`:
+(Registering `Token0Hook()` in `litellm.callbacks` does **not** work with the SDK — LiteLLM only runs pre-call hooks inside the proxy.)
+
+**Proxy** — in `config.yaml`:
 
 ```yaml
 litellm_settings:
-  callbacks: ["token0.litellm_hook.Token0Hook"]
+  callbacks: ["token0.litellm_hook.proxy_handler_instance"]
 ```
 
 ### Use With LangChain

{token0-0.3.2 → token0-0.4.0}/README.md

@@ -22,6 +22,21 @@ Token0 fixes this. It sits between your app and the LLM, analyzes every image an
 
 ---
 
+## When Token0 Helps (Measured)
+
+Real requests through the LiteLLM Proxy integration, GPT-4.1 token formulas:
+
+| Request | What the provider receives | Image tokens |
+|---|---|---|
+| "Classify this image", 4000×3000 photo | 512×384, low detail (40 KB instead of 1.4 MB) | 765 → 85 |
+| "Extract all text", 200dpi document page | OCR text (verified cheaper than the image) | 1,105 → 815 |
+| "Read the text on this page", dense 1080p screenshot | image — OCR text would cost 1,513 | 1,105 → 1,105 |
+| "Click the add to cart button", 1080p screenshot | original dimensions kept, PNG → JPEG | 1,105 → 1,105 |
+
+- **Biggest wins:** simple tasks on large photos (classify, yes/no, count) and document pages.
+- **No token savings on agent screenshots, by design:** UI/computer-use prompts keep the original dimensions so the coordinates the model returns still map onto the screen; only the upload gets smaller.
+- **OCR is verified, not assumed:** Token0 extracts the text, and if it would cost more than the (optimized) image, it sends the image instead. OCR runs on CPU via EasyOCR and adds seconds per image, so it suits batch/document pipelines more than latency-sensitive calls.
+
 ## How It Works
 
 ```
@@ -30,7 +45,7 @@ Your App → Token0 Proxy → [Analyze → Classify → Route → Transform →
          Database (logs every optimization decision + savings)
 ```
 
-Token0 applies **11 optimizations** automatically:
+Token0 applies **12 optimizations** automatically:
 
 ### Core Optimizations (Free Tier)
 
@@ -58,6 +73,8 @@ Token0 applies **11 optimizations** automatically:
 
 **11. Saliency-Based ROI Cropping** — Detects which region of an image the prompt is asking about and crops to that region before sending to the LLM. "What's the total on this invoice?" → crops to the bottom 40% of the image. "Read the header" → crops to the top 25%. Rule-based spatial keyword matching (zero ML deps). Delivers ~60% additional pixel reduction on document and form images before any other optimization runs.
 
+**12. Accessibility Tree Routing** — UI automation agents often have both a screenshot and an accessibility tree (AXUIElement, Playwright, Chrome DevTools). Token0 accepts both and routes to the cheaper representation automatically. If the tree is complete (no canvas/iframe/opaque elements), the screenshot is dropped and the tree is serialized as compact text — **93-97% token savings** vs a 1080p screenshot. If the tree has opaque nodes (a Figma canvas, a video element), Token0 keeps the screenshot. Supports Playwright/CDP format, macOS AXUIElement format, and pre-serialized strings.
+
 ---
 
 ## Benchmarks
@@ -202,10 +219,58 @@ Using OpenAI's published token formulas on real images and GPT-4.1 pricing ($2.0
 9. **On cloud APIs, total image savings reach 98.9%** when all optimizations are combined with model cascading.
 10. **Video deduplication collapses 60-frame clips to ~10 keyframes** — 13-45% savings on local models, ~83% projected on GPT-4.1.
 11. **Model-aware OCR skip is critical** — ultra-efficient encoders like llama3.2-vision use <50 tokens/image; OCR text output would cost more, not less.
+12. **Accessibility tree routing eliminates screenshot cost entirely** for UI agents — 93-97% savings when the tree is complete; screenshot fallback is automatic when canvas/iframe nodes are detected.
+
+### Accessibility Tree Benchmark (GPT-4o pricing)
+
+UI agents that send both a screenshot and an accessibility tree can route to the cheaper representation automatically.
+
+**Real browser results — Playwright, 1280×720, live pages** (actual reported prompt_tokens):
+
+| Page | Screenshot Tokens | Tree Tokens | Savings | Model |
+|---|---|---|---|---|
+| Hacker News | 750 | 192 | **74.4%** | moondream |
+| Hacker News | 602 | 164 | **72.8%** | llava:7b |
+| GitHub Home | 751 | 560 | **25.4%** | moondream |
+| GitHub Home | 601 | 560 | **6.8%** | llava:7b |
+| Wikipedia | 747 | 747 | **0%** | moondream |
+| Wikipedia | 599 | 1,165 | **-94.5%** | llava:7b — tree too large |
+
+> Wikipedia's rich navigation tree exceeded the screenshot token count on llava:7b — token0 would correctly fall back to the screenshot in this case. Hacker News (minimal DOM) shows the best real-world savings.
+
+**Ollama model results — 7 vision models, synthetic 800×600 screenshots** (actual reported prompt_tokens):
+
+> Synthetic screenshots: PIL-generated images with drawn UI elements (login form, todo list). Not real browser screenshots.
+
+| Model | Screenshot Tokens | Tree Tokens | Savings | Note |
+|---|---|---|---|---|
+| granite3.2-vision | 10,328 | 218 | **97.9%** | High-res encoder |
+| moondream | 1,500 | 168 | **88.8%** | |
+| llava:7b | 1,202 | 160 | **86.7%** | |
+| llava-llama3 | 1,201 | 164 | **86.3%** | |
+| minicpm-v | 704 | 128 | **81.8%** | |
+| gemma3:4b | 566 | 145 | **74.4%** | |
+| llama3.2-vision | 46 | 130 | n/a | Ultra-efficient encoder — tree costs more; screenshot wins |
+
+**Cloud API extrapolation** (tree tokens from Ollama measurements, screenshot tokens from published formulas, 800×600 image):
+
+| Provider | Screenshot Tokens | Tree Tokens (avg) | Savings | At 100K calls/day, saved/mo |
+|---|---|---|---|---|
+| OpenAI GPT-4o | 1,530 | ~80 | **89.6%** | **~$10,282** |
+| Anthropic Claude | 1,280 | ~80 | **87.6%** | **~$10,089** |
+
+> Tree token counts are text-based and provider-agnostic (~4 chars/token). Screenshot tokens use OpenAI tile formula (85 + 170×tiles) and Anthropic pixel formula (w×h/750). Canvas/iframe nodes trigger automatic screenshot fallback — no configuration needed.
+
+Run benchmarks:
+```bash
+python -m benchmarks.bench_ax_tree                        # formula-based projections
+python -m benchmarks.bench_ax_tree_models                 # all 7 Ollama vision models (synthetic)
+python -m benchmarks.bench_ax_tree_real                   # real browser pages via Playwright
+```
 
 ### Additional Test Coverage
 
-Token0 includes **171 unit tests** and benchmarks across multiple suites:
+Token0 includes **216 unit tests** and benchmarks across multiple suites:
 
 | Suite | Tests | What It Validates |
 |---|---|---|
@@ -222,6 +287,7 @@ Token0 includes **171 unit tests** and benchmarks across multiple suites:
 | `pdf` | 8 | PDF detection, decode, text extraction, token estimation |
 | `estimate` | 11 | /v1/estimate endpoint: single image, multiple images, remote URL skip, cost calc |
 | `langchain` | 8 | LangChain callback: import, text passthrough, image optimization, role mapping |
+| `ax_tree` | 22 | AX tree serialize, opaque detection, AXUIElement format, combo routing |
 
 ---
 
@@ -369,6 +435,40 @@ response = client.chat.completions.create(
 # ~83% savings on GPT-4.1
 ```
 
+### Accessibility Tree Support (UI Agents)
+
+If your agent captures both a screenshot and an accessibility tree, send both — Token0 picks the cheaper path automatically:
+
+```python
+import json
+
+# Playwright example
+page = await browser.new_page()
+snapshot = await page.accessibility.snapshot()  # returns a dict
+
+response = client.chat.completions.create(
+    model="gpt-4.1",
+    messages=[{
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "What button should I click to submit the form?"},
+            # Screenshot fallback — only used if tree has canvas/iframe nodes
+            {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
+            # Accessibility tree — token0 routes to this when complete
+            {"type": "accessibility_tree", "accessibility_tree": {
+                "data": snapshot,
+                "source": "playwright"
+            }},
+        ]
+    }],
+    extra_headers={"X-Provider-Key": "sk-..."}
+)
+# GitHub PR page: 2,125 tokens (screenshot) → 132 tokens (tree) — 93.8% savings
+# response.token0.optimizations_applied = ["ax tree → text (1,993 tokens saved vs screenshot)"]
+```
+
+Works with **Playwright**, **macOS AXUIElement**, **Chrome DevTools Protocol**, and pre-serialized strings. Canvas, iframe, and video elements trigger automatic screenshot fallback — no configuration needed.
+
 ### Streaming Support
 
 Token0 supports `stream=true` — images are optimized before streaming begins, then tokens flow word-by-word via SSE:
@@ -395,16 +495,14 @@ for chunk in stream:
 
 ### Use With LiteLLM
 
-Already using [LiteLLM](https://github.com/BerriAI/litellm)? Add Token0 as a callback hook — no proxy needed:
+Already using [LiteLLM](https://github.com/BerriAI/litellm)?
 
-```python
-import litellm
-from token0.litellm_hook import Token0Hook
+**SDK** — swap the import; same arguments as `litellm.completion` / `litellm.acompletion`:
 
-litellm.callbacks = [Token0Hook()]
+```python
+from token0.litellm_hook import completion  # or acompletion
 
-# All your existing litellm calls now get image optimization for free
-response = litellm.completion(
+response = completion(
     model="gpt-4.1",
     messages=[{
         "role": "user",
@@ -415,15 +513,16 @@ response = litellm.completion(
     }]
 )
 
-# Stats available in response metadata
-# response._hidden_params["metadata"]["token0"]["tokens_saved"]
+# Stats: response._hidden_params["token0"]["tokens_saved"]
 ```
 
-Or in LiteLLM proxy `config.yaml`:
+(Registering `Token0Hook()` in `litellm.callbacks` does **not** work with the SDK — LiteLLM only runs pre-call hooks inside the proxy.)
+
+**Proxy** — in `config.yaml`:
 
 ```yaml
 litellm_settings:
-  callbacks: ["token0.litellm_hook.Token0Hook"]
+  callbacks: ["token0.litellm_hook.proxy_handler_instance"]
 ```
 
 ### Use With LangChain