tokenzero 0.1.0__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {tokenzero-0.1.0/tokenzero.egg-info → tokenzero-0.1.2}/PKG-INFO +49 -66
- {tokenzero-0.1.0 → tokenzero-0.1.2}/README.md +48 -65
- {tokenzero-0.1.0 → tokenzero-0.1.2}/mcp_server.py +2 -5
- {tokenzero-0.1.0 → tokenzero-0.1.2}/pyproject.toml +1 -2
- {tokenzero-0.1.0 → tokenzero-0.1.2}/scanner.py +3 -3
- {tokenzero-0.1.0 → tokenzero-0.1.2}/squeeze.py +38 -38
- {tokenzero-0.1.0 → tokenzero-0.1.2}/tests/test_cli.py +1 -1
- {tokenzero-0.1.0 → tokenzero-0.1.2/tokenzero.egg-info}/PKG-INFO +49 -66
- {tokenzero-0.1.0 → tokenzero-0.1.2}/tokenzero.egg-info/entry_points.txt +0 -1
- {tokenzero-0.1.0 → tokenzero-0.1.2}/LICENSE +0 -0
- {tokenzero-0.1.0 → tokenzero-0.1.2}/proxy.py +0 -0
- {tokenzero-0.1.0 → tokenzero-0.1.2}/setup.cfg +0 -0
- {tokenzero-0.1.0 → tokenzero-0.1.2}/tokenzero.egg-info/SOURCES.txt +0 -0
- {tokenzero-0.1.0 → tokenzero-0.1.2}/tokenzero.egg-info/dependency_links.txt +0 -0
- {tokenzero-0.1.0 → tokenzero-0.1.2}/tokenzero.egg-info/requires.txt +0 -0
- {tokenzero-0.1.0 → tokenzero-0.1.2}/tokenzero.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: tokenzero
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: Zero Token Waste - AST Code Skeletonizer & Live AI Monitor for Coding Agents
|
|
5
5
|
License-Expression: MIT
|
|
6
6
|
Keywords: tokenzero,tokens,ai-agents,ast,antigravity,claude,cursor,optimization,mcp
|
|
@@ -19,68 +19,67 @@ Requires-Dist: httpx>=0.27.0
|
|
|
19
19
|
Requires-Dist: tiktoken>=0.7.0
|
|
20
20
|
Dynamic: license-file
|
|
21
21
|
|
|
22
|
-
#
|
|
22
|
+
# TokenZero
|
|
23
23
|
|
|
24
|
-
|
|
24
|
+
[](https://pypi.org/project/tokenzero/)
|
|
25
|
+
[](https://opensource.org/licenses/MIT)
|
|
26
|
+
[](https://www.python.org/downloads/)
|
|
25
27
|
|
|
26
|
-
|
|
28
|
+
**Zero Token Waste for AI Coding Agents (Antigravity, Claude Code, Cursor, Codex).**
|
|
27
29
|
|
|
28
|
-
|
|
30
|
+
TokenZero cuts AI coding token consumption by 70% to 73% without sacrificing intelligence or code accuracy. It acts as an automated AST code skeletonizer, live real-time token monitor, and tool call interceptor.
|
|
29
31
|
|
|
30
|
-
##
|
|
32
|
+
## The Problem TokenZero Solves
|
|
31
33
|
|
|
32
34
|
When AI agents inspect codebases, they routinely load entire 500–2,000 line source files just to read a single 15-line function.
|
|
33
|
-
-
|
|
34
|
-
-
|
|
35
|
-
|
|
36
|
-
---
|
|
35
|
+
- Without TokenZero: Reading 15 functions across 5 files consumes ~40,000–60,000 tokens.
|
|
36
|
+
- With TokenZero: Agents inspect the AST skeletal map and execute targeted partial line-range reads (StartLine & EndLine), consuming ~2,500–5,000 tokens (a 70% to 73% direct reduction).
|
|
37
37
|
|
|
38
|
-
##
|
|
38
|
+
## Features
|
|
39
39
|
|
|
40
|
-
-
|
|
41
|
-
- Python (
|
|
42
|
-
- JavaScript & TypeScript (
|
|
43
|
-
- Go (
|
|
44
|
-
-
|
|
45
|
-
- Intercepts agent tool calls (
|
|
40
|
+
- Multi-Language AST Skeletonizer:
|
|
41
|
+
- Python (.py via Python AST)
|
|
42
|
+
- JavaScript & TypeScript (.js, .jsx, .ts, .tsx, .mjs, .cjs via structural brace-tracking parser)
|
|
43
|
+
- Go (.go)
|
|
44
|
+
- Real-Time Lifecycle Hooks (<2ms):
|
|
45
|
+
- Intercepts agent tool calls (read_file, edit_file, run_command) on-the-fly via .agents/hooks.json.
|
|
46
46
|
- Blocks automated edit popups from IDE diff viewers.
|
|
47
47
|
- Zero performance overhead.
|
|
48
|
-
-
|
|
49
|
-
- Pure terminal table view showing each file inspection, function targeted, lines read, and
|
|
50
|
-
-
|
|
51
|
-
-
|
|
48
|
+
- Live Activity Dashboard (tokenzero watch):
|
|
49
|
+
- Pure terminal table view showing each file inspection, function targeted, lines read, and Tokens Used.
|
|
50
|
+
- ROI & Savings Comparison: Compares unoptimized token consumption against TokenZero usage and displays exact dollar savings.
|
|
51
|
+
- Universal Clean Removal (tokenzero remove):
|
|
52
52
|
- One-command uninstaller that restores the project to its exact original state.
|
|
53
|
-
-
|
|
53
|
+
- Standalone Compression Proxy (tokenzero proxy):
|
|
54
54
|
- OpenAI-compatible local HTTP proxy for automated prompt payload compression.
|
|
55
55
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
## 🚀 Quick Start
|
|
56
|
+
## Quick Start
|
|
59
57
|
|
|
60
58
|
### 1. Installation
|
|
61
59
|
|
|
62
|
-
Install globally using
|
|
60
|
+
Install globally using pipx or standard pip:
|
|
63
61
|
|
|
64
62
|
```bash
|
|
65
|
-
|
|
66
|
-
|
|
63
|
+
pip install tokenzero
|
|
64
|
+
```
|
|
67
65
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
66
|
+
Or via pipx:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
pipx install tokenzero
|
|
72
70
|
```
|
|
73
71
|
|
|
74
72
|
### 2. One-Time Universal Setup
|
|
75
73
|
|
|
76
|
-
Automatically configure
|
|
74
|
+
Automatically configure Google Antigravity, Cursor IDE, Claude Desktop, and Claude Code with 1 command:
|
|
77
75
|
|
|
78
76
|
```bash
|
|
79
77
|
tokenzero setup
|
|
80
78
|
```
|
|
81
|
-
*(Note: If you skip this, running `tokenzero init` or `tokenzero watch` in any project will also auto-configure all your installed AI agents silently on first run!)*
|
|
82
79
|
|
|
83
|
-
|
|
80
|
+
(Note: If you skip this, running tokenzero init or tokenzero watch in any project will also auto-configure all your installed AI agents silently on first run).
|
|
81
|
+
|
|
82
|
+
TokenZero registers the Model Context Protocol (MCP) server across your system. Zero files are added to your project directories.
|
|
84
83
|
|
|
85
84
|
### 3. Initialize in Any Project (Zero-File Mode)
|
|
86
85
|
|
|
@@ -91,9 +90,9 @@ cd /path/to/your/project
|
|
|
91
90
|
tokenzero init
|
|
92
91
|
```
|
|
93
92
|
|
|
94
|
-
TokenZero indexes the AST code skeleton into your global cache (
|
|
93
|
+
TokenZero indexes the AST code skeleton into your global cache (~/.tokenzero/cache/). Your Git repository remains 100% clean with zero untracked files.
|
|
95
94
|
|
|
96
|
-
|
|
95
|
+
(Optional: Use tokenzero init --local if you prefer repository-specific rules with auto .gitignore).
|
|
97
96
|
|
|
98
97
|
### 4. Live Monitor
|
|
99
98
|
|
|
@@ -103,17 +102,9 @@ Open a second terminal or split pane and run:
|
|
|
103
102
|
tokenzero watch
|
|
104
103
|
```
|
|
105
104
|
|
|
106
|
-
Watch live tool calls stream in real time as your AI agent inspects and edits code
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
tokenzero watch
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
Watch live tool calls stream in real time as your AI agent inspects and edits code!
|
|
113
|
-
|
|
114
|
-
---
|
|
105
|
+
Watch live tool calls stream in real time as your AI agent inspects and edits code.
|
|
115
106
|
|
|
116
|
-
##
|
|
107
|
+
## Live Dashboard Preview
|
|
117
108
|
|
|
118
109
|
```text
|
|
119
110
|
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
|
|
@@ -127,16 +118,14 @@ Watch live tool calls stream in real time as your AI agent inspects and edits co
|
|
|
127
118
|
│ 2026-11-15 23:12:49 │ READ │ WRITEUP_2_PAGES.md:L1-L60 │ 60 / 161 lines │ ~420 tokens │
|
|
128
119
|
└─────────────────────┴──────────┴────────────────────────────────────────────────┴──────────────────┴─────────────────┘
|
|
129
120
|
|
|
130
|
-
╭────────────────────────────────
|
|
131
|
-
│ Activity: 28 events across 18 files |
|
|
132
|
-
│
|
|
133
|
-
│
|
|
121
|
+
╭──────────────────────────────── TokenZero ROI & Savings Comparison ───────────────────────────────────╮
|
|
122
|
+
│ Activity: 28 events across 18 files | Actual Cost (Without TokenZero): 29,534 tokens ($0.0738) ➜ │
|
|
123
|
+
│ Used (With TokenZero): 8,064 tokens ($0.0202) │
|
|
124
|
+
│ Net Impact: Saved +21,470 tokens (72.7% reduction) • You Saved $0.0537 USD! │
|
|
134
125
|
╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
135
126
|
```
|
|
136
127
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
## 🛠️ CLI Reference
|
|
128
|
+
## CLI Reference
|
|
140
129
|
|
|
141
130
|
| Command | Description |
|
|
142
131
|
| :--- | :--- |
|
|
@@ -144,16 +133,12 @@ Watch live tool calls stream in real time as your AI agent inspects and edits co
|
|
|
144
133
|
| `tokenzero watch [path]` | Starts live real-time terminal monitor for AI tool events. |
|
|
145
134
|
| `tokenzero stats [path] [--once]` | Prints summary audit report and savings metrics. |
|
|
146
135
|
| `tokenzero mcp` | Starts Model Context Protocol (MCP) stdio server for native AI tool calling. |
|
|
147
|
-
| `tokenzero remove [path]` | Completely cleans up
|
|
148
|
-
| `tokenzero proxy` | Starts local FastAPI token optimization proxy on
|
|
149
|
-
|
|
150
|
-
*(Note: `tokensqueeze` is also retained as a backwards-compatible alias for all commands).*
|
|
136
|
+
| `tokenzero remove [path]` | Completely cleans up .tokenzero/, rules, and hooks from the repo. |
|
|
137
|
+
| `tokenzero proxy` | Starts local FastAPI token optimization proxy on http://127.0.0.1:8080. |
|
|
151
138
|
|
|
152
|
-
|
|
139
|
+
## Model Context Protocol (MCP) Integration
|
|
153
140
|
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
TokenZero supports the standard Model Context Protocol (MCP). To connect TokenZero directly to **Claude Code**, **Claude Desktop**, or **Cursor**:
|
|
141
|
+
TokenZero supports the standard Model Context Protocol (MCP). To connect TokenZero directly to Claude Code, Claude Desktop, or Cursor:
|
|
157
142
|
|
|
158
143
|
Add to your `claude_desktop_config.json` or `.cursor/mcp.json`:
|
|
159
144
|
|
|
@@ -170,11 +155,9 @@ Add to your `claude_desktop_config.json` or `.cursor/mcp.json`:
|
|
|
170
155
|
|
|
171
156
|
Now your AI assistant can invoke:
|
|
172
157
|
- `tokenzero_get_repo_map`: Instant AST symbol inspection before reading code.
|
|
173
|
-
- `tokenzero_read_lines`: Target specific line numbers (
|
|
158
|
+
- `tokenzero_read_lines`: Target specific line numbers (start_line to end_line), saving 70% to 73% tokens.
|
|
174
159
|
- `tokenzero_get_stats`: Live savings metrics and ROI tracking.
|
|
175
160
|
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
## 📄 License
|
|
161
|
+
## License
|
|
179
162
|
|
|
180
163
|
MIT License. Free and open source for all developers and teams.
|
|
@@ -1,65 +1,64 @@
|
|
|
1
|
-
#
|
|
1
|
+
# TokenZero
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://pypi.org/project/tokenzero/)
|
|
4
|
+
[](https://opensource.org/licenses/MIT)
|
|
5
|
+
[](https://www.python.org/downloads/)
|
|
4
6
|
|
|
5
|
-
|
|
7
|
+
**Zero Token Waste for AI Coding Agents (Antigravity, Claude Code, Cursor, Codex).**
|
|
6
8
|
|
|
7
|
-
|
|
9
|
+
TokenZero cuts AI coding token consumption by 70% to 73% without sacrificing intelligence or code accuracy. It acts as an automated AST code skeletonizer, live real-time token monitor, and tool call interceptor.
|
|
8
10
|
|
|
9
|
-
##
|
|
11
|
+
## The Problem TokenZero Solves
|
|
10
12
|
|
|
11
13
|
When AI agents inspect codebases, they routinely load entire 500–2,000 line source files just to read a single 15-line function.
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
|
|
15
|
-
---
|
|
14
|
+
- Without TokenZero: Reading 15 functions across 5 files consumes ~40,000–60,000 tokens.
|
|
15
|
+
- With TokenZero: Agents inspect the AST skeletal map and execute targeted partial line-range reads (StartLine & EndLine), consuming ~2,500–5,000 tokens (a 70% to 73% direct reduction).
|
|
16
16
|
|
|
17
|
-
##
|
|
17
|
+
## Features
|
|
18
18
|
|
|
19
|
-
-
|
|
20
|
-
- Python (
|
|
21
|
-
- JavaScript & TypeScript (
|
|
22
|
-
- Go (
|
|
23
|
-
-
|
|
24
|
-
- Intercepts agent tool calls (
|
|
19
|
+
- Multi-Language AST Skeletonizer:
|
|
20
|
+
- Python (.py via Python AST)
|
|
21
|
+
- JavaScript & TypeScript (.js, .jsx, .ts, .tsx, .mjs, .cjs via structural brace-tracking parser)
|
|
22
|
+
- Go (.go)
|
|
23
|
+
- Real-Time Lifecycle Hooks (<2ms):
|
|
24
|
+
- Intercepts agent tool calls (read_file, edit_file, run_command) on-the-fly via .agents/hooks.json.
|
|
25
25
|
- Blocks automated edit popups from IDE diff viewers.
|
|
26
26
|
- Zero performance overhead.
|
|
27
|
-
-
|
|
28
|
-
- Pure terminal table view showing each file inspection, function targeted, lines read, and
|
|
29
|
-
-
|
|
30
|
-
-
|
|
27
|
+
- Live Activity Dashboard (tokenzero watch):
|
|
28
|
+
- Pure terminal table view showing each file inspection, function targeted, lines read, and Tokens Used.
|
|
29
|
+
- ROI & Savings Comparison: Compares unoptimized token consumption against TokenZero usage and displays exact dollar savings.
|
|
30
|
+
- Universal Clean Removal (tokenzero remove):
|
|
31
31
|
- One-command uninstaller that restores the project to its exact original state.
|
|
32
|
-
-
|
|
32
|
+
- Standalone Compression Proxy (tokenzero proxy):
|
|
33
33
|
- OpenAI-compatible local HTTP proxy for automated prompt payload compression.
|
|
34
34
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
## 🚀 Quick Start
|
|
35
|
+
## Quick Start
|
|
38
36
|
|
|
39
37
|
### 1. Installation
|
|
40
38
|
|
|
41
|
-
Install globally using
|
|
39
|
+
Install globally using pipx or standard pip:
|
|
42
40
|
|
|
43
41
|
```bash
|
|
44
|
-
|
|
45
|
-
|
|
42
|
+
pip install tokenzero
|
|
43
|
+
```
|
|
46
44
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
45
|
+
Or via pipx:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
pipx install tokenzero
|
|
51
49
|
```
|
|
52
50
|
|
|
53
51
|
### 2. One-Time Universal Setup
|
|
54
52
|
|
|
55
|
-
Automatically configure
|
|
53
|
+
Automatically configure Google Antigravity, Cursor IDE, Claude Desktop, and Claude Code with 1 command:
|
|
56
54
|
|
|
57
55
|
```bash
|
|
58
56
|
tokenzero setup
|
|
59
57
|
```
|
|
60
|
-
*(Note: If you skip this, running `tokenzero init` or `tokenzero watch` in any project will also auto-configure all your installed AI agents silently on first run!)*
|
|
61
58
|
|
|
62
|
-
|
|
59
|
+
(Note: If you skip this, running tokenzero init or tokenzero watch in any project will also auto-configure all your installed AI agents silently on first run).
|
|
60
|
+
|
|
61
|
+
TokenZero registers the Model Context Protocol (MCP) server across your system. Zero files are added to your project directories.
|
|
63
62
|
|
|
64
63
|
### 3. Initialize in Any Project (Zero-File Mode)
|
|
65
64
|
|
|
@@ -70,9 +69,9 @@ cd /path/to/your/project
|
|
|
70
69
|
tokenzero init
|
|
71
70
|
```
|
|
72
71
|
|
|
73
|
-
TokenZero indexes the AST code skeleton into your global cache (
|
|
72
|
+
TokenZero indexes the AST code skeleton into your global cache (~/.tokenzero/cache/). Your Git repository remains 100% clean with zero untracked files.
|
|
74
73
|
|
|
75
|
-
|
|
74
|
+
(Optional: Use tokenzero init --local if you prefer repository-specific rules with auto .gitignore).
|
|
76
75
|
|
|
77
76
|
### 4. Live Monitor
|
|
78
77
|
|
|
@@ -82,17 +81,9 @@ Open a second terminal or split pane and run:
|
|
|
82
81
|
tokenzero watch
|
|
83
82
|
```
|
|
84
83
|
|
|
85
|
-
Watch live tool calls stream in real time as your AI agent inspects and edits code
|
|
86
|
-
|
|
87
|
-
```bash
|
|
88
|
-
tokenzero watch
|
|
89
|
-
```
|
|
90
|
-
|
|
91
|
-
Watch live tool calls stream in real time as your AI agent inspects and edits code!
|
|
92
|
-
|
|
93
|
-
---
|
|
84
|
+
Watch live tool calls stream in real time as your AI agent inspects and edits code.
|
|
94
85
|
|
|
95
|
-
##
|
|
86
|
+
## Live Dashboard Preview
|
|
96
87
|
|
|
97
88
|
```text
|
|
98
89
|
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
|
|
@@ -106,16 +97,14 @@ Watch live tool calls stream in real time as your AI agent inspects and edits co
|
|
|
106
97
|
│ 2026-11-15 23:12:49 │ READ │ WRITEUP_2_PAGES.md:L1-L60 │ 60 / 161 lines │ ~420 tokens │
|
|
107
98
|
└─────────────────────┴──────────┴────────────────────────────────────────────────┴──────────────────┴─────────────────┘
|
|
108
99
|
|
|
109
|
-
╭────────────────────────────────
|
|
110
|
-
│ Activity: 28 events across 18 files |
|
|
111
|
-
│
|
|
112
|
-
│
|
|
100
|
+
╭──────────────────────────────── TokenZero ROI & Savings Comparison ───────────────────────────────────╮
|
|
101
|
+
│ Activity: 28 events across 18 files | Actual Cost (Without TokenZero): 29,534 tokens ($0.0738) ➜ │
|
|
102
|
+
│ Used (With TokenZero): 8,064 tokens ($0.0202) │
|
|
103
|
+
│ Net Impact: Saved +21,470 tokens (72.7% reduction) • You Saved $0.0537 USD! │
|
|
113
104
|
╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
114
105
|
```
|
|
115
106
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
## 🛠️ CLI Reference
|
|
107
|
+
## CLI Reference
|
|
119
108
|
|
|
120
109
|
| Command | Description |
|
|
121
110
|
| :--- | :--- |
|
|
@@ -123,16 +112,12 @@ Watch live tool calls stream in real time as your AI agent inspects and edits co
|
|
|
123
112
|
| `tokenzero watch [path]` | Starts live real-time terminal monitor for AI tool events. |
|
|
124
113
|
| `tokenzero stats [path] [--once]` | Prints summary audit report and savings metrics. |
|
|
125
114
|
| `tokenzero mcp` | Starts Model Context Protocol (MCP) stdio server for native AI tool calling. |
|
|
126
|
-
| `tokenzero remove [path]` | Completely cleans up
|
|
127
|
-
| `tokenzero proxy` | Starts local FastAPI token optimization proxy on
|
|
128
|
-
|
|
129
|
-
*(Note: `tokensqueeze` is also retained as a backwards-compatible alias for all commands).*
|
|
115
|
+
| `tokenzero remove [path]` | Completely cleans up .tokenzero/, rules, and hooks from the repo. |
|
|
116
|
+
| `tokenzero proxy` | Starts local FastAPI token optimization proxy on http://127.0.0.1:8080. |
|
|
130
117
|
|
|
131
|
-
|
|
118
|
+
## Model Context Protocol (MCP) Integration
|
|
132
119
|
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
TokenZero supports the standard Model Context Protocol (MCP). To connect TokenZero directly to **Claude Code**, **Claude Desktop**, or **Cursor**:
|
|
120
|
+
TokenZero supports the standard Model Context Protocol (MCP). To connect TokenZero directly to Claude Code, Claude Desktop, or Cursor:
|
|
136
121
|
|
|
137
122
|
Add to your `claude_desktop_config.json` or `.cursor/mcp.json`:
|
|
138
123
|
|
|
@@ -149,11 +134,9 @@ Add to your `claude_desktop_config.json` or `.cursor/mcp.json`:
|
|
|
149
134
|
|
|
150
135
|
Now your AI assistant can invoke:
|
|
151
136
|
- `tokenzero_get_repo_map`: Instant AST symbol inspection before reading code.
|
|
152
|
-
- `tokenzero_read_lines`: Target specific line numbers (
|
|
137
|
+
- `tokenzero_read_lines`: Target specific line numbers (start_line to end_line), saving 70% to 73% tokens.
|
|
153
138
|
- `tokenzero_get_stats`: Live savings metrics and ROI tracking.
|
|
154
139
|
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
## 📄 License
|
|
140
|
+
## License
|
|
158
141
|
|
|
159
142
|
MIT License. Free and open source for all developers and teams.
|
|
@@ -11,7 +11,7 @@ from scanner import scan_project, generate_markdown_map
|
|
|
11
11
|
|
|
12
12
|
INTERNAL_PATTERNS = (
|
|
13
13
|
".tokenzero",
|
|
14
|
-
|
|
14
|
+
|
|
15
15
|
"repo_map.md",
|
|
16
16
|
"audit.jsonl",
|
|
17
17
|
"hooks.json",
|
|
@@ -48,9 +48,6 @@ def get_repo_map_path(target_dir: Path) -> Path:
|
|
|
48
48
|
local_map = target_dir / ".tokenzero" / "REPO_MAP.md"
|
|
49
49
|
if local_map.exists():
|
|
50
50
|
return local_map
|
|
51
|
-
local_ts = target_dir / ".tokensqueeze" / "REPO_MAP.md"
|
|
52
|
-
if local_ts.exists():
|
|
53
|
-
return local_ts
|
|
54
51
|
return get_project_cache_dir(target_dir) / "REPO_MAP.md"
|
|
55
52
|
|
|
56
53
|
def record_activity(target_dir: Path, action: str, file: str, function: str, lines_used: int, total_lines: int):
|
|
@@ -222,7 +219,7 @@ def handle_get_stats(args: dict) -> str:
|
|
|
222
219
|
pass
|
|
223
220
|
|
|
224
221
|
# 2. Local fallback
|
|
225
|
-
for folder in (".tokenzero",
|
|
222
|
+
for folder in (".tokenzero",):
|
|
226
223
|
local_audit = target_dir / folder / "audit.jsonl"
|
|
227
224
|
if local_audit.exists():
|
|
228
225
|
for line in local_audit.read_text(encoding="utf-8").splitlines():
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "tokenzero"
|
|
7
|
-
version = "0.1.
|
|
7
|
+
version = "0.1.2"
|
|
8
8
|
description = "Zero Token Waste - AST Code Skeletonizer & Live AI Monitor for Coding Agents"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
license = "MIT"
|
|
@@ -27,7 +27,6 @@ dependencies = [
|
|
|
27
27
|
|
|
28
28
|
[project.scripts]
|
|
29
29
|
tokenzero = "squeeze:main"
|
|
30
|
-
tokensqueeze = "squeeze:main"
|
|
31
30
|
|
|
32
31
|
[tool.setuptools]
|
|
33
32
|
py-modules = ["squeeze", "scanner", "proxy", "mcp_server"]
|
|
@@ -11,7 +11,7 @@ from typing import Dict, Any, List
|
|
|
11
11
|
|
|
12
12
|
IGNORE_DIRS = {
|
|
13
13
|
".git", ".venv", "venv", "__pycache__", "node_modules",
|
|
14
|
-
".pytest_cache", ".tokenzero", ".
|
|
14
|
+
".pytest_cache", ".tokenzero", ".tokenzero", "dist", "build", ".egg-info",
|
|
15
15
|
".next", ".turbo", "coverage", ".cache"
|
|
16
16
|
}
|
|
17
17
|
|
|
@@ -377,8 +377,8 @@ def scan_project(root_dir: Path) -> List[Dict[str, Any]]:
|
|
|
377
377
|
def generate_markdown_map(scan_data: List[Dict[str, Any]]) -> str:
|
|
378
378
|
"""Converts the parsed AST data into an ultra-clean, dense markdown map."""
|
|
379
379
|
md = [
|
|
380
|
-
"#
|
|
381
|
-
"> Auto-generated by **
|
|
380
|
+
"# Project Architecture & Symbol Map",
|
|
381
|
+
"> Auto-generated by **TokenZero**. Consult this map for line ranges before reading full files.",
|
|
382
382
|
""
|
|
383
383
|
]
|
|
384
384
|
|
|
@@ -26,9 +26,9 @@ AGENT_CONFIGS = [
|
|
|
26
26
|
"file": "GEMINI.md",
|
|
27
27
|
"name": "Google Antigravity",
|
|
28
28
|
"rule": """<!-- BEGIN_TOKENZERO_RULE -->
|
|
29
|
-
#
|
|
29
|
+
# TokenZero Optimization Guide (Active)
|
|
30
30
|
|
|
31
|
-
1. **Check the Map First**: Before reading full source files, consult `.tokenzero/REPO_MAP.md` (or `.
|
|
31
|
+
1. **Check the Map First**: Before reading full source files, consult `.tokenzero/REPO_MAP.md` (or `.tokenzero/REPO_MAP.md`) to see the project structure, function signatures, and exact line ranges.
|
|
32
32
|
2. **Partial File Reads**: Do NOT read whole files (>100 lines) if you only need one class or function. Always use line-range reading (`StartLine` and `EndLine`) based on the line numbers in `REPO_MAP.md`.
|
|
33
33
|
3. **Targeted Edits**: Only inspect the exact function or block being edited to preserve context window and reduce token consumption.
|
|
34
34
|
4. **Do NOT Edit audit.jsonl**: The TokenZero background hook manages audit logging automatically. Never propose edits to `audit.jsonl`.
|
|
@@ -40,7 +40,7 @@ AGENT_CONFIGS = [
|
|
|
40
40
|
"file": "CLAUDE.md",
|
|
41
41
|
"name": "Claude Code",
|
|
42
42
|
"rule": """<!-- BEGIN_TOKENZERO_RULE -->
|
|
43
|
-
#
|
|
43
|
+
# TokenZero Optimization Guide (Active)
|
|
44
44
|
|
|
45
45
|
1. **Check the Map First**: Before reading full source files, consult `.tokenzero/REPO_MAP.md` to see the project structure, function signatures, and exact line ranges.
|
|
46
46
|
2. **Partial File Reads**: Do NOT read whole files (>100 lines) if you only need one class or function. Always use line-range reading based on the line numbers in `REPO_MAP.md`.
|
|
@@ -52,7 +52,7 @@ AGENT_CONFIGS = [
|
|
|
52
52
|
"file": ".cursorrules",
|
|
53
53
|
"name": "Cursor IDE",
|
|
54
54
|
"rule": """<!-- BEGIN_TOKENZERO_RULE -->
|
|
55
|
-
#
|
|
55
|
+
# TokenZero Optimization Guide (Active)
|
|
56
56
|
|
|
57
57
|
1. **Check the Map First**: Before reading full source files, consult `.tokenzero/REPO_MAP.md` to see the project structure, function signatures, and exact line ranges.
|
|
58
58
|
2. **Partial File Reads**: Do NOT read whole files (>100 lines) if you only need one class or function. Always use line-range reading based on the line numbers in `REPO_MAP.md`.
|
|
@@ -63,7 +63,7 @@ AGENT_CONFIGS = [
|
|
|
63
63
|
"file": ".windsurfrules",
|
|
64
64
|
"name": "Windsurf IDE",
|
|
65
65
|
"rule": """<!-- BEGIN_TOKENZERO_RULE -->
|
|
66
|
-
#
|
|
66
|
+
# TokenZero Optimization Guide (Active)
|
|
67
67
|
|
|
68
68
|
1. **Check the Map First**: Before reading full source files, consult `.tokenzero/REPO_MAP.md` to see the project structure, function signatures, and exact line ranges.
|
|
69
69
|
2. **Partial File Reads**: Do NOT read whole files (>100 lines) if you only need one class or function. Always use line-range reading based on the line numbers in `REPO_MAP.md`.
|
|
@@ -74,7 +74,7 @@ AGENT_CONFIGS = [
|
|
|
74
74
|
"file": ".github/copilot-instructions.md",
|
|
75
75
|
"name": "GitHub Copilot",
|
|
76
76
|
"rule": """<!-- BEGIN_TOKENZERO_RULE -->
|
|
77
|
-
#
|
|
77
|
+
# TokenZero Optimization Guide (Active)
|
|
78
78
|
|
|
79
79
|
1. **Check the Map First**: Before reading full source files, consult `.tokenzero/REPO_MAP.md` to see the project structure, function signatures, and exact line ranges.
|
|
80
80
|
2. **Partial File Reads**: Do NOT read whole files (>100 lines) if you only need one class or function. Always use line-range reading based on the line numbers in `REPO_MAP.md`.
|
|
@@ -90,7 +90,7 @@ HOOK_CONFIG = {
|
|
|
90
90
|
"hooks": [
|
|
91
91
|
{
|
|
92
92
|
"type": "command",
|
|
93
|
-
"command": "PATH=\"$HOME/.local/bin:$PATH\" tokenzero hook",
|
|
93
|
+
"command": "PATH=\"$HOME/.local/bin:$PATH\" tokenzero hook || exit 0",
|
|
94
94
|
"timeout": 5
|
|
95
95
|
}
|
|
96
96
|
]
|
|
@@ -101,7 +101,7 @@ HOOK_CONFIG = {
|
|
|
101
101
|
|
|
102
102
|
INTERNAL_PATTERNS = (
|
|
103
103
|
".tokenzero",
|
|
104
|
-
".
|
|
104
|
+
".tokenzero",
|
|
105
105
|
"repo_map.md",
|
|
106
106
|
"audit.jsonl",
|
|
107
107
|
"hooks.json",
|
|
@@ -142,7 +142,7 @@ def get_repo_map_path(target_dir: Path) -> Path:
|
|
|
142
142
|
local_map = target_dir / ".tokenzero" / "REPO_MAP.md"
|
|
143
143
|
if local_map.exists():
|
|
144
144
|
return local_map
|
|
145
|
-
local_ts = target_dir / ".
|
|
145
|
+
local_ts = target_dir / ".tokenzero" / "REPO_MAP.md"
|
|
146
146
|
if local_ts.exists():
|
|
147
147
|
return local_ts
|
|
148
148
|
return get_project_cache_dir(target_dir) / "REPO_MAP.md"
|
|
@@ -150,7 +150,7 @@ def get_repo_map_path(target_dir: Path) -> Path:
|
|
|
150
150
|
def get_config_dir(target_dir: Path) -> Path:
|
|
151
151
|
"""Returns local .tokenzero dir if exists, else global cache."""
|
|
152
152
|
tz = target_dir / ".tokenzero"
|
|
153
|
-
ts = target_dir / ".
|
|
153
|
+
ts = target_dir / ".tokenzero"
|
|
154
154
|
if tz.exists():
|
|
155
155
|
return tz
|
|
156
156
|
if ts.exists():
|
|
@@ -283,7 +283,7 @@ def get_audit_entries(target_dir: Path):
|
|
|
283
283
|
pass
|
|
284
284
|
|
|
285
285
|
# 2. Local fallback if legacy audit exists
|
|
286
|
-
for folder in (".tokenzero", ".
|
|
286
|
+
for folder in (".tokenzero", ".tokenzero"):
|
|
287
287
|
local_audit = target_dir / folder / "audit.jsonl"
|
|
288
288
|
if local_audit.exists():
|
|
289
289
|
try:
|
|
@@ -338,7 +338,7 @@ def render_dashboard_layout(target_dir: Path, spinner_frame: str) -> Layout:
|
|
|
338
338
|
table.add_row(
|
|
339
339
|
datetime.now().strftime("%H:%M:%S"),
|
|
340
340
|
"[dim]WAIT[/dim]",
|
|
341
|
-
"[dim]
|
|
341
|
+
"[dim]Listening for AI agent tool calls...[/dim]",
|
|
342
342
|
"-",
|
|
343
343
|
"-"
|
|
344
344
|
)
|
|
@@ -389,14 +389,14 @@ def render_dashboard_layout(target_dir: Path, spinner_frame: str) -> Layout:
|
|
|
389
389
|
|
|
390
390
|
summary_table.add_row(
|
|
391
391
|
f"[dim]Activity:[/dim] [bold white]{len(entries)} events[/bold white] across [bold white]{len(files_touched)} files[/bold white] | "
|
|
392
|
-
f"
|
|
393
|
-
f"
|
|
392
|
+
f"[dim]Unoptimized:[/dim] [white]{without_tokens:,}[/white] tokens (${dollars_without:.4f}) ➜ "
|
|
393
|
+
f"[dim]Optimized:[/dim] [green]{total_tokens_used:,}[/green] tokens (${dollars_with:.4f})"
|
|
394
394
|
)
|
|
395
395
|
summary_table.add_row(
|
|
396
|
-
f"
|
|
396
|
+
f"[bold green]Net Savings: {total_tokens_saved:,} tokens ({pct_saved:.1f}% reduction) | Cost Saved: ${dollars_saved:.4f} USD[/bold green]"
|
|
397
397
|
)
|
|
398
398
|
|
|
399
|
-
layout["footer"].update(Panel(summary_table, title="
|
|
399
|
+
layout["footer"].update(Panel(summary_table, title="TokenZero Metrics & Savings", border_style="cyan"))
|
|
400
400
|
|
|
401
401
|
return layout
|
|
402
402
|
|
|
@@ -405,7 +405,7 @@ def watch_dashboard(target_dir: Path):
|
|
|
405
405
|
setup_tokenzero(silent=True)
|
|
406
406
|
map_file = get_repo_map_path(target_dir)
|
|
407
407
|
if not map_file.exists():
|
|
408
|
-
console.print(f"[
|
|
408
|
+
console.print(f"[dim]TokenZero cache not found for {target_dir.name}. Initializing stealth mode...[/dim]")
|
|
409
409
|
init_tokenzero(target_dir, local=False)
|
|
410
410
|
|
|
411
411
|
console.clear()
|
|
@@ -419,19 +419,19 @@ def watch_dashboard(target_dir: Path):
|
|
|
419
419
|
idx = (idx + 1) % len(spinner_chars)
|
|
420
420
|
live.update(render_dashboard_layout(target_dir, spinner_chars[idx]))
|
|
421
421
|
except KeyboardInterrupt:
|
|
422
|
-
console.print("\n[
|
|
422
|
+
console.print("\n[dim]TokenZero live monitor stopped.[/dim]")
|
|
423
423
|
|
|
424
424
|
def show_stats_once(target_dir: Path):
|
|
425
425
|
"""Static one-time stats print."""
|
|
426
426
|
console.print(render_dashboard_layout(target_dir, "●"))
|
|
427
427
|
|
|
428
428
|
def init_tokenzero(target_dir: Path, local: bool = False):
|
|
429
|
-
console.print(f"\n[bold
|
|
429
|
+
console.print(f"\n[bold]TokenZero[/bold] initializing in: {target_dir}")
|
|
430
430
|
|
|
431
431
|
# 1. Scan codebase (Python, JavaScript, TypeScript, Go)
|
|
432
432
|
scan_results = scan_project(target_dir)
|
|
433
433
|
if not scan_results:
|
|
434
|
-
console.print("[yellow]
|
|
434
|
+
console.print("[yellow]Warning: No supported source files (.py, .js, .ts, .jsx, .tsx, .go) found.[/yellow]")
|
|
435
435
|
return
|
|
436
436
|
|
|
437
437
|
map_content = generate_markdown_map(scan_results)
|
|
@@ -449,7 +449,7 @@ def init_tokenzero(target_dir: Path, local: bool = False):
|
|
|
449
449
|
with open(gitignore_path, "a", encoding="utf-8") as f:
|
|
450
450
|
if existing_gi and not existing_gi.endswith("\n"):
|
|
451
451
|
f.write("\n")
|
|
452
|
-
f.write("\n# TokenZero (Auto-ignored)\n.tokenzero/\n.
|
|
452
|
+
f.write("\n# TokenZero (Auto-ignored)\n.tokenzero/\n.tokenzero/\n")
|
|
453
453
|
|
|
454
454
|
# Inject Rules into supported agent files
|
|
455
455
|
for cfg in AGENT_CONFIGS:
|
|
@@ -503,7 +503,7 @@ def init_tokenzero(target_dir: Path, local: bool = False):
|
|
|
503
503
|
|
|
504
504
|
mode_label = "Local (Auto-.gitignore)" if local else "[bold green]Zero-File Stealth Mode (0 files in project)[/bold green]"
|
|
505
505
|
|
|
506
|
-
table = Table(title="
|
|
506
|
+
table = Table(title="TokenZero Configuration", header_style="bold")
|
|
507
507
|
table.add_column("Component", style="cyan")
|
|
508
508
|
table.add_column("Status / Impact", style="bold white")
|
|
509
509
|
|
|
@@ -516,7 +516,7 @@ def init_tokenzero(target_dir: Path, local: bool = False):
|
|
|
516
516
|
|
|
517
517
|
console.print()
|
|
518
518
|
console.print(table)
|
|
519
|
-
console.print("\n[
|
|
519
|
+
console.print("\n[dim]Ready. Run '[cyan]tokenzero watch[/cyan]' to monitor live tool calls.[/dim]")
|
|
520
520
|
|
|
521
521
|
def ensure_antigravity_plugin() -> tuple:
|
|
522
522
|
"""Installs or updates the global Antigravity plugin in ~/.gemini/config/plugins/tokenzero/"""
|
|
@@ -535,7 +535,7 @@ def ensure_antigravity_plugin() -> tuple:
|
|
|
535
535
|
rules_dir = plugin_dir / "rules"
|
|
536
536
|
rules_dir.mkdir(exist_ok=True)
|
|
537
537
|
(rules_dir / "AGENTS.md").write_text("""<!-- BEGIN_TOKENZERO_RULE -->
|
|
538
|
-
#
|
|
538
|
+
# TokenZero Optimization Guide (Active)
|
|
539
539
|
|
|
540
540
|
1. **Check the Map First**: Before reading full source files, consult TokenZero REPO_MAP to see the project structure, function signatures, and exact line ranges.
|
|
541
541
|
2. **Partial File Reads**: Do NOT read whole files (>100 lines) if you only need one class or function. Always use line-range reading (`StartLine` and `EndLine`).
|
|
@@ -553,7 +553,7 @@ def setup_tokenzero(silent: bool = False) -> list:
|
|
|
553
553
|
Configures Claude Desktop, Claude Code, Cursor IDE, etc.
|
|
554
554
|
"""
|
|
555
555
|
if not silent:
|
|
556
|
-
console.print("\n[bold
|
|
556
|
+
console.print("\n[bold]TokenZero Universal Integration[/bold]\n")
|
|
557
557
|
results = []
|
|
558
558
|
|
|
559
559
|
mcp_config_entry = {
|
|
@@ -617,7 +617,7 @@ def setup_tokenzero(silent: bool = False) -> list:
|
|
|
617
617
|
results.append(("Google Antigravity", "~/.gemini/config/plugins/", f"[yellow]Skipped ({path_or_err})[/yellow]"))
|
|
618
618
|
|
|
619
619
|
if not silent:
|
|
620
|
-
table = Table(title="
|
|
620
|
+
table = Table(title="Integration Status", header_style="bold")
|
|
621
621
|
table.add_column("AI Agent / IDE", style="cyan")
|
|
622
622
|
table.add_column("Configuration Target", style="dim")
|
|
623
623
|
table.add_column("Status", style="bold white")
|
|
@@ -626,18 +626,18 @@ def setup_tokenzero(silent: bool = False) -> list:
|
|
|
626
626
|
table.add_row(agent, path, status)
|
|
627
627
|
|
|
628
628
|
console.print(table)
|
|
629
|
-
console.print("\n[
|
|
629
|
+
console.print("\n[dim]All configured AI coding assistants now route tool calls through TokenZero.[/dim]")
|
|
630
630
|
console.print("[dim]Project repositories remain 100% clean with 0 files added.[/dim]\n")
|
|
631
631
|
|
|
632
632
|
return results
|
|
633
633
|
|
|
634
634
|
def remove_tokenzero(target_dir: Path):
|
|
635
635
|
"""Completely removes TokenZero config, maps, rules, and hooks from a project."""
|
|
636
|
-
console.print(f"\n[bold
|
|
636
|
+
console.print(f"\n[bold]TokenZero:[/bold] Deactivating from {target_dir}")
|
|
637
637
|
removed_items = []
|
|
638
638
|
|
|
639
639
|
# 1. Remove directories
|
|
640
|
-
for d in (".tokenzero", ".
|
|
640
|
+
for d in (".tokenzero", ".tokenzero"):
|
|
641
641
|
folder = target_dir / d
|
|
642
642
|
if folder.exists():
|
|
643
643
|
shutil.rmtree(folder)
|
|
@@ -686,7 +686,7 @@ def remove_tokenzero(target_dir: Path):
|
|
|
686
686
|
try:
|
|
687
687
|
data = json.loads(hooks_file.read_text(encoding="utf-8"))
|
|
688
688
|
data.pop("tokenzero-live", None)
|
|
689
|
-
|
|
689
|
+
|
|
690
690
|
if data:
|
|
691
691
|
hooks_file.write_text(json.dumps(data, indent=2), encoding="utf-8")
|
|
692
692
|
removed_items.append(".agents/hooks.json")
|
|
@@ -704,7 +704,7 @@ def remove_tokenzero(target_dir: Path):
|
|
|
704
704
|
if gitignore_path.exists():
|
|
705
705
|
try:
|
|
706
706
|
gi_content = gitignore_path.read_text(encoding="utf-8")
|
|
707
|
-
cleaned_gi = re.sub(r"\n*# TokenZero \(Auto-ignored\)\n\.tokenzero/\n\.
|
|
707
|
+
cleaned_gi = re.sub(r"\n*# TokenZero \(Auto-ignored\)\n\.tokenzero/\n\.tokenzero/\n*", "\n", gi_content).strip()
|
|
708
708
|
if cleaned_gi != gi_content.strip():
|
|
709
709
|
if cleaned_gi:
|
|
710
710
|
gitignore_path.write_text(cleaned_gi + "\n", encoding="utf-8")
|
|
@@ -715,14 +715,14 @@ def remove_tokenzero(target_dir: Path):
|
|
|
715
715
|
pass
|
|
716
716
|
|
|
717
717
|
if removed_items:
|
|
718
|
-
console.print(f"\n[
|
|
718
|
+
console.print(f"\n[green]TokenZero optimizations deactivated successfully.[/green]")
|
|
719
719
|
console.print(f"[dim]Codebase restored to original state ({len(removed_items)} components cleaned).[/dim]\n")
|
|
720
720
|
else:
|
|
721
|
-
console.print(f"[
|
|
721
|
+
console.print(f"[dim]No TokenZero configurations found in {target_dir.name}.[/dim]\n")
|
|
722
722
|
|
|
723
723
|
def remove_global_configs() -> list:
|
|
724
724
|
"""Completely uninstalls TokenZero from all global AI editors, plugins, and cache."""
|
|
725
|
-
console.print("\n[bold
|
|
725
|
+
console.print("\n[bold]TokenZero Global Cleanup[/bold]\n")
|
|
726
726
|
cleaned = []
|
|
727
727
|
|
|
728
728
|
# 1. Remove Google Antigravity Global Plugin
|
|
@@ -786,7 +786,7 @@ def remove_global_configs() -> list:
|
|
|
786
786
|
except Exception as e:
|
|
787
787
|
cleaned.append(("Global Cache & Telemetry", str(GLOBAL_DIR), f"[yellow]Error ({e})[/yellow]"))
|
|
788
788
|
|
|
789
|
-
table = Table(title="
|
|
789
|
+
table = Table(title="Global Cleanup Status", header_style="bold")
|
|
790
790
|
table.add_column("AI Agent / Resource", style="cyan")
|
|
791
791
|
table.add_column("Location", style="dim")
|
|
792
792
|
table.add_column("Result", style="bold white")
|
|
@@ -795,12 +795,12 @@ def remove_global_configs() -> list:
|
|
|
795
795
|
table.add_row(agent, path, status)
|
|
796
796
|
|
|
797
797
|
console.print(table)
|
|
798
|
-
console.print("\n[
|
|
798
|
+
console.print("\n[green]All TokenZero global integrations and local caches have been removed.[/green]\n")
|
|
799
799
|
return cleaned
|
|
800
800
|
|
|
801
801
|
def main():
|
|
802
802
|
import argparse
|
|
803
|
-
parser = argparse.ArgumentParser(description="
|
|
803
|
+
parser = argparse.ArgumentParser(description="TokenZero - AST Code Skeletonizer & Token Optimizer for AI Coding Agents")
|
|
804
804
|
subparsers = parser.add_subparsers(dest="command", help="Available commands")
|
|
805
805
|
|
|
806
806
|
# init command
|
|
@@ -865,7 +865,7 @@ def main():
|
|
|
865
865
|
elif args.command == "proxy":
|
|
866
866
|
import uvicorn
|
|
867
867
|
from proxy import app
|
|
868
|
-
console.print("
|
|
868
|
+
console.print("TokenZero Proxy listening on http://127.0.0.1:8080")
|
|
869
869
|
uvicorn.run(app, host="127.0.0.1", port=8080, log_level="warning")
|
|
870
870
|
elif args.command == "watch":
|
|
871
871
|
target_path = Path(getattr(args, "path", ".")).resolve()
|
|
@@ -107,7 +107,7 @@ class TestTokenZeroScanner(unittest.TestCase):
|
|
|
107
107
|
py_file.write_text("def helper(): pass\n")
|
|
108
108
|
results = scan_project(Path(self.test_dir))
|
|
109
109
|
md = generate_markdown_map(results)
|
|
110
|
-
self.assertIn("#
|
|
110
|
+
self.assertIn("# Project Architecture & Symbol Map", md)
|
|
111
111
|
self.assertIn("utils.py", md)
|
|
112
112
|
self.assertIn("helper", md)
|
|
113
113
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: tokenzero
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: Zero Token Waste - AST Code Skeletonizer & Live AI Monitor for Coding Agents
|
|
5
5
|
License-Expression: MIT
|
|
6
6
|
Keywords: tokenzero,tokens,ai-agents,ast,antigravity,claude,cursor,optimization,mcp
|
|
@@ -19,68 +19,67 @@ Requires-Dist: httpx>=0.27.0
|
|
|
19
19
|
Requires-Dist: tiktoken>=0.7.0
|
|
20
20
|
Dynamic: license-file
|
|
21
21
|
|
|
22
|
-
#
|
|
22
|
+
# TokenZero
|
|
23
23
|
|
|
24
|
-
|
|
24
|
+
[](https://pypi.org/project/tokenzero/)
|
|
25
|
+
[](https://opensource.org/licenses/MIT)
|
|
26
|
+
[](https://www.python.org/downloads/)
|
|
25
27
|
|
|
26
|
-
|
|
28
|
+
**Zero Token Waste for AI Coding Agents (Antigravity, Claude Code, Cursor, Codex).**
|
|
27
29
|
|
|
28
|
-
|
|
30
|
+
TokenZero cuts AI coding token consumption by 70% to 73% without sacrificing intelligence or code accuracy. It acts as an automated AST code skeletonizer, live real-time token monitor, and tool call interceptor.
|
|
29
31
|
|
|
30
|
-
##
|
|
32
|
+
## The Problem TokenZero Solves
|
|
31
33
|
|
|
32
34
|
When AI agents inspect codebases, they routinely load entire 500–2,000 line source files just to read a single 15-line function.
|
|
33
|
-
-
|
|
34
|
-
-
|
|
35
|
-
|
|
36
|
-
---
|
|
35
|
+
- Without TokenZero: Reading 15 functions across 5 files consumes ~40,000–60,000 tokens.
|
|
36
|
+
- With TokenZero: Agents inspect the AST skeletal map and execute targeted partial line-range reads (StartLine & EndLine), consuming ~2,500–5,000 tokens (a 70% to 73% direct reduction).
|
|
37
37
|
|
|
38
|
-
##
|
|
38
|
+
## Features
|
|
39
39
|
|
|
40
|
-
-
|
|
41
|
-
- Python (
|
|
42
|
-
- JavaScript & TypeScript (
|
|
43
|
-
- Go (
|
|
44
|
-
-
|
|
45
|
-
- Intercepts agent tool calls (
|
|
40
|
+
- Multi-Language AST Skeletonizer:
|
|
41
|
+
- Python (.py via Python AST)
|
|
42
|
+
- JavaScript & TypeScript (.js, .jsx, .ts, .tsx, .mjs, .cjs via structural brace-tracking parser)
|
|
43
|
+
- Go (.go)
|
|
44
|
+
- Real-Time Lifecycle Hooks (<2ms):
|
|
45
|
+
- Intercepts agent tool calls (read_file, edit_file, run_command) on-the-fly via .agents/hooks.json.
|
|
46
46
|
- Blocks automated edit popups from IDE diff viewers.
|
|
47
47
|
- Zero performance overhead.
|
|
48
|
-
-
|
|
49
|
-
- Pure terminal table view showing each file inspection, function targeted, lines read, and
|
|
50
|
-
-
|
|
51
|
-
-
|
|
48
|
+
- Live Activity Dashboard (tokenzero watch):
|
|
49
|
+
- Pure terminal table view showing each file inspection, function targeted, lines read, and Tokens Used.
|
|
50
|
+
- ROI & Savings Comparison: Compares unoptimized token consumption against TokenZero usage and displays exact dollar savings.
|
|
51
|
+
- Universal Clean Removal (tokenzero remove):
|
|
52
52
|
- One-command uninstaller that restores the project to its exact original state.
|
|
53
|
-
-
|
|
53
|
+
- Standalone Compression Proxy (tokenzero proxy):
|
|
54
54
|
- OpenAI-compatible local HTTP proxy for automated prompt payload compression.
|
|
55
55
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
## 🚀 Quick Start
|
|
56
|
+
## Quick Start
|
|
59
57
|
|
|
60
58
|
### 1. Installation
|
|
61
59
|
|
|
62
|
-
Install globally using
|
|
60
|
+
Install globally using pipx or standard pip:
|
|
63
61
|
|
|
64
62
|
```bash
|
|
65
|
-
|
|
66
|
-
|
|
63
|
+
pip install tokenzero
|
|
64
|
+
```
|
|
67
65
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
66
|
+
Or via pipx:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
pipx install tokenzero
|
|
72
70
|
```
|
|
73
71
|
|
|
74
72
|
### 2. One-Time Universal Setup
|
|
75
73
|
|
|
76
|
-
Automatically configure
|
|
74
|
+
Automatically configure Google Antigravity, Cursor IDE, Claude Desktop, and Claude Code with 1 command:
|
|
77
75
|
|
|
78
76
|
```bash
|
|
79
77
|
tokenzero setup
|
|
80
78
|
```
|
|
81
|
-
*(Note: If you skip this, running `tokenzero init` or `tokenzero watch` in any project will also auto-configure all your installed AI agents silently on first run!)*
|
|
82
79
|
|
|
83
|
-
|
|
80
|
+
(Note: If you skip this, running tokenzero init or tokenzero watch in any project will also auto-configure all your installed AI agents silently on first run).
|
|
81
|
+
|
|
82
|
+
TokenZero registers the Model Context Protocol (MCP) server across your system. Zero files are added to your project directories.
|
|
84
83
|
|
|
85
84
|
### 3. Initialize in Any Project (Zero-File Mode)
|
|
86
85
|
|
|
@@ -91,9 +90,9 @@ cd /path/to/your/project
|
|
|
91
90
|
tokenzero init
|
|
92
91
|
```
|
|
93
92
|
|
|
94
|
-
TokenZero indexes the AST code skeleton into your global cache (
|
|
93
|
+
TokenZero indexes the AST code skeleton into your global cache (~/.tokenzero/cache/). Your Git repository remains 100% clean with zero untracked files.
|
|
95
94
|
|
|
96
|
-
|
|
95
|
+
(Optional: Use tokenzero init --local if you prefer repository-specific rules with auto .gitignore).
|
|
97
96
|
|
|
98
97
|
### 4. Live Monitor
|
|
99
98
|
|
|
@@ -103,17 +102,9 @@ Open a second terminal or split pane and run:
|
|
|
103
102
|
tokenzero watch
|
|
104
103
|
```
|
|
105
104
|
|
|
106
|
-
Watch live tool calls stream in real time as your AI agent inspects and edits code
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
tokenzero watch
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
Watch live tool calls stream in real time as your AI agent inspects and edits code!
|
|
113
|
-
|
|
114
|
-
---
|
|
105
|
+
Watch live tool calls stream in real time as your AI agent inspects and edits code.
|
|
115
106
|
|
|
116
|
-
##
|
|
107
|
+
## Live Dashboard Preview
|
|
117
108
|
|
|
118
109
|
```text
|
|
119
110
|
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
|
|
@@ -127,16 +118,14 @@ Watch live tool calls stream in real time as your AI agent inspects and edits co
|
|
|
127
118
|
│ 2026-11-15 23:12:49 │ READ │ WRITEUP_2_PAGES.md:L1-L60 │ 60 / 161 lines │ ~420 tokens │
|
|
128
119
|
└─────────────────────┴──────────┴────────────────────────────────────────────────┴──────────────────┴─────────────────┘
|
|
129
120
|
|
|
130
|
-
╭────────────────────────────────
|
|
131
|
-
│ Activity: 28 events across 18 files |
|
|
132
|
-
│
|
|
133
|
-
│
|
|
121
|
+
╭──────────────────────────────── TokenZero ROI & Savings Comparison ───────────────────────────────────╮
|
|
122
|
+
│ Activity: 28 events across 18 files | Actual Cost (Without TokenZero): 29,534 tokens ($0.0738) ➜ │
|
|
123
|
+
│ Used (With TokenZero): 8,064 tokens ($0.0202) │
|
|
124
|
+
│ Net Impact: Saved +21,470 tokens (72.7% reduction) • You Saved $0.0537 USD! │
|
|
134
125
|
╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
135
126
|
```
|
|
136
127
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
## 🛠️ CLI Reference
|
|
128
|
+
## CLI Reference
|
|
140
129
|
|
|
141
130
|
| Command | Description |
|
|
142
131
|
| :--- | :--- |
|
|
@@ -144,16 +133,12 @@ Watch live tool calls stream in real time as your AI agent inspects and edits co
|
|
|
144
133
|
| `tokenzero watch [path]` | Starts live real-time terminal monitor for AI tool events. |
|
|
145
134
|
| `tokenzero stats [path] [--once]` | Prints summary audit report and savings metrics. |
|
|
146
135
|
| `tokenzero mcp` | Starts Model Context Protocol (MCP) stdio server for native AI tool calling. |
|
|
147
|
-
| `tokenzero remove [path]` | Completely cleans up
|
|
148
|
-
| `tokenzero proxy` | Starts local FastAPI token optimization proxy on
|
|
149
|
-
|
|
150
|
-
*(Note: `tokensqueeze` is also retained as a backwards-compatible alias for all commands).*
|
|
136
|
+
| `tokenzero remove [path]` | Completely cleans up .tokenzero/, rules, and hooks from the repo. |
|
|
137
|
+
| `tokenzero proxy` | Starts local FastAPI token optimization proxy on http://127.0.0.1:8080. |
|
|
151
138
|
|
|
152
|
-
|
|
139
|
+
## Model Context Protocol (MCP) Integration
|
|
153
140
|
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
TokenZero supports the standard Model Context Protocol (MCP). To connect TokenZero directly to **Claude Code**, **Claude Desktop**, or **Cursor**:
|
|
141
|
+
TokenZero supports the standard Model Context Protocol (MCP). To connect TokenZero directly to Claude Code, Claude Desktop, or Cursor:
|
|
157
142
|
|
|
158
143
|
Add to your `claude_desktop_config.json` or `.cursor/mcp.json`:
|
|
159
144
|
|
|
@@ -170,11 +155,9 @@ Add to your `claude_desktop_config.json` or `.cursor/mcp.json`:
|
|
|
170
155
|
|
|
171
156
|
Now your AI assistant can invoke:
|
|
172
157
|
- `tokenzero_get_repo_map`: Instant AST symbol inspection before reading code.
|
|
173
|
-
- `tokenzero_read_lines`: Target specific line numbers (
|
|
158
|
+
- `tokenzero_read_lines`: Target specific line numbers (start_line to end_line), saving 70% to 73% tokens.
|
|
174
159
|
- `tokenzero_get_stats`: Live savings metrics and ROI tracking.
|
|
175
160
|
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
## 📄 License
|
|
161
|
+
## License
|
|
179
162
|
|
|
180
163
|
MIT License. Free and open source for all developers and teams.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|