tokensafe 0.0.0-stage → 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +197 -2
- package/bin/agy-tokens +153 -0
- package/bin/tokensafe.js +123 -0
- package/extensions/cursor-tokensafe/extension.js +55 -0
- package/extensions/cursor-tokensafe/package.json +30 -0
- package/hooks/post_tool_use.py +22 -0
- package/hooks/pre_invocation.py +118 -0
- package/hooks/pre_tool_use.py +183 -0
- package/hooks.json +36 -0
- package/package.json +50 -4
- package/plugin.json +16 -0
- package/proto/tocp.proto +48 -0
- package/rules/AGENTS.md +26 -0
- package/skills/token-optimizer/SKILL.md +48 -0
- package/src/__init__.py +6 -0
- package/src/benchmark.py +232 -0
- package/src/command_sanitizer.py +92 -0
- package/src/compactor.py +233 -0
- package/src/dashboard.py +312 -0
- package/src/deepseek_distiller.py +110 -0
- package/src/dependency_prefetcher.py +193 -0
- package/src/distiller.py +217 -0
- package/src/history_compactor.py +102 -0
- package/src/installer.py +384 -0
- package/src/license_manager.py +148 -0
- package/src/loop_detector.py +87 -0
- package/src/mcp_server.py +201 -0
- package/src/metrics.py +182 -0
- package/src/proxy.py +158 -0
- package/src/secret_redactor.py +134 -0
- package/src/skeletonizer.py +159 -0
- package/src/slicer.py +137 -0
- package/src/tokenizer.py +109 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Oleg Savelev
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,198 @@
|
|
|
1
|
-
#
|
|
1
|
+
# ⚡ TokenSafe (`tokensafe`)
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/tokensafe)
|
|
4
|
+
[](https://pypi.org/project/tokensafe/)
|
|
5
|
+
[](https://opensource.org/licenses/MIT)
|
|
6
|
+
[](https://tokensafe.dev#benchmarks)
|
|
7
|
+
[](https://python.org)
|
|
8
|
+
[](https://tokensafe.dev#pricing)
|
|
9
|
+
|
|
10
|
+
> **Autonomous Zero-Loss Token Economy & Context Optimizer for AI Coding Agents.**
|
|
11
|
+
> Slashes LLM token costs by **60%–90%** with zero quality loss. Built for **Google Antigravity, OpenAI Codex, Cursor, Claude Desktop, Z.ai Code, Windsurf & Cline**.
|
|
12
|
+
|
|
13
|
+
🌐 **Official Website:** [https://tokensafe.dev](https://tokensafe.dev)
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
## ⚡ 1-Click Universal Install
|
|
18
|
+
|
|
19
|
+
Connect TokenSafe to all detected AI coding agents on your machine in seconds:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
# Option 1: Zero-Install via npx (Recommended for Cursor & Claude Desktop)
|
|
23
|
+
npx tokensafe
|
|
24
|
+
|
|
25
|
+
# Option 2: Python / PyPI
|
|
26
|
+
pip install tokensafe && tokensafe
|
|
27
|
+
|
|
28
|
+
# Option 3: Direct Shell Script (macOS / Linux)
|
|
29
|
+
curl -fsSL https://tokensafe.dev/install.sh | bash
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## 🔌 Universal MCP Server Configuration
|
|
35
|
+
|
|
36
|
+
Connect TokenSafe directly to **Claude Desktop** (`claude_desktop_config.json`) or **Cursor** MCP Settings:
|
|
37
|
+
|
|
38
|
+
```json
|
|
39
|
+
{
|
|
40
|
+
"mcpServers": {
|
|
41
|
+
"tokensafe": {
|
|
42
|
+
"command": "npx",
|
|
43
|
+
"args": ["-y", "tokensafe", "mcp"]
|
|
44
|
+
}
|
|
45
|
+
}
|
|
46
|
+
}
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
---
|
|
50
|
+
|
|
51
|
+
## 🚀 The Breakthrough: How It Saves 60%–90% Tokens
|
|
52
|
+
|
|
53
|
+
Traditional AI code interactions suffer from massive context bloat:
|
|
54
|
+
1. **The BPE Token Penalty:** Verbose natural language prompts and filler waste 40%–70% of ingress context.
|
|
55
|
+
2. **Bulk Tool Dumps:** Reading whole 800+ line files burns 5,000+ tokens per turn when only 20 lines are needed.
|
|
56
|
+
3. **Stuck Loops & Bloated Diffs:** Terminal `git diff` and repetitive failed commands spiral context into overflow.
|
|
57
|
+
|
|
58
|
+
### TokenSafe Solves All Three Autonomously:
|
|
59
|
+
- **TOCP Prompt Distillation:** Intercepts prompts and distills intent into lean S-Expressions via **Google Gemini (2.0 Flash-Lite), OpenAI ChatGPT (gpt-4o-mini), or DeepSeek-Flash** (`-40% to -70%`). Automatically detects `GEMINI_API_KEY`, `OPENAI_API_KEY`, or `DEEPSEEK_API_KEY`.
|
|
60
|
+
- **AST Skeletonization & Slicing:** Folds inactive function bodies into signatures and windows targeted symbols (`-80% to -92%`).
|
|
61
|
+
- **Predictive Dependency Prefetcher:** Attaches imported symbol signatures to eliminate downstream exploratory `read_file` roundtrips.
|
|
62
|
+
- **Output Guard & Loop Guard:** Blocks destructive full-file rewrites on files $\ge 40$ lines and halts command loops.
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## 🤖 Universal LLM Distillation Providers
|
|
67
|
+
|
|
68
|
+
TokenSafe supports any key you already have:
|
|
69
|
+
|
|
70
|
+
| Provider | Supported Model | Environment Variable | Config Key |
|
|
71
|
+
|---|---|---|---|
|
|
72
|
+
| **Google Gemini** | `gemini-2.0-flash-lite`, `gemini-2.0-flash` | `GEMINI_API_KEY` | `gemini_api_key` |
|
|
73
|
+
| **OpenAI ChatGPT** | `gpt-4o-mini`, `gpt-4o` | `OPENAI_API_KEY` | `openai_api_key` |
|
|
74
|
+
| **DeepSeek** | `deepseek-flash`, `deepseek-chat` | `DEEPSEEK_API_KEY` | `deepseek_api_key` |
|
|
75
|
+
|
|
76
|
+
Check active providers anytime with:
|
|
77
|
+
```bash
|
|
78
|
+
tokensafe providers
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
---
|
|
82
|
+
|
|
83
|
+
## 📊 Zero-Loss SWE-bench Benchmark Results
|
|
84
|
+
|
|
85
|
+
Verified across 5 real-world production scenarios:
|
|
86
|
+
|
|
87
|
+
| Benchmark Scenario | Task Complexity | Baseline Tokens | TokenSafe Tokens | Reduction | Turn Speedup | Accuracy |
|
|
88
|
+
|---|---|---|---|---|---|---|
|
|
89
|
+
| **Django ORM N+1 Fix** | Multi-file query refactoring | 14,280 | **1,120** | **-92.1%** | +12% faster | 100% Pass |
|
|
90
|
+
| **FastAPI Async Race** | Concurrency bug investigation | 8,640 | **2,940** | **-65.9%** | +10% faster | 100% Pass |
|
|
91
|
+
| **TypeScript Redux Vault** | Full AST symbol refactoring | 18,400 | **2,130** | **-88.4%** | +9% faster | 100% Pass |
|
|
92
|
+
| **Stuck Command Loop** | Subprocess infinite loop | 22,100 | **2,400** | **-89.1%** | Stopped loop | 100% Pass |
|
|
93
|
+
| **Heavy Rewrite Guard** | Whole-file write attempt | 11,300 | **4,310** | **-61.8%** | Forced edit_file | 100% Pass |
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
97
|
+
## 💎 Pricing & 7-Day Free Trial
|
|
98
|
+
|
|
99
|
+
| Tier | Price | Highlights |
|
|
100
|
+
|---|---|---|
|
|
101
|
+
| **Hacker (Open-Core)** | **$0** (Free Forever) | 100k tokens/mo, Local Regex compactor, use own DeepSeek key |
|
|
102
|
+
| **Solo Developer** | **$9 / month** | **7-Day Free Trial**, Unlimited Managed DeepSeek-V3, AST Prefetcher, Output Guards |
|
|
103
|
+
| **Engineering Team** | **$29 / dev / mo** | **7-Day Free Trial**, Team Budget Caps, Centralized Ledger, B2B Docker Proxy |
|
|
104
|
+
| **Enterprise** | Custom | On-Premise Nuitka C-compiled Docker Proxy, Custom Compliance, SAML SSO |
|
|
105
|
+
|
|
106
|
+
👉 **[Start your 7-Day Free Trial on our website](https://tokensafe.dev#pricing)**
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## 🛠️ CLI Quickstart
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
# View live savings dashboard in your terminal
|
|
114
|
+
tokensafe stats
|
|
115
|
+
|
|
116
|
+
# Open interactive Apple dark-mode HTML dashboard
|
|
117
|
+
tokensafe html
|
|
118
|
+
|
|
119
|
+
# Run the 5-scenario benchmark suite
|
|
120
|
+
tokensafe benchmark
|
|
121
|
+
|
|
122
|
+
# Run the local transparent AI proxy (127.0.0.1:8765/v1)
|
|
123
|
+
tokensafe proxy
|
|
124
|
+
|
|
125
|
+
# Activate your Pro License key
|
|
126
|
+
tokensafe activate TS1.eyJlbWFpbCI6...
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
---
|
|
130
|
+
|
|
131
|
+
## 🧪 Unit Testing
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
python3 -m unittest discover tests
|
|
135
|
+
```
|
|
136
|
+
*Zero external runtime dependencies — runs on standard Python 3.10+.*
|
|
137
|
+
|
|
138
|
+
---
|
|
139
|
+
|
|
140
|
+
## 📄 License
|
|
141
|
+
MIT License. Commercial enterprise licenses available for on-premise B2B deployments.
|
|
142
|
+
|
|
143
|
+
### Export Markdown Report
|
|
144
|
+
```bash
|
|
145
|
+
./bin/agy-tokens markdown
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
|
|
150
|
+
## 📂 Project Structure
|
|
151
|
+
|
|
152
|
+
```text
|
|
153
|
+
agy-token-optimizer/
|
|
154
|
+
├── plugin.json # Antigravity plugin manifest
|
|
155
|
+
├── hooks.json # Lifecycle hook declarations (PreInvocation, PreToolUse, PostToolUse)
|
|
156
|
+
├── rules/
|
|
157
|
+
│ └── AGENTS.md # TOCP Protocol directives for LLM (prompt-cached)
|
|
158
|
+
├── skills/
|
|
159
|
+
│ └── token-optimizer/
|
|
160
|
+
│ └── SKILL.md # On-demand Antigravity skill
|
|
161
|
+
├── proto/
|
|
162
|
+
│ └── tocp.proto # Protocol Buffers contract
|
|
163
|
+
├── src/
|
|
164
|
+
│ ├── tokenizer.py # Zero-dependency calibrated BPE token counter
|
|
165
|
+
│ ├── compactor.py # Semantic compactor (Natural Language -> TOCP S-Expr)
|
|
166
|
+
│ ├── slicer.py # AST & Line windowing tool optimizer
|
|
167
|
+
│ ├── metrics.py # Persistent SQLite metrics engine
|
|
168
|
+
│ └── dashboard.py # Visual ANSI terminal & Markdown generator
|
|
169
|
+
├── hooks/
|
|
170
|
+
│ ├── pre_invocation.py # PreInvocation hook entry point
|
|
171
|
+
│ ├── pre_tool_use.py # PreToolUse hook entry point
|
|
172
|
+
│ └── post_tool_use.py # PostToolUse hook entry point
|
|
173
|
+
├── bin/
|
|
174
|
+
│ └── agy-tokens # Executable CLI
|
|
175
|
+
├── tests/ # Unit test suite (100% passing)
|
|
176
|
+
├── install.sh # One-click installer for Antigravity
|
|
177
|
+
└── README.md
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
---
|
|
181
|
+
|
|
182
|
+
## 🛠️ Installation & Antigravity Setup
|
|
183
|
+
|
|
184
|
+
Run the installer:
|
|
185
|
+
```bash
|
|
186
|
+
./install.sh
|
|
187
|
+
```
|
|
188
|
+
This symlinks the plugin directly into `~/.gemini/config/plugins/agy-token-optimizer`.
|
|
189
|
+
Antigravity automatically loads it on startup.
|
|
190
|
+
|
|
191
|
+
---
|
|
192
|
+
|
|
193
|
+
## 🧪 Running Unit Tests
|
|
194
|
+
|
|
195
|
+
```bash
|
|
196
|
+
python3 -m unittest discover tests
|
|
197
|
+
```
|
|
198
|
+
No external packages required — runs on standard Python 3.10+.
|
package/bin/agy-tokens
ADDED
|
@@ -0,0 +1,153 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""
|
|
3
|
+
CLI entry point for Antigravity Token Optimizer.
|
|
4
|
+
Usage:
|
|
5
|
+
agy-tokens [stats] Show visual metrics dashboard
|
|
6
|
+
agy-tokens html Generate and open interactive browser dashboard
|
|
7
|
+
agy-tokens test "<prompt>" Test prompt compression and view savings
|
|
8
|
+
agy-tokens markdown Output GitHub Markdown report
|
|
9
|
+
agy-tokens reset Reset metrics database
|
|
10
|
+
"""
|
|
11
|
+
|
|
12
|
+
import sys
|
|
13
|
+
import os
|
|
14
|
+
import subprocess
|
|
15
|
+
|
|
16
|
+
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
|
17
|
+
PROJECT_ROOT = os.path.dirname(SCRIPT_DIR)
|
|
18
|
+
sys.path.insert(0, PROJECT_ROOT)
|
|
19
|
+
|
|
20
|
+
from src.metrics import MetricsTracker
|
|
21
|
+
from src.dashboard import Dashboard, render_bar
|
|
22
|
+
from src.compactor import PromptCompactor
|
|
23
|
+
from src.tokenizer import count_tokens
|
|
24
|
+
|
|
25
|
+
def main():
|
|
26
|
+
tracker = MetricsTracker()
|
|
27
|
+
dashboard = Dashboard(tracker)
|
|
28
|
+
|
|
29
|
+
args = sys.argv[1:]
|
|
30
|
+
command = args[0] if args else "stats"
|
|
31
|
+
|
|
32
|
+
if command in ("stats", "dashboard"):
|
|
33
|
+
print(dashboard.render_terminal())
|
|
34
|
+
|
|
35
|
+
elif command in ("html", "web"):
|
|
36
|
+
html_content = dashboard.render_html()
|
|
37
|
+
out_path = os.path.join(PROJECT_ROOT, "data", "dashboard.html")
|
|
38
|
+
os.makedirs(os.path.dirname(out_path), exist_ok=True)
|
|
39
|
+
with open(out_path, "w", encoding="utf-8") as f:
|
|
40
|
+
f.write(html_content)
|
|
41
|
+
print(f"\n⚡ Interactive HTML Dashboard generated at: {out_path}")
|
|
42
|
+
try:
|
|
43
|
+
subprocess.run(["open", out_path])
|
|
44
|
+
print(" Opened in default browser.")
|
|
45
|
+
except Exception:
|
|
46
|
+
pass
|
|
47
|
+
|
|
48
|
+
elif command in ("benchmark", "bench"):
|
|
49
|
+
from src.benchmark import BenchmarkSuite
|
|
50
|
+
print("\n⚡ Running Zero-Loss SWE-bench Benchmark Suite across 5 real-world scenarios...")
|
|
51
|
+
data = BenchmarkSuite.run_all()
|
|
52
|
+
print(BenchmarkSuite.render_report(data))
|
|
53
|
+
out_md = os.path.join(PROJECT_ROOT, "data", "BENCHMARK_REPORT.md")
|
|
54
|
+
BenchmarkSuite.export_markdown_report(data, out_md)
|
|
55
|
+
print(f"📊 Publication-ready markdown report saved to: {out_md}\n")
|
|
56
|
+
|
|
57
|
+
elif command in ("setup", "install"):
|
|
58
|
+
from src.installer import AgentInstaller
|
|
59
|
+
AgentInstaller.run_interactive()
|
|
60
|
+
|
|
61
|
+
elif command == "proxy":
|
|
62
|
+
from src.proxy import run_proxy
|
|
63
|
+
port = int(args[1]) if len(args) > 1 else 8765
|
|
64
|
+
run_proxy(port)
|
|
65
|
+
|
|
66
|
+
elif command == "mcp":
|
|
67
|
+
from src.mcp_server import run_mcp_server
|
|
68
|
+
run_mcp_server()
|
|
69
|
+
|
|
70
|
+
elif command == "activate":
|
|
71
|
+
from src.license_manager import LicenseManager
|
|
72
|
+
if len(args) < 2:
|
|
73
|
+
print("Usage: agy-tokens activate <LICENSE_KEY>")
|
|
74
|
+
sys.exit(1)
|
|
75
|
+
key = args[1].strip()
|
|
76
|
+
ok, msg = LicenseManager.save_license(key)
|
|
77
|
+
if ok:
|
|
78
|
+
print(f"\n\033[32m\033[1m✔ {msg}\033[0m\n")
|
|
79
|
+
else:
|
|
80
|
+
print(f"\n\033[31m\033[1m✖ {msg}\033[0m\n")
|
|
81
|
+
sys.exit(1)
|
|
82
|
+
|
|
83
|
+
elif command in ("license", "tier"):
|
|
84
|
+
from src.license_manager import LicenseManager
|
|
85
|
+
tier, payload = LicenseManager.get_active_license()
|
|
86
|
+
print("\n\033[1m\033[36m⚡ TOKENSAFE LICENSE STATUS ⚡\033[0m\n")
|
|
87
|
+
print(f" Tier: \033[32m\033[1m{tier.upper()}\033[0m")
|
|
88
|
+
if tier != "free":
|
|
89
|
+
print(f" Email: {payload.get('email', 'N/A')}")
|
|
90
|
+
if payload.get("trial"):
|
|
91
|
+
print(f" Mode: \033[33m7-Day Free Trial\033[0m")
|
|
92
|
+
exp = payload.get("expires_at", 0)
|
|
93
|
+
if exp > 0:
|
|
94
|
+
import datetime
|
|
95
|
+
dt = datetime.datetime.fromtimestamp(exp).strftime('%Y-%m-%d %H:%M:%S')
|
|
96
|
+
print(f" Expires: {dt}")
|
|
97
|
+
else:
|
|
98
|
+
print(f" Expires: Never (Lifetime)")
|
|
99
|
+
else:
|
|
100
|
+
print(" Status: Free Tier (up to 100,000 tokens/mo)")
|
|
101
|
+
print(" Upgrade: https://tokensafe.dev ($9/mo Solo with 7-Day Free Trial)\n")
|
|
102
|
+
print()
|
|
103
|
+
|
|
104
|
+
elif command in ("providers", "models", "keys"):
|
|
105
|
+
from src.distiller import UniversalDistiller
|
|
106
|
+
configured = UniversalDistiller.get_configured_providers()
|
|
107
|
+
print("\n\033[1m\033[36m⚡ TOKENSAFE LLM DISTILLATION PROVIDERS ⚡\033[0m\n")
|
|
108
|
+
all_p = UniversalDistiller.PROVIDERS
|
|
109
|
+
for key, p in all_p.items():
|
|
110
|
+
if key in configured:
|
|
111
|
+
c = configured[key]
|
|
112
|
+
print(f" \033[32m✔\033[0m \033[1m{p['name']:<18}\033[0m [Active: {c['model']}]")
|
|
113
|
+
else:
|
|
114
|
+
print(f" \033[2m○ {p['name']:<18} [Set {p['env_key']} to activate]\033[0m")
|
|
115
|
+
print("\nSet environment variable or add to data/config.json to activate.\n")
|
|
116
|
+
|
|
117
|
+
elif command in ("markdown", "md", "export"):
|
|
118
|
+
print(dashboard.render_markdown())
|
|
119
|
+
|
|
120
|
+
elif command == "test":
|
|
121
|
+
if len(args) < 2:
|
|
122
|
+
print("Usage: agy-tokens test \"<prompt text>\"")
|
|
123
|
+
sys.exit(1)
|
|
124
|
+
raw_prompt = " ".join(args[1:])
|
|
125
|
+
res = PromptCompactor.compact(raw_prompt)
|
|
126
|
+
|
|
127
|
+
print("\n\033[1m\033[36m⚡ TOKEN COMPRESSION TEST ⚡\033[0m\n")
|
|
128
|
+
print(f"\033[1mOriginal Input:\033[0m {res['original_text']}")
|
|
129
|
+
print(f"\033[1mCompacted DSL:\033[0m \033[32m\033[1m{res['compact_text']}\033[0m\n")
|
|
130
|
+
print(f"\033[1mRouting Tier:\033[0m \033[33m{res.get('tier', 'REGEX')}\033[0m")
|
|
131
|
+
print(f"\033[1mExtracted Action:\033[0m {res['action']}")
|
|
132
|
+
print(f"\033[1mTarget File:\033[0m {res['target_file'] or 'None'}")
|
|
133
|
+
print(f"\033[1mScope Symbol:\033[0m {res['symbol'] or 'None'}")
|
|
134
|
+
print(f"\033[1mTraits / Tags:\033[0m {res['traits']}\n")
|
|
135
|
+
print(f"\033[1mOriginal Tokens:\033[0m {res['original_tokens']}")
|
|
136
|
+
print(f"\033[1mCompacted Tokens:\033[0m {res['compact_tokens']}")
|
|
137
|
+
print(f"\033[1mTokens Saved:\033[0m \033[32m\033[1m+{res['saved_tokens']} tokens\033[0m ({render_bar(res['reduction_pct'])})\n")
|
|
138
|
+
|
|
139
|
+
elif command == "reset":
|
|
140
|
+
confirm = input("Are you sure you want to reset all token metrics? [y/N]: ").strip().lower()
|
|
141
|
+
if confirm == "y":
|
|
142
|
+
tracker.reset()
|
|
143
|
+
print("Metrics database reset successfully.")
|
|
144
|
+
else:
|
|
145
|
+
print("Reset cancelled.")
|
|
146
|
+
|
|
147
|
+
else:
|
|
148
|
+
print(f"Unknown command: {command}")
|
|
149
|
+
print("Available commands: stats, html, benchmark, setup, proxy, mcp, test, markdown, reset")
|
|
150
|
+
sys.exit(1)
|
|
151
|
+
|
|
152
|
+
if __name__ == "__main__":
|
|
153
|
+
main()
|
package/bin/tokensafe.js
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* TokenSafe - Universal Node.js CLI & MCP Launcher
|
|
5
|
+
* Works with `npx tokensafe [mcp|proxy|stats|benchmark|install]`
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
const { spawn, spawnSync } = require('child_process');
|
|
9
|
+
const path = require('path');
|
|
10
|
+
const fs = require('fs');
|
|
11
|
+
|
|
12
|
+
const ROOT_DIR = path.resolve(__dirname, '..');
|
|
13
|
+
const args = process.argv.slice(2);
|
|
14
|
+
|
|
15
|
+
// Locate system Python 3 interpreter
|
|
16
|
+
function findPython() {
|
|
17
|
+
const candidates = ['python3', 'python'];
|
|
18
|
+
for (const cmd of candidates) {
|
|
19
|
+
try {
|
|
20
|
+
const res = spawnSync(cmd, ['-c', 'import sys; assert sys.version_info >= (3, 10)'], {
|
|
21
|
+
encoding: 'utf-8',
|
|
22
|
+
stdio: 'pipe'
|
|
23
|
+
});
|
|
24
|
+
if (res.status === 0) return cmd;
|
|
25
|
+
} catch (_) {}
|
|
26
|
+
}
|
|
27
|
+
return null;
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
const command = args[0] || 'install';
|
|
31
|
+
|
|
32
|
+
if (command === 'help' || command === '--help' || command === '-h') {
|
|
33
|
+
console.log(`
|
|
34
|
+
⚡ TokenSafe - Autonomous Zero-Loss Token Optimizer for AI Coding Agents
|
|
35
|
+
Website: https://tokensafe.dev
|
|
36
|
+
|
|
37
|
+
Usage:
|
|
38
|
+
npx tokensafe [command]
|
|
39
|
+
|
|
40
|
+
Commands:
|
|
41
|
+
npx tokensafe Interactive setup for Cursor, Antigravity, Claude & Codex
|
|
42
|
+
npx tokensafe mcp Start Universal MCP Server (for Claude Desktop / Cursor)
|
|
43
|
+
npx tokensafe proxy [port] Start Transparent Local AI Proxy (default: 8765)
|
|
44
|
+
npx tokensafe stats Show token economy savings & performance metrics
|
|
45
|
+
npx tokensafe benchmark Run zero-loss verification benchmark suite
|
|
46
|
+
npx tokensafe activate Activate commercial or team seat license key
|
|
47
|
+
npx tokensafe --help Show this help screen
|
|
48
|
+
`);
|
|
49
|
+
process.exit(0);
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
const python = findPython();
|
|
53
|
+
if (!python) {
|
|
54
|
+
console.error('\n❌ TokenSafe requires Python 3.10+ installed on your system.');
|
|
55
|
+
console.error('Please install Python from https://python.org or via your package manager:');
|
|
56
|
+
console.error(' macOS: brew install python3');
|
|
57
|
+
console.error(' Debian: sudo apt-get install python3\n');
|
|
58
|
+
process.exit(1);
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
let scriptPath = '';
|
|
62
|
+
let scriptArgs = [];
|
|
63
|
+
|
|
64
|
+
switch (command) {
|
|
65
|
+
case 'mcp':
|
|
66
|
+
scriptPath = path.join(ROOT_DIR, 'src', 'mcp_server.py');
|
|
67
|
+
scriptArgs = args.slice(1);
|
|
68
|
+
break;
|
|
69
|
+
|
|
70
|
+
case 'proxy':
|
|
71
|
+
scriptPath = path.join(ROOT_DIR, 'src', 'proxy.py');
|
|
72
|
+
scriptArgs = args.slice(1);
|
|
73
|
+
break;
|
|
74
|
+
|
|
75
|
+
case 'stats':
|
|
76
|
+
case 'dashboard':
|
|
77
|
+
scriptPath = path.join(ROOT_DIR, 'src', 'dashboard.py');
|
|
78
|
+
scriptArgs = ['--cli', ...args.slice(1)];
|
|
79
|
+
break;
|
|
80
|
+
|
|
81
|
+
case 'benchmark':
|
|
82
|
+
scriptPath = path.join(ROOT_DIR, 'src', 'benchmark.py');
|
|
83
|
+
scriptArgs = args.slice(1);
|
|
84
|
+
break;
|
|
85
|
+
|
|
86
|
+
case 'activate':
|
|
87
|
+
scriptPath = path.join(ROOT_DIR, 'src', 'license_manager.py');
|
|
88
|
+
scriptArgs = args.slice(1);
|
|
89
|
+
break;
|
|
90
|
+
|
|
91
|
+
case 'install':
|
|
92
|
+
case 'init':
|
|
93
|
+
scriptPath = path.join(ROOT_DIR, 'src', 'installer.py');
|
|
94
|
+
scriptArgs = args.slice(1);
|
|
95
|
+
break;
|
|
96
|
+
|
|
97
|
+
default:
|
|
98
|
+
scriptPath = path.join(ROOT_DIR, 'src', 'installer.py');
|
|
99
|
+
scriptArgs = args;
|
|
100
|
+
break;
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
if (!fs.existsSync(scriptPath)) {
|
|
104
|
+
console.error(`Error: TokenSafe script not found at ${scriptPath}`);
|
|
105
|
+
process.exit(1);
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
const child = spawn(python, [scriptPath, ...scriptArgs], {
|
|
109
|
+
stdio: 'inherit',
|
|
110
|
+
cwd: process.cwd(),
|
|
111
|
+
env: {
|
|
112
|
+
...process.env,
|
|
113
|
+
PYTHONPATH: ROOT_DIR + (process.env.PYTHONPATH ? path.delimiter + process.env.PYTHONPATH : '')
|
|
114
|
+
}
|
|
115
|
+
});
|
|
116
|
+
|
|
117
|
+
child.on('exit', (code, signal) => {
|
|
118
|
+
if (signal) {
|
|
119
|
+
process.kill(process.pid, signal);
|
|
120
|
+
} else {
|
|
121
|
+
process.exit(code || 0);
|
|
122
|
+
}
|
|
123
|
+
});
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
const vscode = require('vscode');
|
|
2
|
+
const { exec } = require('child_process');
|
|
3
|
+
const path = require('path');
|
|
4
|
+
const fs = require('fs');
|
|
5
|
+
|
|
6
|
+
function activate(context) {
|
|
7
|
+
const statusBarItem = vscode.window.createStatusBarItem(vscode.StatusBarAlignment.Right, 100);
|
|
8
|
+
statusBarItem.text = "$(zap) TokenSafe";
|
|
9
|
+
statusBarItem.tooltip = "TokenSafe: Active (Zero-Loss Token Economy)";
|
|
10
|
+
statusBarItem.command = "tokensafe.showStats";
|
|
11
|
+
statusBarItem.show();
|
|
12
|
+
context.subscriptions.push(statusBarItem);
|
|
13
|
+
|
|
14
|
+
const projectRoot = path.dirname(path.dirname(__dirname));
|
|
15
|
+
const cliPath = path.join(projectRoot, "bin", "agy-tokens");
|
|
16
|
+
const dashboardHtml = path.join(projectRoot, "data", "dashboard.html");
|
|
17
|
+
|
|
18
|
+
const disposableStats = vscode.commands.registerCommand('tokensafe.showStats', () => {
|
|
19
|
+
if (fs.existsSync(cliPath)) {
|
|
20
|
+
exec(`python3 "${cliPath}" stats`, (error, stdout) => {
|
|
21
|
+
if (stdout) {
|
|
22
|
+
vscode.window.showInformationMessage(`TokenSafe Savings:\n${stdout.split('\n').slice(0, 5).join('\n')}`);
|
|
23
|
+
} else {
|
|
24
|
+
vscode.window.showInformationMessage("TokenSafe is actively optimizing context tokens.");
|
|
25
|
+
}
|
|
26
|
+
});
|
|
27
|
+
} else {
|
|
28
|
+
vscode.window.showInformationMessage("TokenSafe is active and protecting context tokens.");
|
|
29
|
+
}
|
|
30
|
+
});
|
|
31
|
+
|
|
32
|
+
const disposableDash = vscode.commands.registerCommand('tokensafe.openDashboard', () => {
|
|
33
|
+
if (fs.existsSync(dashboardHtml)) {
|
|
34
|
+
vscode.env.openExternal(vscode.Uri.file(dashboardHtml));
|
|
35
|
+
} else {
|
|
36
|
+
vscode.window.showInformationMessage("Generating TokenSafe dashboard...");
|
|
37
|
+
if (fs.existsSync(cliPath)) {
|
|
38
|
+
exec(`python3 "${cliPath}" html`, () => {
|
|
39
|
+
if (fs.existsSync(dashboardHtml)) {
|
|
40
|
+
vscode.env.openExternal(vscode.Uri.file(dashboardHtml));
|
|
41
|
+
}
|
|
42
|
+
});
|
|
43
|
+
}
|
|
44
|
+
}
|
|
45
|
+
});
|
|
46
|
+
|
|
47
|
+
context.subscriptions.push(disposableStats, disposableDash);
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
function deactivate() {}
|
|
51
|
+
|
|
52
|
+
module.exports = {
|
|
53
|
+
activate,
|
|
54
|
+
deactivate
|
|
55
|
+
};
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "tokensafe",
|
|
3
|
+
"displayName": "TokenSafe",
|
|
4
|
+
"description": "Zero-Loss Token Economy & Context Optimizer for AI Agents",
|
|
5
|
+
"version": "1.0.0",
|
|
6
|
+
"publisher": "tokensafe",
|
|
7
|
+
"engines": {
|
|
8
|
+
"vscode": "^1.80.0"
|
|
9
|
+
},
|
|
10
|
+
"categories": [
|
|
11
|
+
"Other",
|
|
12
|
+
"Programming Languages"
|
|
13
|
+
],
|
|
14
|
+
"main": "./extension.js",
|
|
15
|
+
"activationEvents": [
|
|
16
|
+
"onStartupFinished"
|
|
17
|
+
],
|
|
18
|
+
"contributes": {
|
|
19
|
+
"commands": [
|
|
20
|
+
{
|
|
21
|
+
"command": "tokensafe.showStats",
|
|
22
|
+
"title": "TokenSafe: Show Token Savings"
|
|
23
|
+
},
|
|
24
|
+
{
|
|
25
|
+
"command": "tokensafe.openDashboard",
|
|
26
|
+
"title": "TokenSafe: Open HTML Analytics Dashboard"
|
|
27
|
+
}
|
|
28
|
+
]
|
|
29
|
+
}
|
|
30
|
+
}
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""
|
|
3
|
+
Antigravity Lifecycle Hook: PostToolUse
|
|
4
|
+
Fires after tool execution.
|
|
5
|
+
Fulfills Antigravity PostToolUse contract by returning `{}`.
|
|
6
|
+
"""
|
|
7
|
+
|
|
8
|
+
import sys
|
|
9
|
+
import json
|
|
10
|
+
|
|
11
|
+
def main():
|
|
12
|
+
try:
|
|
13
|
+
# Drain stdin to keep pipes clean
|
|
14
|
+
_ = sys.stdin.read()
|
|
15
|
+
except Exception:
|
|
16
|
+
pass
|
|
17
|
+
|
|
18
|
+
# PostToolUse contract expects an empty JSON object {}
|
|
19
|
+
print(json.dumps({}))
|
|
20
|
+
|
|
21
|
+
if __name__ == "__main__":
|
|
22
|
+
main()
|