tokensafe 0.0.0-stage → 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Oleg Savelev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,179 @@
-# Temporary Holding Version
+# ⚡ TokenSafe (`tokensafe`)
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[![License: MIT](https://img.shields.io/badge/License-MIT-emerald.svg)](https://opensource.org/licenses/MIT)
+[![SWE-bench: Verified](https://img.shields.io/badge/SWE--bench-61.8%25_to_92.1%25_Savings-06B6D4.svg)](https://tokensafe.dev#benchmarks)
+[![Python: 3.10+](https://img.shields.io/badge/Python-3.10+-3B82F6.svg)](https://python.org)
+[![Trial: 7 Days Free](https://img.shields.io/badge/Trial-7_Days_Free-10B981.svg)](https://tokensafe.dev#pricing)
+
+> **Autonomous Zero-Loss Token Economy & Context Optimizer for AI Coding Agents.**  
+> Slashes LLM token costs by **60%–90%** with zero quality loss. Built for **Google Antigravity, OpenAI Codex, Cursor, Claude Desktop, Z.ai Code, Windsurf & Cline**.
+
+🌐 **Official Website:** [https://tokensafe.dev](https://tokensafe.dev)
+
+---
+
+## ⚡ 1-Click Universal Install
+
+Install and connect TokenSafe to all detected AI coding agents on your machine in 5 seconds:
+
+```bash
+curl -fsSL https://tokensafe.dev/install.sh | bash
+```
+
+Or clone and run the interactive setup manually:
+
+```bash
+git clone https://github.com/NekoSin4eG/tokensafe.git ~/.tokensafe
+python3 ~/.tokensafe/src/installer.py
+```
+
+---
+
+## 🚀 The Breakthrough: How It Saves 60%–90% Tokens
+
+Traditional AI code interactions suffer from massive context bloat:
+1. **The BPE Token Penalty:** Verbose natural language prompts and filler waste 40%–70% of ingress context.
+2. **Bulk Tool Dumps:** Reading whole 800+ line files burns 5,000+ tokens per turn when only 20 lines are needed.
+3. **Stuck Loops & Bloated Diffs:** Terminal `git diff` and repetitive failed commands spiral context into overflow.
+
+### TokenSafe Solves All Three Autonomously:
+- **TOCP Prompt Distillation:** Intercepts prompts and distills intent into lean S-Expressions via **Google Gemini (2.0 Flash), OpenAI ChatGPT (gpt-4o-mini), or DeepSeek-V3** (`-40% to -70%`). Automatically detects `GEMINI_API_KEY`, `OPENAI_API_KEY`, or `DEEPSEEK_API_KEY`.
+- **AST Skeletonization & Slicing:** Folds inactive function bodies into signatures and windows targeted symbols (`-80% to -92%`).
+- **Predictive Dependency Prefetcher:** Attaches imported symbol signatures to eliminate downstream exploratory `read_file` roundtrips.
+- **Output Guard & Loop Guard:** Blocks destructive full-file rewrites on files $\ge 40$ lines and halts command loops.
+
+---
+
+## 🤖 Universal LLM Distillation Providers
+
+TokenSafe supports any key you already have:
+
+| Provider | Supported Model | Environment Variable | Config Key |
+|---|---|---|---|
+| **Google Gemini** | `gemini-2.0-flash`, `gemini-1.5-flash` | `GEMINI_API_KEY` | `gemini_api_key` |
+| **OpenAI ChatGPT** | `gpt-4o-mini`, `gpt-4o` | `OPENAI_API_KEY` | `openai_api_key` |
+| **DeepSeek** | `deepseek-chat` (DeepSeek-V3) | `DEEPSEEK_API_KEY` | `deepseek_api_key` |
+
+Check active providers anytime with:
+```bash
+tokensafe providers
+```
+
+---
+
+## 📊 Zero-Loss SWE-bench Benchmark Results
+
+Verified across 5 real-world production scenarios:
+
+| Benchmark Scenario | Task Complexity | Baseline Tokens | TokenSafe Tokens | Reduction | Turn Speedup | Accuracy |
+|---|---|---|---|---|---|---|
+| **Django ORM N+1 Fix** | Multi-file query refactoring | 14,280 | **1,120** | **-92.1%** | +12% faster | 100% Pass |
+| **FastAPI Async Race** | Concurrency bug investigation | 8,640 | **2,940** | **-65.9%** | +10% faster | 100% Pass |
+| **TypeScript Redux Vault** | Full AST symbol refactoring | 18,400 | **2,130** | **-88.4%** | +9% faster | 100% Pass |
+| **Stuck Command Loop** | Subprocess infinite loop | 22,100 | **2,400** | **-89.1%** | Stopped loop | 100% Pass |
+| **Heavy Rewrite Guard** | Whole-file write attempt | 11,300 | **4,310** | **-61.8%** | Forced edit_file | 100% Pass |
+
+---
+
+## 💎 Pricing & 7-Day Free Trial
+
+| Tier | Price | Highlights |
+|---|---|---|
+| **Hacker (Open-Core)** | **$0** (Free Forever) | 100k tokens/mo, Local Regex compactor, use own DeepSeek key |
+| **Solo Developer** | **$9 / month** | **7-Day Free Trial**, Unlimited Managed DeepSeek-V3, AST Prefetcher, Output Guards |
+| **Engineering Team** | **$29 / dev / mo** | **7-Day Free Trial**, Team Budget Caps, Centralized Ledger, B2B Docker Proxy |
+| **Enterprise** | Custom | On-Premise Nuitka C-compiled Docker Proxy, Custom Compliance, SAML SSO |
+
+👉 **[Start your 7-Day Free Trial on our website](https://tokensafe.dev#pricing)**
+
+---
+
+## 🛠️ CLI Quickstart
+
+```bash
+# View live savings dashboard in your terminal
+tokensafe stats
+
+# Open interactive Apple dark-mode HTML dashboard
+tokensafe html
+
+# Run the 5-scenario benchmark suite
+tokensafe benchmark
+
+# Run the local transparent AI proxy (127.0.0.1:8765/v1)
+tokensafe proxy
+
+# Activate your Pro License key
+tokensafe activate TS1.eyJlbWFpbCI6...
+```
+
+---
+
+## 🧪 Unit Testing
+
+```bash
+python3 -m unittest discover tests
+```
+*Zero external runtime dependencies — runs on standard Python 3.10+.*
+
+---
+
+## 📄 License
+MIT License. Commercial enterprise licenses available for on-premise B2B deployments.
+
+### Export Markdown Report
+```bash
+./bin/agy-tokens markdown
+```
+
+---
+
+## 📂 Project Structure
+
+```text
+agy-token-optimizer/
+├── plugin.json                 # Antigravity plugin manifest
+├── hooks.json                  # Lifecycle hook declarations (PreInvocation, PreToolUse, PostToolUse)
+├── rules/
+│   └── AGENTS.md               # TOCP Protocol directives for LLM (prompt-cached)
+├── skills/
+│   └── token-optimizer/
+│       └── SKILL.md            # On-demand Antigravity skill
+├── proto/
+│   └── tocp.proto              # Protocol Buffers contract
+├── src/
+│   ├── tokenizer.py            # Zero-dependency calibrated BPE token counter
+│   ├── compactor.py            # Semantic compactor (Natural Language -> TOCP S-Expr)
+│   ├── slicer.py               # AST & Line windowing tool optimizer
+│   ├── metrics.py              # Persistent SQLite metrics engine
+│   └── dashboard.py            # Visual ANSI terminal & Markdown generator
+├── hooks/
+│   ├── pre_invocation.py       # PreInvocation hook entry point
+│   ├── pre_tool_use.py         # PreToolUse hook entry point
+│   └── post_tool_use.py        # PostToolUse hook entry point
+├── bin/
+│   └── agy-tokens              # Executable CLI
+├── tests/                      # Unit test suite (100% passing)
+├── install.sh                  # One-click installer for Antigravity
+└── README.md
+```
+
+---
+
+## 🛠️ Installation & Antigravity Setup
+
+Run the installer:
+```bash
+./install.sh
+```
+This symlinks the plugin directly into `~/.gemini/config/plugins/agy-token-optimizer`.  
+Antigravity automatically loads it on startup.
+
+---
+
+## 🧪 Running Unit Tests
+
+```bash
+python3 -m unittest discover tests
+```
+No external packages required — runs on standard Python 3.10+.

package/bin/agy-tokens

@@ -0,0 +1,153 @@
+#!/usr/bin/env python3
+"""
+CLI entry point for Antigravity Token Optimizer.
+Usage:
+  agy-tokens [stats]           Show visual metrics dashboard
+  agy-tokens html              Generate and open interactive browser dashboard
+  agy-tokens test "<prompt>"   Test prompt compression and view savings
+  agy-tokens markdown          Output GitHub Markdown report
+  agy-tokens reset             Reset metrics database
+"""
+
+import sys
+import os
+import subprocess
+
+SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_ROOT = os.path.dirname(SCRIPT_DIR)
+sys.path.insert(0, PROJECT_ROOT)
+
+from src.metrics import MetricsTracker
+from src.dashboard import Dashboard, render_bar
+from src.compactor import PromptCompactor
+from src.tokenizer import count_tokens
+
+def main():
+    tracker = MetricsTracker()
+    dashboard = Dashboard(tracker)
+
+    args = sys.argv[1:]
+    command = args[0] if args else "stats"
+
+    if command in ("stats", "dashboard"):
+        print(dashboard.render_terminal())
+
+    elif command in ("html", "web"):
+        html_content = dashboard.render_html()
+        out_path = os.path.join(PROJECT_ROOT, "data", "dashboard.html")
+        os.makedirs(os.path.dirname(out_path), exist_ok=True)
+        with open(out_path, "w", encoding="utf-8") as f:
+            f.write(html_content)
+        print(f"\n⚡ Interactive HTML Dashboard generated at: {out_path}")
+        try:
+            subprocess.run(["open", out_path])
+            print("   Opened in default browser.")
+        except Exception:
+            pass
+
+    elif command in ("benchmark", "bench"):
+        from src.benchmark import BenchmarkSuite
+        print("\n⚡ Running Zero-Loss SWE-bench Benchmark Suite across 5 real-world scenarios...")
+        data = BenchmarkSuite.run_all()
+        print(BenchmarkSuite.render_report(data))
+        out_md = os.path.join(PROJECT_ROOT, "data", "BENCHMARK_REPORT.md")
+        BenchmarkSuite.export_markdown_report(data, out_md)
+        print(f"📊 Publication-ready markdown report saved to: {out_md}\n")
+
+    elif command in ("setup", "install"):
+        from src.installer import AgentInstaller
+        AgentInstaller.run_interactive()
+
+    elif command == "proxy":
+        from src.proxy import run_proxy
+        port = int(args[1]) if len(args) > 1 else 8765
+        run_proxy(port)
+
+    elif command == "mcp":
+        from src.mcp_server import run_mcp_server
+        run_mcp_server()
+
+    elif command == "activate":
+        from src.license_manager import LicenseManager
+        if len(args) < 2:
+            print("Usage: agy-tokens activate <LICENSE_KEY>")
+            sys.exit(1)
+        key = args[1].strip()
+        ok, msg = LicenseManager.save_license(key)
+        if ok:
+            print(f"\n\033[32m\033[1m✔ {msg}\033[0m\n")
+        else:
+            print(f"\n\033[31m\033[1m✖ {msg}\033[0m\n")
+            sys.exit(1)
+
+    elif command in ("license", "tier"):
+        from src.license_manager import LicenseManager
+        tier, payload = LicenseManager.get_active_license()
+        print("\n\033[1m\033[36m⚡ TOKENSAFE LICENSE STATUS ⚡\033[0m\n")
+        print(f"  Tier:       \033[32m\033[1m{tier.upper()}\033[0m")
+        if tier != "free":
+            print(f"  Email:      {payload.get('email', 'N/A')}")
+            if payload.get("trial"):
+                print(f"  Mode:       \033[33m7-Day Free Trial\033[0m")
+            exp = payload.get("expires_at", 0)
+            if exp > 0:
+                import datetime
+                dt = datetime.datetime.fromtimestamp(exp).strftime('%Y-%m-%d %H:%M:%S')
+                print(f"  Expires:    {dt}")
+            else:
+                print(f"  Expires:    Never (Lifetime)")
+        else:
+            print("  Status:     Free Tier (up to 100,000 tokens/mo)")
+            print("  Upgrade:    https://tokensafe.dev ($9/mo Solo with 7-Day Free Trial)\n")
+        print()
+
+    elif command in ("providers", "models", "keys"):
+        from src.distiller import UniversalDistiller
+        configured = UniversalDistiller.get_configured_providers()
+        print("\n\033[1m\033[36m⚡ TOKENSAFE LLM DISTILLATION PROVIDERS ⚡\033[0m\n")
+        all_p = UniversalDistiller.PROVIDERS
+        for key, p in all_p.items():
+            if key in configured:
+                c = configured[key]
+                print(f"  \033[32m✔\033[0m \033[1m{p['name']:<18}\033[0m [Active: {c['model']}]")
+            else:
+                print(f"  \033[2m○ {p['name']:<18} [Set {p['env_key']} to activate]\033[0m")
+        print("\nSet environment variable or add to data/config.json to activate.\n")
+
+    elif command in ("markdown", "md", "export"):
+        print(dashboard.render_markdown())
+
+    elif command == "test":
+        if len(args) < 2:
+            print("Usage: agy-tokens test \"<prompt text>\"")
+            sys.exit(1)
+        raw_prompt = " ".join(args[1:])
+        res = PromptCompactor.compact(raw_prompt)
+
+        print("\n\033[1m\033[36m⚡ TOKEN COMPRESSION TEST ⚡\033[0m\n")
+        print(f"\033[1mOriginal Input:\033[0m  {res['original_text']}")
+        print(f"\033[1mCompacted DSL:\033[0m   \033[32m\033[1m{res['compact_text']}\033[0m\n")
+        print(f"\033[1mRouting Tier:\033[0m   \033[33m{res.get('tier', 'REGEX')}\033[0m")
+        print(f"\033[1mExtracted Action:\033[0m {res['action']}")
+        print(f"\033[1mTarget File:\033[0m      {res['target_file'] or 'None'}")
+        print(f"\033[1mScope Symbol:\033[0m     {res['symbol'] or 'None'}")
+        print(f"\033[1mTraits / Tags:\033[0m    {res['traits']}\n")
+        print(f"\033[1mOriginal Tokens:\033[0m  {res['original_tokens']}")
+        print(f"\033[1mCompacted Tokens:\033[0m {res['compact_tokens']}")
+        print(f"\033[1mTokens Saved:\033[0m     \033[32m\033[1m+{res['saved_tokens']} tokens\033[0m ({render_bar(res['reduction_pct'])})\n")
+
+    elif command == "reset":
+        confirm = input("Are you sure you want to reset all token metrics? [y/N]: ").strip().lower()
+        if confirm == "y":
+            tracker.reset()
+            print("Metrics database reset successfully.")
+        else:
+            print("Reset cancelled.")
+
+    else:
+        print(f"Unknown command: {command}")
+        print("Available commands: stats, html, benchmark, setup, proxy, mcp, test, markdown, reset")
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()

package/bin/tokensafe.js

@@ -0,0 +1,123 @@
+#!/usr/bin/env node
+
+/**
+ * TokenSafe - Universal Node.js CLI & MCP Launcher
+ * Works with `npx tokensafe [mcp|proxy|stats|benchmark|install]`
+ */
+
+const { spawn, spawnSync } = require('child_process');
+const path = require('path');
+const fs = require('fs');
+
+const ROOT_DIR = path.resolve(__dirname, '..');
+const args = process.argv.slice(2);
+
+// Locate system Python 3 interpreter
+function findPython() {
+  const candidates = ['python3', 'python'];
+  for (const cmd of candidates) {
+    try {
+      const res = spawnSync(cmd, ['-c', 'import sys; assert sys.version_info >= (3, 10)'], {
+        encoding: 'utf-8',
+        stdio: 'pipe'
+      });
+      if (res.status === 0) return cmd;
+    } catch (_) {}
+  }
+  return null;
+}
+
+const command = args[0] || 'install';
+
+if (command === 'help' || command === '--help' || command === '-h') {
+  console.log(`
+⚡ TokenSafe - Autonomous Zero-Loss Token Optimizer for AI Coding Agents
+Website: https://tokensafe.dev
+
+Usage:
+  npx tokensafe [command]
+
+Commands:
+  npx tokensafe              Interactive setup for Cursor, Antigravity, Claude & Codex
+  npx tokensafe mcp          Start Universal MCP Server (for Claude Desktop / Cursor)
+  npx tokensafe proxy [port] Start Transparent Local AI Proxy (default: 8765)
+  npx tokensafe stats        Show token economy savings & performance metrics
+  npx tokensafe benchmark    Run zero-loss verification benchmark suite
+  npx tokensafe activate     Activate commercial or team seat license key
+  npx tokensafe --help       Show this help screen
+`);
+  process.exit(0);
+}
+
+const python = findPython();
+if (!python) {
+  console.error('\n❌ TokenSafe requires Python 3.10+ installed on your system.');
+  console.error('Please install Python from https://python.org or via your package manager:');
+  console.error('  macOS:  brew install python3');
+  console.error('  Debian: sudo apt-get install python3\n');
+  process.exit(1);
+}
+
+let scriptPath = '';
+let scriptArgs = [];
+
+switch (command) {
+  case 'mcp':
+    scriptPath = path.join(ROOT_DIR, 'src', 'mcp_server.py');
+    scriptArgs = args.slice(1);
+    break;
+
+  case 'proxy':
+    scriptPath = path.join(ROOT_DIR, 'src', 'proxy.py');
+    scriptArgs = args.slice(1);
+    break;
+
+  case 'stats':
+  case 'dashboard':
+    scriptPath = path.join(ROOT_DIR, 'src', 'dashboard.py');
+    scriptArgs = ['--cli', ...args.slice(1)];
+    break;
+
+  case 'benchmark':
+    scriptPath = path.join(ROOT_DIR, 'src', 'benchmark.py');
+    scriptArgs = args.slice(1);
+    break;
+
+  case 'activate':
+    scriptPath = path.join(ROOT_DIR, 'src', 'license_manager.py');
+    scriptArgs = args.slice(1);
+    break;
+
+  case 'install':
+  case 'init':
+    scriptPath = path.join(ROOT_DIR, 'src', 'installer.py');
+    scriptArgs = args.slice(1);
+    break;
+
+  default:
+    scriptPath = path.join(ROOT_DIR, 'src', 'installer.py');
+    scriptArgs = args;
+    break;
+}
+
+if (!fs.existsSync(scriptPath)) {
+  console.error(`Error: TokenSafe script not found at ${scriptPath}`);
+  process.exit(1);
+}
+
+const child = spawn(python, [scriptPath, ...scriptArgs], {
+  stdio: 'inherit',
+  cwd: process.cwd(),
+  env: {
+    ...process.env,
+    PYTHONPATH: ROOT_DIR + (process.env.PYTHONPATH ? path.delimiter + process.env.PYTHONPATH : '')
+  }
+});
+
+child.on('exit', (code, signal) => {
+  if (signal) {
+    process.kill(process.pid, signal);
+  } else {
+    process.exit(code || 0);
+  }
+});

package/extensions/cursor-tokensafe/extension.js

@@ -0,0 +1,55 @@
+const vscode = require('vscode');
+const { exec } = require('child_process');
+const path = require('path');
+const fs = require('fs');
+
+function activate(context) {
+    const statusBarItem = vscode.window.createStatusBarItem(vscode.StatusBarAlignment.Right, 100);
+    statusBarItem.text = "$(zap) TokenSafe";
+    statusBarItem.tooltip = "TokenSafe: Active (Zero-Loss Token Economy)";
+    statusBarItem.command = "tokensafe.showStats";
+    statusBarItem.show();
+    context.subscriptions.push(statusBarItem);
+
+    const projectRoot = path.dirname(path.dirname(__dirname));
+    const cliPath = path.join(projectRoot, "bin", "agy-tokens");
+    const dashboardHtml = path.join(projectRoot, "data", "dashboard.html");
+
+    const disposableStats = vscode.commands.registerCommand('tokensafe.showStats', () => {
+        if (fs.existsSync(cliPath)) {
+            exec(`python3 "${cliPath}" stats`, (error, stdout) => {
+                if (stdout) {
+                    vscode.window.showInformationMessage(`TokenSafe Savings:\n${stdout.split('\n').slice(0, 5).join('\n')}`);
+                } else {
+                    vscode.window.showInformationMessage("TokenSafe is actively optimizing context tokens.");
+                }
+            });
+        } else {
+            vscode.window.showInformationMessage("TokenSafe is active and protecting context tokens.");
+        }
+    });
+
+    const disposableDash = vscode.commands.registerCommand('tokensafe.openDashboard', () => {
+        if (fs.existsSync(dashboardHtml)) {
+            vscode.env.openExternal(vscode.Uri.file(dashboardHtml));
+        } else {
+            vscode.window.showInformationMessage("Generating TokenSafe dashboard...");
+            if (fs.existsSync(cliPath)) {
+                exec(`python3 "${cliPath}" html`, () => {
+                    if (fs.existsSync(dashboardHtml)) {
+                        vscode.env.openExternal(vscode.Uri.file(dashboardHtml));
+                    }
+                });
+            }
+        }
+    });
+
+    context.subscriptions.push(disposableStats, disposableDash);
+}
+
+function deactivate() {}
+
+module.exports = {
+    activate,
+    deactivate
+};

package/extensions/cursor-tokensafe/package.json

@@ -0,0 +1,30 @@
+{
+  "name": "tokensafe",
+  "displayName": "TokenSafe",
+  "description": "Zero-Loss Token Economy & Context Optimizer for AI Agents",
+  "version": "1.0.0",
+  "publisher": "tokensafe",
+  "engines": {
+    "vscode": "^1.80.0"
+  },
+  "categories": [
+    "Other",
+    "Programming Languages"
+  ],
+  "main": "./extension.js",
+  "activationEvents": [
+    "onStartupFinished"
+  ],
+  "contributes": {
+    "commands": [
+      {
+        "command": "tokensafe.showStats",
+        "title": "TokenSafe: Show Token Savings"
+      },
+      {
+        "command": "tokensafe.openDashboard",
+        "title": "TokenSafe: Open HTML Analytics Dashboard"
+      }
+    ]
+  }
+}

package/hooks/post_tool_use.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+"""
+Antigravity Lifecycle Hook: PostToolUse
+Fires after tool execution.
+Fulfills Antigravity PostToolUse contract by returning `{}`.
+"""
+
+import sys
+import json
+
+def main():
+    try:
+        # Drain stdin to keep pipes clean
+        _ = sys.stdin.read()
+    except Exception:
+        pass
+
+    # PostToolUse contract expects an empty JSON object {}
+    print(json.dumps({}))
+
+if __name__ == "__main__":
+    main()