greedy-token 0.2.1__tar.gz

greedy_token-0.2.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Stanislav Vasenkov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

greedy_token-0.2.1/PKG-INFO

@@ -0,0 +1,151 @@
+Metadata-Version: 2.4
+Name: greedy-token
+Version: 0.2.1
+Summary: Route dev tasks through tool → python → ollama → RAG before escalating to Cursor
+Author: Stanislav Vasenkov
+License: MIT
+Project-URL: Homepage, https://github.com/svasenkov/greedy-token
+Project-URL: Repository, https://github.com/svasenkov/greedy-token
+Keywords: cli,llm,cursor,token-optimization,rag,ollama
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: tiktoken>=0.7
+Dynamic: license-file
+
+# greedy-token
+
+Route dev tasks through **tool → python → ollama → RAG** before escalating to Cursor/Claude.
+
+```
+Your task  →  greedy-token  →  rg/jq | scripts | Ollama | docs/rag | Cursor
+```
+
+## Cursor vs greedy-token (token comparison)
+
+Measured with `greedy-token audit-context` and `greedy-token estimate` against [zero-design-system](https://github.com/svasenkov/zero-design-system) (tiktoken `cl100k_base`; order of magnitude, not API billing).
+
+### Cursor context overhead (every new chat)
+
+| Context | Tokens | Charged when |
+|---------|-------:|--------------|
+| Always-on rules (`.cursor/rules/*.mdc`) | 2,524 | every chat |
+| Skills on disk (`.cursor/skills/*/SKILL.md`) | 26,386 | if agent loads skill |
+| Sampled docs (`CONTEXT.md`, `migration-prompts.md`) | 3,349 | if referenced |
+| **Sampled set total** | **32,259** | full agent context |
+| Naive agent baseline (`rules + 6k overhead + task`) | ~8,530 | default Cursor path |
+
+Rules ≥ 1024 tokens → stable prefix is cache-friendly for Claude API prompt caching.
+
+### Task routing: naive Cursor vs greedy-token
+
+| Task | Naive Cursor | greedy-token route | Est. tokens | Saved vs Cursor | Savings | Command |
+|------|-------------:|--------------------|------------:|----------------:|--------:|---------|
+| `какой -D flag для baseUrl в e2e config` | ~8,534 | **rag** (95%) | 1,810 | ~6,724 | ~79% | `greedy-token rag "baseUrl -D flag"` |
+| `ADR 002 baseUrl pattern` | ~8,530 | **rag** (61%) | 1,806 | ~6,724 | ~79% | `greedy-token rag "ADR 002 baseUrl"` |
+| `find baseUrl in e2e properties` | ~8,532 | **tool** (59%) | 0 | ~8,532 | ~100% | `greedy-token run "…" --execute` → `rg` |
+| `sync phase-manifest и skills-map` | ~8,532 | **python** (65%) | 0 | ~8,532 | ~100% | `greedy-token scripts --run check-meta-sync --execute` |
+| `rsync template-project в monorepo` | ~8,533 | **python** (60%) | 0 | ~8,533 | ~100% | dry-run script; run manually |
+| `batch inventory template-project` | ~8,532 | **ollama** (66%) | 0 cloud | ~8,532 | ~100% cloud | `scripts/ollama/batch-inventory.sh` (local LLM) |
+| `refactor header layout and wire nav links` | ~8,535 | **cursor** (82%) | 8,535 | 0 | — | new Cursor chat + skill from `docs/skills-map.md` |
+
+**Tier order:** `tool (rg/jq) → python → ollama → rag → cursor` — first match wins. Ollama tier is skipped when unavailable.
+
+**Takeaway:** lookup / search / sync / bulk tasks save **~6.7k–8.5k tokens per request**; wiring and architecture correctly stay on Cursor.
+
+```bash
+greedy-token audit-context                    # your workspace overhead
+greedy-token estimate "your task here"        # route + savings before opening a chat
+```
+
+Install once, point at your workspace root, route every task through the cheapest tier that can handle it.
+
+## Install
+
+```bash
+pip install greedy-token
+# or editable: pip install -e .
+# or from git: pip install git+https://github.com/svasenkov/greedy-token.git
+```
+
+`tiktoken` (exact BPE counts via `cl100k_base`) is a required dependency. If install fails
+on an unsupported platform, use a Python version with a prebuilt `tiktoken` wheel or install
+from source with a Rust toolchain.
+
+### PyPI publish (maintainer)
+
+1. Create project **greedy-token** on [pypi.org](https://pypi.org/manage/projects/)
+2. Add trusted publisher: Owner `svasenkov`, repo `greedy-token`, workflow `publish.yml`
+3. Publish: GitHub → Releases → re-run workflow or new tag
+
+## Workspace root
+
+`greedy-token` runs against a project directory (monorepo, app repo, etc.):
+
+```bash
+export GREEDY_TOKEN_ROOT=/path/to/your-workspace
+```
+
+Auto-detect works when the workspace has `docs/phase-manifest.json` and `scripts/check-meta-sync.sh` (e.g. [zero-design-system](https://github.com/svasenkov/zero-design-system)). Otherwise set `GREEDY_TOKEN_ROOT` explicitly.
+
+## Commands
+
+| Command | Purpose |
+|---------|---------|
+| `greedy-token route "…"` | Recommend: tool \| python \| ollama \| rag \| cursor + scoring |
+| `greedy-token estimate "…"` | Token-aware estimate: complexity, est_tokens, tier scan |
+| `greedy-token run "…" [--execute]` | Route + dry-run / **read-only** execute |
+| `greedy-token scripts --list` | List workspace script wrappers |
+| `greedy-token scripts --run ID [--execute]` | Dry-run / execute read-only wrapper |
+| `greedy-token audit-context` | Size of always-on rules/skills (tokens) |
+| `greedy-token tokens PATH…` | Count tokens in files/directories |
+| `greedy-token rag QUERY` | Search chunks in `docs/rag/` |
+| `greedy-token compress` | Short prompt version (stdin; `--ollama` for LLM) |
+
+## Tier order
+
+```
+tool (rg/jq) → python → ollama → rag → cursor
+```
+
+First matching tier wins. Ollama is **optional** — if unavailable, the tier is skipped.
+
+## Examples
+
+```bash
+greedy-token route "find baseUrl"
+# → tool (rg)
+
+greedy-token estimate "refactor header layout"
+# → cursor, complexity=high
+
+greedy-token route "batch inventory template-project"
+# → ollama
+
+greedy-token route "sync phase-manifest and skills-map"
+# → python
+
+greedy-token route "ADR 002 baseUrl pattern"
+# → rag
+```
+
+## Environment
+
+| Var | Default |
+|-----|---------|
+| `GREEDY_TOKEN_ROOT` | auto-detect or required |
+| `OLLAMA_URL` | `http://localhost:11434` |
+| `OLLAMA_MODEL` | `qwen2.5-coder:14b` |
+
+## `--execute`
+
+Read-only only: `rg`, `jq`, `check-meta-sync.sh`. Rsync/migrate/ollama — dry-run; run manually.
+
+## Route config
+
+`src/greedy_token/config/routes.yaml` — customize patterns and commands for your workspace.
+
+## License
+
+MIT

greedy_token-0.2.1/README.md

@@ -0,0 +1,135 @@
+# greedy-token
+
+Route dev tasks through **tool → python → ollama → RAG** before escalating to Cursor/Claude.
+
+```
+Your task  →  greedy-token  →  rg/jq | scripts | Ollama | docs/rag | Cursor
+```
+
+## Cursor vs greedy-token (token comparison)
+
+Measured with `greedy-token audit-context` and `greedy-token estimate` against [zero-design-system](https://github.com/svasenkov/zero-design-system) (tiktoken `cl100k_base`; order of magnitude, not API billing).
+
+### Cursor context overhead (every new chat)
+
+| Context | Tokens | Charged when |
+|---------|-------:|--------------|
+| Always-on rules (`.cursor/rules/*.mdc`) | 2,524 | every chat |
+| Skills on disk (`.cursor/skills/*/SKILL.md`) | 26,386 | if agent loads skill |
+| Sampled docs (`CONTEXT.md`, `migration-prompts.md`) | 3,349 | if referenced |
+| **Sampled set total** | **32,259** | full agent context |
+| Naive agent baseline (`rules + 6k overhead + task`) | ~8,530 | default Cursor path |
+
+Rules ≥ 1024 tokens → stable prefix is cache-friendly for Claude API prompt caching.
+
+### Task routing: naive Cursor vs greedy-token
+
+| Task | Naive Cursor | greedy-token route | Est. tokens | Saved vs Cursor | Savings | Command |
+|------|-------------:|--------------------|------------:|----------------:|--------:|---------|
+| `какой -D flag для baseUrl в e2e config` | ~8,534 | **rag** (95%) | 1,810 | ~6,724 | ~79% | `greedy-token rag "baseUrl -D flag"` |
+| `ADR 002 baseUrl pattern` | ~8,530 | **rag** (61%) | 1,806 | ~6,724 | ~79% | `greedy-token rag "ADR 002 baseUrl"` |
+| `find baseUrl in e2e properties` | ~8,532 | **tool** (59%) | 0 | ~8,532 | ~100% | `greedy-token run "…" --execute` → `rg` |
+| `sync phase-manifest и skills-map` | ~8,532 | **python** (65%) | 0 | ~8,532 | ~100% | `greedy-token scripts --run check-meta-sync --execute` |
+| `rsync template-project в monorepo` | ~8,533 | **python** (60%) | 0 | ~8,533 | ~100% | dry-run script; run manually |
+| `batch inventory template-project` | ~8,532 | **ollama** (66%) | 0 cloud | ~8,532 | ~100% cloud | `scripts/ollama/batch-inventory.sh` (local LLM) |
+| `refactor header layout and wire nav links` | ~8,535 | **cursor** (82%) | 8,535 | 0 | — | new Cursor chat + skill from `docs/skills-map.md` |
+
+**Tier order:** `tool (rg/jq) → python → ollama → rag → cursor` — first match wins. Ollama tier is skipped when unavailable.
+
+**Takeaway:** lookup / search / sync / bulk tasks save **~6.7k–8.5k tokens per request**; wiring and architecture correctly stay on Cursor.
+
+```bash
+greedy-token audit-context                    # your workspace overhead
+greedy-token estimate "your task here"        # route + savings before opening a chat
+```
+
+Install once, point at your workspace root, route every task through the cheapest tier that can handle it.
+
+## Install
+
+```bash
+pip install greedy-token
+# or editable: pip install -e .
+# or from git: pip install git+https://github.com/svasenkov/greedy-token.git
+```
+
+`tiktoken` (exact BPE counts via `cl100k_base`) is a required dependency. If install fails
+on an unsupported platform, use a Python version with a prebuilt `tiktoken` wheel or install
+from source with a Rust toolchain.
+
+### PyPI publish (maintainer)
+
+1. Create project **greedy-token** on [pypi.org](https://pypi.org/manage/projects/)
+2. Add trusted publisher: Owner `svasenkov`, repo `greedy-token`, workflow `publish.yml`
+3. Publish: GitHub → Releases → re-run workflow or new tag
+
+## Workspace root
+
+`greedy-token` runs against a project directory (monorepo, app repo, etc.):
+
+```bash
+export GREEDY_TOKEN_ROOT=/path/to/your-workspace
+```
+
+Auto-detect works when the workspace has `docs/phase-manifest.json` and `scripts/check-meta-sync.sh` (e.g. [zero-design-system](https://github.com/svasenkov/zero-design-system)). Otherwise set `GREEDY_TOKEN_ROOT` explicitly.
+
+## Commands
+
+| Command | Purpose |
+|---------|---------|
+| `greedy-token route "…"` | Recommend: tool \| python \| ollama \| rag \| cursor + scoring |
+| `greedy-token estimate "…"` | Token-aware estimate: complexity, est_tokens, tier scan |
+| `greedy-token run "…" [--execute]` | Route + dry-run / **read-only** execute |
+| `greedy-token scripts --list` | List workspace script wrappers |
+| `greedy-token scripts --run ID [--execute]` | Dry-run / execute read-only wrapper |
+| `greedy-token audit-context` | Size of always-on rules/skills (tokens) |
+| `greedy-token tokens PATH…` | Count tokens in files/directories |
+| `greedy-token rag QUERY` | Search chunks in `docs/rag/` |
+| `greedy-token compress` | Short prompt version (stdin; `--ollama` for LLM) |
+
+## Tier order
+
+```
+tool (rg/jq) → python → ollama → rag → cursor
+```
+
+First matching tier wins. Ollama is **optional** — if unavailable, the tier is skipped.
+
+## Examples
+
+```bash
+greedy-token route "find baseUrl"
+# → tool (rg)
+
+greedy-token estimate "refactor header layout"
+# → cursor, complexity=high
+
+greedy-token route "batch inventory template-project"
+# → ollama
+
+greedy-token route "sync phase-manifest and skills-map"
+# → python
+
+greedy-token route "ADR 002 baseUrl pattern"
+# → rag
+```
+
+## Environment
+
+| Var | Default |
+|-----|---------|
+| `GREEDY_TOKEN_ROOT` | auto-detect or required |
+| `OLLAMA_URL` | `http://localhost:11434` |
+| `OLLAMA_MODEL` | `qwen2.5-coder:14b` |
+
+## `--execute`
+
+Read-only only: `rg`, `jq`, `check-meta-sync.sh`. Rsync/migrate/ollama — dry-run; run manually.
+
+## Route config
+
+`src/greedy_token/config/routes.yaml` — customize patterns and commands for your workspace.
+
+## License
+
+MIT

greedy_token-0.2.1/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "greedy-token"
+version = "0.2.1"
+description = "Route dev tasks through tool → python → ollama → RAG before escalating to Cursor"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Stanislav Vasenkov" }]
+keywords = ["cli", "llm", "cursor", "token-optimization", "rag", "ollama"]
+dependencies = [
+  "PyYAML>=6.0",
+  "tiktoken>=0.7",
+]
+
+[project.scripts]
+greedy-token = "greedy_token.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/svasenkov/greedy-token"
+Repository = "https://github.com/svasenkov/greedy-token"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+greedy_token = ["config/*.yaml"]

greedy_token-0.2.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

greedy_token-0.2.1/src/greedy_token/__init__.py

@@ -0,0 +1,3 @@
+"""Greedy-token: route dev tasks before expensive LLM calls."""
+
+__version__ = "0.2.0"

greedy_token-0.2.1/src/greedy_token/__main__.py

@@ -0,0 +1,4 @@
+from greedy_token.cli import main
+
+if __name__ == "__main__":
+    main()

greedy_token-0.2.1/src/greedy_token/cli.py

@@ -0,0 +1,205 @@
+from __future__ import annotations
+
+import argparse
+import sys
+
+from greedy_token.context_audit import audit_context, render_audit
+from greedy_token.estimator import estimate_task, format_estimate
+from greedy_token.executors import execute_plan, plan_run
+from greedy_token.paths import find_monorepo_root
+from greedy_token.prompt_compress import compress_prompt, format_dual
+from greedy_token.rag_search import format_hits, search_rag
+from greedy_token.router import format_decision, route_task
+from greedy_token.tokens import TokenEstimate, collect_paths, count_files, format_size_table
+from greedy_token.wrappers import WRAPPERS, ollama_status_line, resolve_wrapper_command
+
+
+def cmd_route(args: argparse.Namespace) -> int:
+    root = find_monorepo_root()
+    decision = route_task(args.task, root)
+    print(format_decision(decision, args.task, root))
+    return 0
+
+
+def cmd_estimate(args: argparse.Namespace) -> int:
+    root = find_monorepo_root()
+    estimate = estimate_task(args.task, root)
+    print(format_estimate(estimate, args.task, root))
+    return 0
+
+
+def cmd_run(args: argparse.Namespace) -> int:
+    root = find_monorepo_root()
+    decision = route_task(args.task, root)
+    plan = plan_run(decision, args.task, root)
+    print(f"Route: {decision.target} ({decision.route_id})")
+    print(f"Complexity: {decision.complexity}  Est. tokens: {decision.est_tokens:,}")
+    print()
+    if args.execute:
+        code, out = execute_plan(plan)
+        print(out)
+        return code
+    print(plan.dry_run_output)
+    if plan.command:
+        if plan.executable:
+            print("\n(read-only — add --execute to run)")
+        else:
+            print("\n(not read-only — dry-run only)")
+    return 0
+
+
+def cmd_audit_context(_: argparse.Namespace) -> int:
+    items = audit_context()
+    print(render_audit(items))
+    return 0
+
+
+def cmd_tokens(args: argparse.Namespace) -> int:
+    root = find_monorepo_root()
+    paths = collect_paths(args.paths, root)
+    if not paths:
+        print("No files found.", file=sys.stderr)
+        return 1
+    estimates = count_files(paths)
+    rows = []
+    total_chars = 0
+    total_tokens = 0
+    method = "heuristic/4"
+    for p, est in zip(paths, estimates):
+        rel = str(p.relative_to(root)) if p.is_relative_to(root) else str(p)
+        rows.append((rel, est))
+        total_chars += est.chars
+        total_tokens += est.tokens
+        method = est.method
+    rows.sort(key=lambda r: -r[1].tokens)
+    total = TokenEstimate(tokens=total_tokens, chars=total_chars, method=method)
+    print(format_size_table(rows, total))
+    return 0
+
+
+def cmd_rag(args: argparse.Namespace) -> int:
+    root = find_monorepo_root()
+    domains = args.domain.split(",") if args.domain else None
+    hits = search_rag(args.query, root, domains=domains, limit=args.limit)
+    print(format_hits(args.query, hits))
+    return 0
+
+
+def cmd_compress(args: argparse.Namespace) -> int:
+    text = sys.stdin.read()
+    if not text.strip():
+        print("Read prompt from stdin.", file=sys.stderr)
+        return 1
+    short = compress_prompt(text, use_ollama=args.ollama)
+    if args.raw:
+        print(short)
+    else:
+        print(format_dual(text, short))
+    return 0
+
+
+def cmd_scripts(args: argparse.Namespace) -> int:
+    root = find_monorepo_root()
+    if args.list:
+        lines = ["Script wrappers (scripts/ollama | migrate | check-meta-sync):", ""]
+        for wrapper in WRAPPERS.values():
+            ro = "read-only" if wrapper.read_only else "writes"
+            oll = " +ollama" if wrapper.requires_ollama else ""
+            lines.append(f"  {wrapper.id:<20} [{wrapper.category}] {ro}{oll}")
+            lines.append(f"    {wrapper.path}")
+            if wrapper.note:
+                lines.append(f"    {wrapper.note}")
+        lines.append("")
+        lines.append(ollama_status_line())
+        print("\n".join(lines))
+        return 0
+    if args.run:
+        try:
+            cmd = resolve_wrapper_command(args.run, root, extra_args=args.args or "")
+        except (KeyError, FileNotFoundError) as exc:
+            print(exc, file=sys.stderr)
+            return 1
+        wrapper = WRAPPERS[args.run]
+        if args.execute:
+            if not wrapper.read_only:
+                print(
+                    f"Refusing --execute: {args.run} is not read-only.",
+                    file=sys.stderr,
+                )
+                return 1
+            import subprocess
+
+            proc = subprocess.run(cmd, shell=True)
+            return proc.returncode
+        print(cmd)
+        if wrapper.read_only:
+            print("\n(read-only — add --execute to run)")
+        else:
+            print("\n(not read-only — dry-run only)")
+        return 0
+    print("Use scripts --list or scripts --run ID", file=sys.stderr)
+    return 1
+
+
+def build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="greedy-token",
+        description="Task orchestrator: tool | Python | Ollama | RAG | Cursor",
+    )
+    sub = p.add_subparsers(dest="command", required=True)
+
+    r = sub.add_parser("route", help="Recommend executor for a task")
+    r.add_argument("task", help="Natural language task description")
+    r.set_defaults(func=cmd_route)
+
+    est = sub.add_parser("estimate", help="Token-aware route estimate with tier scan")
+    est.add_argument("task", help="Task description")
+    est.set_defaults(func=cmd_estimate)
+
+    run = sub.add_parser("run", help="Route and show/run command")
+    run.add_argument("task", help="Task description")
+    run.add_argument(
+        "--execute",
+        action="store_true",
+        help="Execute read-only tool/python commands only",
+    )
+    run.set_defaults(func=cmd_run)
+
+    sub.add_parser("audit-context", help="Token audit of rules/skills").set_defaults(
+        func=cmd_audit_context
+    )
+
+    t = sub.add_parser("tokens", help="Count tokens in paths")
+    t.add_argument("paths", nargs="+", help="Files or directories")
+    t.set_defaults(func=cmd_tokens)
+
+    rag = sub.add_parser("rag", help="Search docs/rag chunks")
+    rag.add_argument("query", help="Search query")
+    rag.add_argument("--domain", help="Comma-separated domains filter")
+    rag.add_argument("--limit", type=int, default=5)
+    rag.set_defaults(func=cmd_rag)
+
+    c = sub.add_parser("compress", help="Short agent prompt from stdin")
+    c.add_argument("--ollama", action="store_true", help="Use local Ollama")
+    c.add_argument("--raw", action="store_true", help="Print short text only")
+    c.set_defaults(func=cmd_compress)
+
+    scr = sub.add_parser("scripts", help="Wrappers for monorepo scripts")
+    scr.add_argument("--list", action="store_true", help="List script wrappers")
+    scr.add_argument("--run", metavar="ID", help="Wrapper id (e.g. check-meta-sync)")
+    scr.add_argument("args", nargs="?", default="", help="Extra args for script")
+    scr.add_argument(
+        "--execute",
+        action="store_true",
+        help="Run read-only wrapper only",
+    )
+    scr.set_defaults(func=cmd_scripts)
+
+    return p
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    code = args.func(args)
+    raise SystemExit(code)