cto-ai-cli 6.1.0 → 8.0.0

package/README.md

@@ -1,81 +1,141 @@
-# CTO — AI context selection done right
+# CTO — AI Context Selection Engine
 
 [![npm](https://img.shields.io/npm/v/cto-ai-cli.svg)](https://www.npmjs.com/package/cto-ai-cli)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-1133%20passing-brightgreen)](.)
 
-Pick the right files for any AI task. Secrets auto-redacted. Learns from your feedback.
+**The most complete AI context selection engine in open source.** Picks the right code *chunks* (not just files), auto-redacts secrets, learns from feedback. 18 signals. Zero AI dependencies.
 
 ```bash
-# Select context, copy to clipboard
-cto --context "fix the auth middleware" --stdout | pbcopy
-
-# Generate a complete AI prompt
-cto --context "fix the auth middleware" --prompt "Refactor this to use JWT"
+cto --context "fix the seller info cache invalidation on KVS delete" --stdout | pbcopy
+```
 
-# Was the AI output good? Tell CTO so it learns.
-cto --accept
+```
+→ 166 relevant chunks from 59 files (26K tokens, 0 secrets)
+→ Full chain: DeleteEndpoint → Router → UseCase → CacheService → KvsRepository
 ```
 
-74KB package. Zero bloat.
+202KB package · 1,133 tests · 96 source modules · Zero AI dependencies.
 
 ---
 
-## What it does
+## The Problem
 
-When you ask an AI to help with code, it needs the right files as context. Send too few and the AI hallucinates. Send too many and you waste tokens. CTO picks the right ones:
+When developers use AI coding assistants, they need to provide context — the right source files. Today, most teams either:
 
-1. **Matches your task** — TF-IDF/BM25 semantic matching, not keyword guessing
-2. **Ranks by composite score** — `risk × 0.4 + semantic × 0.4 + learner × 0.2`
-3. **Sanitizes output** — API keys, tokens, passwords auto-redacted before they reach any AI
-4. **Learns from feedback** — `--accept` / `--reject` teach it what you actually need
+- **Send everything** → expensive, slow, hits token limits
+- **Pick files manually** → miss dependencies, forget test files, leak secrets
 
-Different tasks → different files. `"fix auth"` and `"add database tests"` return **completely different selections**.
+CTO solves both: it **automatically selects the most relevant files** for any task, **sanitizes secrets** before they reach any AI provider, and **learns from feedback** to get better over time.
 
-## Install
+## Quick Demo
 
 ```bash
-npm i -g cto-ai-cli    # global
-npx cto-ai-cli         # or one-shot
+cto --demo   # Run a live showcase on your project
 ```
 
-## Context Selection
+This runs a self-contained presentation that shows: project analysis, semantic matching proof, secret sanitization, ROI calculation, and benchmark results.
 
-```bash
-# Human-readable summary
-cto --context "refactor the auth middleware"
+## Benchmark Results
+
+**Eval Harness v8.0** — 20-file Java enterprise project, 4 tasks with expert-labeled ground truth:
+
+| Metric | Result |
+|---|---|
+| **Must-have recall** | **100%** (every critical file found) |
+| **Precision** | **38–44%** |
+| **F1** | **55%** |
+| **Noise rate** | **11.3%** |
+
+**Real production repos** (Mercado Libre Java monoliths):
+
+| Repo | Files | Without CTO | With CTO v8.0 |
+|---|---|---|---|
+| fury_supply-seller-info | 219 | 212 files (97%) | **166 chunks from 59 files** |
+| sell-sizechart-middleend | 1,719 | 230 files | **72 chunks from 37 files** |
+| charts-backend | 1,261 | 685 files (54%) | **142 chunks from 16 files** |
 
-# Pipe to clipboard (macOS)
-cto --context "fix login bug" --stdout | pbcopy
+**Internal benchmark** (8 tasks, own codebase):
 
-# Save to file (secrets auto-redacted)
-cto --context "add tests" --output context.md
+| Strategy | Precision | Recall | F1 |
+|---|---|---|---|
+| **CTO + Reranker** | **96.9%** | 100% | 98.4% |
+| TF-IDF only | 54.6% | 87.5% | 62.0% |
+| Random | 7.7% | 6.3% | 2.8% |
 
-# Full AI prompt with instruction
-cto --context "fix login" --prompt "Refactor to use async/await"
+## ROI
 
-# JSON for tooling
-cto --context "debug scoring" --json
+On a typical 130-file TypeScript project:
+
+| Metric | Without CTO | With CTO |
+|---|---|---|
+| Tokens per interaction | 370K (all files) | ~28K (selected) |
+| Cost per interaction (Sonnet) | $1.11 | $0.08 |
+| **Monthly cost (10 devs, 40/day)** | **$8,880** | **$640** |
+| **Annual savings** | — | **~$99,000** |
+
+Plus: fewer hallucinations (right context), zero secret leaks, and the learner gets smarter with every `--accept` / `--reject`.
+
+## How it Works (v8.0 Pipeline)
 
-# Custom token budget
-cto --context "fix auth" --budget 30000
+```
+Task → Query Intent Parser → structured action/entities/layers
+         │
+         ▼
+   BM25 (weighted) ──────┐
+   TF-IDF Embedding ─────┤──→ RRF Fusion ─→ 8-signal Boosting ─→ Reranker
+   Multi-hop (auto) ─────┘          │
+                                    ▼
+                              Selection ─→ Chunk Extraction ─→ Output
+                                              (methods, not files)
 ```
 
-Output includes full file contents in markdown, ready to paste into Claude, ChatGPT, or any AI. **Secrets are automatically redacted** — API keys, tokens, passwords, PII are replaced with `****` before output.
+**10-step pipeline:**
 
-## Feedback Loop
+| # | Step | What it does |
+|---|---|---|
+| 0 | **Query Intent** | Parses "fix cache invalidation on delete" → `action:fix`, `entities:[cache,kvs]`, `layers:[cache]` |
+| 1 | **BM25 + Embedding** | Lexical matching + TF-IDF cosine vectors, merged via Reciprocal Rank Fusion |
+| 2 | **Multi-hop** | Complex queries auto-detected → iterative BM25 expansion via deps + call graph (2 hops) |
+| 3 | **Path IDF Boost** | Query terms in file paths get boosted |
+| 4 | **Layer Boost** | Architectural layer matching (controller, service, repository) |
+| 5 | **Import Boost** | Dependencies of top-ranked files get pulled in |
+| 6 | **Call Graph Boost** | Cross-file method calls traced (Java/TS/Python/Go) |
+| 7 | **Git Co-Change** | Files frequently modified together (Jaccard similarity from commits) |
+| 8 | **Reranker** | 5-signal quality gate: term coverage, specificity, bigram proximity, deps, path |
+| 9 | **Chunk Extraction** | Extracts relevant functions/methods — not whole files. 10x token efficiency |
 
-CTO learns from real feedback, not from itself:
+**No AI is used for selection.** Same input → same output. Deterministic.
+
+## Install
 
 ```bash
-# After using the context and it worked:
-cto --accept
+npm i -g cto-ai-cli    # global
+npx cto-ai-cli         # or one-shot
+```
 
-# If the AI needed files CTO didn't include:
-cto --reject
-cto --reject --missing src/types/auth.ts
+## Context Selection
 
-# See what CTO has learned:
-cto --stats
+```bash
+cto --context "refactor the auth middleware"                 # human-readable summary
+cto --context "fix login bug" --stdout | pbcopy              # pipe to clipboard
+cto --context "add tests" --output context.md                # save to file
+cto --context "fix login" --prompt "Refactor to async/await" # full AI prompt
+cto --context "debug scoring" --json                         # JSON for tooling
+cto --context "fix auth" --budget 30000                      # custom token budget
+```
+
+Output includes full file contents in markdown, ready for Claude, ChatGPT, or any AI. **Secrets are automatically redacted** — API keys, tokens, passwords, PII are replaced with `****` before output.
+
+## Feedback Loop
+
+CTO learns from real feedback, not from itself:
+
+```bash
+cto --accept                         # last selection was good
+cto --reject                         # last selection was bad
+cto --reject --missing src/auth.ts   # this file was missing
+cto --stats                          # see what CTO has learned
 ```
 
 On `--reject`, CTO also detects files you edited after the selection that weren't in the context — those get automatically boosted for next time.
@@ -89,7 +149,74 @@ cto --audit --full-scan      # ignore cache, scan everything
 cto --audit --json           # machine-readable output
 ```
 
-45+ patterns (AWS, Stripe, GitHub, OpenAI, Slack, etc.) plus Shannon entropy analysis. But the real value is that **audit protects context**: every `--stdout`, `--output`, and `--prompt` command auto-sanitizes secrets before output.
+45+ patterns (AWS, Stripe, GitHub, OpenAI, Slack, Cloudflare...) plus Shannon entropy analysis. The real value: **audit protects context** — every `--stdout`, `--output`, and `--prompt` auto-sanitizes secrets before output.
+
+```
+Before:  OPENAI_KEY = "sk-Rk8bN3xYz2Wq5PmL7jCvT1aBcDe"
+After:   OPENAI_KEY = "sk-R********************De"
+```
+
+## AI Gateway (Enterprise)
+
+A transparent HTTP proxy between your developers and AI providers. Automatically injects optimized context, redacts secrets, and tracks costs — without changing developer workflow.
+
+```bash
+cto --gateway                        # Start on port 8787
+cto --gateway --port 9000            # Custom port
+cto --gateway --block-secrets        # Block requests with critical secrets
+cto --gateway --budget-daily 50      # $50/day budget limit
+cto --gateway --budget-monthly 500   # $500/month budget limit
+```
+
+```
+Developer → CTO Gateway → [context injection + sanitization + cost tracking] → AI Provider
+                ↓
+          Dashboard (http://localhost:8787/__cto)
+```
+
+**What the gateway does automatically:**
+- **Injects CTO-selected context** into every AI request (TF-IDF + composite scoring)
+- **Redacts secrets** before they leave the network (45+ patterns)
+- **Tracks costs** per model, per day, per month with budget alerts
+- **Streams responses** with zero-copy SSE passthrough
+- **Serves a live dashboard** at `/__cto` with real-time metrics
+
+Supports OpenAI, Anthropic, Google, and Azure OpenAI. SSRF protection built-in.
+
+## Cross-Repo Context
+
+When working on a task, CTO can pull relevant files from **sibling repositories** — not just the current project.
+
+```bash
+cto --context "fix payment webhook" --auto-repos   # Auto-discover sibling repos
+cto --context "fix payment webhook" --repos shared-types,payment-service
+```
+
+**How it works:**
+1. Discovers sibling repos in parent directory (any dir with `package.json`, `tsconfig.json`, `Cargo.toml`, etc.)
+2. Builds a lightweight TF-IDF index per sibling (reads source files, no full analysis)
+3. Queries each sibling with the task description
+4. Returns ranked matches with repo attribution and content
+
+Real use case: You're fixing a webhook handler in `api-gateway` — CTO finds the `Payment` interface in `shared-types` and the consumer in `notification-service` automatically.
+
+## Cost-Aware Model Routing
+
+CTO analyzes the **actual selected context** (not just the project) to recommend the cheapest model that can handle the task.
+
+```bash
+cto --context "update readme" --route     # → Haiku ($0.08/call, 73% cheaper)
+cto --context "fix auth bug" --route      # → Opus ($1.33/call, critical complexity)
+cto --context "refactor API" --route      # → Sonnet ($0.30/call, balanced)
+```
+
+**Complexity is computed from real signals:**
+- Token density (% of budget used)
+- Risk concentration (top-5 file avg risk vs project max)
+- Directory diversity (cross-cutting = harder)
+- Dependency density among selected files
+
+The gateway also uses this: every proxied request gets a model recommendation in the injected context.
 
 ## MCP Server
 
@@ -107,17 +234,6 @@ Works as an MCP server for AI editors (Windsurf, Claude Desktop, Cursor).
 
 MCP output is also auto-sanitized when `includeContents: true`.
 
-## How it works
-
-1. **Dependency graph** — parses imports, builds adjacency list, identifies hubs
-2. **Risk scoring** — complexity × centrality × recency (continuous, log-scaled)
-3. **TF-IDF/BM25 semantic matching** — task description scored against all file contents + path boosting
-4. **Composite ranking** — `finalScore = risk × 0.4 + semantic × 0.4 + learner × 0.2`
-5. **Greedy allocation** — fills token budget top-down, cascading prune levels (full → signatures → skeleton)
-6. **Bayesian learning** — exponential decay on priors, Wilson score confidence, per-task-type patterns
-
-No AI is used for selection. Same input → same output. Deterministic.
-
 ## Programmatic API
 
 ```typescript
@@ -132,22 +248,107 @@ const selection = await selectContext({
   task: 'fix auth',
   analysis,
   budget: 50_000,
-  semanticScores,  // wired into ranking
+  semanticScores,
 });
 ```
 
-## Honest limitations
+## v8.0 — What's New
+
+### Chunk-Level Retrieval (the big one)
+
+Instead of including entire files, CTO now extracts **only the relevant functions and methods**. A 2000-line file with 1 relevant method → 50 lines included, not 2000.
+
+```
+### src/main/java/com/example/cache/CacheService.java
+```java
+// L15-22: method invalidate
+public void invalidate(String id) {
+    redis.delete("cache:seller:" + id);
+}
+
+// ... lines 23-45 omitted ...
+
+// L46-52: method retrieve
+public SellerDTO retrieve(String id) {
+    return redis.opsForValue().get("cache:seller:" + id);
+}
+```
+
+Supports Java, TypeScript, Python, Go.
+
+### Query Intent Parsing
+
+Before searching, CTO parses your task into structured intent:
+
+```
+"fix the seller cache invalidation on KVS delete"
+  → action: fix
+  → entities: [seller, kvs] (3× weight)
+  → operations: [invalidate, delete] (2× weight)
+  → layers: [cache]
+```
+
+Entities get 3× BM25 weight, operations get 2×. Much better precision on enterprise queries.
+
+### Embedding Search + RRF Fusion
+
+TF-IDF cosine embedding vectors complement BM25 lexical matching. Merged via Reciprocal Rank Fusion (60/40 BM25/embedding). Catches semantic similarity that BM25 misses.
+
+### Cross-File Call Graph
+
+Traces method calls across files: `cacheService.invalidate()` in UseCase → finds `CacheService.java`. Regex-based, works for Java/TS/Python/Go.
+
+### Git Co-Change Signal
+
+Files frequently modified together in git history get boosted. Jaccard similarity from commit co-occurrence.
+
+### Multi-Hop Reasoning
+
+Complex enterprise queries auto-detected. Iterative BM25: top matches → expand via deps + call graph → re-query. Traces full execution chains (4/4 hops).
+
+### Evaluation Harness
+
+Ground truth benchmark with must-have/relevant/noise labels. 100% must-have recall on 4-task Java enterprise benchmark.
+
+## Enterprise Features
+
+- **AI Gateway** — transparent HTTP proxy with context injection, secret redaction, cost tracking
+- **Team Auth** — per-team API keys, JWT (HS256/RS256), rate limiting, OIDC discovery
+- **Policy Engine** — model overrides by task type, cost caps, block rules
+- **Metrics** — Prometheus, Datadog JSON, StatsD UDP
+- **A/B Testing** — context strategy experiments with z-test significance
+- **LSP Bridge** — JSON-RPC 2.0 for VS Code, JetBrains, Neovim
+- **Persistent Index Cache** — 50K-file repos: 5s → <100ms on warm cache
+
+## Competitor Comparison
+
+| Feature | CTO v8 | Cursor | Sourcegraph Cody |
+|---|---|---|---|
+| BM25 retrieval | ✅ | ✅ | ✅ |
+| Embedding search | ✅ TF-IDF cosine+RRF | ✅ | ✅ |
+| Chunk-level retrieval | ✅ 4 langs | ✅ | ✅ |
+| Multi-signal RRF fusion | ✅ 8-signal | ❌ | ❌ |
+| Cross-file call graph | ✅ | ❌ | ❌ |
+| Git co-change signal | ✅ | ❌ | ❌ |
+| Multi-hop reasoning | ✅ | ❌ | ❌ |
+| Query intent parsing | ✅ | ❌ | ❌ |
+| Feedback learning | ✅ | ❌ | ❌ |
+| Secret redaction | ✅ | ❌ | ❌ |
+| **Total signals** | **18** | **~3** | **~5** |
+
+## Honest Limitations
 
-- **TypeScript/JavaScript gets deep analysis.** Other languages get basic file + import analysis.
-- **TF-IDF, not embeddings.** Handles most tasks well but won't understand complex intent.
-- **Learning needs ~5 feedback cycles** to start influencing selection. First runs are pure graph + risk + semantic.
-- **Not compared against Cursor/Copilot internal context.** Our baselines are naive (alphabetical, random).
+- **TypeScript/JavaScript gets AST analysis.** Python/Go/Java/Rust get regex-based parsing (good for graphs + chunking, not AST-precise).
+- **Embeddings are TF-IDF cosine, not neural.** ONNX infrastructure ready — neural model would add ~5-10% recall.
+- **Learning needs ~5 feedback cycles** to start influencing selection. First runs are pure pipeline.
+- **Chunk extraction is regex-based** — works for standard methods/functions, may miss DSLs or deeply nested code.
+- **Benchmarked against naive baselines.** Not compared against Cursor/Copilot internal context engines.
 
 ## Contributing
 
 ```bash
 git clone https://github.com/cto-ai/cto-ai-cli.git && cd cto-ai-cli
-npm install && npm run build && npm test  # 597 tests
+npm install && npm run build && npm test  # 1,133 tests
 ```
 
 ## License