cto-ai-cli 7.1.0 → 8.0.0

package/README.md

@@ -2,17 +2,20 @@
 
 [![npm](https://img.shields.io/npm/v/cto-ai-cli.svg)](https://www.npmjs.com/package/cto-ai-cli)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-606%20passing-brightgreen)](.)
+[![Tests](https://img.shields.io/badge/tests-1133%20passing-brightgreen)](.)
 
-**Pick the right files for any AI task. Secrets auto-redacted. Learns from your feedback.**
+**The most complete AI context selection engine in open source.** Picks the right code *chunks* (not just files), auto-redacts secrets, learns from feedback. 18 signals. Zero AI dependencies.
 
 ```bash
-cto --context "fix the auth middleware" --stdout | pbcopy   # → clipboard
-cto --context "fix auth" --prompt "Refactor to use JWT"     # → AI prompt
-cto --accept                                                 # → learns
+cto --context "fix the seller info cache invalidation on KVS delete" --stdout | pbcopy
 ```
 
-76KB package · 606 tests · Zero AI dependencies.
+```
+→ 166 relevant chunks from 59 files (26K tokens, 0 secrets)
+→ Full chain: DeleteEndpoint → Router → UseCase → CacheService → KvsRepository
+```
+
+202KB package · 1,133 tests · 96 source modules · Zero AI dependencies.
 
 ---
 
@@ -35,18 +38,31 @@ This runs a self-contained presentation that shows: project analysis, semantic m
 
 ## Benchmark Results
 
-Tested against 8 curated tasks with ground truth (known correct files):
+**Eval Harness v8.0** — 20-file Java enterprise project, 4 tasks with expert-labeled ground truth:
 
-| Strategy | Precision | Must-have Recall | F1 |
+| Metric | Result |
+|---|---|
+| **Must-have recall** | **100%** (every critical file found) |
+| **Precision** | **38–44%** |
+| **F1** | **55%** |
+| **Noise rate** | **11.3%** |
+
+**Real production repos** (Mercado Libre Java monoliths):
+
+| Repo | Files | Without CTO | With CTO v8.0 |
 |---|---|---|---|
-| **CTO** | 33.6% | **100.0%** | **48.7%** |
+| fury_supply-seller-info | 219 | 212 files (97%) | **166 chunks from 59 files** |
+| sell-sizechart-middleend | 1,719 | 230 files | **72 chunks from 37 files** |
+| charts-backend | 1,261 | 685 files (54%) | **142 chunks from 16 files** |
+
+**Internal benchmark** (8 tasks, own codebase):
+
+| Strategy | Precision | Recall | F1 |
+|---|---|---|---|
+| **CTO + Reranker** | **96.9%** | 100% | 98.4% |
 | TF-IDF only | 54.6% | 87.5% | 62.0% |
-| Risk-only | 20.8% | 18.8% | 15.0% |
-| Alphabetical | 8.3% | 31.3% | 12.9% |
 | Random | 7.7% | 6.3% | 2.8% |
 
-**CTO never misses a must-have file** (100% recall). 3.8× better F1 than alphabetical. 17× better than random.
-
 ## ROI
 
 On a typical 130-file TypeScript project:
@@ -60,23 +76,34 @@ On a typical 130-file TypeScript project:
 
 Plus: fewer hallucinations (right context), zero secret leaks, and the learner gets smarter with every `--accept` / `--reject`.
 
-## How it Works
+## How it Works (v8.0 Pipeline)
 
 ```
-Task description ──→ TF-IDF/BM25 ──→ Semantic scores ──┐
-                                                         │
-Project files ──→ Dependency graph ──→ Risk scores ──────┤──→ Composite ──→ Greedy ──→ Selection
-                                                         │     ranking      alloc
-Feedback history ──→ Bayesian learner ──→ Boosts ────────┘
+Task → Query Intent Parser → structured action/entities/layers
+         │
+         ▼
+   BM25 (weighted) ──────┐
+   TF-IDF Embedding ─────┤──→ RRF Fusion ─→ 8-signal Boosting ─→ Reranker
+   Multi-hop (auto) ─────┘          │
+                                    ▼
+                              Selection ─→ Chunk Extraction ─→ Output
+                                              (methods, not files)
 ```
 
-1. **Dependency graph** — parses imports, builds adjacency list, identifies hubs
-2. **Risk scoring** — complexity × centrality × recency (continuous, log-scaled)
-3. **TF-IDF/BM25 semantic matching** — task description scored against file contents + path boosting
-4. **Composite ranking** — `finalScore = semantic × 0.55 + risk × 0.25 + learner × 0.2`
-5. **Noise filtering** — files with zero semantic relevance are excluded (benchmark-driven optimization)
-6. **Greedy allocation** — fills token budget top-down, cascading prune levels (full → signatures → skeleton)
-7. **Bayesian learning** — exponential decay, Wilson score confidence, per-task-type patterns
+**10-step pipeline:**
+
+| # | Step | What it does |
+|---|---|---|
+| 0 | **Query Intent** | Parses "fix cache invalidation on delete" → `action:fix`, `entities:[cache,kvs]`, `layers:[cache]` |
+| 1 | **BM25 + Embedding** | Lexical matching + TF-IDF cosine vectors, merged via Reciprocal Rank Fusion |
+| 2 | **Multi-hop** | Complex queries auto-detected → iterative BM25 expansion via deps + call graph (2 hops) |
+| 3 | **Path IDF Boost** | Query terms in file paths get boosted |
+| 4 | **Layer Boost** | Architectural layer matching (controller, service, repository) |
+| 5 | **Import Boost** | Dependencies of top-ranked files get pulled in |
+| 6 | **Call Graph Boost** | Cross-file method calls traced (Java/TS/Python/Go) |
+| 7 | **Git Co-Change** | Files frequently modified together (Jaccard similarity from commits) |
+| 8 | **Reranker** | 5-signal quality gate: term coverage, specificity, bigram proximity, deps, path |
+| 9 | **Chunk Extraction** | Extracts relevant functions/methods — not whole files. 10x token efficiency |
 
 **No AI is used for selection.** Same input → same output. Deterministic.
 
@@ -225,62 +252,103 @@ const selection = await selectContext({
 });
 ```
 
-## v7.0 Enterprise Features
+## v8.0 — What's New
 
-### Precision Reranker (96.9% precision, was 33.6%)
+### Chunk-Level Retrieval (the big one)
 
-Multi-signal reranker between BM25 retrieval and greedy allocation:
-- **Term coverage**: fraction of unique query terms matched per file
-- **Term specificity**: IDF-weighted — rare terms matter more
-- **Bigram proximity**: query terms appearing close together in the file
-- **Dependency signal**: files in the dependency cone of top matches
-- **Quality gate**: adaptive cutoff stops filling budget with noise
+Instead of including entire files, CTO now extracts **only the relevant functions and methods**. A 2000-line file with 1 relevant method → 50 lines included, not 2000.
 
-### Persistent Index Cache
+```
+### src/main/java/com/example/cache/CacheService.java
+```java
+// L15-22: method invalidate
+public void invalidate(String id) {
+    redis.delete("cache:seller:" + id);
+}
+
+// ... lines 23-45 omitted ...
+
+// L46-52: method retrieve
+public SellerDTO retrieve(String id) {
+    return redis.opsForValue().get("cache:seller:" + id);
+}
+```
 
-TF-IDF index persisted to `.cto/index-cache.json` with per-file mtime tracking. Subsequent queries only re-tokenize changed files. 50K-file repos go from 5s → <100ms on warm cache.
+Supports Java, TypeScript, Python, Go.
 
-### Multi-Language Dependency Graphs
+### Query Intent Parsing
 
-Regex-based import parsing for **Python**, **Go**, **Java**, and **Rust** alongside ts-morph for TS/JS. Enables hub detection, risk scoring, and dependency expansion for polyglot codebases.
+Before searching, CTO parses your task into structured intent:
 
-```bash
-# Works on Python, Go, Java, Rust projects — not just TypeScript
-cto --context "fix auth handler" /path/to/go-project
 ```
+"fix the seller cache invalidation on KVS delete"
+  → action: fix
+  → entities: [seller, kvs] (3× weight)
+  → operations: [invalidate, delete] (2× weight)
+  → layers: [cache]
+```
+
+Entities get 3× BM25 weight, operations get 2×. Much better precision on enterprise queries.
 
-### Team Authentication & SSO
+### Embedding Search + RRF Fusion
 
-Per-team API keys, JWT validation (HS256/RS256), rate limiting, model allowlists. Teams stored in `.cto/gateway/teams.json`.
+TF-IDF cosine embedding vectors complement BM25 lexical matching. Merged via Reciprocal Rank Fusion (60/40 BM25/embedding). Catches semantic similarity that BM25 misses.
 
-### Metrics Export
+### Cross-File Call Graph
 
-Prometheus exposition format at `/__cto/metrics`, Datadog JSON, and StatsD UDP. Counters, histograms, gauges for requests, tokens, cost, latency, secrets.
+Traces method calls across files: `cacheService.invalidate()` in UseCase → finds `CacheService.java`. Regex-based, works for Java/TS/Python/Go.
 
-### Per-Team Policy Engine
+### Git Co-Change Signal
 
-Routing rules per team: model overrides by task type, cost caps per request, context budget limits, block rules. Preset policies: `createCostConscious()`, `createSecurityFirst()`.
+Files frequently modified together in git history get boosted. Jaccard similarity from commit co-occurrence.
 
-### Closed-Loop A/B Testing
+### Multi-Hop Reasoning
 
-Real experimentation on context strategies with two-proportion z-test for statistical significance. Deterministic assignment (SHA-256 hashing), auto-conclusion when p < 0.05.
+Complex enterprise queries auto-detected. Iterative BM25: top matches → expand via deps + call graph → re-query. Traces full execution chains (4/4 hops).
 
-### LSP Bridge (IDE Plugin)
+### Evaluation Harness
 
-JSON-RPC 2.0 server over stdin/stdout for any IDE: VS Code, JetBrains, Neovim, Emacs. Custom methods: `cto/selectContext`, `cto/score`, `cto/audit`, `cto/experiments`.
+Ground truth benchmark with must-have/relevant/noise labels. 100% must-have recall on 4-task Java enterprise benchmark.
+
+## Enterprise Features
+
+- **AI Gateway** — transparent HTTP proxy with context injection, secret redaction, cost tracking
+- **Team Auth** — per-team API keys, JWT (HS256/RS256), rate limiting, OIDC discovery
+- **Policy Engine** — model overrides by task type, cost caps, block rules
+- **Metrics** — Prometheus, Datadog JSON, StatsD UDP
+- **A/B Testing** — context strategy experiments with z-test significance
+- **LSP Bridge** — JSON-RPC 2.0 for VS Code, JetBrains, Neovim
+- **Persistent Index Cache** — 50K-file repos: 5s → <100ms on warm cache
+
+## Competitor Comparison
+
+| Feature | CTO v8 | Cursor | Sourcegraph Cody |
+|---|---|---|---|
+| BM25 retrieval | ✅ | ✅ | ✅ |
+| Embedding search | ✅ TF-IDF cosine+RRF | ✅ | ✅ |
+| Chunk-level retrieval | ✅ 4 langs | ✅ | ✅ |
+| Multi-signal RRF fusion | ✅ 8-signal | ❌ | ❌ |
+| Cross-file call graph | ✅ | ❌ | ❌ |
+| Git co-change signal | ✅ | ❌ | ❌ |
+| Multi-hop reasoning | ✅ | ❌ | ❌ |
+| Query intent parsing | ✅ | ❌ | ❌ |
+| Feedback learning | ✅ | ❌ | ❌ |
+| Secret redaction | ✅ | ❌ | ❌ |
+| **Total signals** | **18** | **~3** | **~5** |
 
 ## Honest Limitations
 
-- **TypeScript/JavaScript gets AST analysis.** Python/Go/Java/Rust get regex-based import parsing (good for graphs, not AST-accurate).
-- **BM25 + reranker, not embeddings.** 96.9% precision on our benchmark. No neural model needed.
-- **Learning needs ~5 feedback cycles** to start influencing selection. First runs are pure graph + risk + semantic.
-- **Benchmarked against naive baselines** (alphabetical, random, risk-only, TF-IDF-only). Not compared against Cursor/Copilot internal context engines.
+- **TypeScript/JavaScript gets AST analysis.** Python/Go/Java/Rust get regex-based parsing (good for graphs + chunking, not AST-precise).
+- **Embeddings are TF-IDF cosine, not neural.** ONNX infrastructure ready — neural model would add ~5-10% recall.
+- **Learning needs ~5 feedback cycles** to start influencing selection. First runs are pure pipeline.
+- **Chunk extraction is regex-based** — works for standard methods/functions, may miss DSLs or deeply nested code.
+- **Benchmarked against naive baselines.** Not compared against Cursor/Copilot internal context engines.
 
 ## Contributing
 
 ```bash
 git clone https://github.com/cto-ai/cto-ai-cli.git && cd cto-ai-cli
-npm install && npm run build && npm test  # 776 tests
+npm install && npm run build && npm test  # 1,133 tests
 ```
 
 ## License