cto-ai-cli 5.1.0 → 6.1.0

package/README.md

@@ -1,334 +1,155 @@
-# CTO — Stop sending your entire codebase to AI
+# CTO — AI context selection done right
 
-[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
-[![Tests](https://img.shields.io/badge/tests-550_passing-brightgreen.svg)](#)
-[![Coverage](https://img.shields.io/badge/coverage-91%25-brightgreen.svg)](#)
 [![npm](https://img.shields.io/npm/v/cto-ai-cli.svg)](https://www.npmjs.com/package/cto-ai-cli)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
 
-CTO analyzes your project and selects the **minimum set of files** your AI needs — saving tokens, reducing cost, and producing code that actually compiles.
+Pick the right files for any AI task. Secrets auto-redacted. Learns from your feedback.
 
 ```bash
-npx cto-ai-cli
-```
-
-**Runs in <1 second.** No API keys. No data leaves your machine.
-
----
-
-## The Problem
-
-When you ask an AI to help with code, it needs context. Most approaches:
+# Select context, copy to clipboard
+cto --context "fix the auth middleware" --stdout | pbcopy
 
-- **Send everything** — expensive, noisy, AI gets confused
-- **Send open files** — misses types, dependencies, config
-- **Let the AI pick** — it doesn't know your dependency graph
+# Generate a complete AI prompt
+cto --context "fix the auth middleware" --prompt "Refactor this to use JWT"
 
-The result: AI generates code that **doesn't compile** because it never saw your type definitions.
-
-## The Fix
-
-```bash
-$ npx cto-ai-cli ./my-project
-```
+# Was the AI output good? Tell CTO so it learns.
+cto --accept
 ```
-  ⚡ cto-score — analyzing your project...
-
-  ╔══════════════════════════════════════════════════╗
-  ║                                                  ║
-  ║   🟢 Context Score™  88 / 100   Grade: A-       ║
-  ║                                                  ║
-  ║   Efficiency     ████████████████░░░░  80%       ║
-  ║   Coverage       ████████████████████ 100%       ║
-  ║   Risk Control   ████████████████████ 100%       ║
-  ║   Structure      █░░░░░░░░░░░░░░░░░░   5%       ║
-  ║   Governance     ██████████████████░  90%        ║
-  ║                                                  ║
-  ║   💰 vs. Sending Everything:                     ║
-  ║   Tokens saved: 392K (88%)                       ║
-  ║   Monthly savings: ~$943                         ║
-  ║                                                  ║
-  ╚══════════════════════════════════════════════════╝
-
-  Scanned in 0.6s · 199 files · 443K tokens
-```
-
-### What each number means
 
-| Metric | What it measures | Why it matters |
-|--------|-----------------|----------------|
-| **Context Score (88/100)** | Overall AI-readiness of your project | Higher = AI tools produce better output with your code |
-| **Efficiency (80%)** | How much CTO can compress without losing value | 80% means we send 20% of tokens for the same quality |
-| **Coverage (100%)** | % of important files included in the selection | 100% = every dependency and type file is captured |
-| **Risk Control (100%)** | Are high-risk files (hubs, complex code) prioritized? | Ensures AI sees the files most likely to cause bugs |
-| **Structure (5%)** | How well-organized your codebase is for AI | Low = too many large files, poor modularity |
-| **Governance (90%)** | Audit logging, policy enforcement, secret scanning | Enterprise readiness |
-| **Tokens saved (88%)** | Reduction vs. sending every file | Directly reduces your API costs |
-| **Monthly savings ($943)** | Estimated cost reduction at 800 interactions/month | Based on average GPT-4o pricing |
+74KB package. Zero bloat.
 
 ---
 
-## Quick Start
+## What it does
 
-### Score your project
+When you ask an AI to help with code, it needs the right files as context. Send too few and the AI hallucinates. Send too many and you waste tokens. CTO picks the right ones:
 
-```bash
-npx cto-ai-cli                     # Analyze current directory
-npx cto-ai-cli ./my-project         # Analyze a specific project
-npx cto-ai-cli --json               # Machine-readable JSON output
-```
+1. **Matches your task** — TF-IDF/BM25 semantic matching, not keyword guessing
+2. **Ranks by composite score** — `risk × 0.4 + semantic × 0.4 + learner × 0.2`
+3. **Sanitizes output** — API keys, tokens, passwords auto-redacted before they reach any AI
+4. **Learns from feedback** — `--accept` / `--reject` teach it what you actually need
 
-### Generate optimized context for AI
+Different tasks → different files. `"fix auth"` and `"add database tests"` return **completely different selections**.
 
-```bash
-npx cto-ai-cli --fix
-```
-
-Creates `.cto/context.md` — paste this into any AI chat for optimal context. Also generates `.cto/config.json` and `.cto/.cteignore`.
+## Install
 
 ```bash
-npx cto-ai-cli --context "refactor the auth middleware"
-```
-
-Generates **task-specific** context — only files relevant to auth, including types, dependencies, and related tests.
-
-Example output:
-```
-  📋 Context for: "refactor the auth middleware"
-
-  Selected 12 files (8.2K tokens):
-
-  ┌─ Core (3 files) ─────────────────────────────
-  │  src/middleware/auth.ts          2,100 tokens
-  │  src/types/auth.ts                 450 tokens
-  │  src/config/jwt.ts                 320 tokens
-  │
-  ├─ Dependencies (5 files) ─────────────────────
-  │  src/models/user.ts              1,200 tokens
-  │  src/services/token.ts             890 tokens
-  │  ...
-  │
-  └─ Tests (2 files) ────────────────────────────
-     tests/auth.test.ts              1,800 tokens
-     tests/middleware.test.ts          940 tokens
-
-  Saved to .cto/context.md (8.2K tokens — 97% smaller than full project)
+npm i -g cto-ai-cli    # global
+npx cto-ai-cli         # or one-shot
 ```
 
-### Security audit
+## Context Selection
 
 ```bash
-npx cto-ai-cli --audit
-```
+# Human-readable summary
+cto --context "refactor the auth middleware"
 
-Scans for **API keys, tokens, passwords, and PII** before they end up in an AI prompt. 45+ patterns (AWS, Stripe, GitHub, OpenAI, etc.) plus Shannon entropy analysis for unknown formats.
+# Pipe to clipboard (macOS)
+cto --context "fix login bug" --stdout | pbcopy
 
-```
-  🔴 CRITICAL src/config/stripe.ts:8
-             api-key: sk_l********************yZ
-  🔴 CRITICAL src/config/database.ts:14
-             connection-string: post********************db
-  🟠 HIGH     src/utils/email.ts:22
-             pii: admi**********om
-
-  🚨 3 critical findings. Rotate credentials immediately.
-```
+# Save to file (secrets auto-redacted)
+cto --context "add tests" --output context.md
 
-Run in CI to block PRs with secrets: `CI=true npx cto-ai-cli --audit`
+# Full AI prompt with instruction
+cto --context "fix login" --prompt "Refactor to use async/await"
 
-### Code review intelligence
+# JSON for tooling
+cto --context "debug scoring" --json
 
-```bash
-npx cto-ai-cli --review
+# Custom token budget
+cto --context "fix auth" --budget 30000
 ```
 
-Analyzes your git diff and generates a structured review:
+Output includes full file contents in markdown, ready to paste into Claude, ChatGPT, or any AI. **Secrets are automatically redacted** — API keys, tokens, passwords, PII are replaced with `****` before output.
 
-```
-  📊 Review Quality: 82/100 (B+)
+## Feedback Loop
 
-  Breaking Changes:
-    🔴 Removed export: UserService.findById (used by 4 files)
-    🟡 Changed signature: authenticate(token) → authenticate(token, opts)
-
-  Missing Files:
-    ⚠️  No test file for src/services/auth.ts
-    ⚠️  src/types/user.ts changed but barrel index not updated
-
-  Impact Radius:
-    Direct: 4 files  |  Transitive: 12 files  |  Tests: 3 files
-
-  Saved review prompt to .cto/review-prompt.md
-```
-
-| What it detects | Example |
-|-----------------|--------|
-| **Breaking changes** | Removed exports, changed function signatures, deleted files |
-| **Missing files** | Tests, type files, barrel exports, importers of changed code |
-| **Impact radius** | How many files are affected (direct + transitive via BFS) |
-| **Review quality** | Score based on PR size, focus, breaking changes, completeness |
-
-### Learning mode
+CTO learns from real feedback, not from itself:
 
 ```bash
-npx cto-ai-cli --learn               # View feedback model & stats
-npx cto-ai-cli --predict              # Predict relevant files for a task
-npx cto-ai-cli --learn --json         # Export learning data for team sharing
-```
+# After using the context and it worked:
+cto --accept
 
-CTO learns from your usage patterns over time. Uses **EWMA temporal decay** (recent feedback weighs more) and **Bayesian confidence** (Wilson score — avoids over-trusting sparse data).
+# If the AI needed files CTO didn't include:
+cto --reject
+cto --reject --missing src/types/auth.ts
 
-### Quality gate for CI/CD
-
-```bash
-npx cto-ai-cli --ci                   # Run quality gate (exits 1 on failure)
-npx cto-ai-cli --ci --threshold 80    # Custom minimum score
-npx cto-ai-cli --ci --json            # JSON for pipeline parsing
+# See what CTO has learned:
+cto --stats
 ```
 
-Block merges when context quality drops below your threshold. Tracks baselines and detects regressions.
+On `--reject`, CTO also detects files you edited after the selection that weren't in the context — those get automatically boosted for next time.
 
-### Monorepo support
+## Secret Audit
 
 ```bash
-npx cto-ai-cli --monorepo             # Analyze all packages
-npx cto-ai-cli --monorepo --package api  # Focus on one package
+cto --audit                  # scan all files
+cto --audit --init-hook      # install pre-commit hook
+cto --audit --full-scan      # ignore cache, scan everything
+cto --audit --json           # machine-readable output
 ```
 
-Detects npm/yarn/pnpm workspaces, Turborepo, Nx, and Lerna. Shows cross-package dependencies, isolation scores, and shared package analysis.
+45+ patterns (AWS, Stripe, GitHub, OpenAI, Slack, etc.) plus Shannon entropy analysis. But the real value is that **audit protects context**: every `--stdout`, `--output`, and `--prompt` command auto-sanitizes secrets before output.
 
----
+## MCP Server
 
-## All CLI Flags
+Works as an MCP server for AI editors (Windsurf, Claude Desktop, Cursor).
 
-```bash
-# Analysis
-npx cto-ai-cli [path]                 # Score a project
-npx cto-ai-cli --json                 # JSON output
-npx cto-ai-cli --benchmark            # CTO vs naive vs random comparison
-npx cto-ai-cli --compare              # Compare vs popular OSS projects
-npx cto-ai-cli --report               # Markdown report + badge
-
-# Context generation
-npx cto-ai-cli --fix                  # Auto-generate .cto/context.md
-npx cto-ai-cli --context "task"       # Task-specific context
-
-# Security
-npx cto-ai-cli --audit                # Secret & PII detection
-npx cto-ai-cli --audit --full-scan    # Scan all files (ignore cache)
-npx cto-ai-cli --audit --init-hook    # Install pre-commit hook
-
-# Code review
-npx cto-ai-cli --review               # PR review analysis
-npx cto-ai-cli --review --json        # Review data as JSON
-
-# Learning
-npx cto-ai-cli --learn                # Feedback model dashboard
-npx cto-ai-cli --predict              # File predictions for a task
-npx cto-ai-cli --learn --json         # Export learning data
-
-# CI/CD
-npx cto-ai-cli --ci                   # Quality gate
-npx cto-ai-cli --ci --threshold 80    # Custom threshold
-
-# Monorepo
-npx cto-ai-cli --monorepo             # Full monorepo analysis
-npx cto-ai-cli --monorepo --package X # Single package
-
-# Gateway (AI proxy)
-npx cto-gateway                       # Start proxy server
-npx cto-gateway --budget-daily 10     # With budget enforcement
-```
-
----
-
-## MCP Server (for AI Editors)
+**3 tools:** `cto_select_context`, `cto_audit_secrets`, `cto_explain`
 
-CTO works as an [MCP server](https://modelcontextprotocol.io/) — plug it into Claude, Windsurf, or Cursor.
-
-**Windsurf** — add to `~/.codeium/windsurf/mcp_config.json`:
 ```json
-{
-  "mcpServers": {
-    "cto": { "command": "cto-mcp" }
-  }
-}
-```
+// Windsurf: ~/.codeium/windsurf/mcp_config.json
+{ "mcpServers": { "cto": { "command": "cto-mcp" } } }
 
-**Claude Desktop:**
-```json
-{
-  "mcpServers": {
-    "cto": { "command": "npx", "args": ["-y", "cto-ai-cli", "--mcp"] }
-  }
-}
+// Claude Desktop
+{ "mcpServers": { "cto": { "command": "npx", "args": ["-y", "cto-ai-cli"] } } }
 ```
 
-Tools available: `cto_analyze`, `cto_select_context`, `cto_score`, `cto_benchmark`, `cto_risk`, and more.
+MCP output is also auto-sanitized when `includeContents: true`.
 
----
+## How it works
+
+1. **Dependency graph** — parses imports, builds adjacency list, identifies hubs
+2. **Risk scoring** — complexity × centrality × recency (continuous, log-scaled)
+3. **TF-IDF/BM25 semantic matching** — task description scored against all file contents + path boosting
+4. **Composite ranking** — `finalScore = risk × 0.4 + semantic × 0.4 + learner × 0.2`
+5. **Greedy allocation** — fills token budget top-down, cascading prune levels (full → signatures → skeleton)
+6. **Bayesian learning** — exponential decay on priors, Wilson score confidence, per-task-type patterns
+
+No AI is used for selection. Same input → same output. Deterministic.
 
 ## Programmatic API
 
 ```typescript
-import { analyzeProject, computeContextScore, selectContext } from 'cto-ai-cli';
+import { analyzeProject, selectContext, buildIndex, query } from 'cto-ai-cli';
 
-// Analyze a project
 const analysis = await analyzeProject('./my-project');
+const index = buildIndex(files);
+const semanticScores = query(index, 'fix auth', 50)
+  .map(m => ({ filePath: m.filePath, score: m.score }));
 
-// Get the Context Score
-const score = await computeContextScore(analysis);
-console.log(`Score: ${score.overall}/100 (${score.grade})`);
-console.log(`Tokens saved: ${score.comparison.savedPercent}%`);
-
-// Select optimal files for a task
 const selection = await selectContext({
-  task: 'refactor the auth middleware',
+  task: 'fix auth',
   analysis,
-  budget: 50_000,  // 50K token budget
+  budget: 50_000,
+  semanticScores,  // wired into ranking
 });
-
-console.log(`Selected ${selection.files.length} files`);
-console.log(`Coverage: ${selection.coverage.score}%`);
-for (const file of selection.files) {
-  console.log(`  ${file.relativePath} (${file.tokens} tokens, risk: ${file.riskScore})`);
-}
 ```
 
----
-
-## How It Works
-
-1. **Scan** — walks your project, parses imports, builds a dependency graph
-2. **Score** — computes risk for each file (complexity, hub score, centrality, recency)
-3. **Select** — deterministic greedy algorithm: picks highest-risk files first within token budget
-4. **Prove** — measures coverage (% of important files included), compares vs naive strategies
+## Honest limitations
 
-No AI is used for selection. Same input always produces the same output. Fully reproducible.
-
----
-
-## Honest Limitations
-
-- **TypeScript/JavaScript gets the deepest analysis.** Other languages (Python, Go, Rust, Java) get basic file + import analysis.
-- **Benchmarks use simple baselines** (alphabetical, random). We haven't compared against Cursor's or Copilot's internal context selection.
-- **Savings are estimates** based on average API pricing. Actual savings depend on your model and usage.
-- **Risk scoring uses a complexity proxy** instead of real git churn data (planned improvement).
-
----
+- **TypeScript/JavaScript gets deep analysis.** Other languages get basic file + import analysis.
+- **TF-IDF, not embeddings.** Handles most tasks well but won't understand complex intent.
+- **Learning needs ~5 feedback cycles** to start influencing selection. First runs are pure graph + risk + semantic.
+- **Not compared against Cursor/Copilot internal context.** Our baselines are naive (alphabetical, random).
 
 ## Contributing
 
 ```bash
-git clone https://github.com/cto-ai/cto-ai-cli.git
-cd cto-ai-cli
-npm install
-npm run build
-npm test              # 376 tests
-npm run typecheck     # strict TypeScript, zero errors
+git clone https://github.com/cto-ai/cto-ai-cli.git && cd cto-ai-cli
+npm install && npm run build && npm test  # 597 tests
 ```
 
-Full API docs, MCP server reference, and architecture are in [DOCS.md](DOCS.md).
-
 ## License
 
 [MIT](LICENSE)