aiforcecli 0.4.0 → 0.4.1

package/README.md

@@ -1,305 +0,0 @@
-# aiforcecli
-
-**One interface over every coding agent — that you can trust and afford.**
-
-`aiforcecli` wraps multiple coding agents — **Claude Code**, **OpenAI Codex**, **Aider**
-(which itself routes to DeepSeek, Mistral/Codestral, Qwen, Ollama, OpenRouter, and more), and
-**Antigravity** (Google's `agy`, for Gemini models) — behind a single normalized interface, and
-adds the things a single-agent CLI structurally can't:
-
-- 🧭 **Advise** — instantly recommend the best agent+model for a task (public benchmarks blended with *your* results), before you spend a cent.
-- 🏁 **Race** — run several agents in parallel in isolated git worktrees, **verify each against your tests**, and apply the winner.
-- 🔁 **Self-heal** — run → verify → on failure feed the error back and escalate to a stronger agent until green.
-- 📊 **Bench / Eval** — record every outcome and learn which agent wins which task, per dollar, on *your* codebase.
-- 💸 **First-class FinOps** — every run's token usage and cost recorded locally; hard budgets enforced before and during a run.
-
-`aiforcecli` does **not** reimplement any agent. It shells out to the agent CLIs you already
-have installed and normalizes their I/O behind one adapter interface — so it improves as the
-agents do, and you keep your existing auth and subscriptions.
-
-## 60-second quickstart
-
-```bash
-# 1. Install
-npm install -g aiforcecli          # or: npx aiforcecli <command>
-
-# 2. Make sure at least one underlying agent CLI is installed
-npm install -g @anthropic-ai/claude-code   # Claude Code
-npm install -g @openai/codex               # Codex
-aiforcecli agents                                  # shows install status
-
-# 3. Scaffold config in your project
-cd my-project
-aiforcecli init
-
-# 4. Run a task
-aiforcecli run "add a health check endpoint and a test for it"
-
-# 5. See what it cost
-aiforcecli cost
-```
-
-That's it — you're productive on day one.
-
-## Commands
-
-| Command | What it does |
-| --- | --- |
-| `aiforcecli run "<task>"` | Run a task with an agent. With no `--agent`, `aiforcecli` **auto-routes** to the best agent+model for the task within budget. Add `--heal` to **verify and self-heal** (see below). Flags: `--agent claude-code\|codex\|aider\|antigravity`, `--model <m>`, `--cwd <path>`, `--budget <usd>`, `--route deterministic\|bayesian`, `--explore`/`--no-explore` (bayesian), `--explain`, `--resume <sessionId>`, `--json`, `--heal`, `--max-attempts <n>`, `--verify <cmd>`, `--skip-verify`. |
-| `aiforcecli advise "<task>"` | **Recommend** which agent + model to use — instantly, without running anything. Blends a public benchmark scorecard with your private outcomes. `--cwd <path>`, `--budget <usd>` (preview under a cap), `--explore`, `--json`. |
-| `aiforcecli race "<task>"` | Run the task on **several agents in parallel** (isolated git worktrees), verify each, and apply the winner. Flags: `--agents claude-code,codex`, `--budget <usd>` (total, split across agents), `--select cheapest\|fastest\|first-pass`, `--verify <cmd>`, `--keep`, `--json`. |
-| `aiforcecli eval` | Run the **private eval suite** to calibrate `advise` on your codebase. `--dir <path>`, `--agents <ids>`, `--json`. |
-| `aiforcecli bench` | **Leaderboard** from your local history: which agent/model wins which task class, per dollar. `--since 7d\|24h\|<ISO>`, `--by-task`, `--json`. |
-| `aiforcecli agents` | List agents and whether their CLI is installed (`--json`). |
-| `aiforcecli cost` | Report spend. `--since 7d\|24h\|<ISO>`, `--by day\|agent\|project`, `--json`. |
-| `aiforcecli init` | Scaffold `aiforcecli.config.json` and install bundled assets. `--force` to overwrite. |
-| `aiforcecli mcp` | (Phase 3) Start the MCP server. |
-
-## How it works
-
-```
-CLI ── orchestrator ── AgentAdapter ── subprocess ── claude / codex
-            │
-            ├── BudgetEnforcer  (pre-run daily cap + mid-run per-run cap)
-            └── UsageStore      (SQLite: every run tagged agent/project/time/promptHash)
-```
-
-Each adapter parses its agent's **native** output (Claude Code `--output-format
-stream-json`, Codex `exec --json` JSONL) and maps it onto one normalized event shape:
-
-```ts
-type NormalizedEvent =
-  | { type: 'token';     text: string }
-  | { type: 'message';   role: 'assistant' | 'user'; text: string }
-  | { type: 'tool_call'; name: string; input?: unknown; id?: string }
-  | { type: 'usage';     usage: Usage; cumulative: boolean }
-  | { type: 'error';     message: string; fatal: boolean };
-```
-
-A run resolves to `{ message, usage: { inputTokens, outputTokens, costUsd, costSource },
-exitCode, sessionId, aborted }`.
-
-## Routing
-
-Instead of hardcoding which agent and model to call, omit `--agent` and `aiforcecli`
-picks for you. It classifies the task (a transparent keyword + length
-heuristic), then selects the **most relevant** model whose estimated cost fits
-the effective budget — the tightest of `--budget`, `maxCostPerRunUsd`, and the
-remaining daily headroom (`dailyCapUsd − spent today`):
-
-```
-CLI ── router ── classifyTask  (light | standard | heavy)
-          │
-          ├── model catalog  (agent, model, tier, price)
-          └── effectiveBudget (min of --budget / per-run cap / daily headroom)
-```
-
-- A model that **meets** the task tier and fits the budget is preferred.
-- If budget is too tight, it **downgrades** to the most capable model that fits.
-- If nothing fits, it picks the cheapest and warns — the budget enforcer still
-  aborts the run if it overruns.
-
-```bash
-aiforcecli run "fix a typo in the README"                  # → cheap, light model
-aiforcecli run "refactor the auth architecture" --budget 5 # → top model within $5
-aiforcecli run "<task>" --explain                           # show the decision, don't run
-```
-
-An explicit `--agent` (and/or `--model`) always wins and skips routing. Tune via
-the `routing` block in config: `enabled`, `prefer` (tie-break agent order),
-`only` (restrict to catalog keys), and `models` (add custom catalog entries).
-
-## Advise — pick the right agent before you start
-
-Before a complex project, `advise` gives you an instant, no-cost recommendation of which agent and
-model to use — the cheap predictor that complements `race` (proof) and `run --heal` (verify):
-
-```bash
-aiforcecli advise "migrate the auth module from sessions to JWT across the API"
-```
-
-```
-Task brief
-  type        security  (auth, security)
-  complexity  heavy
-  codebase    TypeScript · 412 files · tests: yes (npm test)
-  budget      $4.21 headroom
-
-Recommendation
-  → claude-code · opus    confidence 82%
-    expected ~$0.90 · ~88% one-shot pass
-  Why
-    • public eval: claude-code·opus ~88 on security
-    • your history: 91% pass over 9 security run(s)
-  Runner-up  codex·gpt-5-codex  ~$0.40 · ~80% — cheaper
-  Run it:    aiforcecli run "<task>" --agent claude-code --model opus --heal
-```
-
-**How the recommendation is scored.** A transparent blend:
-
-```
-capability = posterior(prior = public scorecard, evidence = your private pass rate)
-score      = wCapability·capability + wCost·costFit + wSpeed·speedFit
-```
-
-- **Public scorecard** (`src/advise/scorecard.ts`) — curated, dated priors per agent·model × task
-  type, distilled from public coding benchmarks. A *prior*, not gospel; it goes stale, so your own
-  data overrides it.
-- **Private outcomes** — your recorded `--heal`/`race`/`eval` results. Capability is a Bayesian
-  blend: the public prior dominates until you have evidence, then your numbers take over
-  (`advise.privatePseudocount` sets how fast). Confidence rises with your sample size.
-
-### The learning policy (contextual bandit)
-
-Under the hood the recommendation is a **contextual bandit**. Each arm `(agent, model)` has a Beta
-posterior over its pass probability — seeded by the public prior, updated by your verified outcomes
-for that task type. `advise` shows the **policy mix**: how often the policy would pick each arm.
-
-- **Exploit (default):** recommend the top expected arm.
-- **Explore (`advise --explore`, or `advise.explore: true` for routed `run`):** pick via **Thompson
-  Sampling** — sample each arm's posterior and take the winner — so uncertain-but-promising arms get
-  tried occasionally. That's how the policy *discovers* which agent quietly wins on your code instead
-  of only exploiting the current best.
-
-**Off-policy logging (so it keeps learning):** every verified run records its decision **context**,
-the chosen arm's **propensity** (selection probability), and a scalar **reward** (pass, minus
-cost/latency/attempt penalties, plus acceptance). That turns your run history into a labeled dataset
-for unbiased off-policy improvement — the foundation for a learned escalation/route policy next.
-The reward signal is automatic and objective because it comes from **your tests**, which is what
-makes the learning real rather than guesswork.
-
-**Calibrate it to your codebase** with the private eval suite. Drop task cases as JSON files in
-`.aiforcecli/evals/`:
-
-```json
-{ "prompt": "fix the off-by-one in parseRange", "verify": "npm test", "taskType": "bugfix" }
-```
-
-Then run them against every candidate agent·model (each in an isolated worktree, verified):
-
-```bash
-aiforcecli eval                      # all installed agents × all cases
-aiforcecli eval --agents claude-code # restrict the field
-```
-
-Results are recorded as private evidence, so `advise` (and `bench`) get smarter about *your* repo
-with every run. `eval` runs real agents — it costs money and time, so run it occasionally.
-
-## Verify, self-heal & race
-
-aiforcecli sits *above* the agents, so it can do something they can't: run the work, **verify it
-against your tests**, and act on the result. Verification is auto-detected from the project
-(`npm test`, `pytest`, `cargo test`, `go test ./...`, …) and overridable via `verify.command` or
-`--verify "<cmd>"`.
-
-**Self-healing runs** — `aiforcecli run "<task>" --heal`
-
-```
-run → verify → ✗ fail → retry same agent with the failure → ✗ → escalate to a stronger agent → ✓
-```
-
-After the run, aiforcecli runs your verify command. On failure it feeds the output back to the agent
-(resuming the session), and if it still fails it **escalates to a stronger agent/model** — looping
-until green, `heal.maxAttempts` is reached, or the budget runs out. A budget breach never escalates.
-
-**Race** — `aiforcecli race "<task>" --agents claude-code,codex`
-
-Runs the task on every agent **in parallel**, each in its own throwaway **git worktree** (your
-uncommitted changes are carried in, so they build on your current work). Each result is verified, and
-the **cheapest passing** diff (or `--select fastest|first-pass`) is applied to your tree — the tests
-decide, not vibes. With no verify command, you get each candidate's diff and pick one. `--budget` is
-the *total* and is split across racers; `--keep` leaves the worktrees for inspection.
-
-A fresh worktree only has git-tracked files, so the repo's `node_modules` is **symlinked** into each
-one automatically — otherwise `npm test` couldn't find its runner. For other ecosystems whose deps
-are gitignored, list them under `race.linkPaths` (e.g. `[".venv", "target", "vendor"]`).
-
-```bash
-aiforcecli race "fix the failing auth test" --agents claude-code,codex --budget 2
-#   claude-code · sonnet   ✓ pass   42s   $0.31
-#   codex · gpt-5-codex    ✗ fail   51s   $0.27
-#   → applied claude-code's diff (cheapest passing)   total race cost: $0.58
-```
-
-**Leaderboard** — `aiforcecli bench`
-
-Every `--heal` and `race` run records an *outcome* (passed? won? cost? duration?). `bench` reports,
-from your own history, which agent/model actually wins which task class per dollar — turning routing
-from anecdote into evidence. Opt in to `telemetry` to contribute anonymized outcomes to a shared
-leaderboard (off by default; never sends prompts, paths, or output).
-
-## FinOps & budgets
-
-- **Cost source of truth:** `aiforcecli` uses the cost the agent CLI reports when available
-  (Claude Code reports `total_cost_usd`). When a CLI reports tokens only (Codex), cost is
-  **computed** from a local pricing table — `costSource` records which, so reports are
-  never silently wrong.
-- **Window caps (`dailyCapUsd`, `weeklyCapUsd`, `monthlyCapUsd`):** checked *before* a run
-  starts. If spend in any window already meets its cap (today / this calendar week from Monday /
-  this calendar month), the run is refused (exit code `2`). They also tighten the routing budget.
-- **Per-run cap (`maxCostPerRunUsd`, or `--budget`):** enforced *during* the run. As usage
-  events arrive, if cumulative cost crosses the cap the subprocess is aborted
-  (SIGTERM → SIGKILL).
-  - **Honest limitation:** agents that only report usage at the *end* of a run can only
-    be caught after the fact. Mid-run enforcement is best-effort for those.
-- **Never hangs:** every run has a wall-clock `timeoutMs` and an `inactivityTimeoutMs`
-  watchdog; a stuck subprocess is always killed.
-
-## Configuration
-
-`aiforcecli.config.json` (or `.ts`) is discovered at two levels, deep-merged (**project wins**), then
-CLI flags win over both:
-
-1. **User level** — your defaults, applied in *every* folder. Put the file here once; you don't need
-   one per project. The directory is platform-specific:
-   - **Windows:** `%APPDATA%\aiforcecli\Config\aiforcecli.config.json`
-   - **macOS:** `~/Library/Preferences/aiforcecli/aiforcecli.config.json`
-   - **Linux:** `~/.config/aiforcecli/aiforcecli.config.json` (or `$XDG_CONFIG_HOME`)
-2. **Project level** — `aiforcecli.config.json` in the working directory, for per-project overrides only.
-
-The CLI command itself (installed globally via `npm i -g aiforcecli`, or `npm link` for a dev build)
-is separate from config — installing it once is enough; settings come from the files above. See
-[`aiforcecli.config.example.json`](./aiforcecli.config.example.json) and the `examples/` directory.
-
-| Block | Key | Default | Purpose |
-| --- | --- | --- | --- |
-| `agents.<id>` | `enabled`, `model`, `bin`, `defaultFlags`, `allowedTools` | `enabled:true` | Per-agent settings. `enabled:false` removes the agent everywhere. Plus binary path, forced model, default flags, tool allow-list. |
-| `defaultAgent` | | `claude-code` | Agent used by `run` when routing is off and no `--agent`. |
-| `routing` | `enabled`, `strategy`, `prefer`, `only`, `models` | `enabled:true`, `strategy:deterministic` | Auto-routing. `strategy`: `deterministic` (budget-aware tier router) or `bayesian` (learned engine; honors `advise.explore`). `only` restricts the catalog, `models` adds entries. Per-run override: `run --route`. |
-| `verify` | `enabled`, `command`, `timeoutMs` | `enabled:true`, 300 s | How results are verified; `command` overrides auto-detection. |
-| `heal` | `enabled`, `maxAttempts`, `escalate` | `false`, `3`, `true` | Self-healing loop behavior for `run --heal`. |
-| `race` | `agents`, `select`, `keepWorktrees`, `linkPaths` | `select:cheapest` | Defaults for `race`; `linkPaths` symlinks extra dep dirs into worktrees. |
-| `advise` | `weights`, `privatePseudocount`, `explore` | `0.7/0.2/0.1`, `5`, `false` | Recommendation scoring weights, prior strength, and Thompson-Sampling exploration. |
-| `eval` | `dir`, `models` | `.aiforcecli/evals` | Private eval suite location and which catalog models to evaluate. |
-| `telemetry` | `enabled`, `endpoint` | `false` | Opt-in anonymized outcome upload for the shared leaderboard. |
-| `budgets` | `maxCostPerRunUsd`, `dailyCapUsd`, `weeklyCapUsd`, `monthlyCapUsd` | — | Per-run cap + daily/weekly/monthly spend caps (refuse to start once a window cap is hit). |
-| top-level | `timeoutMs`, `inactivityTimeoutMs` | 600 s, 120 s | Wall-clock and inactivity watchdogs for every run. |
-
-## Development
-
-```bash
-npm install
-npm run build       # tsup -> dist/ (ESM)
-npm test            # vitest — adapter parsers run against recorded fixtures
-npm run typecheck
-```
-
-Adapter parser tests use **recorded real outputs** from each CLI under
-`test/fixtures/`. To refresh them, re-capture with:
-
-```bash
-claude -p "say hi" --output-format stream-json --verbose > test/fixtures/claude-code/hello.stream.jsonl
-codex exec --json --sandbox workspace-write "say hi"     > test/fixtures/codex/hello.jsonl
-```
-
-## Documentation
-
-- [Feature Reference (FEATURES)](./docs/FEATURES.md) — every feature in depth: public eval, the RL/bandit policy, race, heal, FinOps, config, data model.
-- [Product Requirements (PRD)](./docs/PRD.md) — problem, users, features, scope, success metrics.
-- [Business Requirements (BRD)](./docs/BRD.md) — market, value proposition, model, KPIs.
-
-## Roadmap
-
-- **Shipped:** adapters (claude-code, codex), config, FinOps + budgets, budget-aware routing, `run` (+ `--heal`), `race`, `advise` (**contextual-bandit policy + off-policy logging**), `eval`, `bench`, `cost`, `agents`, `init`.
-- **Next:** learned escalation/route policy trained from the logged (context, arm, reward, propensity) data; surface agent fatal errors in `race`/`eval`; richer task classification (`--deep`); bundled `assets/` via `init`.
-- **Later:** `aiforcecli mcp` MCP server with auth + hard budget cap; hosted shared leaderboard + federated global/local policy; team/org budgets and reporting.