agent-ultramode 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +194 -0
  3. package/cli.mjs +262 -0
  4. package/package.json +46 -0
  5. package/ultra.ts +381 -0
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 maverick-tr (https://github.com/maverick-tr)
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md ADDED
@@ -0,0 +1,194 @@
1
+ # ultra
2
+
3
+ Took the [LLM-as-a-Verifier](https://github.com/llm-as-a-verifier/llm-as-a-verifier) paper (Kwok et al., 2026) and turned it into a simple `/ultra` command for coding agents.
4
+
5
+ Instead of shipping the agent's first attempt, `ultra` runs your task **N times in isolated git worktrees**, in parallel, then uses the **same model** as a verifier to pick the best result. When it is confident (or when most attempts succeeded) it applies the winning diff to your working tree; when the attempts diverge and it is not sure, it hands you the top candidates. No cross-model dependency, no extra services, no first-attempt lottery.
6
+
7
+ And unlike most "add a verifier" posts, I benchmarked it before believing it. The receipts are below, good and bad.
8
+
9
+ ## Demo
10
+
11
+ <video src="https://github.com/maverick-tr/agent-ultramode/raw/main/ultramode.mp4" controls muted playsinline width="100%"></video>
12
+
13
+ > For a reliable inline player on GitHub, edit this README on github.com and drag `ultramode.mp4` into the editor; GitHub inserts a `https://github.com/.../assets/...` URL that plays inline. The file is committed at [`ultramode.mp4`](./ultramode.mp4).
14
+
15
+ ## The receipts
16
+
17
+ Benchmarked on **Terminal-Bench** with **DeepSeek V4 Flash 0731** as both the agent and the verifier, running inside **opencode** (N=5 attempts, same-model verifier). Verified end to end with both **OpenCode** and **Claude Code** as the agent (the numbers above are the OpenCode run; a Claude Code number will be added once benchmarked). Works with any agent, not just opencode.
18
+
19
+ | Slice | base@1 (single shot) | ultra (best-of-N + verify) | oracle@5 (ceiling) |
20
+ |---|:---:|:---:|:---:|
21
+ | All 15 tasks | 24% | **33%** | 40% |
22
+ | 4 recoverable tasks | 40% | **75%** | rescued 3 of 4 |
23
+
24
+ Overall the verifier captured about **56% of the pass@1 to pass@5 headroom**. On the tasks with real variance, where best-of-N can actually help, it took the passing attempt on `chess-best-move`, `new-encrypt-command`, and `decommissioning-service` (missing only `jupyter-notebook-server`). A 40 to 75 jump on recoverable tasks is not a rounding error.
25
+
26
+ ## How the competition works
27
+
28
+ ```text
29
+ your task
30
+ |
31
+ +----------+-----------+-----------+----------+ N attempts, in parallel,
32
+ | | | | | each in its own git worktree
33
+ worktree1 worktree2 worktree3 worktree4 ... + a lean sandbox (no MCP,
34
+ | | | | | no plugins, isolated state)
35
+ agent agent agent agent agent
36
+ | | | | |
37
+ diff1 diff2 diff3 diff4 diffN
38
+ +----------+-----------+-----------+----------+
39
+ |
40
+ v
41
+ Probabilistic Pivot Tournament (same model, acting as verifier)
42
+ 1. ring pass each diff judged once vs a neighbour -> seed pivots
43
+ 2. pivot duels field vs the top pivots, K reasoned votes per duel
44
+ 3. score normalised win-rate; confidence = top1 minus top2
45
+ |
46
+ v
47
+ decide
48
+ confident, or most attempts changed something -> APPLY the winner
49
+ diverged and not confident -> show top candidates
50
+ ```
51
+
52
+ 1. **Fan out.** N detached git worktrees off `HEAD`; the agent runs the full task in each, in parallel, isolated so the attempts never collide.
53
+ 2. **Run each attempt lean.** Every attempt is a full agent process, but stripped to essentials (no MCP servers, no other plugins, its own private state). That is what makes N real agents in parallel take seconds instead of minutes.
54
+ 3. **Verify.** The tournament ranks the N diffs by same-model reasoned votes. Confidence is the win-rate margin between the top two.
55
+ 4. **Decide.** Confident or a clear majority produced a change: apply the winner to your working tree (uncommitted, you review before committing). Otherwise: present the top candidates instead of guessing.
56
+
57
+ Progress streams live: phase titles in the web UI tool card, and toasts in the terminal.
58
+
59
+ ## Install
60
+
61
+ ```sh
62
+ opencode plugin agent-ultramode
63
+ ```
64
+
65
+ That is the whole setup. With no options it drafts and verifies with your **current session model**, so `/ultra <task>` just works:
66
+
67
+ ```text
68
+ /ultra fix the failing test in foo/bar
69
+ ```
70
+
71
+ `ultra` branches attempts off `HEAD`, so run it inside a git repo with at least one commit.
72
+
73
+ **Pin a specific model (optional).** To draft and verify with a model other than the session one:
74
+
75
+ ```jsonc
76
+ { "plugin": [ ["agent-ultramode", { "model": "myprovider/my-model" }] ] }
77
+ ```
78
+
79
+ **Run a different agent (Claude Code, Grok, Pi, Codex, ...).** The `agent` option is the command run once per attempt, with `{task}` substituted and the cwd set to an isolated worktree. Point it at any CLI that edits files and exits:
80
+
81
+ ```jsonc
82
+ ["agent-ultramode", { "agent": "claude -p --dangerously-skip-permissions \"{task}\"" }] // Claude Code (verified)
83
+ ["agent-ultramode", { "agent": "pi \"{task}\"" }] // pi
84
+ ["agent-ultramode", { "agent": "grok build \"{task}\"" }] // grok
85
+ ```
86
+
87
+ Adjust each agent's flags for headless, file-editing runs (for example Claude Code's permission mode). The best-of-N and verify loop stays the same, which makes ultra-mode easy to push onto any agent you use.
88
+
89
+ ## Configure
90
+
91
+ Every option also reads from an `ULTRA_*` env var.
92
+
93
+ | Option | Default | Meaning |
94
+ |---|---|---|
95
+ | `model` | your current session model | `"provider/model-id"` used to run attempts and to verify |
96
+ | `agent` | run the current model | the command run once per attempt; `{task}` substituted, cwd is an isolated worktree |
97
+ | `n` | `4` | number of attempts (2 to 8) |
98
+ | `k` | `3` | reasoned votes per verifier duel |
99
+ | `conf` | `0.34` | confidence margin at or above which the winner is applied |
100
+ | `baseURL` / `apiKey` | from the provider | override the OpenAI-compatible endpoint / key for the verifier |
101
+ | `effort` | `"none"` | `reasoning_effort` for the verifier calls (kept low so judging stays fast) |
102
+ | `concurrency` | `6` | how many attempts run at once |
103
+ | `agentTimeout` | `600000` | per-attempt timeout in ms |
104
+
105
+ ## Use without opencode (the CLI)
106
+
107
+ Same loop, no opencode host required. Install and run it in any git repo:
108
+
109
+ ```sh
110
+ npm i -g agent-ultramode # or: npx agent-ultramode "<task>"
111
+ ```
112
+
113
+ ```sh
114
+ agent-ultramode "fix the failing test in foo/bar"
115
+ ```
116
+
117
+ By default each attempt runs `opencode run` (and it tells you clearly if opencode is not installed). Point `--agent` at anything else that edits files:
118
+
119
+ ```sh
120
+ agent-ultramode -t "add rate limiting to /login" \
121
+ --agent 'claude -p --dangerously-skip-permissions "{task}"' \
122
+ --verify-model gpt-4o-mini
123
+ ```
124
+
125
+ **Multiple models in one pass.** Pass `--agent` more than once to spread the attempts across different models, judged by one neutral verifier:
126
+
127
+ ```sh
128
+ agent-ultramode -t "design and implement the data migration" \
129
+ --agent 'claude -p --dangerously-skip-permissions "{task}"' \
130
+ --agent 'opencode run --model grok "{task}"' \
131
+ --agent 'opencode run --model deepseek-v4-flash "{task}"' \
132
+ --n 6 --verify-model gpt-4o-mini
133
+ ```
134
+
135
+ The 6 attempts round-robin across the three agents, and the verifier picks the best regardless of which model produced it. Good for one hard task where you want several strong models to take a shot.
136
+
137
+ The verifier needs an OpenAI-compatible endpoint: by default it reads `OPENAI_API_KEY` and `OPENAI_BASE_URL`, or pass `--api-key` / `--base-url` / `--verify-model`. Run `agent-ultramode --help` for all options.
138
+
139
+ ## Why I built this, and the honest story
140
+
141
+ I wanted to know if best-of-N verification could squeeze real quality out of a cheap-but-capable model without reaching for a bigger, pricier one. So, ran it on real Terminal-Bench tasks and let the numbers decide. What I found:
142
+
143
+ - **Planning-first best-of-N does nothing for execution tasks.** Drafting N plans, picking the best, then executing once: **0 out of 5** outcomes changed on Terminal-Bench, at 3 to 10 times the cost. It amplifies effort, not capability. I dropped it.
144
+ - **Best-of-N over full trajectories plus a same-model verifier does work.** This is the version that ships.
145
+ - **A fancier verifier did not help.** A multi-criteria checklist tied the plain holistic judge and sometimes did worse. Same-model verification hits a ceiling that more prompting does not break.
146
+ - **Confidence is a real signal, but a noisy one.** Useful, not a magic gate.
147
+
148
+ It significantly helps on the tasks that matter, and I can point at the per-task data.
149
+
150
+ ## The honest caveats (because benchmarks lie by omission)
151
+
152
+ 1. **Confidence is a noisy signal at this scale.** The one miss (`jupyter`) had confidence 0.25, higher than two tasks it got right (0.17, 0.21). Some beyond-capability tasks even show high confidence while being pure failures. "Trust when confident" is directionally real, not a clean gate.
153
+ 2. **It cannot help beyond-capability tasks.** 9 of the 15 never passed in 5 tries, and no verifier manufactures a solution that was never there.
154
+ 3. **n=15 (4 with real variance) is a credible but modest sample.** Enough to headline honestly, not enough to over-claim precision.
155
+
156
+ ## What this likely means at scale
157
+
158
+ The 15 tasks were deliberately failure-skewed, so the +9 points is not what you would see on the whole benchmark. On the full set the model already passes about 83%, so most tasks have no headroom. My honest estimate for a full run is a **modest +2 to +5 points on average**, because the dramatic gains only land on the minority of tasks the model *sometimes* solves. Average lift small, per-recoverable-task lift large. Same fact, two views.
159
+
160
+ ## Roadmap
161
+
162
+ - [x] **Standalone CLI** (`npx agent-ultramode`) so the loop runs anywhere, with any agent, no opencode required.
163
+ - [x] **Multiple models in one pass** (repeatable `--agent`): spread attempts across different models, one neutral verifier picks the best.
164
+ - [x] **OpenCode and Claude Code** verified end to end.
165
+ - [ ] **First-class agent integrations** with tuned defaults, and a benchmark number, for Claude Code, Grok, Pi, and Codex.
166
+ - [ ] Native slash-command or MCP packaging per agent. Contributions welcome.
167
+
168
+ ## Credits
169
+
170
+ The verification method is the **Probabilistic Pivot Tournament** from **LLM-as-a-Verifier** (Kwok et al., 2026), reimplemented from the paper. The `K`-sample reward estimate and the apply policy are our adaptations. The self-verification ceiling we ran into is documented in the cross-model and weak-verifier literature, worth reading before you assume same-model verification is a free lunch:
171
+
172
+ - Paper: [arXiv:2607.05391](https://arxiv.org/abs/2607.05391)
173
+ - Repo: [llm-as-a-verifier/llm-as-a-verifier](https://github.com/llm-as-a-verifier/llm-as-a-verifier)
174
+ - On the ceiling: [LLM-as-a-Jury (cross-model)](https://arxiv.org/html/2607.10139), [Weaver: weak verifiers](https://arxiv.org/html/2506.18203), [Generative Verifiers](https://arxiv.org/pdf/2408.15240)
175
+
176
+ ```bibtex
177
+ @misc{kwok2026llmasaverifiergeneralpurposeverificationframework,
178
+ title={LLM-as-a-Verifier: A General-Purpose Verification Framework},
179
+ author={Jacky Kwok and Shulu Li and Pranav Atreya and Yuejiang Liu and
180
+ Yixing Jiang and Chelsea Finn and Marco Pavone and Ion Stoica
181
+ and Azalia Mirhoseini},
182
+ year={2026},
183
+ eprint={2607.05391},
184
+ archivePrefix={arXiv},
185
+ primaryClass={cs.AI},
186
+ url={https://arxiv.org/abs/2607.05391}
187
+ }
188
+ ```
189
+
190
+ Built for [opencode](https://opencode.ai). Zero heavy dependencies: opencode's bundled `@opencode-ai/plugin`, the global `fetch`, and an OpenAI-compatible endpoint.
191
+
192
+ ## License and author
193
+
194
+ MIT, by [maverick-tr](https://github.com/maverick-tr). Built on the method from [llm-as-a-verifier](https://github.com/llm-as-a-verifier/llm-as-a-verifier) (Kwok et al., 2026). See [LICENSE](./LICENSE).
package/cli.mjs ADDED
@@ -0,0 +1,262 @@
1
+ #!/usr/bin/env node
2
+ // agent-ultramode: best-of-N for coding agents with a same-model verifier, as a CLI.
3
+ //
4
+ // Runs your task N times in isolated git worktrees, verifies the diffs with a
5
+ // Probabilistic Pivot Tournament (the same-model verifier from llm-as-a-verifier),
6
+ // and applies the winner. No opencode host required. Defaults the per-attempt
7
+ // agent to `opencode run`, but `--agent` takes any CLI that edits files.
8
+ //
9
+ // Node 18+ (uses global fetch and node:util parseArgs). Zero dependencies.
10
+ import { execFile } from "node:child_process"
11
+ import { promisify } from "node:util"
12
+ import { mkdtemp, rm, writeFile, mkdir } from "node:fs/promises"
13
+ import { tmpdir } from "node:os"
14
+ import { join } from "node:path"
15
+ import { parseArgs } from "node:util"
16
+
17
+ const execFileP = promisify(execFile)
18
+ const int = (v, def, lo, hi) => { const n = parseInt(String(v ?? ""), 10); return Number.isFinite(n) ? Math.max(lo, Math.min(hi, n)) : def }
19
+ const err = (m) => process.stderr.write(m + "\n")
20
+
21
+ function printHelp() {
22
+ err(`agent-ultramode - best-of-N for coding agents with a same-model verifier
23
+
24
+ USAGE
25
+ agent-ultramode --task "<what to do>" [options]
26
+ agent-ultramode "<what to do>" [options]
27
+
28
+ OPTIONS
29
+ -t, --task <text> the task to solve (or pass it as positional words)
30
+ -a, --agent <cmd> command run per attempt; "{task}" substituted, cwd is an isolated
31
+ worktree. Repeatable: pass several to spread attempts across
32
+ different models in one pass (round-robin). default: opencode run "{task}"
33
+ -n, --n <int> number of attempts, 2 to 8 (default 4)
34
+ --k <int> reasoned votes per verifier duel (default 3)
35
+ --conf <float> confidence margin to auto-apply (default 0.34)
36
+ -m, --verify-model <id> model id for the verifier (default $ULTRA_VERIFY_MODEL or gpt-4o-mini)
37
+ --base-url <url> verifier OpenAI-compatible endpoint (default $OPENAI_BASE_URL or OpenAI)
38
+ --api-key <key> verifier api key (default $OPENAI_API_KEY)
39
+ --repo <path> repo to run in (default: current directory)
40
+ --concurrency <int> attempts to run at once (default 6)
41
+ --effort <level> verifier reasoning_effort (default none)
42
+ -h, --help show this help
43
+
44
+ EXAMPLES
45
+ # default: opencode as the per-attempt agent, verify with your OpenAI key
46
+ agent-ultramode "fix the failing test in foo/bar"
47
+
48
+ # use Claude Code as the agent
49
+ agent-ultramode -t "add rate limiting to /login" \\
50
+ --agent 'claude -p --dangerously-skip-permissions "{task}"' \\
51
+ --verify-model gpt-4o-mini
52
+
53
+ # point the verifier at any OpenAI-compatible endpoint
54
+ agent-ultramode "..." --base-url http://localhost:8000/v1 --api-key x --verify-model my-model
55
+ `)
56
+ }
57
+
58
+ async function git(cwd, ...args) {
59
+ const { stdout } = await execFileP("git", args, { cwd, maxBuffer: 32 * 1024 * 1024 })
60
+ return stdout
61
+ }
62
+
63
+ function pLimit(max) {
64
+ let active = 0
65
+ const q = []
66
+ const next = () => { active--; q.shift()?.() }
67
+ return async (fn) => {
68
+ if (active >= max) await new Promise((r) => q.push(r))
69
+ active++
70
+ try { return await fn() } finally { next() }
71
+ }
72
+ }
73
+
74
+ async function chat(C, content) {
75
+ const body = { model: C.verifyModel, messages: [{ role: "user", content }], max_tokens: 12000, temperature: 0.9 }
76
+ if (C.effort) body.reasoning_effort = C.effort
77
+ const res = await fetch(`${C.baseURL.replace(/\/$/, "")}/chat/completions`, {
78
+ method: "POST",
79
+ headers: { Authorization: `Bearer ${C.apiKey}`, "Content-Type": "application/json" },
80
+ body: JSON.stringify(body),
81
+ })
82
+ if (!res.ok) throw new Error(`verifier HTTP ${res.status}`)
83
+ const j = await res.json()
84
+ const m = j?.choices?.[0]?.message ?? {}
85
+ return `${m.content ?? ""}\n${m.reasoning_content ?? ""}`
86
+ }
87
+
88
+ async function judge(C, limit, task, a, b) {
89
+ const prompt =
90
+ `A coding task and two candidate solutions (agent log tail + the diff each produced). Decide which ` +
91
+ `more likely FULLY and correctly accomplishes the task. Be skeptical: prefer concrete evidence of a ` +
92
+ `correct, complete change with no errors; a diff that looks plausible but is incomplete or wrong should ` +
93
+ `lose.\n\nTASK:\n${task}\n\n=== A ===\n${a}\n\n=== B ===\n${b}\n\n` +
94
+ `Reason briefly, then end with exactly one line: 'FINAL: A' or 'FINAL: B'.`
95
+ try {
96
+ const out = (await limit(() => chat(C, prompt))).toUpperCase()
97
+ const ms = out.match(/FINAL:\s*([AB])/g)
98
+ return ms ? ms[ms.length - 1].slice(-1) : null
99
+ } catch { return null }
100
+ }
101
+
102
+ async function tournament(C, task, summaries) {
103
+ const n = summaries.length
104
+ const ids = Array.from({ length: n }, (_, i) => i)
105
+ if (n <= 1) return { ranked: ids, conf: 1 }
106
+ const limit = pLimit(C.cc)
107
+ const order = [...ids]
108
+ for (let i = order.length - 1; i > 0; i--) { const j = Math.floor(Math.random() * (i + 1)); [order[i], order[j]] = [order[j], order[i]] }
109
+ const ring = Object.fromEntries(ids.map((i) => [i, 0]))
110
+ await Promise.all(order.map(async (id, i) => {
111
+ const b = order[(i + 1) % n]
112
+ const v = await judge(C, limit, task, summaries[id], summaries[b])
113
+ ring[v === "B" ? b : id]++
114
+ }))
115
+ const pivots = [...ids].sort((x, y) => ring[y] - ring[x]).slice(0, Math.min(2, n))
116
+ const nonp = ids.filter((i) => !pivots.includes(i))
117
+ const pairs = []
118
+ for (const np of nonp) for (const pv of pivots) pairs.push([np, pv])
119
+ for (let i = 0; i < pivots.length; i++) for (let j = i + 1; j < pivots.length; j++) pairs.push([pivots[i], pivots[j]])
120
+ const mass = Object.fromEntries(ids.map((i) => [i, 0]))
121
+ const games = Object.fromEntries(ids.map((i) => [i, 0]))
122
+ await Promise.all(pairs.map(async ([x, y]) => {
123
+ const votes = await Promise.all(Array.from({ length: C.k }, () => judge(C, limit, task, summaries[x], summaries[y])))
124
+ const na = votes.filter((v) => v === "A").length
125
+ const nb = votes.filter((v) => v === "B").length
126
+ const px = na + nb === 0 ? 0.5 : na / (na + nb)
127
+ mass[x] += px; mass[y] += 1 - px; games[x]++; games[y]++
128
+ }))
129
+ const ratio = (i) => (games[i] ? mass[i] / games[i] : ring[i] / Math.max(1, n - 1))
130
+ const ranked = [...ids].sort((a, b) => ratio(b) - ratio(a))
131
+ return { ranked, conf: ratio(ranked[0]) - ratio(ranked[1]) }
132
+ }
133
+
134
+ async function main() {
135
+ const { values, positionals } = parseArgs({
136
+ allowPositionals: true,
137
+ options: {
138
+ task: { type: "string", short: "t" },
139
+ agent: { type: "string", short: "a", multiple: true },
140
+ n: { type: "string", short: "n" },
141
+ k: { type: "string" },
142
+ conf: { type: "string" },
143
+ "verify-model": { type: "string", short: "m" },
144
+ "base-url": { type: "string" },
145
+ "api-key": { type: "string" },
146
+ repo: { type: "string" },
147
+ concurrency: { type: "string" },
148
+ effort: { type: "string" },
149
+ help: { type: "boolean", short: "h" },
150
+ },
151
+ })
152
+
153
+ if (values.help) { printHelp(); process.exit(0) }
154
+ const task = (values.task || positionals.join(" ")).trim()
155
+ if (!task) { printHelp(); err("\nagent-ultramode: pass a --task (or task words) to run."); process.exit(1) }
156
+
157
+ const agents = values.agent && values.agent.length ? values.agent : ['opencode run "{task}"']
158
+ const n = int(values.n, 4, 2, 8)
159
+ const k = int(values.k, 3, 1, 7)
160
+ const conf = Number(values.conf ?? 0.34)
161
+ const cc = int(values.concurrency, 6, 1, 12)
162
+ const agentTimeout = 600000
163
+ const C = {
164
+ verifyModel: values["verify-model"] || process.env.ULTRA_VERIFY_MODEL || process.env.OPENAI_MODEL || "gpt-4o-mini",
165
+ baseURL: values["base-url"] || process.env.OPENAI_BASE_URL || process.env.ULTRA_BASE_URL || "https://api.openai.com/v1",
166
+ apiKey: values["api-key"] || process.env.OPENAI_API_KEY || process.env.ULTRA_API_KEY || "",
167
+ effort: values.effort ?? process.env.ULTRA_EFFORT ?? "none",
168
+ k, cc,
169
+ }
170
+
171
+ // Friendly check: an opencode agent must exist on PATH.
172
+ const binOf = (a) => a.trim().split(/\s+/)[0]
173
+ for (const b of [...new Set(agents.map(binOf))]) {
174
+ if (b !== "opencode") continue
175
+ try { await execFileP(b, ["--version"], { timeout: 10000 }) }
176
+ catch {
177
+ err(`\nagent-ultramode: '${b}' is not installed or not on your PATH.\n` +
178
+ `Install it, or pass --agent to use a different agent, for example:\n` +
179
+ ` --agent 'claude -p --dangerously-skip-permissions "{task}"'\n`)
180
+ process.exit(1)
181
+ }
182
+ }
183
+ if (!C.apiKey) {
184
+ err(`\nagent-ultramode: no verifier api key. The verifier needs an OpenAI-compatible model to judge attempts.\n` +
185
+ `Set OPENAI_API_KEY (and OPENAI_BASE_URL / --verify-model for a non-OpenAI endpoint), or pass --api-key.\n`)
186
+ process.exit(1)
187
+ }
188
+
189
+ const repoArg = values.repo || process.cwd()
190
+ let repo
191
+ try { repo = (await git(repoArg, "rev-parse", "--show-toplevel")).trim() } catch { err(`\nagent-ultramode: '${repoArg}' is not inside a git repo.`); process.exit(1) }
192
+ let base
193
+ try { base = (await git(repo, "rev-parse", "HEAD")).trim() } catch { err(`\nagent-ultramode: the repo has no commits yet (need a HEAD to branch attempts from).`); process.exit(1) }
194
+
195
+ const work = await mkdtemp(join(tmpdir(), "ultramode-"))
196
+ const worktrees = []
197
+ try {
198
+ const label = agents.length > 1 ? `${agents.length} agents (round-robin)` : `'${binOf(agents[0])}'`
199
+ err(`>>> fan-out: ${n} attempts off ${base.slice(0, 8)} with ${label} ...`)
200
+ for (let i = 0; i < n; i++) {
201
+ const wt = join(work, `attempt-${i}`)
202
+ await git(repo, "worktree", "add", "--detach", wt, base)
203
+ worktrees.push(wt)
204
+ }
205
+ const limit = pLimit(Math.max(1, Math.min(cc, n)))
206
+ const esc = task.replace(/"/g, '\\"')
207
+ const outs = await Promise.all(worktrees.map((wt, i) => limit(async () => {
208
+ const cmd = agents[i % agents.length].replace("{task}", esc)
209
+ const d = join(work, `xdg-${i}`)
210
+ await mkdir(d, { recursive: true }).catch(() => {})
211
+ const env = {
212
+ ...process.env,
213
+ XDG_DATA_HOME: d, XDG_STATE_HOME: d, XDG_CACHE_HOME: d,
214
+ OPENCODE_DISABLE_DEFAULT_PLUGINS: "1", OPENCODE_DISABLE_AUTOUPDATE: "1", OPENCODE_DISABLE_MODELS_FETCH: "1",
215
+ }
216
+ let log = ""
217
+ try {
218
+ const { stdout, stderr } = await execFileP("bash", ["-lc", `exec </dev/null; ${cmd}`], { cwd: wt, env, timeout: agentTimeout, maxBuffer: 32 * 1024 * 1024 })
219
+ log = (stdout || "") + (stderr || "")
220
+ } catch (e) { log = `agent error: ${e?.message || e}` }
221
+ await git(wt, "add", "-A").catch(() => {})
222
+ const diff = await git(wt, "diff", "--cached").catch(() => "")
223
+ err(` attempt ${i} [${binOf(agents[i % agents.length])}]: ${diff ? diff.length + " diff chars" : "no changes"}`)
224
+ return { diff, summary: `AGENT LOG (tail):\n${log.slice(-1500)}\n\nDIFF:\n${diff.slice(0, 6000) || "(no changes)"}` }
225
+ })))
226
+
227
+ const diffs = outs.map((o) => o.diff)
228
+ const summaries = outs.map((o) => o.summary)
229
+ if (diffs.every((d) => !d.trim())) {
230
+ console.log("agent-ultramode: none of the attempts made any changes. Try a more specific task, or check that your agent can edit files headlessly.")
231
+ return
232
+ }
233
+
234
+ err(`>>> verify: probabilistic pivot tournament (${C.verifyModel}) ...`)
235
+ const { ranked, conf: margin } = await tournament(C, task, summaries)
236
+ const best = ranked[0]
237
+ const nonEmpty = diffs.filter((d) => d.trim()).length
238
+ const majority = nonEmpty >= Math.max(2, Math.ceil(n / 2))
239
+
240
+ if ((margin >= conf || majority) && diffs[best].trim()) {
241
+ const patch = join(work, "winner.patch")
242
+ await writeFile(patch, diffs[best])
243
+ const why = margin >= conf ? `confidence ${margin.toFixed(2)}` : `top pick of ${nonEmpty}/${n}, margin ${margin.toFixed(2)}`
244
+ try {
245
+ await git(repo, "apply", "--3way", patch)
246
+ console.log(`\n🏆 applied the best of ${n} attempts (${why}). The change is in your working tree; review it, then commit.`)
247
+ } catch (e) {
248
+ console.log(`\nverified the best attempt (${why}) but the patch did not apply cleanly (${e?.message || e}). The diff:\n\n${diffs[best].slice(0, 8000)}`)
249
+ }
250
+ } else {
251
+ console.log(`\nattempts diverged and the verifier was not confident (margin ${margin.toFixed(2)}); applied nothing. Top candidates:\n`)
252
+ ranked.slice(0, 3).forEach((idx, r) => {
253
+ console.log(`# ${r + 1} attempt ${idx} (${diffs[idx].length} chars)\n${diffs[idx].slice(0, 4000) || "(no changes)"}\n`)
254
+ })
255
+ }
256
+ } finally {
257
+ for (const wt of worktrees) await git(repo, "worktree", "remove", "--force", wt).catch(() => {})
258
+ await rm(work, { recursive: true, force: true }).catch(() => {})
259
+ }
260
+ }
261
+
262
+ main().catch((e) => { err(`agent-ultramode: ${e?.message || e}`); process.exit(1) })
package/package.json ADDED
@@ -0,0 +1,46 @@
1
+ {
2
+ "name": "agent-ultramode",
3
+ "version": "0.1.0",
4
+ "description": "Best-of-N for coding agents with a same-model verifier: run your task N times in isolated git worktrees and apply the verified winner. A /ultra command for opencode.",
5
+ "module": "ultra.ts",
6
+ "main": "ultra.ts",
7
+ "exports": {
8
+ ".": "./ultra.ts"
9
+ },
10
+ "type": "module",
11
+ "bin": {
12
+ "agent-ultramode": "cli.mjs"
13
+ },
14
+ "engines": {
15
+ "node": ">=18"
16
+ },
17
+ "files": [
18
+ "ultra.ts",
19
+ "cli.mjs",
20
+ "README.md",
21
+ "LICENSE"
22
+ ],
23
+ "keywords": [
24
+ "opencode",
25
+ "opencode-plugin",
26
+ "plugin",
27
+ "best-of-n",
28
+ "verifier",
29
+ "llm-as-a-verifier",
30
+ "coding-agent",
31
+ "ultra"
32
+ ],
33
+ "license": "MIT",
34
+ "author": "maverick-tr (https://github.com/maverick-tr)",
35
+ "repository": {
36
+ "type": "git",
37
+ "url": "git+https://github.com/maverick-tr/agent-ultramode.git"
38
+ },
39
+ "devDependencies": {
40
+ "@opencode-ai/plugin": "^1.0.153",
41
+ "@types/bun": "latest"
42
+ },
43
+ "peerDependencies": {
44
+ "typescript": "^5"
45
+ }
46
+ }
package/ultra.ts ADDED
@@ -0,0 +1,381 @@
1
+ /**
2
+ * ultra - best-of-N trajectories + confidence-gated same-model verifier, for opencode.
3
+ *
4
+ * `/ultra <task>` runs the task N times in isolated git worktrees, ranks the results
5
+ * with the same model acting as a verifier (a Probabilistic Pivot Tournament, from
6
+ * llm-as-a-verifier), then:
7
+ * high confidence -> applies the winning diff to your working tree
8
+ * low confidence -> shows you the top candidates and applies nothing
9
+ * (a low-confidence pick is a coin flip and should not be applied silently).
10
+ *
11
+ * This is the version validated on Terminal-Bench: plan-first best-of-N gave no edge,
12
+ * but best-of-N over full trajectories + this verifier lifted the recoverable tasks
13
+ * from 40% to 75% (n=15, same model, no cross-model dependency).
14
+ *
15
+ * Install (opencode.json):
16
+ * { "plugin": [ ["file:///abs/path/ultra.ts", { "model": "provider/model", "n": 5 }] ] }
17
+ * Then: /ultra fix the failing test in foo/bar
18
+ *
19
+ * Zero deps: node built-ins + global fetch + an OpenAI-compatible endpoint.
20
+ */
21
+ import type { Plugin } from "@opencode-ai/plugin"
22
+ import { tool } from "@opencode-ai/plugin"
23
+ import { execFile } from "node:child_process"
24
+ import { promisify } from "node:util"
25
+ import { mkdtemp, rm, writeFile, mkdir } from "node:fs/promises"
26
+ import { existsSync } from "node:fs"
27
+ import { tmpdir, homedir } from "node:os"
28
+ import { join } from "node:path"
29
+
30
+ const execFileP = promisify(execFile)
31
+ type Opts = Record<string, any>
32
+ const VERSION = "0821g"
33
+
34
+ interface Cfg {
35
+ url: string
36
+ key: string
37
+ model: string
38
+ provider: string
39
+ effort: string
40
+ agent: string
41
+ n: number
42
+ k: number
43
+ topk: number
44
+ conf: number
45
+ judgeTokens: number
46
+ cc: number
47
+ agentTimeoutMs: number
48
+ }
49
+
50
+ const num = (v: any, def: number, lo: number, hi: number): number => {
51
+ const n = Number.parseInt(String(v ?? ""), 10)
52
+ return Number.isFinite(n) ? Math.max(lo, Math.min(hi, n)) : def
53
+ }
54
+
55
+ function buildCfg(opts: Opts): Cfg {
56
+ // Endpoint + model default to the CURRENT SESSION model (captured in the chat.params hook
57
+ // below); plugin options and ULTRA_* env vars override. The per-attempt agent defaults to
58
+ // opencode's own run command, and any other CLI works via the agent option.
59
+ const ref = String(opts.model ?? process.env.ULTRA_MODEL ?? "")
60
+ const s = ref.indexOf("/")
61
+ const provider = s >= 0 ? ref.slice(0, s) : String(opts.provider ?? process.env.ULTRA_PROVIDER ?? "")
62
+ return {
63
+ url: String(opts.baseURL ?? process.env.ULTRA_BASE_URL ?? ""),
64
+ key: String(opts.apiKey ?? process.env.ULTRA_API_KEY ?? ""),
65
+ model: s >= 0 ? ref.slice(s + 1) : ref,
66
+ provider,
67
+ // "none" keeps the verifier fast and non-truncating.
68
+ effort: String(opts.effort ?? process.env.ULTRA_EFFORT ?? "none"),
69
+ // the agent to run in each worktree; {task} is substituted, CWD = the worktree.
70
+ agent: String(opts.agent ?? process.env.ULTRA_AGENT ?? 'opencode run "{task}"'),
71
+ n: num(opts.n ?? process.env.ULTRA_N, 4, 2, 8),
72
+ k: num(opts.k ?? process.env.ULTRA_K, 3, 1, 7),
73
+ topk: 2,
74
+ conf: Number(opts.conf ?? process.env.ULTRA_CONF ?? 0.34),
75
+ judgeTokens: num(opts.judgeTokens ?? process.env.ULTRA_JUDGE_TOKENS, 12000, 500, 200000),
76
+ cc: num(opts.concurrency ?? process.env.ULTRA_CC, 6, 1, 12),
77
+ agentTimeoutMs: num(opts.agentTimeout ?? process.env.ULTRA_AGENT_TIMEOUT, 600000, 30000, 1800000),
78
+ }
79
+ }
80
+
81
+ function pLimit(max: number) {
82
+ let active = 0
83
+ const q: (() => void)[] = []
84
+ const next = () => { active--; q.shift()?.() }
85
+ return async <T>(fn: () => Promise<T>): Promise<T> => {
86
+ if (active >= max) await new Promise<void>((r) => q.push(r))
87
+ active++
88
+ try { return await fn() } finally { next() }
89
+ }
90
+ }
91
+
92
+ async function git(cwd: string, ...args: string[]): Promise<string> {
93
+ const { stdout } = await execFileP("git", args, { cwd, maxBuffer: 32 * 1024 * 1024 })
94
+ return stdout
95
+ }
96
+
97
+ // one non-streaming verifier call; a failure is swallowed by the caller (null vote).
98
+ async function chat(C: Cfg, content: string): Promise<string> {
99
+ const body: Record<string, any> = {
100
+ model: C.model,
101
+ messages: [{ role: "user", content }],
102
+ max_tokens: C.judgeTokens,
103
+ temperature: 0.9,
104
+ }
105
+ if (C.effort) body.reasoning_effort = C.effort
106
+ const res = await fetch(`${C.url.replace(/\/$/, "")}/chat/completions`, {
107
+ method: "POST",
108
+ headers: { Authorization: `Bearer ${C.key}`, "Content-Type": "application/json" },
109
+ body: JSON.stringify(body),
110
+ })
111
+ if (!res.ok) throw new Error(`HTTP ${res.status}`)
112
+ const j: any = await res.json()
113
+ const m = j?.choices?.[0]?.message ?? {}
114
+ return `${m.content ?? ""}\n${m.reasoning_content ?? ""}`
115
+ }
116
+
117
+ async function judge(C: Cfg, limit: ReturnType<typeof pLimit>, task: string, a: string, b: string): Promise<"A" | "B" | null> {
118
+ const prompt =
119
+ `A coding task and two candidate solutions (agent log tail + the diff each produced). Decide which ` +
120
+ `more likely FULLY and correctly accomplishes the task. Be skeptical: prefer concrete evidence of a ` +
121
+ `correct, complete change with no errors; a diff that looks plausible but is incomplete or wrong should ` +
122
+ `lose.\n\nTASK:\n${task}\n\n=== A ===\n${a}\n\n=== B ===\n${b}\n\n` +
123
+ `Reason briefly, then end with exactly one line: 'FINAL: A' or 'FINAL: B'.`
124
+ try {
125
+ const out = (await limit(() => chat(C, prompt))).toUpperCase()
126
+ const ms = out.match(/FINAL:\s*([AB])/g)
127
+ return ms ? (ms[ms.length - 1].slice(-1) as "A" | "B") : null
128
+ } catch {
129
+ return null
130
+ }
131
+ }
132
+
133
+ // Probabilistic Pivot Tournament -> ranked indices + confidence (top1-top2 win-ratio margin).
134
+ async function tournament(C: Cfg, task: string, summaries: string[]): Promise<{ ranked: number[]; conf: number }> {
135
+ const n = summaries.length
136
+ const ids = Array.from({ length: n }, (_, i) => i)
137
+ if (n <= 1) return { ranked: ids, conf: 1 }
138
+ const limit = pLimit(C.cc)
139
+ const order = [...ids]
140
+ for (let i = order.length - 1; i > 0; i--) { const j = Math.floor(Math.random() * (i + 1)); [order[i], order[j]] = [order[j], order[i]] }
141
+ const ring: Record<number, number> = Object.fromEntries(ids.map((i) => [i, 0]))
142
+ await Promise.all(order.map(async (id, i) => {
143
+ const b = order[(i + 1) % n]
144
+ const v = await judge(C, limit, task, summaries[id], summaries[b])
145
+ ring[v === "B" ? b : id]++
146
+ }))
147
+ const pivots = [...ids].sort((x, y) => ring[y] - ring[x]).slice(0, Math.min(C.topk, n))
148
+ const nonp = ids.filter((i) => !pivots.includes(i))
149
+ const pairs: [number, number][] = []
150
+ for (const np of nonp) for (const pv of pivots) pairs.push([np, pv])
151
+ for (let i = 0; i < pivots.length; i++) for (let j = i + 1; j < pivots.length; j++) pairs.push([pivots[i], pivots[j]])
152
+ const mass: Record<number, number> = Object.fromEntries(ids.map((i) => [i, 0]))
153
+ const games: Record<number, number> = Object.fromEntries(ids.map((i) => [i, 0]))
154
+ await Promise.all(pairs.map(async ([x, y]) => {
155
+ const votes = await Promise.all(Array.from({ length: C.k }, () => judge(C, limit, task, summaries[x], summaries[y])))
156
+ const na = votes.filter((v) => v === "A").length
157
+ const nb = votes.filter((v) => v === "B").length
158
+ const px = na + nb === 0 ? 0.5 : na / (na + nb)
159
+ mass[x] += px; mass[y] += 1 - px; games[x]++; games[y]++
160
+ }))
161
+ const ratio = (i: number) => (games[i] ? mass[i] / games[i] : ring[i] / Math.max(1, n - 1))
162
+ const ranked = [...ids].sort((a, b) => ratio(b) - ratio(a))
163
+ return { ranked, conf: ratio(ranked[0]) - ratio(ranked[1]) }
164
+ }
165
+
166
+ // ---- lean sandbox for sub-agent attempts ------------------------------------------------
167
+ // Sub-agents run under XDG_CONFIG_HOME -> a provider-only config so they skip the user's MCP
168
+ // servers and plugins (the slow part). Deps install once into the sandbox and cache forever.
169
+ function leanConfig(C: Cfg): string {
170
+ return JSON.stringify({
171
+ $schema: "https://opencode.ai/config.json",
172
+ model: `${C.provider || "provider"}/${C.model}`,
173
+ permission: { edit: "allow", bash: "allow", webfetch: "allow" },
174
+ provider: {
175
+ [C.provider || "provider"]: {
176
+ npm: "@ai-sdk/openai-compatible",
177
+ name: C.provider || "provider",
178
+ options: { baseURL: C.url, apiKey: C.key },
179
+ models: { [C.model]: { name: C.model } },
180
+ },
181
+ },
182
+ }, null, 2)
183
+ }
184
+
185
+ function appFromAgent(_agent: string): string {
186
+ // The lean config is written under XDG/opencode/; opencode reads it, other agents ignore XDG.
187
+ return "opencode"
188
+ }
189
+
190
+ async function ensureLeanSandbox(C: Cfg, app: string): Promise<{ home: string; needsWarm: boolean }> {
191
+ const home = join(homedir(), ".cache", "agent-ultramode")
192
+ const appDir = join(home, app)
193
+ await mkdir(appDir, { recursive: true })
194
+ await writeFile(join(appDir, `${app}.json`), leanConfig(C))
195
+ return { home, needsWarm: !existsSync(join(appDir, "node_modules")) }
196
+ }
197
+
198
+ export const Ultra: Plugin = async (_input, options) => {
199
+ const C = buildCfg((options as Opts) ?? {})
200
+ const current = { model: "", url: "", key: "" }
201
+ const client = (_input as any)?.client
202
+ // load marker: lets us confirm which build actually loaded.
203
+ writeFile(join(homedir(), ".cache", "agent-ultramode-loaded.txt"), `ultra ${VERSION} loaded ${new Date().toISOString()}\n`).catch(() => {})
204
+
205
+ const ultra = tool({
206
+ description:
207
+ "Best-of-N with a verifier. Runs a coding/terminal task N times in isolated git worktrees, ranks the " +
208
+ "attempts with the same model as a verifier, and applies the winning diff when confident (otherwise " +
209
+ "reports the top candidates). Use for a task worth spending compute to get right the first time.",
210
+ args: {
211
+ task: tool.schema.string().describe("The task to solve N times and verify. Be specific."),
212
+ },
213
+ async execute(args, ctx) {
214
+ const task = String((args as any)?.task ?? "").trim()
215
+ if (!task) return "Pass a `task` to run."
216
+ const eff: Cfg = { ...C, model: C.model || current.model, url: C.url || current.url, key: C.key || current.key }
217
+ if (!eff.url || !eff.model) return "ultra could not resolve a verifier model. Set `model` in the plugin options, or send a message first so it can use your current model."
218
+
219
+ let dir = (ctx as any).worktree || (ctx as any).directory || ""
220
+ if (!dir) {
221
+ // some hosts omit directory/worktree on the tool context; resolve the repo root from cwd.
222
+ try { dir = (await git(process.cwd(), "rev-parse", "--show-toplevel")).trim() } catch { dir = process.cwd() }
223
+ }
224
+ let base: string
225
+ try { base = (await git(dir, "rev-parse", "HEAD")).trim() } catch { return "ultra needs a git repo with a HEAD to branch attempts from (run it inside your project)." }
226
+
227
+ const work = await mkdtemp(join(tmpdir(), "ultra-"))
228
+ const worktrees: string[] = []
229
+ const diffs: string[] = []
230
+ const summaries: string[] = []
231
+ const status = (title: string, body?: string) => {
232
+ try { ctx.metadata({ title, metadata: body ? { output: body } : {} }) } catch {}
233
+ }
234
+ // ctx.metadata renders in the web UI tool card; toasts render in the terminal TUI.
235
+ const toast = (message: string, variant: "info" | "success" | "warning" = "info") => {
236
+ try { client?.tui?.showToast?.({ body: { message: `ultra: ${message}`, variant, duration: 4000 } })?.catch?.(() => {}) } catch {}
237
+ }
238
+ try {
239
+ toast(`v${VERSION}: starting ${eff.n} attempts`)
240
+ // Each attempt runs the agent in a LEAN sandbox: XDG_CONFIG_HOME points opencode at a
241
+ // provider-only config (no MCP, no plugins, permission:allow), so an
242
+ // attempt is a fast model turn instead of booting the whole environment N times.
243
+ const app = appFromAgent(eff.agent)
244
+ const { home: leanHome, needsWarm } = await ensureLeanSandbox(eff, app)
245
+ // Shared: config dir (warm node_modules + lean config). Per-run: XDG_DATA/STATE/CACHE,
246
+ // so each sub-run gets its OWN opencode.db. One shared DB across the main session + N
247
+ // parallel runs deadlocks on the SQLite write lock (all idle, nothing written).
248
+ const baseEnv = {
249
+ ...process.env,
250
+ XDG_CONFIG_HOME: leanHome,
251
+ OPENCODE_DISABLE_DEFAULT_PLUGINS: "1",
252
+ OPENCODE_DISABLE_AUTOUPDATE: "1",
253
+ OPENCODE_DISABLE_MODELS_FETCH: "1",
254
+ }
255
+ const isolatedEnv = async (tag: string) => {
256
+ const d = join(work, `xdg-${tag}`)
257
+ await mkdir(d, { recursive: true }).catch(() => {})
258
+ return { ...baseEnv, XDG_DATA_HOME: d, XDG_STATE_HOME: d, XDG_CACHE_HOME: d }
259
+ }
260
+ const cmd = eff.agent.replace("{task}", task.replace(/"/g, '\\"'))
261
+
262
+ if (needsWarm) {
263
+ // First run on this machine installs the sandbox deps once (cached afterwards).
264
+ status(`ultra: one-time sandbox setup (installing ${app} sandbox deps, ~1-2 min)...`)
265
+ const warm = await mkdtemp(join(tmpdir(), "ultra-warm-"))
266
+ await execFileP("bash", ["-lc", "git init -q"], { cwd: warm }).catch(() => {})
267
+ const warmCmd = eff.agent.replace("{task}", "reply with exactly: ready. do not create or edit files.")
268
+ await execFileP("bash", ["-lc", `exec </dev/null; ${warmCmd}`], { cwd: warm, env: await isolatedEnv("warm"), timeout: eff.agentTimeoutMs, maxBuffer: 8 * 1024 * 1024 }).catch(() => {})
269
+ await rm(warm, { recursive: true, force: true }).catch(() => {})
270
+ }
271
+
272
+ // Create N detached worktrees off HEAD (serial: git worktree add takes a repo lock).
273
+ status(`ultra: preparing ${eff.n} isolated worktrees...`)
274
+ for (let i = 0; i < eff.n; i++) {
275
+ const wt = join(work, `attempt-${i}`)
276
+ await git(dir, "worktree", "add", "--detach", wt, base)
277
+ worktrees.push(wt)
278
+ }
279
+
280
+ // Run the agent in each worktree IN PARALLEL (lean sandbox keeps this cheap).
281
+ const limit = pLimit(Math.max(1, Math.min(eff.cc, eff.n)))
282
+ let done = 0
283
+ status(`ultra: running ${eff.n} attempts in parallel...`)
284
+ toast(`running ${eff.n} attempts in parallel`)
285
+ const outs = await Promise.all(worktrees.map((wt, i) => limit(async () => {
286
+ const env = await isolatedEnv(String(i))
287
+ let log = ""
288
+ try {
289
+ const { stdout, stderr } = await execFileP("bash", ["-lc", `exec </dev/null; ${cmd}`], { cwd: wt, env, timeout: eff.agentTimeoutMs, maxBuffer: 32 * 1024 * 1024, signal: ctx.abort })
290
+ log = (stdout || "") + (stderr || "")
291
+ } catch (e: any) { log = `agent error: ${e?.message || e}` }
292
+ await git(wt, "add", "-A").catch(() => {})
293
+ const diff = await git(wt, "diff", "--cached").catch(() => "")
294
+ done++
295
+ status(`ultra: ${done}/${eff.n} attempts done`, `attempt ${i}: ${diff ? diff.length + " diff chars" : "no changes"}`)
296
+ return { diff, summary: `AGENT LOG (tail):\n${log.slice(-1500)}\n\nDIFF:\n${diff.slice(0, 6000) || "(no changes)"}` }
297
+ })))
298
+ for (const o of outs) { diffs.push(o.diff); summaries.push(o.summary) }
299
+
300
+ if (diffs.every((d) => !d.trim())) {
301
+ return `ultra ran ${eff.n} attempts but none made any changes (all diffs empty). The sub-agent could not act: the task may be too ambiguous, or the sandbox model could not reach its endpoint. Try a more specific task, or check the model endpoint.`
302
+ }
303
+
304
+ status("ultra: verifying...")
305
+ toast(`verifying ${eff.n} candidates`)
306
+ const { ranked, conf } = await tournament(eff, task, summaries)
307
+ const best = ranked[0]
308
+
309
+ // Normalised added-lines of a diff, so we can tell when attempts AGREE on the same change.
310
+ const nonEmpty = diffs.filter((d) => d.trim()).length
311
+ // Apply the verifier's TOP pick when it is confident, OR when a majority of attempts
312
+ // produced a change (the task was solvable; the top-ranked diff is a good starting point
313
+ // and you review it before committing). Only mostly-empty runs are handed back as candidates.
314
+ const majority = nonEmpty >= Math.max(2, Math.ceil(eff.n / 2))
315
+ if ((conf >= eff.conf || majority) && diffs[best].trim()) {
316
+ const patch = join(work, "winner.patch")
317
+ await writeFile(patch, diffs[best])
318
+ const confNote = conf >= eff.conf ? `high confidence ${conf.toFixed(2)}` : `top pick of ${nonEmpty}/${eff.n}, margin ${conf.toFixed(2)}`
319
+ try {
320
+ await git(dir, "apply", "--3way", patch)
321
+ toast("applied the winning change", "success")
322
+ return `🏆 ultra applied the best of ${eff.n} attempts (${confNote}). The winning change is now in your working tree; review it before committing.`
323
+ } catch (e: any) {
324
+ return `ultra picked the best attempt (${confNote}) but the patch did not apply cleanly (${e?.message || e}). The diff:\n\n\`\`\`diff\n${diffs[best].slice(0, 6000)}\n\`\`\``
325
+ }
326
+ }
327
+ const top = ranked.slice(0, 3).map((i, r) => `### Candidate ${r + 1} (attempt ${i})\n\`\`\`diff\n${diffs[i].slice(0, 4000) || "(no changes)"}\n\`\`\``).join("\n\n")
328
+ return `ultra ran ${eff.n} attempts but most produced no usable change, so it applied nothing. Top candidates:\n\n${top}`
329
+ } finally {
330
+ for (const wt of worktrees) await git(dir, "worktree", "remove", "--force", wt).catch(() => {})
331
+ await rm(work, { recursive: true, force: true }).catch(() => {})
332
+ }
333
+ },
334
+ })
335
+
336
+ return {
337
+ tool: { ultra },
338
+ "chat.params": async (input: any) => {
339
+ try {
340
+ const id = input?.model?.id ?? input?.model?.modelID
341
+ const o = input?.provider?.options
342
+ if (id && o?.baseURL) { current.model = id; current.url = o.baseURL; if (o.apiKey) current.key = o.apiKey }
343
+ } catch {}
344
+ },
345
+ config: async (cfg: any) => {
346
+ cfg.command = {
347
+ ultra: {
348
+ description: "Best-of-N + verifier: run the task N times in isolated worktrees, apply the best. /ultra <task>",
349
+ template:
350
+ "Call the `ultra` tool exactly once, with `task` set to the request below. It runs the task several " +
351
+ "times in isolated git worktrees and verifies the results. When it returns, report to the user exactly " +
352
+ "what it did (applied the winner, or listed candidates) verbatim; do not re-do the task yourself.\n\nTask: $ARGUMENTS",
353
+ },
354
+ ...(cfg.command ?? {}),
355
+ }
356
+ // Register an "ultra" primary agent so it shows in the tab / agent list; selecting it
357
+ // routes each request through best-of-N. Sub-attempts run the default agent, so no recursion.
358
+ cfg.agent = {
359
+ ultra: {
360
+ description: "Best-of-N mode: run the request N times in isolated worktrees and apply the verified winner.",
361
+ mode: "primary",
362
+ color: "#A855F7",
363
+ prompt:
364
+ "You are in ULTRA mode (best-of-N with a same-model verifier). For the user's coding or terminal " +
365
+ "request, call the `ultra` tool exactly once with the full request as `task`. When it returns, report " +
366
+ "exactly what it did (applied the winner, or listed the candidates) verbatim; do not do the task yourself.",
367
+ },
368
+ ...(cfg.agent ?? {}),
369
+ }
370
+ try {
371
+ if (C.provider && cfg?.provider?.[C.provider]) {
372
+ const p = cfg.provider[C.provider]
373
+ if (!C.url && p.options?.baseURL) C.url = p.options.baseURL
374
+ if (!C.key && p.options?.apiKey) C.key = p.options.apiKey
375
+ }
376
+ } catch {}
377
+ },
378
+ }
379
+ }
380
+
381
+ export default Ultra