agent-ultramode 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +194 -0
- package/cli.mjs +262 -0
- package/package.json +46 -0
- package/ultra.ts +381 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 maverick-tr (https://github.com/maverick-tr)
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,194 @@
|
|
|
1
|
+
# ultra
|
|
2
|
+
|
|
3
|
+
Took the [LLM-as-a-Verifier](https://github.com/llm-as-a-verifier/llm-as-a-verifier) paper (Kwok et al., 2026) and turned it into a simple `/ultra` command for coding agents.
|
|
4
|
+
|
|
5
|
+
Instead of shipping the agent's first attempt, `ultra` runs your task **N times in isolated git worktrees**, in parallel, then uses the **same model** as a verifier to pick the best result. When it is confident (or when most attempts succeeded) it applies the winning diff to your working tree; when the attempts diverge and it is not sure, it hands you the top candidates. No cross-model dependency, no extra services, no first-attempt lottery.
|
|
6
|
+
|
|
7
|
+
And unlike most "add a verifier" posts, I benchmarked it before believing it. The receipts are below, good and bad.
|
|
8
|
+
|
|
9
|
+
## Demo
|
|
10
|
+
|
|
11
|
+
<video src="https://github.com/maverick-tr/agent-ultramode/raw/main/ultramode.mp4" controls muted playsinline width="100%"></video>
|
|
12
|
+
|
|
13
|
+
> For a reliable inline player on GitHub, edit this README on github.com and drag `ultramode.mp4` into the editor; GitHub inserts a `https://github.com/.../assets/...` URL that plays inline. The file is committed at [`ultramode.mp4`](./ultramode.mp4).
|
|
14
|
+
|
|
15
|
+
## The receipts
|
|
16
|
+
|
|
17
|
+
Benchmarked on **Terminal-Bench** with **DeepSeek V4 Flash 0731** as both the agent and the verifier, running inside **opencode** (N=5 attempts, same-model verifier). Verified end to end with both **OpenCode** and **Claude Code** as the agent (the numbers above are the OpenCode run; a Claude Code number will be added once benchmarked). Works with any agent, not just opencode.
|
|
18
|
+
|
|
19
|
+
| Slice | base@1 (single shot) | ultra (best-of-N + verify) | oracle@5 (ceiling) |
|
|
20
|
+
|---|:---:|:---:|:---:|
|
|
21
|
+
| All 15 tasks | 24% | **33%** | 40% |
|
|
22
|
+
| 4 recoverable tasks | 40% | **75%** | rescued 3 of 4 |
|
|
23
|
+
|
|
24
|
+
Overall the verifier captured about **56% of the pass@1 to pass@5 headroom**. On the tasks with real variance, where best-of-N can actually help, it took the passing attempt on `chess-best-move`, `new-encrypt-command`, and `decommissioning-service` (missing only `jupyter-notebook-server`). A 40 to 75 jump on recoverable tasks is not a rounding error.
|
|
25
|
+
|
|
26
|
+
## How the competition works
|
|
27
|
+
|
|
28
|
+
```text
|
|
29
|
+
your task
|
|
30
|
+
|
|
|
31
|
+
+----------+-----------+-----------+----------+ N attempts, in parallel,
|
|
32
|
+
| | | | | each in its own git worktree
|
|
33
|
+
worktree1 worktree2 worktree3 worktree4 ... + a lean sandbox (no MCP,
|
|
34
|
+
| | | | | no plugins, isolated state)
|
|
35
|
+
agent agent agent agent agent
|
|
36
|
+
| | | | |
|
|
37
|
+
diff1 diff2 diff3 diff4 diffN
|
|
38
|
+
+----------+-----------+-----------+----------+
|
|
39
|
+
|
|
|
40
|
+
v
|
|
41
|
+
Probabilistic Pivot Tournament (same model, acting as verifier)
|
|
42
|
+
1. ring pass each diff judged once vs a neighbour -> seed pivots
|
|
43
|
+
2. pivot duels field vs the top pivots, K reasoned votes per duel
|
|
44
|
+
3. score normalised win-rate; confidence = top1 minus top2
|
|
45
|
+
|
|
|
46
|
+
v
|
|
47
|
+
decide
|
|
48
|
+
confident, or most attempts changed something -> APPLY the winner
|
|
49
|
+
diverged and not confident -> show top candidates
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
1. **Fan out.** N detached git worktrees off `HEAD`; the agent runs the full task in each, in parallel, isolated so the attempts never collide.
|
|
53
|
+
2. **Run each attempt lean.** Every attempt is a full agent process, but stripped to essentials (no MCP servers, no other plugins, its own private state). That is what makes N real agents in parallel take seconds instead of minutes.
|
|
54
|
+
3. **Verify.** The tournament ranks the N diffs by same-model reasoned votes. Confidence is the win-rate margin between the top two.
|
|
55
|
+
4. **Decide.** Confident or a clear majority produced a change: apply the winner to your working tree (uncommitted, you review before committing). Otherwise: present the top candidates instead of guessing.
|
|
56
|
+
|
|
57
|
+
Progress streams live: phase titles in the web UI tool card, and toasts in the terminal.
|
|
58
|
+
|
|
59
|
+
## Install
|
|
60
|
+
|
|
61
|
+
```sh
|
|
62
|
+
opencode plugin agent-ultramode
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
That is the whole setup. With no options it drafts and verifies with your **current session model**, so `/ultra <task>` just works:
|
|
66
|
+
|
|
67
|
+
```text
|
|
68
|
+
/ultra fix the failing test in foo/bar
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
`ultra` branches attempts off `HEAD`, so run it inside a git repo with at least one commit.
|
|
72
|
+
|
|
73
|
+
**Pin a specific model (optional).** To draft and verify with a model other than the session one:
|
|
74
|
+
|
|
75
|
+
```jsonc
|
|
76
|
+
{ "plugin": [ ["agent-ultramode", { "model": "myprovider/my-model" }] ] }
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
**Run a different agent (Claude Code, Grok, Pi, Codex, ...).** The `agent` option is the command run once per attempt, with `{task}` substituted and the cwd set to an isolated worktree. Point it at any CLI that edits files and exits:
|
|
80
|
+
|
|
81
|
+
```jsonc
|
|
82
|
+
["agent-ultramode", { "agent": "claude -p --dangerously-skip-permissions \"{task}\"" }] // Claude Code (verified)
|
|
83
|
+
["agent-ultramode", { "agent": "pi \"{task}\"" }] // pi
|
|
84
|
+
["agent-ultramode", { "agent": "grok build \"{task}\"" }] // grok
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Adjust each agent's flags for headless, file-editing runs (for example Claude Code's permission mode). The best-of-N and verify loop stays the same, which makes ultra-mode easy to push onto any agent you use.
|
|
88
|
+
|
|
89
|
+
## Configure
|
|
90
|
+
|
|
91
|
+
Every option also reads from an `ULTRA_*` env var.
|
|
92
|
+
|
|
93
|
+
| Option | Default | Meaning |
|
|
94
|
+
|---|---|---|
|
|
95
|
+
| `model` | your current session model | `"provider/model-id"` used to run attempts and to verify |
|
|
96
|
+
| `agent` | run the current model | the command run once per attempt; `{task}` substituted, cwd is an isolated worktree |
|
|
97
|
+
| `n` | `4` | number of attempts (2 to 8) |
|
|
98
|
+
| `k` | `3` | reasoned votes per verifier duel |
|
|
99
|
+
| `conf` | `0.34` | confidence margin at or above which the winner is applied |
|
|
100
|
+
| `baseURL` / `apiKey` | from the provider | override the OpenAI-compatible endpoint / key for the verifier |
|
|
101
|
+
| `effort` | `"none"` | `reasoning_effort` for the verifier calls (kept low so judging stays fast) |
|
|
102
|
+
| `concurrency` | `6` | how many attempts run at once |
|
|
103
|
+
| `agentTimeout` | `600000` | per-attempt timeout in ms |
|
|
104
|
+
|
|
105
|
+
## Use without opencode (the CLI)
|
|
106
|
+
|
|
107
|
+
Same loop, no opencode host required. Install and run it in any git repo:
|
|
108
|
+
|
|
109
|
+
```sh
|
|
110
|
+
npm i -g agent-ultramode # or: npx agent-ultramode "<task>"
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
```sh
|
|
114
|
+
agent-ultramode "fix the failing test in foo/bar"
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
By default each attempt runs `opencode run` (and it tells you clearly if opencode is not installed). Point `--agent` at anything else that edits files:
|
|
118
|
+
|
|
119
|
+
```sh
|
|
120
|
+
agent-ultramode -t "add rate limiting to /login" \
|
|
121
|
+
--agent 'claude -p --dangerously-skip-permissions "{task}"' \
|
|
122
|
+
--verify-model gpt-4o-mini
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
**Multiple models in one pass.** Pass `--agent` more than once to spread the attempts across different models, judged by one neutral verifier:
|
|
126
|
+
|
|
127
|
+
```sh
|
|
128
|
+
agent-ultramode -t "design and implement the data migration" \
|
|
129
|
+
--agent 'claude -p --dangerously-skip-permissions "{task}"' \
|
|
130
|
+
--agent 'opencode run --model grok "{task}"' \
|
|
131
|
+
--agent 'opencode run --model deepseek-v4-flash "{task}"' \
|
|
132
|
+
--n 6 --verify-model gpt-4o-mini
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
The 6 attempts round-robin across the three agents, and the verifier picks the best regardless of which model produced it. Good for one hard task where you want several strong models to take a shot.
|
|
136
|
+
|
|
137
|
+
The verifier needs an OpenAI-compatible endpoint: by default it reads `OPENAI_API_KEY` and `OPENAI_BASE_URL`, or pass `--api-key` / `--base-url` / `--verify-model`. Run `agent-ultramode --help` for all options.
|
|
138
|
+
|
|
139
|
+
## Why I built this, and the honest story
|
|
140
|
+
|
|
141
|
+
I wanted to know if best-of-N verification could squeeze real quality out of a cheap-but-capable model without reaching for a bigger, pricier one. So, ran it on real Terminal-Bench tasks and let the numbers decide. What I found:
|
|
142
|
+
|
|
143
|
+
- **Planning-first best-of-N does nothing for execution tasks.** Drafting N plans, picking the best, then executing once: **0 out of 5** outcomes changed on Terminal-Bench, at 3 to 10 times the cost. It amplifies effort, not capability. I dropped it.
|
|
144
|
+
- **Best-of-N over full trajectories plus a same-model verifier does work.** This is the version that ships.
|
|
145
|
+
- **A fancier verifier did not help.** A multi-criteria checklist tied the plain holistic judge and sometimes did worse. Same-model verification hits a ceiling that more prompting does not break.
|
|
146
|
+
- **Confidence is a real signal, but a noisy one.** Useful, not a magic gate.
|
|
147
|
+
|
|
148
|
+
It significantly helps on the tasks that matter, and I can point at the per-task data.
|
|
149
|
+
|
|
150
|
+
## The honest caveats (because benchmarks lie by omission)
|
|
151
|
+
|
|
152
|
+
1. **Confidence is a noisy signal at this scale.** The one miss (`jupyter`) had confidence 0.25, higher than two tasks it got right (0.17, 0.21). Some beyond-capability tasks even show high confidence while being pure failures. "Trust when confident" is directionally real, not a clean gate.
|
|
153
|
+
2. **It cannot help beyond-capability tasks.** 9 of the 15 never passed in 5 tries, and no verifier manufactures a solution that was never there.
|
|
154
|
+
3. **n=15 (4 with real variance) is a credible but modest sample.** Enough to headline honestly, not enough to over-claim precision.
|
|
155
|
+
|
|
156
|
+
## What this likely means at scale
|
|
157
|
+
|
|
158
|
+
The 15 tasks were deliberately failure-skewed, so the +9 points is not what you would see on the whole benchmark. On the full set the model already passes about 83%, so most tasks have no headroom. My honest estimate for a full run is a **modest +2 to +5 points on average**, because the dramatic gains only land on the minority of tasks the model *sometimes* solves. Average lift small, per-recoverable-task lift large. Same fact, two views.
|
|
159
|
+
|
|
160
|
+
## Roadmap
|
|
161
|
+
|
|
162
|
+
- [x] **Standalone CLI** (`npx agent-ultramode`) so the loop runs anywhere, with any agent, no opencode required.
|
|
163
|
+
- [x] **Multiple models in one pass** (repeatable `--agent`): spread attempts across different models, one neutral verifier picks the best.
|
|
164
|
+
- [x] **OpenCode and Claude Code** verified end to end.
|
|
165
|
+
- [ ] **First-class agent integrations** with tuned defaults, and a benchmark number, for Claude Code, Grok, Pi, and Codex.
|
|
166
|
+
- [ ] Native slash-command or MCP packaging per agent. Contributions welcome.
|
|
167
|
+
|
|
168
|
+
## Credits
|
|
169
|
+
|
|
170
|
+
The verification method is the **Probabilistic Pivot Tournament** from **LLM-as-a-Verifier** (Kwok et al., 2026), reimplemented from the paper. The `K`-sample reward estimate and the apply policy are our adaptations. The self-verification ceiling we ran into is documented in the cross-model and weak-verifier literature, worth reading before you assume same-model verification is a free lunch:
|
|
171
|
+
|
|
172
|
+
- Paper: [arXiv:2607.05391](https://arxiv.org/abs/2607.05391)
|
|
173
|
+
- Repo: [llm-as-a-verifier/llm-as-a-verifier](https://github.com/llm-as-a-verifier/llm-as-a-verifier)
|
|
174
|
+
- On the ceiling: [LLM-as-a-Jury (cross-model)](https://arxiv.org/html/2607.10139), [Weaver: weak verifiers](https://arxiv.org/html/2506.18203), [Generative Verifiers](https://arxiv.org/pdf/2408.15240)
|
|
175
|
+
|
|
176
|
+
```bibtex
|
|
177
|
+
@misc{kwok2026llmasaverifiergeneralpurposeverificationframework,
|
|
178
|
+
title={LLM-as-a-Verifier: A General-Purpose Verification Framework},
|
|
179
|
+
author={Jacky Kwok and Shulu Li and Pranav Atreya and Yuejiang Liu and
|
|
180
|
+
Yixing Jiang and Chelsea Finn and Marco Pavone and Ion Stoica
|
|
181
|
+
and Azalia Mirhoseini},
|
|
182
|
+
year={2026},
|
|
183
|
+
eprint={2607.05391},
|
|
184
|
+
archivePrefix={arXiv},
|
|
185
|
+
primaryClass={cs.AI},
|
|
186
|
+
url={https://arxiv.org/abs/2607.05391}
|
|
187
|
+
}
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
Built for [opencode](https://opencode.ai). Zero heavy dependencies: opencode's bundled `@opencode-ai/plugin`, the global `fetch`, and an OpenAI-compatible endpoint.
|
|
191
|
+
|
|
192
|
+
## License and author
|
|
193
|
+
|
|
194
|
+
MIT, by [maverick-tr](https://github.com/maverick-tr). Built on the method from [llm-as-a-verifier](https://github.com/llm-as-a-verifier/llm-as-a-verifier) (Kwok et al., 2026). See [LICENSE](./LICENSE).
|
package/cli.mjs
ADDED
|
@@ -0,0 +1,262 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// agent-ultramode: best-of-N for coding agents with a same-model verifier, as a CLI.
|
|
3
|
+
//
|
|
4
|
+
// Runs your task N times in isolated git worktrees, verifies the diffs with a
|
|
5
|
+
// Probabilistic Pivot Tournament (the same-model verifier from llm-as-a-verifier),
|
|
6
|
+
// and applies the winner. No opencode host required. Defaults the per-attempt
|
|
7
|
+
// agent to `opencode run`, but `--agent` takes any CLI that edits files.
|
|
8
|
+
//
|
|
9
|
+
// Node 18+ (uses global fetch and node:util parseArgs). Zero dependencies.
|
|
10
|
+
import { execFile } from "node:child_process"
|
|
11
|
+
import { promisify } from "node:util"
|
|
12
|
+
import { mkdtemp, rm, writeFile, mkdir } from "node:fs/promises"
|
|
13
|
+
import { tmpdir } from "node:os"
|
|
14
|
+
import { join } from "node:path"
|
|
15
|
+
import { parseArgs } from "node:util"
|
|
16
|
+
|
|
17
|
+
const execFileP = promisify(execFile)
|
|
18
|
+
const int = (v, def, lo, hi) => { const n = parseInt(String(v ?? ""), 10); return Number.isFinite(n) ? Math.max(lo, Math.min(hi, n)) : def }
|
|
19
|
+
const err = (m) => process.stderr.write(m + "\n")
|
|
20
|
+
|
|
21
|
+
function printHelp() {
|
|
22
|
+
err(`agent-ultramode - best-of-N for coding agents with a same-model verifier
|
|
23
|
+
|
|
24
|
+
USAGE
|
|
25
|
+
agent-ultramode --task "<what to do>" [options]
|
|
26
|
+
agent-ultramode "<what to do>" [options]
|
|
27
|
+
|
|
28
|
+
OPTIONS
|
|
29
|
+
-t, --task <text> the task to solve (or pass it as positional words)
|
|
30
|
+
-a, --agent <cmd> command run per attempt; "{task}" substituted, cwd is an isolated
|
|
31
|
+
worktree. Repeatable: pass several to spread attempts across
|
|
32
|
+
different models in one pass (round-robin). default: opencode run "{task}"
|
|
33
|
+
-n, --n <int> number of attempts, 2 to 8 (default 4)
|
|
34
|
+
--k <int> reasoned votes per verifier duel (default 3)
|
|
35
|
+
--conf <float> confidence margin to auto-apply (default 0.34)
|
|
36
|
+
-m, --verify-model <id> model id for the verifier (default $ULTRA_VERIFY_MODEL or gpt-4o-mini)
|
|
37
|
+
--base-url <url> verifier OpenAI-compatible endpoint (default $OPENAI_BASE_URL or OpenAI)
|
|
38
|
+
--api-key <key> verifier api key (default $OPENAI_API_KEY)
|
|
39
|
+
--repo <path> repo to run in (default: current directory)
|
|
40
|
+
--concurrency <int> attempts to run at once (default 6)
|
|
41
|
+
--effort <level> verifier reasoning_effort (default none)
|
|
42
|
+
-h, --help show this help
|
|
43
|
+
|
|
44
|
+
EXAMPLES
|
|
45
|
+
# default: opencode as the per-attempt agent, verify with your OpenAI key
|
|
46
|
+
agent-ultramode "fix the failing test in foo/bar"
|
|
47
|
+
|
|
48
|
+
# use Claude Code as the agent
|
|
49
|
+
agent-ultramode -t "add rate limiting to /login" \\
|
|
50
|
+
--agent 'claude -p --dangerously-skip-permissions "{task}"' \\
|
|
51
|
+
--verify-model gpt-4o-mini
|
|
52
|
+
|
|
53
|
+
# point the verifier at any OpenAI-compatible endpoint
|
|
54
|
+
agent-ultramode "..." --base-url http://localhost:8000/v1 --api-key x --verify-model my-model
|
|
55
|
+
`)
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
async function git(cwd, ...args) {
|
|
59
|
+
const { stdout } = await execFileP("git", args, { cwd, maxBuffer: 32 * 1024 * 1024 })
|
|
60
|
+
return stdout
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
function pLimit(max) {
|
|
64
|
+
let active = 0
|
|
65
|
+
const q = []
|
|
66
|
+
const next = () => { active--; q.shift()?.() }
|
|
67
|
+
return async (fn) => {
|
|
68
|
+
if (active >= max) await new Promise((r) => q.push(r))
|
|
69
|
+
active++
|
|
70
|
+
try { return await fn() } finally { next() }
|
|
71
|
+
}
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
async function chat(C, content) {
|
|
75
|
+
const body = { model: C.verifyModel, messages: [{ role: "user", content }], max_tokens: 12000, temperature: 0.9 }
|
|
76
|
+
if (C.effort) body.reasoning_effort = C.effort
|
|
77
|
+
const res = await fetch(`${C.baseURL.replace(/\/$/, "")}/chat/completions`, {
|
|
78
|
+
method: "POST",
|
|
79
|
+
headers: { Authorization: `Bearer ${C.apiKey}`, "Content-Type": "application/json" },
|
|
80
|
+
body: JSON.stringify(body),
|
|
81
|
+
})
|
|
82
|
+
if (!res.ok) throw new Error(`verifier HTTP ${res.status}`)
|
|
83
|
+
const j = await res.json()
|
|
84
|
+
const m = j?.choices?.[0]?.message ?? {}
|
|
85
|
+
return `${m.content ?? ""}\n${m.reasoning_content ?? ""}`
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
async function judge(C, limit, task, a, b) {
|
|
89
|
+
const prompt =
|
|
90
|
+
`A coding task and two candidate solutions (agent log tail + the diff each produced). Decide which ` +
|
|
91
|
+
`more likely FULLY and correctly accomplishes the task. Be skeptical: prefer concrete evidence of a ` +
|
|
92
|
+
`correct, complete change with no errors; a diff that looks plausible but is incomplete or wrong should ` +
|
|
93
|
+
`lose.\n\nTASK:\n${task}\n\n=== A ===\n${a}\n\n=== B ===\n${b}\n\n` +
|
|
94
|
+
`Reason briefly, then end with exactly one line: 'FINAL: A' or 'FINAL: B'.`
|
|
95
|
+
try {
|
|
96
|
+
const out = (await limit(() => chat(C, prompt))).toUpperCase()
|
|
97
|
+
const ms = out.match(/FINAL:\s*([AB])/g)
|
|
98
|
+
return ms ? ms[ms.length - 1].slice(-1) : null
|
|
99
|
+
} catch { return null }
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
async function tournament(C, task, summaries) {
|
|
103
|
+
const n = summaries.length
|
|
104
|
+
const ids = Array.from({ length: n }, (_, i) => i)
|
|
105
|
+
if (n <= 1) return { ranked: ids, conf: 1 }
|
|
106
|
+
const limit = pLimit(C.cc)
|
|
107
|
+
const order = [...ids]
|
|
108
|
+
for (let i = order.length - 1; i > 0; i--) { const j = Math.floor(Math.random() * (i + 1)); [order[i], order[j]] = [order[j], order[i]] }
|
|
109
|
+
const ring = Object.fromEntries(ids.map((i) => [i, 0]))
|
|
110
|
+
await Promise.all(order.map(async (id, i) => {
|
|
111
|
+
const b = order[(i + 1) % n]
|
|
112
|
+
const v = await judge(C, limit, task, summaries[id], summaries[b])
|
|
113
|
+
ring[v === "B" ? b : id]++
|
|
114
|
+
}))
|
|
115
|
+
const pivots = [...ids].sort((x, y) => ring[y] - ring[x]).slice(0, Math.min(2, n))
|
|
116
|
+
const nonp = ids.filter((i) => !pivots.includes(i))
|
|
117
|
+
const pairs = []
|
|
118
|
+
for (const np of nonp) for (const pv of pivots) pairs.push([np, pv])
|
|
119
|
+
for (let i = 0; i < pivots.length; i++) for (let j = i + 1; j < pivots.length; j++) pairs.push([pivots[i], pivots[j]])
|
|
120
|
+
const mass = Object.fromEntries(ids.map((i) => [i, 0]))
|
|
121
|
+
const games = Object.fromEntries(ids.map((i) => [i, 0]))
|
|
122
|
+
await Promise.all(pairs.map(async ([x, y]) => {
|
|
123
|
+
const votes = await Promise.all(Array.from({ length: C.k }, () => judge(C, limit, task, summaries[x], summaries[y])))
|
|
124
|
+
const na = votes.filter((v) => v === "A").length
|
|
125
|
+
const nb = votes.filter((v) => v === "B").length
|
|
126
|
+
const px = na + nb === 0 ? 0.5 : na / (na + nb)
|
|
127
|
+
mass[x] += px; mass[y] += 1 - px; games[x]++; games[y]++
|
|
128
|
+
}))
|
|
129
|
+
const ratio = (i) => (games[i] ? mass[i] / games[i] : ring[i] / Math.max(1, n - 1))
|
|
130
|
+
const ranked = [...ids].sort((a, b) => ratio(b) - ratio(a))
|
|
131
|
+
return { ranked, conf: ratio(ranked[0]) - ratio(ranked[1]) }
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
async function main() {
|
|
135
|
+
const { values, positionals } = parseArgs({
|
|
136
|
+
allowPositionals: true,
|
|
137
|
+
options: {
|
|
138
|
+
task: { type: "string", short: "t" },
|
|
139
|
+
agent: { type: "string", short: "a", multiple: true },
|
|
140
|
+
n: { type: "string", short: "n" },
|
|
141
|
+
k: { type: "string" },
|
|
142
|
+
conf: { type: "string" },
|
|
143
|
+
"verify-model": { type: "string", short: "m" },
|
|
144
|
+
"base-url": { type: "string" },
|
|
145
|
+
"api-key": { type: "string" },
|
|
146
|
+
repo: { type: "string" },
|
|
147
|
+
concurrency: { type: "string" },
|
|
148
|
+
effort: { type: "string" },
|
|
149
|
+
help: { type: "boolean", short: "h" },
|
|
150
|
+
},
|
|
151
|
+
})
|
|
152
|
+
|
|
153
|
+
if (values.help) { printHelp(); process.exit(0) }
|
|
154
|
+
const task = (values.task || positionals.join(" ")).trim()
|
|
155
|
+
if (!task) { printHelp(); err("\nagent-ultramode: pass a --task (or task words) to run."); process.exit(1) }
|
|
156
|
+
|
|
157
|
+
const agents = values.agent && values.agent.length ? values.agent : ['opencode run "{task}"']
|
|
158
|
+
const n = int(values.n, 4, 2, 8)
|
|
159
|
+
const k = int(values.k, 3, 1, 7)
|
|
160
|
+
const conf = Number(values.conf ?? 0.34)
|
|
161
|
+
const cc = int(values.concurrency, 6, 1, 12)
|
|
162
|
+
const agentTimeout = 600000
|
|
163
|
+
const C = {
|
|
164
|
+
verifyModel: values["verify-model"] || process.env.ULTRA_VERIFY_MODEL || process.env.OPENAI_MODEL || "gpt-4o-mini",
|
|
165
|
+
baseURL: values["base-url"] || process.env.OPENAI_BASE_URL || process.env.ULTRA_BASE_URL || "https://api.openai.com/v1",
|
|
166
|
+
apiKey: values["api-key"] || process.env.OPENAI_API_KEY || process.env.ULTRA_API_KEY || "",
|
|
167
|
+
effort: values.effort ?? process.env.ULTRA_EFFORT ?? "none",
|
|
168
|
+
k, cc,
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
// Friendly check: an opencode agent must exist on PATH.
|
|
172
|
+
const binOf = (a) => a.trim().split(/\s+/)[0]
|
|
173
|
+
for (const b of [...new Set(agents.map(binOf))]) {
|
|
174
|
+
if (b !== "opencode") continue
|
|
175
|
+
try { await execFileP(b, ["--version"], { timeout: 10000 }) }
|
|
176
|
+
catch {
|
|
177
|
+
err(`\nagent-ultramode: '${b}' is not installed or not on your PATH.\n` +
|
|
178
|
+
`Install it, or pass --agent to use a different agent, for example:\n` +
|
|
179
|
+
` --agent 'claude -p --dangerously-skip-permissions "{task}"'\n`)
|
|
180
|
+
process.exit(1)
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
if (!C.apiKey) {
|
|
184
|
+
err(`\nagent-ultramode: no verifier api key. The verifier needs an OpenAI-compatible model to judge attempts.\n` +
|
|
185
|
+
`Set OPENAI_API_KEY (and OPENAI_BASE_URL / --verify-model for a non-OpenAI endpoint), or pass --api-key.\n`)
|
|
186
|
+
process.exit(1)
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
const repoArg = values.repo || process.cwd()
|
|
190
|
+
let repo
|
|
191
|
+
try { repo = (await git(repoArg, "rev-parse", "--show-toplevel")).trim() } catch { err(`\nagent-ultramode: '${repoArg}' is not inside a git repo.`); process.exit(1) }
|
|
192
|
+
let base
|
|
193
|
+
try { base = (await git(repo, "rev-parse", "HEAD")).trim() } catch { err(`\nagent-ultramode: the repo has no commits yet (need a HEAD to branch attempts from).`); process.exit(1) }
|
|
194
|
+
|
|
195
|
+
const work = await mkdtemp(join(tmpdir(), "ultramode-"))
|
|
196
|
+
const worktrees = []
|
|
197
|
+
try {
|
|
198
|
+
const label = agents.length > 1 ? `${agents.length} agents (round-robin)` : `'${binOf(agents[0])}'`
|
|
199
|
+
err(`>>> fan-out: ${n} attempts off ${base.slice(0, 8)} with ${label} ...`)
|
|
200
|
+
for (let i = 0; i < n; i++) {
|
|
201
|
+
const wt = join(work, `attempt-${i}`)
|
|
202
|
+
await git(repo, "worktree", "add", "--detach", wt, base)
|
|
203
|
+
worktrees.push(wt)
|
|
204
|
+
}
|
|
205
|
+
const limit = pLimit(Math.max(1, Math.min(cc, n)))
|
|
206
|
+
const esc = task.replace(/"/g, '\\"')
|
|
207
|
+
const outs = await Promise.all(worktrees.map((wt, i) => limit(async () => {
|
|
208
|
+
const cmd = agents[i % agents.length].replace("{task}", esc)
|
|
209
|
+
const d = join(work, `xdg-${i}`)
|
|
210
|
+
await mkdir(d, { recursive: true }).catch(() => {})
|
|
211
|
+
const env = {
|
|
212
|
+
...process.env,
|
|
213
|
+
XDG_DATA_HOME: d, XDG_STATE_HOME: d, XDG_CACHE_HOME: d,
|
|
214
|
+
OPENCODE_DISABLE_DEFAULT_PLUGINS: "1", OPENCODE_DISABLE_AUTOUPDATE: "1", OPENCODE_DISABLE_MODELS_FETCH: "1",
|
|
215
|
+
}
|
|
216
|
+
let log = ""
|
|
217
|
+
try {
|
|
218
|
+
const { stdout, stderr } = await execFileP("bash", ["-lc", `exec </dev/null; ${cmd}`], { cwd: wt, env, timeout: agentTimeout, maxBuffer: 32 * 1024 * 1024 })
|
|
219
|
+
log = (stdout || "") + (stderr || "")
|
|
220
|
+
} catch (e) { log = `agent error: ${e?.message || e}` }
|
|
221
|
+
await git(wt, "add", "-A").catch(() => {})
|
|
222
|
+
const diff = await git(wt, "diff", "--cached").catch(() => "")
|
|
223
|
+
err(` attempt ${i} [${binOf(agents[i % agents.length])}]: ${diff ? diff.length + " diff chars" : "no changes"}`)
|
|
224
|
+
return { diff, summary: `AGENT LOG (tail):\n${log.slice(-1500)}\n\nDIFF:\n${diff.slice(0, 6000) || "(no changes)"}` }
|
|
225
|
+
})))
|
|
226
|
+
|
|
227
|
+
const diffs = outs.map((o) => o.diff)
|
|
228
|
+
const summaries = outs.map((o) => o.summary)
|
|
229
|
+
if (diffs.every((d) => !d.trim())) {
|
|
230
|
+
console.log("agent-ultramode: none of the attempts made any changes. Try a more specific task, or check that your agent can edit files headlessly.")
|
|
231
|
+
return
|
|
232
|
+
}
|
|
233
|
+
|
|
234
|
+
err(`>>> verify: probabilistic pivot tournament (${C.verifyModel}) ...`)
|
|
235
|
+
const { ranked, conf: margin } = await tournament(C, task, summaries)
|
|
236
|
+
const best = ranked[0]
|
|
237
|
+
const nonEmpty = diffs.filter((d) => d.trim()).length
|
|
238
|
+
const majority = nonEmpty >= Math.max(2, Math.ceil(n / 2))
|
|
239
|
+
|
|
240
|
+
if ((margin >= conf || majority) && diffs[best].trim()) {
|
|
241
|
+
const patch = join(work, "winner.patch")
|
|
242
|
+
await writeFile(patch, diffs[best])
|
|
243
|
+
const why = margin >= conf ? `confidence ${margin.toFixed(2)}` : `top pick of ${nonEmpty}/${n}, margin ${margin.toFixed(2)}`
|
|
244
|
+
try {
|
|
245
|
+
await git(repo, "apply", "--3way", patch)
|
|
246
|
+
console.log(`\n🏆 applied the best of ${n} attempts (${why}). The change is in your working tree; review it, then commit.`)
|
|
247
|
+
} catch (e) {
|
|
248
|
+
console.log(`\nverified the best attempt (${why}) but the patch did not apply cleanly (${e?.message || e}). The diff:\n\n${diffs[best].slice(0, 8000)}`)
|
|
249
|
+
}
|
|
250
|
+
} else {
|
|
251
|
+
console.log(`\nattempts diverged and the verifier was not confident (margin ${margin.toFixed(2)}); applied nothing. Top candidates:\n`)
|
|
252
|
+
ranked.slice(0, 3).forEach((idx, r) => {
|
|
253
|
+
console.log(`# ${r + 1} attempt ${idx} (${diffs[idx].length} chars)\n${diffs[idx].slice(0, 4000) || "(no changes)"}\n`)
|
|
254
|
+
})
|
|
255
|
+
}
|
|
256
|
+
} finally {
|
|
257
|
+
for (const wt of worktrees) await git(repo, "worktree", "remove", "--force", wt).catch(() => {})
|
|
258
|
+
await rm(work, { recursive: true, force: true }).catch(() => {})
|
|
259
|
+
}
|
|
260
|
+
}
|
|
261
|
+
|
|
262
|
+
main().catch((e) => { err(`agent-ultramode: ${e?.message || e}`); process.exit(1) })
|
package/package.json
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "agent-ultramode",
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Best-of-N for coding agents with a same-model verifier: run your task N times in isolated git worktrees and apply the verified winner. A /ultra command for opencode.",
|
|
5
|
+
"module": "ultra.ts",
|
|
6
|
+
"main": "ultra.ts",
|
|
7
|
+
"exports": {
|
|
8
|
+
".": "./ultra.ts"
|
|
9
|
+
},
|
|
10
|
+
"type": "module",
|
|
11
|
+
"bin": {
|
|
12
|
+
"agent-ultramode": "cli.mjs"
|
|
13
|
+
},
|
|
14
|
+
"engines": {
|
|
15
|
+
"node": ">=18"
|
|
16
|
+
},
|
|
17
|
+
"files": [
|
|
18
|
+
"ultra.ts",
|
|
19
|
+
"cli.mjs",
|
|
20
|
+
"README.md",
|
|
21
|
+
"LICENSE"
|
|
22
|
+
],
|
|
23
|
+
"keywords": [
|
|
24
|
+
"opencode",
|
|
25
|
+
"opencode-plugin",
|
|
26
|
+
"plugin",
|
|
27
|
+
"best-of-n",
|
|
28
|
+
"verifier",
|
|
29
|
+
"llm-as-a-verifier",
|
|
30
|
+
"coding-agent",
|
|
31
|
+
"ultra"
|
|
32
|
+
],
|
|
33
|
+
"license": "MIT",
|
|
34
|
+
"author": "maverick-tr (https://github.com/maverick-tr)",
|
|
35
|
+
"repository": {
|
|
36
|
+
"type": "git",
|
|
37
|
+
"url": "git+https://github.com/maverick-tr/agent-ultramode.git"
|
|
38
|
+
},
|
|
39
|
+
"devDependencies": {
|
|
40
|
+
"@opencode-ai/plugin": "^1.0.153",
|
|
41
|
+
"@types/bun": "latest"
|
|
42
|
+
},
|
|
43
|
+
"peerDependencies": {
|
|
44
|
+
"typescript": "^5"
|
|
45
|
+
}
|
|
46
|
+
}
|
package/ultra.ts
ADDED
|
@@ -0,0 +1,381 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* ultra - best-of-N trajectories + confidence-gated same-model verifier, for opencode.
|
|
3
|
+
*
|
|
4
|
+
* `/ultra <task>` runs the task N times in isolated git worktrees, ranks the results
|
|
5
|
+
* with the same model acting as a verifier (a Probabilistic Pivot Tournament, from
|
|
6
|
+
* llm-as-a-verifier), then:
|
|
7
|
+
* high confidence -> applies the winning diff to your working tree
|
|
8
|
+
* low confidence -> shows you the top candidates and applies nothing
|
|
9
|
+
* (a low-confidence pick is a coin flip and should not be applied silently).
|
|
10
|
+
*
|
|
11
|
+
* This is the version validated on Terminal-Bench: plan-first best-of-N gave no edge,
|
|
12
|
+
* but best-of-N over full trajectories + this verifier lifted the recoverable tasks
|
|
13
|
+
* from 40% to 75% (n=15, same model, no cross-model dependency).
|
|
14
|
+
*
|
|
15
|
+
* Install (opencode.json):
|
|
16
|
+
* { "plugin": [ ["file:///abs/path/ultra.ts", { "model": "provider/model", "n": 5 }] ] }
|
|
17
|
+
* Then: /ultra fix the failing test in foo/bar
|
|
18
|
+
*
|
|
19
|
+
* Zero deps: node built-ins + global fetch + an OpenAI-compatible endpoint.
|
|
20
|
+
*/
|
|
21
|
+
import type { Plugin } from "@opencode-ai/plugin"
|
|
22
|
+
import { tool } from "@opencode-ai/plugin"
|
|
23
|
+
import { execFile } from "node:child_process"
|
|
24
|
+
import { promisify } from "node:util"
|
|
25
|
+
import { mkdtemp, rm, writeFile, mkdir } from "node:fs/promises"
|
|
26
|
+
import { existsSync } from "node:fs"
|
|
27
|
+
import { tmpdir, homedir } from "node:os"
|
|
28
|
+
import { join } from "node:path"
|
|
29
|
+
|
|
30
|
+
const execFileP = promisify(execFile)
|
|
31
|
+
type Opts = Record<string, any>
|
|
32
|
+
const VERSION = "0821g"
|
|
33
|
+
|
|
34
|
+
interface Cfg {
|
|
35
|
+
url: string
|
|
36
|
+
key: string
|
|
37
|
+
model: string
|
|
38
|
+
provider: string
|
|
39
|
+
effort: string
|
|
40
|
+
agent: string
|
|
41
|
+
n: number
|
|
42
|
+
k: number
|
|
43
|
+
topk: number
|
|
44
|
+
conf: number
|
|
45
|
+
judgeTokens: number
|
|
46
|
+
cc: number
|
|
47
|
+
agentTimeoutMs: number
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
const num = (v: any, def: number, lo: number, hi: number): number => {
|
|
51
|
+
const n = Number.parseInt(String(v ?? ""), 10)
|
|
52
|
+
return Number.isFinite(n) ? Math.max(lo, Math.min(hi, n)) : def
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
function buildCfg(opts: Opts): Cfg {
|
|
56
|
+
// Endpoint + model default to the CURRENT SESSION model (captured in the chat.params hook
|
|
57
|
+
// below); plugin options and ULTRA_* env vars override. The per-attempt agent defaults to
|
|
58
|
+
// opencode's own run command, and any other CLI works via the agent option.
|
|
59
|
+
const ref = String(opts.model ?? process.env.ULTRA_MODEL ?? "")
|
|
60
|
+
const s = ref.indexOf("/")
|
|
61
|
+
const provider = s >= 0 ? ref.slice(0, s) : String(opts.provider ?? process.env.ULTRA_PROVIDER ?? "")
|
|
62
|
+
return {
|
|
63
|
+
url: String(opts.baseURL ?? process.env.ULTRA_BASE_URL ?? ""),
|
|
64
|
+
key: String(opts.apiKey ?? process.env.ULTRA_API_KEY ?? ""),
|
|
65
|
+
model: s >= 0 ? ref.slice(s + 1) : ref,
|
|
66
|
+
provider,
|
|
67
|
+
// "none" keeps the verifier fast and non-truncating.
|
|
68
|
+
effort: String(opts.effort ?? process.env.ULTRA_EFFORT ?? "none"),
|
|
69
|
+
// the agent to run in each worktree; {task} is substituted, CWD = the worktree.
|
|
70
|
+
agent: String(opts.agent ?? process.env.ULTRA_AGENT ?? 'opencode run "{task}"'),
|
|
71
|
+
n: num(opts.n ?? process.env.ULTRA_N, 4, 2, 8),
|
|
72
|
+
k: num(opts.k ?? process.env.ULTRA_K, 3, 1, 7),
|
|
73
|
+
topk: 2,
|
|
74
|
+
conf: Number(opts.conf ?? process.env.ULTRA_CONF ?? 0.34),
|
|
75
|
+
judgeTokens: num(opts.judgeTokens ?? process.env.ULTRA_JUDGE_TOKENS, 12000, 500, 200000),
|
|
76
|
+
cc: num(opts.concurrency ?? process.env.ULTRA_CC, 6, 1, 12),
|
|
77
|
+
agentTimeoutMs: num(opts.agentTimeout ?? process.env.ULTRA_AGENT_TIMEOUT, 600000, 30000, 1800000),
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
function pLimit(max: number) {
|
|
82
|
+
let active = 0
|
|
83
|
+
const q: (() => void)[] = []
|
|
84
|
+
const next = () => { active--; q.shift()?.() }
|
|
85
|
+
return async <T>(fn: () => Promise<T>): Promise<T> => {
|
|
86
|
+
if (active >= max) await new Promise<void>((r) => q.push(r))
|
|
87
|
+
active++
|
|
88
|
+
try { return await fn() } finally { next() }
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
async function git(cwd: string, ...args: string[]): Promise<string> {
|
|
93
|
+
const { stdout } = await execFileP("git", args, { cwd, maxBuffer: 32 * 1024 * 1024 })
|
|
94
|
+
return stdout
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
// one non-streaming verifier call; a failure is swallowed by the caller (null vote).
|
|
98
|
+
async function chat(C: Cfg, content: string): Promise<string> {
|
|
99
|
+
const body: Record<string, any> = {
|
|
100
|
+
model: C.model,
|
|
101
|
+
messages: [{ role: "user", content }],
|
|
102
|
+
max_tokens: C.judgeTokens,
|
|
103
|
+
temperature: 0.9,
|
|
104
|
+
}
|
|
105
|
+
if (C.effort) body.reasoning_effort = C.effort
|
|
106
|
+
const res = await fetch(`${C.url.replace(/\/$/, "")}/chat/completions`, {
|
|
107
|
+
method: "POST",
|
|
108
|
+
headers: { Authorization: `Bearer ${C.key}`, "Content-Type": "application/json" },
|
|
109
|
+
body: JSON.stringify(body),
|
|
110
|
+
})
|
|
111
|
+
if (!res.ok) throw new Error(`HTTP ${res.status}`)
|
|
112
|
+
const j: any = await res.json()
|
|
113
|
+
const m = j?.choices?.[0]?.message ?? {}
|
|
114
|
+
return `${m.content ?? ""}\n${m.reasoning_content ?? ""}`
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
async function judge(C: Cfg, limit: ReturnType<typeof pLimit>, task: string, a: string, b: string): Promise<"A" | "B" | null> {
|
|
118
|
+
const prompt =
|
|
119
|
+
`A coding task and two candidate solutions (agent log tail + the diff each produced). Decide which ` +
|
|
120
|
+
`more likely FULLY and correctly accomplishes the task. Be skeptical: prefer concrete evidence of a ` +
|
|
121
|
+
`correct, complete change with no errors; a diff that looks plausible but is incomplete or wrong should ` +
|
|
122
|
+
`lose.\n\nTASK:\n${task}\n\n=== A ===\n${a}\n\n=== B ===\n${b}\n\n` +
|
|
123
|
+
`Reason briefly, then end with exactly one line: 'FINAL: A' or 'FINAL: B'.`
|
|
124
|
+
try {
|
|
125
|
+
const out = (await limit(() => chat(C, prompt))).toUpperCase()
|
|
126
|
+
const ms = out.match(/FINAL:\s*([AB])/g)
|
|
127
|
+
return ms ? (ms[ms.length - 1].slice(-1) as "A" | "B") : null
|
|
128
|
+
} catch {
|
|
129
|
+
return null
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
// Probabilistic Pivot Tournament -> ranked indices + confidence (top1-top2 win-ratio margin).
|
|
134
|
+
async function tournament(C: Cfg, task: string, summaries: string[]): Promise<{ ranked: number[]; conf: number }> {
|
|
135
|
+
const n = summaries.length
|
|
136
|
+
const ids = Array.from({ length: n }, (_, i) => i)
|
|
137
|
+
if (n <= 1) return { ranked: ids, conf: 1 }
|
|
138
|
+
const limit = pLimit(C.cc)
|
|
139
|
+
const order = [...ids]
|
|
140
|
+
for (let i = order.length - 1; i > 0; i--) { const j = Math.floor(Math.random() * (i + 1)); [order[i], order[j]] = [order[j], order[i]] }
|
|
141
|
+
const ring: Record<number, number> = Object.fromEntries(ids.map((i) => [i, 0]))
|
|
142
|
+
await Promise.all(order.map(async (id, i) => {
|
|
143
|
+
const b = order[(i + 1) % n]
|
|
144
|
+
const v = await judge(C, limit, task, summaries[id], summaries[b])
|
|
145
|
+
ring[v === "B" ? b : id]++
|
|
146
|
+
}))
|
|
147
|
+
const pivots = [...ids].sort((x, y) => ring[y] - ring[x]).slice(0, Math.min(C.topk, n))
|
|
148
|
+
const nonp = ids.filter((i) => !pivots.includes(i))
|
|
149
|
+
const pairs: [number, number][] = []
|
|
150
|
+
for (const np of nonp) for (const pv of pivots) pairs.push([np, pv])
|
|
151
|
+
for (let i = 0; i < pivots.length; i++) for (let j = i + 1; j < pivots.length; j++) pairs.push([pivots[i], pivots[j]])
|
|
152
|
+
const mass: Record<number, number> = Object.fromEntries(ids.map((i) => [i, 0]))
|
|
153
|
+
const games: Record<number, number> = Object.fromEntries(ids.map((i) => [i, 0]))
|
|
154
|
+
await Promise.all(pairs.map(async ([x, y]) => {
|
|
155
|
+
const votes = await Promise.all(Array.from({ length: C.k }, () => judge(C, limit, task, summaries[x], summaries[y])))
|
|
156
|
+
const na = votes.filter((v) => v === "A").length
|
|
157
|
+
const nb = votes.filter((v) => v === "B").length
|
|
158
|
+
const px = na + nb === 0 ? 0.5 : na / (na + nb)
|
|
159
|
+
mass[x] += px; mass[y] += 1 - px; games[x]++; games[y]++
|
|
160
|
+
}))
|
|
161
|
+
const ratio = (i: number) => (games[i] ? mass[i] / games[i] : ring[i] / Math.max(1, n - 1))
|
|
162
|
+
const ranked = [...ids].sort((a, b) => ratio(b) - ratio(a))
|
|
163
|
+
return { ranked, conf: ratio(ranked[0]) - ratio(ranked[1]) }
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
// ---- lean sandbox for sub-agent attempts ------------------------------------------------
|
|
167
|
+
// Sub-agents run under XDG_CONFIG_HOME -> a provider-only config so they skip the user's MCP
|
|
168
|
+
// servers and plugins (the slow part). Deps install once into the sandbox and cache forever.
|
|
169
|
+
function leanConfig(C: Cfg): string {
|
|
170
|
+
return JSON.stringify({
|
|
171
|
+
$schema: "https://opencode.ai/config.json",
|
|
172
|
+
model: `${C.provider || "provider"}/${C.model}`,
|
|
173
|
+
permission: { edit: "allow", bash: "allow", webfetch: "allow" },
|
|
174
|
+
provider: {
|
|
175
|
+
[C.provider || "provider"]: {
|
|
176
|
+
npm: "@ai-sdk/openai-compatible",
|
|
177
|
+
name: C.provider || "provider",
|
|
178
|
+
options: { baseURL: C.url, apiKey: C.key },
|
|
179
|
+
models: { [C.model]: { name: C.model } },
|
|
180
|
+
},
|
|
181
|
+
},
|
|
182
|
+
}, null, 2)
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
function appFromAgent(_agent: string): string {
|
|
186
|
+
// The lean config is written under XDG/opencode/; opencode reads it, other agents ignore XDG.
|
|
187
|
+
return "opencode"
|
|
188
|
+
}
|
|
189
|
+
|
|
190
|
+
async function ensureLeanSandbox(C: Cfg, app: string): Promise<{ home: string; needsWarm: boolean }> {
|
|
191
|
+
const home = join(homedir(), ".cache", "agent-ultramode")
|
|
192
|
+
const appDir = join(home, app)
|
|
193
|
+
await mkdir(appDir, { recursive: true })
|
|
194
|
+
await writeFile(join(appDir, `${app}.json`), leanConfig(C))
|
|
195
|
+
return { home, needsWarm: !existsSync(join(appDir, "node_modules")) }
|
|
196
|
+
}
|
|
197
|
+
|
|
198
|
+
export const Ultra: Plugin = async (_input, options) => {
|
|
199
|
+
const C = buildCfg((options as Opts) ?? {})
|
|
200
|
+
const current = { model: "", url: "", key: "" }
|
|
201
|
+
const client = (_input as any)?.client
|
|
202
|
+
// load marker: lets us confirm which build actually loaded.
|
|
203
|
+
writeFile(join(homedir(), ".cache", "agent-ultramode-loaded.txt"), `ultra ${VERSION} loaded ${new Date().toISOString()}\n`).catch(() => {})
|
|
204
|
+
|
|
205
|
+
const ultra = tool({
|
|
206
|
+
description:
|
|
207
|
+
"Best-of-N with a verifier. Runs a coding/terminal task N times in isolated git worktrees, ranks the " +
|
|
208
|
+
"attempts with the same model as a verifier, and applies the winning diff when confident (otherwise " +
|
|
209
|
+
"reports the top candidates). Use for a task worth spending compute to get right the first time.",
|
|
210
|
+
args: {
|
|
211
|
+
task: tool.schema.string().describe("The task to solve N times and verify. Be specific."),
|
|
212
|
+
},
|
|
213
|
+
async execute(args, ctx) {
|
|
214
|
+
const task = String((args as any)?.task ?? "").trim()
|
|
215
|
+
if (!task) return "Pass a `task` to run."
|
|
216
|
+
const eff: Cfg = { ...C, model: C.model || current.model, url: C.url || current.url, key: C.key || current.key }
|
|
217
|
+
if (!eff.url || !eff.model) return "ultra could not resolve a verifier model. Set `model` in the plugin options, or send a message first so it can use your current model."
|
|
218
|
+
|
|
219
|
+
let dir = (ctx as any).worktree || (ctx as any).directory || ""
|
|
220
|
+
if (!dir) {
|
|
221
|
+
// some hosts omit directory/worktree on the tool context; resolve the repo root from cwd.
|
|
222
|
+
try { dir = (await git(process.cwd(), "rev-parse", "--show-toplevel")).trim() } catch { dir = process.cwd() }
|
|
223
|
+
}
|
|
224
|
+
let base: string
|
|
225
|
+
try { base = (await git(dir, "rev-parse", "HEAD")).trim() } catch { return "ultra needs a git repo with a HEAD to branch attempts from (run it inside your project)." }
|
|
226
|
+
|
|
227
|
+
const work = await mkdtemp(join(tmpdir(), "ultra-"))
|
|
228
|
+
const worktrees: string[] = []
|
|
229
|
+
const diffs: string[] = []
|
|
230
|
+
const summaries: string[] = []
|
|
231
|
+
const status = (title: string, body?: string) => {
|
|
232
|
+
try { ctx.metadata({ title, metadata: body ? { output: body } : {} }) } catch {}
|
|
233
|
+
}
|
|
234
|
+
// ctx.metadata renders in the web UI tool card; toasts render in the terminal TUI.
|
|
235
|
+
const toast = (message: string, variant: "info" | "success" | "warning" = "info") => {
|
|
236
|
+
try { client?.tui?.showToast?.({ body: { message: `ultra: ${message}`, variant, duration: 4000 } })?.catch?.(() => {}) } catch {}
|
|
237
|
+
}
|
|
238
|
+
try {
|
|
239
|
+
toast(`v${VERSION}: starting ${eff.n} attempts`)
|
|
240
|
+
// Each attempt runs the agent in a LEAN sandbox: XDG_CONFIG_HOME points opencode at a
|
|
241
|
+
// provider-only config (no MCP, no plugins, permission:allow), so an
|
|
242
|
+
// attempt is a fast model turn instead of booting the whole environment N times.
|
|
243
|
+
const app = appFromAgent(eff.agent)
|
|
244
|
+
const { home: leanHome, needsWarm } = await ensureLeanSandbox(eff, app)
|
|
245
|
+
// Shared: config dir (warm node_modules + lean config). Per-run: XDG_DATA/STATE/CACHE,
|
|
246
|
+
// so each sub-run gets its OWN opencode.db. One shared DB across the main session + N
|
|
247
|
+
// parallel runs deadlocks on the SQLite write lock (all idle, nothing written).
|
|
248
|
+
const baseEnv = {
|
|
249
|
+
...process.env,
|
|
250
|
+
XDG_CONFIG_HOME: leanHome,
|
|
251
|
+
OPENCODE_DISABLE_DEFAULT_PLUGINS: "1",
|
|
252
|
+
OPENCODE_DISABLE_AUTOUPDATE: "1",
|
|
253
|
+
OPENCODE_DISABLE_MODELS_FETCH: "1",
|
|
254
|
+
}
|
|
255
|
+
const isolatedEnv = async (tag: string) => {
|
|
256
|
+
const d = join(work, `xdg-${tag}`)
|
|
257
|
+
await mkdir(d, { recursive: true }).catch(() => {})
|
|
258
|
+
return { ...baseEnv, XDG_DATA_HOME: d, XDG_STATE_HOME: d, XDG_CACHE_HOME: d }
|
|
259
|
+
}
|
|
260
|
+
const cmd = eff.agent.replace("{task}", task.replace(/"/g, '\\"'))
|
|
261
|
+
|
|
262
|
+
if (needsWarm) {
|
|
263
|
+
// First run on this machine installs the sandbox deps once (cached afterwards).
|
|
264
|
+
status(`ultra: one-time sandbox setup (installing ${app} sandbox deps, ~1-2 min)...`)
|
|
265
|
+
const warm = await mkdtemp(join(tmpdir(), "ultra-warm-"))
|
|
266
|
+
await execFileP("bash", ["-lc", "git init -q"], { cwd: warm }).catch(() => {})
|
|
267
|
+
const warmCmd = eff.agent.replace("{task}", "reply with exactly: ready. do not create or edit files.")
|
|
268
|
+
await execFileP("bash", ["-lc", `exec </dev/null; ${warmCmd}`], { cwd: warm, env: await isolatedEnv("warm"), timeout: eff.agentTimeoutMs, maxBuffer: 8 * 1024 * 1024 }).catch(() => {})
|
|
269
|
+
await rm(warm, { recursive: true, force: true }).catch(() => {})
|
|
270
|
+
}
|
|
271
|
+
|
|
272
|
+
// Create N detached worktrees off HEAD (serial: git worktree add takes a repo lock).
|
|
273
|
+
status(`ultra: preparing ${eff.n} isolated worktrees...`)
|
|
274
|
+
for (let i = 0; i < eff.n; i++) {
|
|
275
|
+
const wt = join(work, `attempt-${i}`)
|
|
276
|
+
await git(dir, "worktree", "add", "--detach", wt, base)
|
|
277
|
+
worktrees.push(wt)
|
|
278
|
+
}
|
|
279
|
+
|
|
280
|
+
// Run the agent in each worktree IN PARALLEL (lean sandbox keeps this cheap).
|
|
281
|
+
const limit = pLimit(Math.max(1, Math.min(eff.cc, eff.n)))
|
|
282
|
+
let done = 0
|
|
283
|
+
status(`ultra: running ${eff.n} attempts in parallel...`)
|
|
284
|
+
toast(`running ${eff.n} attempts in parallel`)
|
|
285
|
+
const outs = await Promise.all(worktrees.map((wt, i) => limit(async () => {
|
|
286
|
+
const env = await isolatedEnv(String(i))
|
|
287
|
+
let log = ""
|
|
288
|
+
try {
|
|
289
|
+
const { stdout, stderr } = await execFileP("bash", ["-lc", `exec </dev/null; ${cmd}`], { cwd: wt, env, timeout: eff.agentTimeoutMs, maxBuffer: 32 * 1024 * 1024, signal: ctx.abort })
|
|
290
|
+
log = (stdout || "") + (stderr || "")
|
|
291
|
+
} catch (e: any) { log = `agent error: ${e?.message || e}` }
|
|
292
|
+
await git(wt, "add", "-A").catch(() => {})
|
|
293
|
+
const diff = await git(wt, "diff", "--cached").catch(() => "")
|
|
294
|
+
done++
|
|
295
|
+
status(`ultra: ${done}/${eff.n} attempts done`, `attempt ${i}: ${diff ? diff.length + " diff chars" : "no changes"}`)
|
|
296
|
+
return { diff, summary: `AGENT LOG (tail):\n${log.slice(-1500)}\n\nDIFF:\n${diff.slice(0, 6000) || "(no changes)"}` }
|
|
297
|
+
})))
|
|
298
|
+
for (const o of outs) { diffs.push(o.diff); summaries.push(o.summary) }
|
|
299
|
+
|
|
300
|
+
if (diffs.every((d) => !d.trim())) {
|
|
301
|
+
return `ultra ran ${eff.n} attempts but none made any changes (all diffs empty). The sub-agent could not act: the task may be too ambiguous, or the sandbox model could not reach its endpoint. Try a more specific task, or check the model endpoint.`
|
|
302
|
+
}
|
|
303
|
+
|
|
304
|
+
status("ultra: verifying...")
|
|
305
|
+
toast(`verifying ${eff.n} candidates`)
|
|
306
|
+
const { ranked, conf } = await tournament(eff, task, summaries)
|
|
307
|
+
const best = ranked[0]
|
|
308
|
+
|
|
309
|
+
// Normalised added-lines of a diff, so we can tell when attempts AGREE on the same change.
|
|
310
|
+
const nonEmpty = diffs.filter((d) => d.trim()).length
|
|
311
|
+
// Apply the verifier's TOP pick when it is confident, OR when a majority of attempts
|
|
312
|
+
// produced a change (the task was solvable; the top-ranked diff is a good starting point
|
|
313
|
+
// and you review it before committing). Only mostly-empty runs are handed back as candidates.
|
|
314
|
+
const majority = nonEmpty >= Math.max(2, Math.ceil(eff.n / 2))
|
|
315
|
+
if ((conf >= eff.conf || majority) && diffs[best].trim()) {
|
|
316
|
+
const patch = join(work, "winner.patch")
|
|
317
|
+
await writeFile(patch, diffs[best])
|
|
318
|
+
const confNote = conf >= eff.conf ? `high confidence ${conf.toFixed(2)}` : `top pick of ${nonEmpty}/${eff.n}, margin ${conf.toFixed(2)}`
|
|
319
|
+
try {
|
|
320
|
+
await git(dir, "apply", "--3way", patch)
|
|
321
|
+
toast("applied the winning change", "success")
|
|
322
|
+
return `🏆 ultra applied the best of ${eff.n} attempts (${confNote}). The winning change is now in your working tree; review it before committing.`
|
|
323
|
+
} catch (e: any) {
|
|
324
|
+
return `ultra picked the best attempt (${confNote}) but the patch did not apply cleanly (${e?.message || e}). The diff:\n\n\`\`\`diff\n${diffs[best].slice(0, 6000)}\n\`\`\``
|
|
325
|
+
}
|
|
326
|
+
}
|
|
327
|
+
const top = ranked.slice(0, 3).map((i, r) => `### Candidate ${r + 1} (attempt ${i})\n\`\`\`diff\n${diffs[i].slice(0, 4000) || "(no changes)"}\n\`\`\``).join("\n\n")
|
|
328
|
+
return `ultra ran ${eff.n} attempts but most produced no usable change, so it applied nothing. Top candidates:\n\n${top}`
|
|
329
|
+
} finally {
|
|
330
|
+
for (const wt of worktrees) await git(dir, "worktree", "remove", "--force", wt).catch(() => {})
|
|
331
|
+
await rm(work, { recursive: true, force: true }).catch(() => {})
|
|
332
|
+
}
|
|
333
|
+
},
|
|
334
|
+
})
|
|
335
|
+
|
|
336
|
+
return {
|
|
337
|
+
tool: { ultra },
|
|
338
|
+
"chat.params": async (input: any) => {
|
|
339
|
+
try {
|
|
340
|
+
const id = input?.model?.id ?? input?.model?.modelID
|
|
341
|
+
const o = input?.provider?.options
|
|
342
|
+
if (id && o?.baseURL) { current.model = id; current.url = o.baseURL; if (o.apiKey) current.key = o.apiKey }
|
|
343
|
+
} catch {}
|
|
344
|
+
},
|
|
345
|
+
config: async (cfg: any) => {
|
|
346
|
+
cfg.command = {
|
|
347
|
+
ultra: {
|
|
348
|
+
description: "Best-of-N + verifier: run the task N times in isolated worktrees, apply the best. /ultra <task>",
|
|
349
|
+
template:
|
|
350
|
+
"Call the `ultra` tool exactly once, with `task` set to the request below. It runs the task several " +
|
|
351
|
+
"times in isolated git worktrees and verifies the results. When it returns, report to the user exactly " +
|
|
352
|
+
"what it did (applied the winner, or listed candidates) verbatim; do not re-do the task yourself.\n\nTask: $ARGUMENTS",
|
|
353
|
+
},
|
|
354
|
+
...(cfg.command ?? {}),
|
|
355
|
+
}
|
|
356
|
+
// Register an "ultra" primary agent so it shows in the tab / agent list; selecting it
|
|
357
|
+
// routes each request through best-of-N. Sub-attempts run the default agent, so no recursion.
|
|
358
|
+
cfg.agent = {
|
|
359
|
+
ultra: {
|
|
360
|
+
description: "Best-of-N mode: run the request N times in isolated worktrees and apply the verified winner.",
|
|
361
|
+
mode: "primary",
|
|
362
|
+
color: "#A855F7",
|
|
363
|
+
prompt:
|
|
364
|
+
"You are in ULTRA mode (best-of-N with a same-model verifier). For the user's coding or terminal " +
|
|
365
|
+
"request, call the `ultra` tool exactly once with the full request as `task`. When it returns, report " +
|
|
366
|
+
"exactly what it did (applied the winner, or listed the candidates) verbatim; do not do the task yourself.",
|
|
367
|
+
},
|
|
368
|
+
...(cfg.agent ?? {}),
|
|
369
|
+
}
|
|
370
|
+
try {
|
|
371
|
+
if (C.provider && cfg?.provider?.[C.provider]) {
|
|
372
|
+
const p = cfg.provider[C.provider]
|
|
373
|
+
if (!C.url && p.options?.baseURL) C.url = p.options.baseURL
|
|
374
|
+
if (!C.key && p.options?.apiKey) C.key = p.options.apiKey
|
|
375
|
+
}
|
|
376
|
+
} catch {}
|
|
377
|
+
},
|
|
378
|
+
}
|
|
379
|
+
}
|
|
380
|
+
|
|
381
|
+
export default Ultra
|