mini-coder 0.5.12 → 0.5.13

package/BENCHMARK.md

@@ -1,10 +1,16 @@
+# CORE GOAL
+
+**Terminal-Bench is a signal, not the product. The target is a better coding agent, not a higher benchmark score from benchmark-shaped patches.**
+
+- Do not add fixes to fix an issue with specific terminal bench evals, focus on improving the agent's behaviour.
+- Benchmarks run headless mode in one shot. Make sure you use `tmux` often to check that the TUI and multiturn UX is still
+  good.
+
 # Benchmark workflow
 
 Use this as the default tuning loop for `mini-coder` on Terminal-Bench.
 
-The goal is not to rerun the whole benchmark after every change. The goal is to get fast enough feedback that small prompt / loop / tool-use changes can be judged quickly, then promote only the promising ones to bigger runs.
-
-Terminal-Bench is a signal, not the product. The target is a better coding agent, not a higher benchmark score from benchmark-shaped patches.
+The goal is not to rerun the whole benchmark after every change. The goal is to get fast enough feedback that small changes can be judged quickly, then promote only the promising ones to bigger runs.
 
 ## Principles
 
@@ -19,39 +25,8 @@ Terminal-Bench is a signal, not the product. The target is a better coding agent
 
 ## Suites
 
-### 1. Guardrail suite
-
-Use this to catch regressions on known-good behavior.
-
-Tasks:
-
-- `cancel-async-tasks`
-- `bn-fit-modify`
-- `sparql-university`
-- `extract-elf`
-
-Settings:
-
-- `1` attempt
-- `2` concurrent
-- `0` retries
-
-Estimated wall time:
-
-- about `8–10 minutes`
-
-### 2. Fast suite
-
-Use this after almost every small change.
-
-Tasks:
-
-- `polyglot-rust-c`
-- `overfull-hbox`
-- `gcode-to-text`
-- `mteb-retrieve`
-- `dna-insert`
-- `torch-tensor-parallelism`
+There is a full 89 test baseline run with 2 attemps in the teminal-bench folder. Use the evals in it
+to determine your fast evals to start your optimization process and iterations.
 
 Settings:
 
@@ -59,67 +34,6 @@ Settings:
 - `2` concurrent
 - `0` retries
 
-Estimated wall time:
-
-- about `44 minutes`
-
-### 3. Focused suite
-
-Use this only if a change looks good on the fast suite.
-
-Tasks:
-
-- `polyglot-rust-c`
-- `overfull-hbox`
-- `gcode-to-text`
-- `mteb-retrieve`
-- `torch-tensor-parallelism`
-- `filter-js-from-html`
-- `configure-git-webserver`
-- `qemu-alpine-ssh`
-
-Settings:
-
-- `2` attempts
-- `2` concurrent
-- `0` retries
-
-Estimated wall time:
-
-- about `1.1 hours`
-
-### 4. Broad promotion suite
-
-Use this after 1–3 promising changes, or overnight.
-
-Options:
-
-- rerun recent failures
-- wider local failure suite
-- full leaderboard-style run
-
-Typical settings for rerunning recent failures:
-
-- `4` attempts
-- `2` concurrent
-- `0` retries
-
-## Baseline procedure
-
-Before changing code:
-
-1. run the guardrail suite
-2. run the fast suite
-3. run the fast suite again
-
-That gives a fresh local baseline for:
-
-- pass count
-- runtime
-- variance / noise
-
-Do not compare a new change only against an old run from days ago if a fresh same-HEAD baseline is available.
-
 ## Experiment quality bar
 
 Before changing code, write the hypothesis in two layers:
@@ -133,13 +47,6 @@ Only run an experiment if you can answer all of these:
 - why should that help outside Terminal-Bench?
 - what would make this change obviously overfit?
 
-Good experiment themes:
-
-- verification-equivalence before completion
-- artifact-grounded verification of final outputs
-- preferring a task-named local source of truth over approximations
-- reducing shell thrash before first meaningful verification
-
 Reject or redesign experiments that:
 
 - depend on benchmark-specific task names, file names, package names, or tool names in product logic
@@ -162,108 +69,13 @@ For each change:
 5. form **one** narrow hypothesis
 6. make **one** small change
 7. run:
-   - guardrails
    - fast suite
-8. compare to baseline
+8. compare to baseline or your reference run
 9. decide:
    - keep
    - revert
    - refine
 
-Only run the focused suite if:
-
-- the fast suite improved, and
-- the guardrails did not regress
-
-Only run the broad promotion suite if:
-
-- the focused suite also looks good, or
-- enough promising changes have accumulated to justify it
-
-## Decision rules
-
-Treat the fast suite as a noisy but useful signal.
-
-For the 12-trial fast suite:
-
-- `+2` or more passes: probably meaningful improvement
-- `-2` or more passes: probably meaningful regression
-- `±1`: likely noise unless it repeats
-
-Keep a change if:
-
-- the fast suite improves materially
-- guardrails do not regress
-- the result repeats on another fast-suite rerun
-- the mechanism is still a general agent-quality improvement, not just a benchmark-specific patch
-
-Revert a change if:
-
-- it clearly loses fast-suite passes, or
-- it regresses guardrails
-
-Call it inconclusive if:
-
-- the delta is tiny and does not repeat
-
-Also prefer:
-
-- a smaller, more general improvement over a larger but obviously benchmark-specific patch
-- a clearly exercised mechanism over a theory that never fired in the target runs
-
-## Failure buckets to optimize against
-
-Use these as the main buckets when inspecting logs:
-
-### Exact-contract / cleanup misses
-
-Examples:
-
-- right artifact, wrong path
-- right output plus extra junk
-- correct core work, but violates a strict file / in-place requirement
-
-### Verification-equivalence / premature completion
-
-Examples:
-
-- waited too long to run a meaningful check
-- never checked the exact contract
-- stopped after a weaker non-equivalent check
-- verified the intended design instead of the written artifact
-
-### Source-of-truth selection gaps
-
-Examples:
-
-- task names an exact local tool/package/interface, but the agent uses an approximation instead
-- lower-level library reasoning replaces package-local or task-local semantics
-
-### Over-exploration / shell thrash
-
-Examples:
-
-- too many shell / read steps before first write
-- repeated discovery commands without narrowing the problem
-
-### Persistent correctness gaps
-
-Examples:
-
-- `torch-tensor-parallelism`
-- `filter-js-from-html`
-- `configure-git-webserver`
-
-### Long-horizon / timeout-heavy tasks
-
-These are useful for promotion runs, not tight inner loops.
-Examples:
-
-- `gpt2-codegolf`
-- `query-optimize`
-- `winning-avg-corewars`
-- `train-fasttext`
-
 ## Behavior analysis requirements
 
 Behavior analysis depends on structured agent logs.
@@ -281,128 +93,15 @@ Per trial, keep:
 - structured `agent/mini-coder.ndjson`
 - timestamps
 
-With those artifacts, analyze things like:
-
-- time to first tool call
-- time to first edit
-- number of `shell` / `read` / `grep` / `edit` calls
-- whether the agent ran a verifier-like shell command
-- whether it used the named local source of truth when one was available
-- whether it verified the final artifact it actually wrote
-- whether a dynamic intervention visibly exercised
-- whether it had a local pass before ending
-- whether it kept changing files after a pass-worthy state
-- whether it left extra artifacts
-
-## Commands
-
-### Guardrail suite
-
-```bash
-GUARDRAIL_TASKS=(
-  cancel-async-tasks
-  bn-fit-modify
-  sparql-university
-  extract-elf
-)
-
-cmd=(
-  harbor run -y
-  --job-name local-guardrails-$(date +%F__%H-%M-%S)
-  --jobs-dir "$PWD/terminal-bench/jobs"
-  --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
-  --model openai-codex/gpt-5.4
-  --dataset terminal-bench@2.0
-  --n-attempts 1
-  --n-concurrent 2
-  --max-retries 0
-)
-
-for t in "${GUARDRAIL_TASKS[@]}"; do
-  cmd+=(--include-task-name "$t")
-done
-
-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
-```
-
-### Fast suite
-
-```bash
-FAST_TASKS=(
-  polyglot-rust-c
-  overfull-hbox
-  gcode-to-text
-  mteb-retrieve
-  dna-insert
-  torch-tensor-parallelism
-)
-
-cmd=(
-  harbor run -y
-  --job-name local-fast-$(date +%F__%H-%M-%S)
-  --jobs-dir "$PWD/terminal-bench/jobs"
-  --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
-  --model openai-codex/gpt-5.4
-  --dataset terminal-bench@2.0
-  --n-attempts 2
-  --n-concurrent 2
-  --max-retries 0
-)
-
-for t in "${FAST_TASKS[@]}"; do
-  cmd+=(--include-task-name "$t")
-done
-
-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
-```
-
-### Focused suite
-
-```bash
-FOCUS_TASKS=(
-  polyglot-rust-c
-  overfull-hbox
-  gcode-to-text
-  mteb-retrieve
-  torch-tensor-parallelism
-  filter-js-from-html
-  configure-git-webserver
-  qemu-alpine-ssh
-)
-
-cmd=(
-  harbor run -y
-  --job-name local-focus-$(date +%F__%H-%M-%S)
-  --jobs-dir "$PWD/terminal-bench/jobs"
-  --agent-import-path mini_coder_local_agent:MiniCoderLocalAgent
-  --model openai-codex/gpt-5.4
-  --dataset terminal-bench@2.0
-  --n-attempts 2
-  --n-concurrent 2
-  --max-retries 0
-)
-
-for t in "${FOCUS_TASKS[@]}"; do
-  cmd+=(--include-task-name "$t")
-done
-
-PYTHONPATH="$PWD/terminal-bench" "${cmd[@]}"
-```
-
 ## Minimal experiment log format
 
+You are running in a loop, make sure to keep your progress tracked so you
+can continue between loop iterations, this is to avoid context pressure.:w
 Keep this in `PROGRESS.md`, a final summary for each completed change:
 
 - benchmark symptom
 - general behavior gap
 - why this should help outside Terminal-Bench
-- overfitting risk / why this is still general
-- trigger evidence, if relevant
 - hypothesis
-- files changed
-- local verification
-- guardrail result
-- fast-suite result
-- focused-suite result, if run
-- mechanism exercised?
+- verification method
 - keep / revert / refine (Make the decision very visible in the file).

package/PROGRESS.md

@@ -1,5 +1,4 @@
 # CURRENT TASK PROGRESS
 
-- Benchmark loop stopped.
-- In-progress experiment changes reverted.
 - Progress reset for a fresh benchmark restart.
+- Benchmark for baseline is running (command in `benchmark-baseline.sh`)

package/README.md

@@ -39,7 +39,7 @@ $ mc
 
 ## Tools
 
-Six built-in tools, plus a conditional read-only image tool:
+Six built-in tools, plus a conditional read-only image tool and any configured MCP tools:
 
 - **`shell`** — runs commands in the user's shell. Returns stdout, stderr, and exit code. Large output is truncated to protect model context.
 - **`read`** — reads UTF-8 text files from disk, optionally by line window.
@@ -48,8 +48,7 @@ Six built-in tools, plus a conditional read-only image tool:
 - **`todoWrite`** — creates or updates the session todo list incrementally and returns the full current snapshot.
 - **`todoRead`** — returns the full current session todo list snapshot.
 - **`readImage`** — reads PNG, JPEG, GIF, and WebP files as model input. Only registered when the active model supports images.
-
-Plugins can add more tools, but the core stays intentionally small.
+- **Configured MCP tools** — tools discovered from `settings.json` Streamable HTTP MCP servers. Imported tool names are prefixed with the server name, for example `docs__search`.
 
 ## Features
 
@@ -58,25 +57,27 @@ Plugins can add more tools, but the core stays intentionally small.
 - **Session persistence** — SQLite-backed sessions with undo, fork, resume, and cumulative usage stats. Sessions are scoped to the working directory.
 - **Reasoning and verbosity controls** — toggle thinking visibility and verbose tool rendering on demand. Preferences persist across launches.
 - **[AGENTS.md](https://agents.md) support** — project-specific instructions discovered root-to-leaf, with `~/.agents/` for global instructions.
-- **[Agent Skills](https://agentskills.io)** — skill catalogs exposed in the prompt. `/skill:name` injects a skill body into the next user message.
-- **Plugins** — optional tools, integrations, theme overrides, and prompt suffixes without bloating the core.
+- **[Agent Skills](https://agentskills.io)** — skill catalogs exposed in the prompt. `/skill:name` injects a skill body into the next user message, and `/skill` opens a picker that fills in the selected skill reference without submitting.
+- **Settings-driven MCP tools** — connect Streamable HTTP MCP servers from `~/.config/mini-coder/settings.json` and expose their tools directly in the core runtime.
 
 ## Commands
 
-| Command      | Description                                                                                            |
-| ------------ | ------------------------------------------------------------------------------------------------------ |
-| `/model`     | Switch models and save the choice as the global default.                                               |
-| `/session`   | Open the session picker for the current working directory.                                             |
-| `/new`       | Start a fresh session and reset the running token and cost totals.                                     |
-| `/fork`      | Fork the current chat into a new session, keep the original, and add a UI-only `Forked session.` note. |
-| `/undo`      | Remove the last conversational turn without touching filesystem changes.                               |
-| `/reasoning` | Show or hide model thinking. The setting is saved and restored on launch.                              |
-| `/verbose`   | Expand shell output plus edit previews and edit errors in the conversation log.                        |
-| `/todo`      | Show the current session todo list in the conversation log as a UI-only checklist block.               |
-| `/login`     | Sign in with a supported OAuth provider.                                                               |
-| `/logout`    | Remove saved OAuth credentials for a logged-in provider.                                               |
-| `/effort`    | Choose low, medium, high, or xhigh reasoning effort.                                                   |
-| `/help`      | Show commands, current toggles, loaded AGENTS.md files, skills, and plugins.                           |
+| Command       | Description                                                                                                                                                       |
+| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `/model`      | Switch models and save the choice as the global default.                                                                                                          |
+| `/session`    | Open the session picker for the current working directory.                                                                                                        |
+| `/new`        | Start a fresh session and reset the running token and cost totals.                                                                                                |
+| `/fork`       | Fork the current chat into a new session, keep the original, and add a UI-only `Forked session.` note.                                                            |
+| `/undo`       | Remove the last conversational turn without touching filesystem changes.                                                                                          |
+| `/reasoning`  | Show or hide model thinking. The setting is saved and restored on launch.                                                                                         |
+| `/verbose`    | Toggle compact vs full rendering for verbose-aware tool previews/results, including shell, read, grep, edit previews/errors, and MCP tool blocks.                 |
+| `/mcp`        | Open the MCP server picker and toggle configured servers on or off for future turns.                                                                              |
+| `/todo`       | Show the current session todo list in the conversation log as a UI-only checklist block.                                                                          |
+| `/login`      | Sign in with a supported OAuth provider.                                                                                                                          |
+| `/logout`     | Remove saved OAuth credentials for a logged-in provider.                                                                                                          |
+| `/effort`     | Choose low, medium, high, or xhigh reasoning effort.                                                                                                              |
+| `/help`       | Show commands, current toggles, loaded AGENTS.md files, skills, and MCP servers with on/off state.                                                                |
+| `/skill:name` | Insert a discovered skill into the next message. Submit `/skill` or pick `/skill:name` from slash-command autocomplete to choose a skill without auto-submitting. |
 
 ## Key bindings
 
@@ -85,7 +86,7 @@ Plugins can add more tools, but the core stays intentionally small.
 | `Enter`       | Submit message                                                                                             |
 | `Shift+Enter` | Insert newline                                                                                             |
 | `Escape`      | Dismiss the overlay without changing the draft; otherwise interrupt the running turn; otherwise do nothing |
-| `Tab`         | Autocomplete a path, or open the command picker when the draft starts with `/`                             |
+| `Tab`         | Autocomplete a path, or open the command picker when the draft starts with `/` without clearing the draft  |
 | `Ctrl+R`      | Search global raw input history                                                                            |
 | `Ctrl+C`      | Graceful exit                                                                                              |
 | `Ctrl+D`      | Graceful exit when the input is empty                                                                      |
@@ -105,9 +106,40 @@ $ printf '%s\n' 'fix the failing tests' | mc
 - Starts when `-p/--prompt` is provided or when stdin or stdout is not a TTY.
 - If stdout is redirected but stdin is still interactive, pass `-p`; headless mode will not fall back to an interactive prompt.
 - Uses the same parser as the TUI for plain text, `/skill:name`, and standalone image paths.
+- Without `--json`, keeps stdout script-friendly by writing only the final assistant text there, while lightweight assistant commentary snippets from tool-use turns go to stderr.
 - With `--json`, writes NDJSON events for completed assistant/tool-result messages plus `done` / `error` / `aborted` outcomes; queued `user_message` events may also appear. Streaming deltas are omitted.
 - Headless runs still persist like normal sessions and show up in `/session` history for that working directory.
-- Interactive slash commands such as `/model`, `/session`, and `/help` are not available in headless mode.
+- Interactive slash commands such as `/skill` without a name, `/model`, `/session`, `/mcp`, and `/help` are not available in headless mode.
+
+## Settings
+
+Global defaults live in `~/.config/mini-coder/settings.json`.
+
+```json
+{
+  "customProviders": [
+    {
+      "name": "lm-studio",
+      "baseUrl": "http://127.0.0.1:1234/v1"
+    }
+  ],
+  "mcp": {
+    "servers": [
+      {
+        "name": "docs",
+        "url": "http://127.0.0.1:8787/mcp",
+        "enabled": true
+      }
+    ]
+  }
+}
+```
+
+- `mcp.servers` currently supports Streamable HTTP MCP endpoints.
+- Each server `name` becomes the imported tool prefix, so a remote `search` tool appears as `docs__search`.
+- MCP servers with `enabled: true` connect at startup; disabled ones stay disconnected until you turn them back on.
+- Invalid MCP URLs are skipped immediately, and enabled servers that are unreachable are skipped with a warning.
+- `/mcp` can enable or disable configured MCP servers during the current app run, and that on/off state is persisted.
 
 ## Docs
 

package/benchmark-baseline.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+cd -- "$(dirname -- "${BASH_SOURCE[0]}")"
+
+PYTHONPATH="$PWD/terminal-bench" harbor run -y \
+  --job-name benchmark-baseline-full-$(date +%F__%H-%M-%S) \
+  --jobs-dir "$PWD/terminal-bench/jobs" \
+  --agent-import-path mini_coder_agent:MiniCoderAgent \
+  --ak version=0.5.12 \
+  --model openai-codex/gpt-5.4 \
+  --dataset terminal-bench@2.0 \
+  --n-attempts 2 \
+  --n-concurrent 2 \
+  --max-retries 0