s-app-scoder-agent 0.4.0 → 0.6.0

package/README.md

@@ -50,14 +50,17 @@ npx app-scoder-agent "your task"
 ## Quick Start
 
 ```bash
-# Set your API key
+# One-time setup wizard — writes .scoderrc.json (base URL, key, model, build cmd)
+scoder init
+
+# …or set your API key manually
 export SCODER_API_KEY=sk-...          # any OpenAI-compatible key (OpenAI, Groq, OpenRouter, Ollama, …)
 export OPENAI_BASE_URL=https://api.openai.com/v1  # optional, for custom endpoints
 
 # Run a task
 scoder "add a ping function to src/network.ts"
 
-# Interactive REPL
+# Interactive REPL (supports /help, /clear, /cost, /model)
 scoder
 ```
 
@@ -80,9 +83,22 @@ You (task) → LLM plans → Tools execute code → Build runs → Success? →
 | `edit_file` | Find-and-replace (unique match required) |
 | `list_files` | Recursive directory listing |
 | `run_command` | Execute any shell command (subject to permission policy) |
+| `web_fetch` | Fetch a public HTTP(S) URL → readable text (HTML→text, JSON pretty-printed) |
 
 File writes (`write_file` / `edit_file`) are symlink-safe: paths are resolved via `realpath` and cannot escape the working directory, even through a malicious symlink. `read_file` refuses files larger than 2MB with guidance to use `grep`/`head`/`tail`.
 
+### Web access (`web_fetch`)
+
+The agent can read the internet through the built-in `web_fetch` tool — it fetches exact URLs (it cannot search) and returns readable plain text: HTML is converted to text (scripts/styles stripped, entities decoded, title extracted), JSON is pretty-printed, binary content is refused. The system prompt instructs the agent *when to fetch* (external/API docs, URLs in error messages, current versions, changelogs) and when not to (answer available locally).
+
+Safety defaults:
+
+- **SSRF protection** — `localhost`, private and link-local addresses (10.x, 172.16/12, 192.168.x, 127.x, 169.254.x, IPv6 ULA/link-local) are blocked. Set `SCODER_WEB_ALLOW_PRIVATE=1` to allow local dev servers.
+- Only `http`/`https`; up to 5 redirects — every hop is re-validated against the private-address block list.
+- 30 s timeout, SIGINT-abortable; body capped at 512 KB; extracted text capped at 20k chars (configurable via `max_chars`).
+- HTML is converted to plain text (no JS execution); images/archives/PDFs are refused.
+- Read-only — the tool never writes locally, so it is allowed even in `readonly` permission mode.
+
 ## Configuration
 
 ### Environment variables
@@ -96,6 +112,9 @@ File writes (`write_file` / `edit_file`) are symlink-safe: paths are resolved vi
 | `SCODER_MAX_STEPS` | `40` | Max loop iterations before giving up |
 | `SCODER_PERMISSION` | `auto` | Permission mode: `auto` \| `ask` \| `readonly` |
 | `SCODER_STREAM` | `1` | `0` to disable streaming LLM output |
+| `SCODER_STREAM` | `1` | `0` to disable streaming LLM output |
+| `SCODER_SERVER_TOKEN` | — | `scoder serve` only: require `Authorization: Bearer <token>` |
+| `SCODER_WEB_ALLOW_PRIVATE` | — | Set to `1` to let `web_fetch` reach localhost/private addresses (local dev) |
 | `SCODER_NO_APIKEY_WARNING` | — | Set to `1` to suppress the apiKey-in-rc warning |
 
 ### Per-project config file (.scoderrc.json)
@@ -225,7 +244,61 @@ Sessions older than 7 days are pruned automatically on the next run. Use `--no-s
 
 ## Streaming
 
-LLM output is streamed token-by-token by default (`SCODER_STREAM=0` to disable). If a provider doesn't support streaming, scoder automatically falls back to non-streaming requests.
+scoder streams everything it can, live, as it happens:
+
+- **LLM tokens** — assistant text renders character-by-character (OpenAI-compatible SSE and native Anthropic streaming). If a provider doesn't support streaming, scoder automatically falls back to non-streaming requests. Disable with `SCODER_STREAM=0`.
+- **Tool output** — `run_command` and build output print live (dimmed block) while the command runs, instead of appearing only after it finishes. Live output is capped at 20k chars; the full output still goes to the LLM.
+- **SIGINT-safe** — Ctrl+C kills the running command immediately; streamed rendering never blocks the event loop.
+- Output honors `NO_COLOR` and degrades to plain text when piped (CI-friendly).
+
+## Full-Screen TUI (`scoder tui`)
+
+**Mission Control for the build loop** — a terminal UI built with [Ink](https://github.com/vadimdemedes/ink) (React for terminals), driven by the same `AgentEvent` stream the HTTP API exposes:
+
+```bash
+scoder tui                     # start an interactive full-screen session
+scoder tui "fix the bug"       # auto-submit an initial task
+```
+
+Features:
+
+- **Build Rail** (top bar) — brand, build lamp (`● build ok` / `● build fail` / `no build yet`), permission mode, and a step fuel gauge (`▰▰▰▱▱▱▱▱▱▱`) that fills as the agent burns through its step budget
+- **Chat pane** — your messages, streaming assistant text, and every tool call with live stdout/stderr preview (`✔/✘` status, spinner while running), prefixed with a flight-recorder step number (`03▸`)
+- **Diff cards** (`✎`) — every file write/edit renders as a unified diff (`+` green / `-` red / `@@` cyan hunk headers); the same diff also flows out of the HTTP SSE stream
+- **Status bar** — model, token usage (input/output/cached), keybind hints
+- **Slash commands** — `/help`, `/clear`, `/cost`, `/model`, `/exit` (identical to the REPL)
+- **Multi-turn** — keep typing follow-ups after a task finishes; the conversation context carries over (same as the REPL)
+- **Scrollback** — `PgUp`/`PgDn` page through history; `Esc` returns to the live tail (press again to abort)
+- **Keybinds** — `Enter` send · `PgUp`/`PgDn` scroll · `Esc` reset scroll / abort · `Ctrl+C` quit
+- Non-TTY environments (CI, pipes) automatically fall back to the classic REPL
+- `permission: "ask"` is not supported in the TUI (no readline prompts on the alt screen) — it falls back to `auto`
+
+## HTTP Server Mode (`scoder serve`)
+
+Run scoder as a local HTTP service — submit tasks programmatically and consume progress as Server-Sent Events:
+
+```bash
+scoder serve                    # http://127.0.0.1:0 (random port, printed at startup)
+scoder serve --port 4114        # fixed port
+SCODER_SERVER_TOKEN=secret scoder serve --host 127.0.0.1
+```
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/health` | GET | Liveness: `{ ok, running, runs }` |
+| `/api/runs` | POST | Start a run: `{ "task": "...", "workdir?", "buildCmd?", "model?", "provider?", "maxSteps?", "permission?" }` → `202 { id, eventsUrl }` |
+| `/api/runs` | GET | List runs with status/timing |
+| `/api/runs/<id>` | GET | Run status + final result |
+| `/api/runs/<id>/events` | GET | **SSE stream** — full event replay, then live, closes on finish |
+| `/api/runs/<id>/abort` | POST | Abort a running run (equivalent to SIGINT) |
+
+SSE event types: `run_start`, `step`, `llm_delta` (assistant text), `tool_call`, `tool_output` (live stdout/stderr chunks), `tool_result`, `llm_usage`, `build`, `finish`. The stream replays every event from the beginning, then stays live, and closes after `finish` — safe to (re)connect at any time.
+
+Notes:
+
+- Runs are always non-interactive; `permission: "ask"` is rejected — use `auto` or `readonly`.
+- Binds `127.0.0.1` by default. Set `SCODER_SERVER_TOKEN` to require `Authorization: Bearer <token>` on every request.
+- Max 4 concurrent runs (extra requests get `429`); per-run overrides: `workdir`, `buildCmd`, `model`, `provider`, `maxSteps`, `permission`.
 
 ## Logging
 
@@ -268,6 +341,9 @@ scoder detects the build command automatically:
 ```
 scoder [options] "<task>"     Run a task autonomously until the build passes
 scoder                       Start interactive session
+scoder init                  Setup wizard — writes .scoderrc.json (+ connection test)
+scoder tui [task]            Full-screen terminal UI (Mission Control)
+scoder serve [--port] [--host]   HTTP server mode (see above)
 scoder --help | --version     Help and version
 ```
 
@@ -283,6 +359,9 @@ scoder --help | --version     Help and version
 | `-y` | Non-interactive mode (denies commands needing confirmation) |
 | `--provider <name>` | `openai` \| `anthropic` (env SCODER_PROVIDER; auto-detected) |
 | `--no-save` | Disable session auto-save to `.scoder/sessions/` |
+| `--log-file` | Write a structured JSONL event log to `.scoder/logs/` |
+| `--port <n>` | (`serve`) TCP port, `0` = random free port |
+| `--host <addr>` | (`serve`) Bind address (default `127.0.0.1`) |
 
 ## Examples
 
@@ -305,6 +384,15 @@ scoder -m claude-sonnet-4-5 "refactor the parser module"
 
 # CI: non-interactive, no session saving
 scoder --no-save -y "run the full migration"
+
+# Run as an HTTP service and consume progress via SSE
+scoder serve --port 4114 &
+curl -s -X POST localhost:4114/api/runs -H 'Content-Type: application/json' \
+  -d '{"task": "add input validation to the CLI"}'
+curl -N http://127.0.0.1:4114/api/runs/<id>/events     # live progress
+
+# Fetch external docs during a task (agent decides when, via web_fetch)
+scoder "upgrade express to v5 — check the official migration guide first"
 ```
 
 ## Architecture
@@ -312,32 +400,40 @@ scoder --no-save -y "run the full migration"
 ```
 ┌─────────────────────────────────────────────┐
 │  CLI (src/index.ts)                         │
-│   arg parsing, REPL, SIGINT handling        │
-└─────────────────┬───────────────────────────┘
-                  │
-┌─────────────────▼───────────────────────────┐
-│  Agent (src/agent.ts)                       │
-│   Closed-loop orchestrator                  │
-│   history trimming, error injection         │
-└───────┬───────────────────────┬─────────────┘
-        │                       │
-┌───────▼───────┐    ┌─────────▼──────────┐
-│  LLM Client   │    │  Tool Registry       │
-│ (src/llm.ts)  │    │ (src/tools.ts)       │
-│ OpenAI compat │    │ read/write/run/list │
-└───────────────┘    └──────────────────────┘
-        │
-        ▼
-┌─────────────────────────────────────────────┐
-│  Build Checker (src/build.ts)               │
-│   run build → exit 0 = stop condition met   │
-└─────────────────────────────────────────────┘
+│   arg parsing, REPL, serve, tui, SIGINT     │
+└───────┬─────────────────────┬───────────────┘
+        │                     │
+┌───────▼───────────┐  ┌──────▼─────────────────────┐
+│  TUI (src/tui/)   │  │  Agent (src/agent.ts)      │
+│  Ink/React full-  │  │  Closed-loop orchestrator  │
+│  screen chat UI,  │─▶│  history trimming, error   │
+│  renders the same │  │  injection                 │
+│  AgentEvent stream│  │  emits AgentEvent          │
+└───────────────────┘  └──┬──────────┬──────────────┘
+                          │          │
+                          │          └────────────────────┐
+                          ▼                               ▼
+┌───────────────────────────────┐   ┌──────────────────────────┐
+│  LLM Client (src/llm.ts)      │   │  Server (src/server.ts)  │
+│  OpenAI-compatible + native   │   │  HTTP API + SSE          │
+│  Anthropic provider           │   │  runs in background      │
+└──────────────┬────────────────┘   └──────────────────────────┘
+               ▼
+┌────────────────────────────────────────┐
+│  Tool Registry (src/tools.ts)          │
+│  + web_fetch (src/webfetch.ts)         │
+└──────────────┬─────────────────────────┘
+               ▼
+┌───────────────────────────────┐
+│  Build Checker (src/build.ts) │
+│  run build → exit 0 = stop    │
+└───────────────────────────────┘
 ```
 
 ## Integration Testing
 
 ```bash
-npm test          # runs: lint → typecheck → build → 116 unit tests + e2e + MCP smoke
+npm test          # runs: build → unit tests (node --test) → e2e closed-loop → MCP smoke
 npm run lint      # eslint flat config (typescript-eslint + eqeqeq, prefer-const...)
 npm run typecheck # tsc --noEmit
 ```
@@ -345,17 +441,20 @@ npm run typecheck # tsc --noEmit
 Test suites (run with `npm test` — auto-discovers `test/*.test.mjs`):
 - `test/permissions.test.mjs` — catastrophic patterns, mode classification, allowlist word-boundary, timeout-denies
 - `test/permission-prompt.test.mjs` — interactive prompt timeout behavior
-- `test/tools.test.mjs` — safePath traversal, symlink escape, file-size cap, write/edit/list
+- `test/tools.test.mjs` — safePath traversal, symlink escape, file-size cap, write/edit/list, live `run_command` output streaming
+- `test/webfetch.test.mjs` — HTML→text, JSON, redirects, SSRF guard, binary refusal, truncation
+- `test/server.test.mjs` — HTTP API: run lifecycle via SSE, abort, auth, live `tool_output` events
+- `test/tui.test.mjs` — full-screen TUI in a real PTY (via `test/pty-run.py`): closed loop render + slash commands (`/help`, `/cost`) + clean exit
+- `test/tui-state.test.mjs` — event reducer unit tests: diff items, build lamp, abort marking, slash-command parsing & help formatting
 - `test/core.test.mjs` — unified diff, retry backoff, context trimming
 - `test/config.test.mjs` — env/rc/cli precedence, logFile mapping, permission lists
 - `test/session.test.mjs` — session store, smart prune, resolve
+- `test/cost.test.mjs` — token usage & cost aggregation
 - `test/anthropic.test.mjs` — message translation + native SSE streaming
-- `test/llm.test.mjs` — OpenAI-compatible chat + SSE streaming + redaction
+- `test/llm.test.mjs` — OpenAI-compatible chat + SSE streaming + redaction + wire-format normalization
 - `test/run-e2e.mjs` — closed-loop integration (mock LLM server)
 - `test/run-mcp-test.mjs` — MCP smoke (mock stdio server)
 
-Coverage (lines): **90.3%** overall — see `npm run coverage`.
-
 ## Exit Codes
 
 | Code | Meaning |

package/dist/agent.js

@@ -28,7 +28,13 @@ RULES:
 6. Never invent requirements beyond the given task.
 7. IMPORTANT: if a run_command is denied by permission policy, you will receive
    an error message. Use this as a signal to find a safer alternative or to
-   communicate the limitation to the user.`;
+   communicate the limitation to the user.
+8. You have web access ONLY through web_fetch (it fetches exact URLs, it cannot
+   search). Use it when the task depends on external or fast-changing knowledge:
+   library/API documentation, a URL found in an error message or code comment,
+   current package versions, changelogs, release notes, standards. Skip it when
+   the answer is in the working directory or is stable general knowledge. After
+   fetching, ground your edits in what the page actually says.`;
 export class Agent {
     config;
     client;
@@ -48,6 +54,10 @@ export class Agent {
     mcp = null;
     /** Aborted when abort() is called — cancels in-flight LLM requests (SIGINT). */
     runSignal = new AbortController();
+    /** Optional structured-event sink (used by the HTTP/SSE server frontend). */
+    onEvent = null;
+    /** Set while a tool call is executing, for tool_output event correlation. */
+    currentToolCall = null;
     constructor(config) {
         this.config = config;
         const commonOpts = {
@@ -72,6 +82,7 @@ export class Agent {
             workdir: config.workdir,
             permissions: this.permissions,
             permissionMode: config.permissionMode,
+            onToolOutput: this.handleToolOutput,
         };
         this.session = new SessionStore(config.workdir, config.noSave !== true);
         this.logger = JsonlLogger.forSession(config.workdir, this.session.id, config.logFile);
@@ -92,6 +103,27 @@ export class Agent {
         // terminate its child process immediately (long builds must be interruptible).
         this.ctx.signal = this.runSignal.signal;
     }
+    /** Emit one structured run event to the sink (if attached). */
+    emit(ev) {
+        this.onEvent?.(ev);
+    }
+    /**
+     * Live tool-output handler: renders chunks in the terminal (dimmed block) and
+     * forwards them as `tool_output` events. Opens the stream block lazily on the
+     * first chunk; `endToolStream()` closes it after the tool completes.
+     */
+    handleToolOutput = (name, stream, delta) => {
+        if (!ui.isToolStreamOpen()) {
+            ui.toolStreamStart();
+        }
+        ui.toolStreamChunk(delta);
+        this.emit({ type: "tool_output", name, stream, delta });
+    };
+    /** Close the live tool-output block (if open) before printing the next UI element. */
+    endToolStream() {
+        if (ui.isToolStreamOpen())
+            ui.toolStreamEnd();
+    }
     /** Push one tool-result message back into the conversation. */
     pushToolResult(tc, content) {
         this.messages.push({
@@ -108,6 +140,7 @@ export class Agent {
      */
     rejectToolCall(step, tc, output, logExtra) {
         ui.toolResult(tc.name, false, output);
+        this.emit({ type: "tool_result", step, name: tc.name, ok: false, output });
         this.logger.log({ type: "tool", step, tool: tc.name, ok: false, ...logExtra });
         this.pushToolResult(tc, output);
     }
@@ -115,6 +148,10 @@ export class Agent {
      * Record usage of one LLM call: render token counts, log an `llm_usage`
      * event, and accumulate totals into session state.
      */
+    /** Token usage totals accumulated in this session (for REPL `/cost`). */
+    getUsage() {
+        return { ...this.usageTotals };
+    }
     recordLlmUsage(usage) {
         if (!usage)
             return;
@@ -126,6 +163,10 @@ export class Agent {
         this.usageTotals.outputTokens += output;
         this.usageTotals.cachedTokens += cached;
         ui.info(`tokens: in=${usage.prompt_tokens ?? "?"} out=${usage.completion_tokens ?? "?"}`);
+        this.emit({
+            type: "llm_usage",
+            usage: { input_tokens: input, output_tokens: output, cached_tokens: cached },
+        });
         this.logger.log({
             type: "llm_usage",
             model: this.client.model,
@@ -161,6 +202,7 @@ export class Agent {
                         started = true;
                     }
                     ui.llmStreamChunk(chunk.contentDelta);
+                    this.emit({ type: "llm_delta", delta: chunk.contentDelta });
                     streamedText += chunk.contentDelta;
                 }
                 if (chunk.final)
@@ -217,9 +259,13 @@ export class Agent {
             throw new Error("No API key configured. Set SCODER_API_KEY or OPENAI_API_KEY (or put apiKey in .scoderrc.json).");
         }
         const buildCmd = this.buildCmd;
-        ui.header("TASK");
-        ui.info(task);
-        ui.info(`model=${this.client.model}  workdir=${this.config.workdir}  build="${buildCmd}"  maxSteps=${this.config.maxSteps}`);
+        ui.banner("scoder", [
+            ["task", task],
+            ["model", `${this.client.model} (${this.config.provider}${this.config.stream ? ", stream" : ""})`],
+            ["workdir", this.config.workdir],
+            ["build", buildCmd],
+            ["steps", `max ${this.config.maxSteps}  ·  permission ${this.config.permissionMode}`],
+        ]);
         this.currentTask = task;
         this.resetRunSignal();
         this.resetUsageTotals();
@@ -231,6 +277,14 @@ export class Agent {
             },
             { role: "user", content: task },
         ];
+        this.emit({
+            type: "run_start",
+            task,
+            model: this.client.model,
+            provider: this.config.provider,
+            workdir: this.config.workdir,
+            buildCmd,
+        });
         try {
             await this.connectMcp();
             return await this.runLoop(0, buildCmd, null);
@@ -304,6 +358,7 @@ export class Agent {
         while (steps < this.config.maxSteps && !this.aborted) {
             steps++;
             ui.step(steps, this.config.maxSteps, "thinking...");
+            this.emit({ type: "step", step: steps, maxSteps: this.config.maxSteps });
             const { message, usage } = await this.chatWithStream().catch((err) => {
                 // SIGINT during the LLM call — stop gracefully instead of crashing
                 if (this.aborted) {
@@ -363,6 +418,8 @@ export class Agent {
                     continue;
                 }
                 ui.toolCall(tc.name, args);
+                this.currentToolCall = { step: steps, id: tc.id, name: tc.name };
+                this.emit({ type: "tool_call", step: steps, id: tc.id, name: tc.name, args });
                 if (!tool && !(this.mcp?.hasTool(tc.name))) {
                     this.rejectToolCall(steps, tc, `ERROR: unknown tool "${tc.name}". Available: ${AGENT_TOOLS.map((t) => t.spec.function.name).join(", ")}${this.mcp ? " + MCP tools (mcp__<server>__<tool>)" : ""}`, { error: "unknown tool" });
                     continue;
@@ -426,7 +483,17 @@ export class Agent {
                     this.aborted = true;
                     ui.warn("edit confirmation aborted by user ([q]uit) — stopping the run");
                 }
+                this.endToolStream();
+                this.currentToolCall = null;
                 ui.toolResult(tc.name, result.ok, result.output);
+                this.emit({
+                    type: "tool_result",
+                    step: steps,
+                    name: tc.name,
+                    ok: result.ok,
+                    output: result.output,
+                    ...(result.diff ? { diff: result.diff } : {}),
+                });
                 stepToolResults.set(tc.id, { ok: result.ok, output: result.output });
                 this.pushToolResult(tc, summariseOutput(result.output));
             }
@@ -483,8 +550,10 @@ export class Agent {
     }
     /** Explicitly verify the build (independent of what the LLM claims). */
     async checkBuild(buildCmd) {
-        const res = await runBuild(this.config.workdir, buildCmd);
+        const res = await runBuild(this.config.workdir, buildCmd, (stream, delta) => this.handleToolOutput("build", stream, delta));
+        this.endToolStream();
         ui.buildStatus(res.success, res.output);
+        this.emit({ type: "build", success: res.success, command: buildCmd, output: res.output });
         this.logger.log({ type: "build", success: res.success, command: buildCmd, output: res.output.slice(0, 10_000) });
         return res;
     }
@@ -495,6 +564,7 @@ export class Agent {
         else {
             ui.error(`Task NOT complete. ${summary}`);
         }
+        this.emit({ type: "finish", success, steps, summary: summary.slice(0, 2000) });
         this.logger.log({ type: "finish", success, steps });
         return { success, steps, buildCmd, summary: summary.slice(0, 2000) };
     }

package/dist/build.js

@@ -30,9 +30,10 @@ export function detectBuildCommand(workdir) {
 /**
  * Run the build command. This is the agent's stop condition:
  * exit code 0 == task complete.
+ * `onOutput` (optional) receives stdout/stderr chunks live while the build runs.
  */
-export async function runBuild(workdir, command) {
-    const res = await toolRunCommand(workdir, { command, timeout_secs: 600 });
+export async function runBuild(workdir, command, onOutput) {
+    const res = await toolRunCommand(workdir, { command, timeout_secs: 600 }, undefined, onOutput);
     return {
         success: res.ok,
         command,