runcap 0.2.2 → 0.3.1

package/README.md

@@ -4,15 +4,21 @@
 
 ![Runcap terminal demo: estimate, cap, compress, stop](docs/assets/demo.svg)
 
-**Know what your coding agent will cost before you build it, and set a hard ceiling so it never surprises you.**
+**Your AI coding agent re-reads the same files over and over and quietly burns your money. Runcap estimates the bill before you build, hard-caps the spend so it physically stops at your ceiling, and losslessly compresses every call. Free, MIT, 100% local. Your code and tokens never touch a server.**
 
-Runcap estimates the cost of an agent run as a range, enforces a hard spend ceiling that physically stops the run, and when the agent gets stuck it hands you the exact rescue prompt. Free, MIT, 100% local. Your code and tokens never touch a server.
+On a real OpenAI call, one edited-file re-read dropped from **1,186 to 737 prompt tokens (37.9% saved)** with the model still answering correctly about the changed line. No other proxy does this:
+
+| | Without Runcap | With Runcap |
+|---|---|---|
+| Re-read of an edited file | 1,186 prompt tokens | **737 prompt tokens** |
+| You find out the cost | when the invoice arrives | **before you press go, capped at your ceiling** |
+| When the agent gets stuck | it keeps spending | **run stops, you get the exact rescue prompt** |
 
 > Every other tool here is a rear-view mirror - it shows you the bill *after* you paid it. Runcap estimates the bill *before* you start and caps it. It is a circuit breaker, not a dashboard.
 
 ## Why
 
-Multi-agent coding runs burn roughly **15x more tokens** than a single chat ([Anthropic engineering](https://www.anthropic.com/engineering/built-multi-agent-research-system)). Agents loop on the same error, rewrite plans, and hand you a confident summary while the task is not actually done. You find out what it cost when the invoice - or the subscription limit - arrives.
+**Agents loop on the same error, rewrite plans, and re-read files they just edited - every loop is tokens you pay for.** Multi-agent coding runs burn roughly **15x more tokens** than a single chat ([Anthropic engineering](https://www.anthropic.com/engineering/built-multi-agent-research-system)). They hand you a confident summary while the task is not actually done, and you find out what it cost when the invoice - or the subscription limit - arrives.
 
 Observability tools (Langfuse, Helicone, LangSmith, AgentOps) measure the past. Gateways (LiteLLM, Portkey, OpenRouter) route the present. None of them stop the spend *before* it happens. Runcap does the one thing the rear-view mirror can't:
 
@@ -135,6 +141,12 @@ It's pure Node with **zero ML or native dependencies**, so it installs everywher
 
 The dashboard shows the result as one number: **"You saved $X · N tokens compressed · would have spent $Y."** Disable it with `AIM_COMPRESS=off` if you ever want raw passthrough.
 
+## Loop detection (the "looks productive but stuck" signal)
+
+The hard case in stuck-detection is the agent that keeps producing output but is really circling the same failure, just reworded each time. Plain hashing misses it because the prompt is *similar but never byte-identical* between loops. Because the gateway sees every request, Runcap compares each request's conversation shape against the recent run with the same line-similarity primitive the delta-encoder uses: when several prompts in a row are near-identical (default: 3 prompts at 92%+ similarity) while the conversation never moves forward, it flags `loop.looping` on the event, surfaces a warning in `runcap status`, and fires an alert.
+
+This is a **calculated** signal, not a proven dollar-saving: it tells you *"the agent has sent 3 near-identical prompts in a row with no progress"* so you can step in before the loop burns more budget. Tune or disable it with `AIM_LOOP_DETECT=off`. (Today's [`detectStuck`](src/mission-control.mjs) post-run score is outcome-based: exit code, parsed errors, and zero-diff. The loop signal adds the missing in-flight behavioral signal on top of it.)
+
 ## Pricing table
 
 Costs are calculated from a sourced multi-provider table - Anthropic (Opus / Sonnet / Haiku) and OpenAI (GPT-5 family + legacy GPT-4), with cache-read and batch discounts handled - labeled with source and verification date. When a model is unknown, Runcap says `unknown_price` rather than guessing.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "runcap",
-  "version": "0.2.2",
+  "version": "0.3.1",
   "description": "Cap every agent run before it starts: estimate cost, set a hard ceiling that stops the run, rescue stuck agents. Local, MIT, nothing uploaded.",
   "license": "MIT",
   "type": "module",
@@ -45,8 +45,9 @@
     "acceptance": "node ./scripts/acceptance.mjs",
     "smoke": "node ./bin/runcap.mjs run --label smoke -- npm --prefix examples/broken-ts-app run build",
     "demo:broken": "node ./bin/runcap.mjs run --label broken-ts-demo -- npm --prefix examples/broken-ts-app run build",
-    "test": "node ./scripts/delta-test.mjs && node ./scripts/validate-demo.mjs",
+    "test": "node ./scripts/delta-test.mjs && node ./scripts/loop-test.mjs && node ./scripts/loop-e2e.mjs && node ./scripts/validate-demo.mjs",
     "test:delta": "node ./scripts/delta-test.mjs",
+    "test:loop": "node ./scripts/loop-test.mjs",
     "status": "node ./bin/runcap.mjs status",
     "report": "node ./bin/runcap.mjs report",
     "export": "node ./bin/runcap.mjs export",

package/scripts/loop-e2e.mjs

@@ -0,0 +1,137 @@
+// End-to-end proof that the response-side loop gate works through the REAL
+// gateway over HTTP, not just in unit tests. We stand up a tiny local "upstream"
+// that returns a caller-chosen error string, point the real Runcap gateway at
+// it, and drive near-identical prompts through the wire:
+//   A) error CHANGES each turn (convergence)  -> gateway must NOT flag a loop
+//   B) error STAYS the same each turn (circling) -> gateway MUST flag a loop
+// The gateway records its loop verdict per call in the gateway event log, which
+// we read back to assert the real server behaved correctly.
+//
+// Pure Node, no framework. Exits non-zero on any failure so it can gate CI.
+
+import http from "node:http";
+import os from "node:os";
+import path from "node:path";
+import { mkdtempSync, readFileSync, existsSync } from "node:fs";
+
+// Isolate all gateway state (the .runcap event log lives under cwd) in a
+// throwaway dir so this never touches real data. The gateway writes its event
+// log to ./.runcap, so we chdir into the temp dir before starting it.
+const tmpHome = mkdtempSync(path.join(os.tmpdir(), "runcap-e2e-"));
+process.chdir(tmpHome);
+process.env.AIM_COMPRESS = "off";      // keep the wire bytes predictable
+process.env.AIM_LOOP_DETECT = "on";
+
+// A controllable upstream: returns an OpenAI-shaped completion whose assistant
+// text is whatever error we tell it to via a field in the request body. We use
+// the body (not a header) on purpose: the gateway forwards the request body
+// upstream but rewrites headers, so the body is the channel that actually
+// reaches this stub through the real gateway.
+const upstream = http.createServer((req, res) => {
+  let body = "";
+  req.on("data", (c) => (body += c));
+  req.on("end", () => {
+    let err = "default error";
+    try { err = JSON.parse(body)?.mock_error ?? err; } catch {}
+    const payload = {
+      id: "chatcmpl-stub",
+      object: "chat.completion",
+      created: Math.floor(Date.now() / 1000),
+      model: "stub-model",
+      choices: [{ index: 0, message: { role: "assistant", content: String(err) }, finish_reason: "stop" }],
+      usage: { prompt_tokens: 50, completion_tokens: 10, total_tokens: 60 }
+    };
+    res.writeHead(200, { "content-type": "application/json" });
+    res.end(JSON.stringify(payload));
+  });
+});
+
+async function listen(server, port = 0) {
+  await new Promise((r) => server.listen(port, "127.0.0.1", r));
+  return server.address().port;
+}
+
+let failures = 0;
+function check(name, pass, detail) {
+  if (!pass) failures++;
+  console.log(`${pass ? "PASS" : "FAIL"}  ${name}${detail ? "  — " + detail : ""}`);
+}
+
+const stableTail = [
+  "You are a coding agent. Fix the failing build.",
+  ...Array.from({ length: 40 }, (_, i) => `context line ${i}: prior file content the agent keeps resending`)
+].join("\n");
+
+async function send(port, wording, mockError) {
+  // mock_error rides in the body so it survives the gateway's header rewrite and
+  // reaches the upstream stub, which echoes it back as the assistant response.
+  const body = JSON.stringify({
+    model: "stub-model",
+    mock_error: mockError,
+    messages: [{ role: "user", content: stableTail + "\nLet me try this: " + wording }]
+  });
+  const res = await fetch(`http://127.0.0.1:${port}/v1/chat/completions`, {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body
+  });
+  await res.text();
+}
+
+function readEvents() {
+  const log = path.join(tmpHome, ".runcap", "gateway-events.jsonl");
+  if (!existsSync(log)) return [];
+  return readFileSync(log, "utf8").trim().split("\n").filter(Boolean).map((l) => JSON.parse(l));
+}
+
+// Loop verdicts accumulate across both scenarios in one shared gateway process
+// (the shape history is per-process), so each scenario asserts against only the
+// events it produced. We snapshot the event count before scenario B.
+
+const run = async () => {
+  const upstreamPort = await listen(upstream);
+  process.env.AIM_UPSTREAM_BASE_URL = `http://127.0.0.1:${upstreamPort}/v1`;
+  process.env.AIM_UPSTREAM_API_KEY = "test-key";
+
+  // Import AFTER env is set so the gateway reads our isolated config.
+  const { startEphemeralGateway } = await import("../src/mission-control.mjs");
+  const gw = await startEphemeralGateway();
+  const gwPort = gw.port;
+
+  // Scenario A: same prompt framing, but the error MOVES every turn (convergence).
+  for (const [w, e] of [
+    ["guard the undefined", "TypeError: cannot read property 'id' of undefined"],
+    ["optional chain", "TypeError: cannot read property 'name' of undefined"],
+    ["default to {}", "ReferenceError: parser is not defined"],
+    ["try/catch", "AssertionError: expected 200 but got 404"]
+  ]) {
+    await send(gwPort, w, e);
+  }
+  const afterA = readEvents();
+  const aFlagged = afterA.filter((ev) => ev.loop && ev.loop.looping).length;
+  check("E2E convergence (moving error) is NOT flagged through real gateway", aFlagged === 0,
+    `loops flagged in scenario A=${aFlagged}`);
+
+  // Scenario B: same prompt framing AND the SAME error every turn (circling).
+  const stuck = "TypeError: cannot read property 'id' of undefined";
+  for (const w of ["attempt one", "attempt two reworded", "attempt three reworded", "attempt four reworded", "attempt five reworded"]) {
+    await send(gwPort, w, stuck);
+  }
+  const afterB = readEvents().slice(afterA.length); // only scenario-B events
+  const bFlagged = afterB.filter((ev) => ev.loop && ev.loop.looping).length;
+  check("E2E circling (stuck error) IS flagged through real gateway", bFlagged > 0,
+    `loops flagged in scenario B=${bFlagged}`);
+
+  await gw.close();
+  upstream.close();
+};
+
+run()
+  .then(() => {
+    console.log("\n" + (failures === 0 ? "ALL LOOP E2E TESTS PASSED" : `${failures} LOOP E2E TEST(S) FAILED`));
+    process.exit(failures === 0 ? 0 : 1);
+  })
+  .catch((e) => {
+    console.error("E2E harness error:", e);
+    process.exit(1);
+  });

package/scripts/loop-test.mjs

@@ -0,0 +1,128 @@
+// Loop / circling detection tests, run against the REAL compressor exports.
+// Proves the "looks productive but stuck" signal the gateway emits:
+//   1. Reworded same-failure attempts (similar-but-not-identical prompts) are
+//      flagged as a loop once they repeat enough times.
+//   2. Genuine progress (the conversation tail actually changing) is NOT flagged.
+//   3. A single slow/long legit step is NOT flagged.
+//
+// Pure Node, no test framework. Exits non-zero on any failure so it can gate CI.
+
+import { detectLoop, requestShapeText, responseSignature } from "../src/compressor.mjs";
+
+let failures = 0;
+function check(name, pass, detail) {
+  if (!pass) failures++;
+  console.log(`${pass ? "PASS" : "FAIL"}  ${name}${detail ? "  — " + detail : ""}`);
+}
+
+// A long, stable conversation tail (system + history the agent keeps resending),
+// plus a final attempt line that the agent only REWORDS each loop. This is the
+// exact case that fools cheap hashing: 99% identical, never byte-equal.
+const stableTail = [
+  "You are a coding agent. Fix the failing build.",
+  ...Array.from({ length: 40 }, (_, i) => `context line ${i}: prior file content the agent keeps resending`),
+  "The test still fails with: TypeError: cannot read property 'id' of undefined"
+].join("\n");
+
+function attempt(wording) {
+  return stableTail + "\n" + "Let me try this: " + wording;
+}
+
+// --- Test 1: reworded same-failure attempts are flagged as a loop ---
+{
+  const history = [
+    attempt("guard the undefined with an if check"),
+    attempt("add an optional chain before .id"),
+    attempt("default the object to {} before reading id")
+  ];
+  const current = attempt("wrap the access in a try/catch and read id safely");
+  const r = detectLoop(current, history);
+  check("reworded same-failure attempts flagged as loop", r.looping && r.repeats >= 3,
+    `repeats=${r.repeats}, similarity=${r.similarity}`);
+}
+
+// --- Test 2: real progress is NOT flagged ---
+// Each turn the conversation tail genuinely changes (new files, new errors).
+{
+  const history = [
+    "Fix the build. Error: missing module 'parser'.\n" + "ctx A ".repeat(40),
+    "Installed parser. New error: parser.parse is not a function.\n" + "ctx B ".repeat(40)
+  ];
+  const current = "Fixed the call signature. Now the test passes; writing the next feature.\n" + "ctx C ".repeat(40);
+  const r = detectLoop(current, history);
+  check("genuine progress is NOT flagged as loop", !r.looping,
+    `looping=${r.looping}, repeats=${r.repeats}`);
+}
+
+// --- Test 3: a single slow/long legit step is NOT flagged ---
+// One big request with no prior near-identical history must never trip.
+{
+  const current = attempt("first and only attempt at this step");
+  const r = detectLoop(current, []);
+  check("single long step is NOT flagged", !r.looping && r.repeats === 0,
+    `repeats=${r.repeats}`);
+}
+
+// --- Test 4: two repeats is at_risk but below the warn threshold ---
+{
+  const history = [attempt("try A"), attempt("try B")];
+  const current = attempt("try C");
+  const r = detectLoop(current, history);
+  check("two near-identical repeats not yet a loop (under threshold)", !r.looping && r.repeats === 2,
+    `repeats=${r.repeats}`);
+}
+
+// --- Test 5: requestShapeText pulls the same text from OpenAI and Anthropic shapes ---
+{
+  const openai = requestShapeText({ messages: [{ role: "user", content: "hello world" }] });
+  const anthropic = requestShapeText({ messages: [{ role: "user", content: [{ type: "text", text: "hello world" }] }] });
+  check("requestShapeText normalizes OpenAI and Anthropic content", openai === "hello world" && anthropic === "hello world",
+    `openai="${openai}" anthropic="${anthropic}"`);
+}
+
+// --- Test 6: response-side gate — similar prompts but a MOVING error is NOT a loop ---
+// The edge case raised on the thread: a converging run also sends near-identical
+// prompts (same files, same framing) while it closes in on the fix. The tell is
+// the observation: if the error/test output changes between turns, that's
+// progress, not circling. Prompts are near-identical here, but each response
+// carries a DIFFERENT error, so the gate must keep it from being flagged.
+{
+  const history = [attempt("try A"), attempt("try B"), attempt("try C")];
+  const current = attempt("try D");
+  const responseSignatures = [
+    "TypeError: cannot read property 'id' of undefined",
+    "TypeError: cannot read property 'name' of undefined",
+    "ReferenceError: parser is not defined"
+  ];
+  const currentResponseSignature = "AssertionError: expected 200 but got 404";
+  const r = detectLoop(current, history, { responseSignatures, currentResponseSignature });
+  check("similar prompts but MOVING error are NOT flagged (convergence)", !r.looping && r.responseMoved,
+    `looping=${r.looping}, repeats=${r.repeats}, responseMoved=${r.responseMoved}`);
+}
+
+// --- Test 7: response-side gate — similar prompts AND a STUCK error IS a loop ---
+// Same near-identical prompts, but the identical error keeps coming back. Now
+// both signals agree the run is circling, so it must still be flagged.
+{
+  const history = [attempt("try A"), attempt("try B"), attempt("try C")];
+  const current = attempt("try D");
+  const sameError = "TypeError: cannot read property 'id' of undefined";
+  const responseSignatures = [sameError, sameError, sameError];
+  const currentResponseSignature = sameError;
+  const r = detectLoop(current, history, { responseSignatures, currentResponseSignature });
+  check("similar prompts AND stuck error ARE flagged as loop", r.looping && !r.responseMoved && r.repeats >= 3,
+    `looping=${r.looping}, repeats=${r.repeats}, responseMoved=${r.responseMoved}`);
+}
+
+// --- Test 8: responseSignature extracts the error/text from both provider shapes ---
+{
+  const openai = responseSignature({ choices: [{ message: { content: "boom: it failed" } }] });
+  const anthropic = responseSignature({ content: [{ type: "text", text: "boom: it failed" }] });
+  const errEnvelope = responseSignature({ error: { message: "rate limited" } });
+  check("responseSignature reads OpenAI, Anthropic, and error shapes",
+    openai === "boom: it failed" && anthropic === "boom: it failed" && errEnvelope === "rate limited",
+    `openai="${openai}" anthropic="${anthropic}" err="${errEnvelope}"`);
+}
+
+console.log("\n" + (failures === 0 ? "ALL LOOP TESTS PASSED" : `${failures} LOOP TEST(S) FAILED`));
+process.exit(failures === 0 ? 0 : 1);

package/scripts/make-demo-svg.mjs

@@ -16,27 +16,28 @@ const C = {
 
 const lines = [
   { t: "$ runcap plan --fuel 24 -- \"build a small auth feature and verify it\"", c: C.prompt, at: 0.3 },
-  { t: "Estimate:  $3 - $7   (range, not an oracle)", c: C.text, at: 1.1 },
-  { t: "Recommended cap:  $10", c: C.ok, at: 1.5 },
-  { t: "", c: C.text, at: 1.6 },
-  { t: "$ ANTHROPIC_BASE_URL=http://127.0.0.1:8792/v1 \\", c: C.prompt, at: 2.2 },
-  { t: "    AIM_DAILY_BUDGET_USD=10 runcap gateway", c: C.prompt, at: 2.6 },
-  { t: "gateway up  ·  compression on  ·  hard cap armed", c: C.dim, at: 3.2 },
-  { t: "", c: C.text, at: 3.3 },
-  { t: "→ request   10,144 tokens", c: C.text, at: 3.9 },
-  { t: "→ compressed 1,260 tokens   (JSON + logs trimmed, prose untouched)", c: C.ok, at: 4.6 },
-  { t: "", c: C.text, at: 4.7 },
-  { t: "You saved $7.40  ·  would have spent $18.40  ·  cap $10", c: C.accent, at: 5.4 },
-  { t: "", c: C.text, at: 5.5 },
-  { t: "→ next call would cross the ceiling", c: C.text, at: 6.1 },
-  { t: "HTTP 429  budget_guard  — run stopped before money left your account", c: C.bad, at: 6.8 }
+  { t: "Estimate:  $3 - $7   (range, not an oracle)", c: C.text, at: 1.0 },
+  { t: "Recommended cap:  $10", c: C.ok, at: 1.4 },
+  { t: "", c: C.text, at: 1.5 },
+  { t: "$ ANTHROPIC_BASE_URL=http://127.0.0.1:8792/v1 \\", c: C.prompt, at: 2.0 },
+  { t: "    AIM_DAILY_BUDGET_USD=10 runcap gateway", c: C.prompt, at: 2.3 },
+  { t: "gateway up  ·  compress on  ·  hard cap armed  ·  loop guard on", c: C.dim, at: 2.9 },
+  { t: "", c: C.text, at: 3.0 },
+  { t: "→ request   10,144 tokens", c: C.text, at: 3.5 },
+  { t: "→ compressed 1,260 tokens   (1,186 → 737 on a real call: 37.9% saved)", c: C.ok, at: 4.1 },
+  { t: "", c: C.text, at: 4.2 },
+  { t: "⚠ loop: last 3 prompts 97.7% identical - agent circling the same fail", c: C.violet, at: 5.0 },
+  { t: "   (looks busy, makes no progress, keeps spending)", c: C.dim, at: 5.5 },
+  { t: "", c: C.text, at: 5.6 },
+  { t: "→ next call would cross the ceiling", c: C.text, at: 6.2 },
+  { t: "HTTP 429  budget_guard  - run stopped before money left your account", c: C.bad, at: 6.9 }
 ];
 
-const W = 920, H = 560;
+const W = 920, H = 588;
 const padX = 28, top = 78, lh = 27, fs = 16.5;
 const esc = (s) => s.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
 
-const total = 8.0; // loop length seconds
+const total = 9.0; // loop length seconds
 const rows = lines.map((ln, i) => {
   const y = top + i * lh;
   // fade+slide in at ln.at, hold, then reset at end of loop
@@ -46,7 +47,7 @@ const rows = lines.map((ln, i) => {
   ${esc(ln.t)}</text>`;
 }).join("\n");
 
-const svg = `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 ${W} ${H}" width="${W}" height="${H}" role="img" aria-label="Runcap terminal demo: plan, cap, compress, stop">
+const svg = `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 ${W} ${H}" width="${W}" height="${H}" role="img" aria-label="Runcap terminal demo: plan, cap, compress, detect loop, stop">
   <defs>
     <linearGradient id="brand" x1="0" y1="0" x2="1" y2="0">
       <stop offset="0" stop-color="#22d3ee"/><stop offset="1" stop-color="#34d399"/>
@@ -64,7 +65,7 @@ const svg = `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 ${W} ${H}" wid
     <circle cx="26" cy="28" r="6" fill="#f87171"/>
     <circle cx="48" cy="28" r="6" fill="#fbbf24"/>
     <circle cx="70" cy="28" r="6" fill="#34d399"/>
-    <text x="100" y="33" fill="#8a8a8a" font-family="'JetBrains Mono',monospace" font-size="14">runcap — estimate · cap · compress · rescue</text>
+    <text x="100" y="33" fill="#8a8a8a" font-family="'JetBrains Mono',monospace" font-size="14">runcap · estimate · cap · compress · loop · rescue</text>
     <text x="${W-150}" y="33" fill="url(#brand)" font-family="'JetBrains Mono',monospace" font-weight="700" font-size="15">run·cap</text>
   </g>
   <line x1="0" y1="50" x2="${W}" y2="50" stroke="#1c1c1f"/>