@ducci/jarvis 1.0.31 → 1.0.33

package/docs/findings/015-failed-run-context-strip.md

@@ -0,0 +1,142 @@
+# Finding 015: Failed Runs Leave Tool History in Context (Context Bloat Death Spiral)
+
+**Date:** 2026-03-02
+**Severity:** High — caused 3 consecutive `model_error: Empty choices array` failures; session unusable
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `6123209d-ce5a-44d0-be12-29aac58b4cf3`. Model: `nvidia/nemotron-3-nano-30b-a3b:free`. User requested a ZAP security scanning project.
+
+| Entry | Trigger | Status | messageCount at failure | toolCalls |
+|-------|---------|--------|------------------------|-----------|
+| 1 | "hi all good?" | ok | — | 0 |
+| 2 | ZAP task (run 1) | checkpoint_reached | — | 10 |
+| 3 | handoff resume (run 2) | checkpoint_reached | — | 10 |
+| 4 | handoff resume (run 3) | model_error (empty choices, iter 7) | 22 | 26 |
+| 5 | "Why I get Model returned an empty response?" | model_error (empty choices, iter 3) | 27 | 2 |
+| 6 | "Why I get Model returned an empty response again?!!" | model_error (empty choices, iter 5) | 37 | 4 |
+
+The session ended without producing any result. The user received `'Model returned an empty response.'` three times.
+
+---
+
+## Root Cause 1: Failed runs leave tool call history in session
+
+### What happened
+
+The handoff loop strips tool call messages for `checkpoint_reached` runs:
+
+```js
+session.messages.splice(runStartIndex, session.messages.length - runStartIndex - 1);
+```
+
+Runs that end with `model_error` or `format_error` received **no strip**. Every tool call message (assistant+tool pair, nudge injections) from the failed run remained in `session.messages`, with only a synthetic error note appended afterward.
+
+Run 3 had 26 tool calls across 7 iterations — approximately 13 messages added to the session. These were preserved verbatim. Each subsequent user turn started with more context than the last.
+
+### Message count growth
+
+- Before run 3: ~8 messages (runs 1 and 2 were both checkpoint_reached and stripped correctly)
+- After entry 4 (model_error, no strip): 21 messages + synthetic note = 22
+- After entry 5 (model_error, no strip): 27 messages + synthetic note = 28
+- At entry 6: 37 messages in context
+
+The free model returns `choices: []` when the context exceeds what it can handle. Each failure added more context, making the next failure more likely: a **positive feedback death spiral**.
+
+### Fix
+
+Apply the same splice that checkpoint runs already use:
+
+```js
+if (finalStatus === 'model_error' || finalStatus === 'format_error') {
+  session.messages.splice(runStartIndex, session.messages.length - runStartIndex);
+  // then push synthetic error note as before
+}
+```
+
+The strip runs before the synthetic error note is pushed, returning the session to its pre-run state plus one concise note. The JSONL log preserves all tool results for retrospective inspection via `read_session_log`.
+
+**File**: `src/server/agent.js` — `_runHandleChat`, non-checkpoint break path
+
+---
+
+## Root Cause 2: No detection or escalation for consecutive model_errors
+
+### What happened
+
+After two consecutive `model_error: Empty choices array` entries (4 and 5), no protective mechanism fired. The system continued accepting new user messages and spawning new runs indefinitely.
+
+Existing protection mechanisms all missed this case:
+- `maxHandoffs` — only applies to `checkpoint_reached` runs
+- `consecutiveFailures` — tracks tool failures within a single run
+- Zero-progress detection — only applies to `checkpoint_reached` runs
+
+### Fix
+
+Detect the pattern structurally in `session.messages` before starting each new run: if the last two assistant messages are both synthetic `model_error` notes, the session is in a confirmed failure loop. Escalate to `intervention_required` without running another agent loop.
+
+```js
+function hasConsecutiveModelErrors(messages) {
+  const assistantTail = messages.filter(m => m.role === 'assistant').slice(-2);
+  return (
+    assistantTail.length === 2 &&
+    assistantTail.every(
+      m =>
+        typeof m.content === 'string' &&
+        m.content.startsWith('[System: Previous run failed (model_error)')
+    )
+  );
+}
+```
+
+This uses no additional state: it reads the session history directly. Old sessions are handled correctly. One failure is allowed (transient errors are real); two consecutive failures mean the session cannot self-recover.
+
+Combined with Fix 1, consecutive model_errors in this session would have played out as:
+1. Entry 4 (run 3): model_error → strip → synthetic note. Session back to 9 messages.
+2. Entry 5 (user "Why?"): run 4 starts with 10 messages. If it still fails → strip → synthetic note. Two model_error notes now in session.
+3. Entry 6 (user "Why again?!"): `hasConsecutiveModelErrors` fires → `intervention_required` returned immediately. User gets a clear message: start a new session or switch model.
+
+**File**: `src/server/agent.js` — `hasConsecutiveModelErrors` function + check at top of handoff loop
+
+---
+
+## Root Cause 3: Empty choices error message provides no actionable guidance
+
+### What happened
+
+The `choices.length === 0` path returned:
+
+```
+Model returned an empty response.
+```
+
+When the user asked "why?", the agent — with ZAP tool call context still present — continued ZAP investigation instead of explaining the API failure. The opaque error and the polluted context compounded: the model had no clear signal about what went wrong and no guidance on how to recover.
+
+### Fix
+
+Include the context size and recovery guidance in the response:
+
+```js
+response: `Model returned an empty response (${preparedMessages.length} messages in context). This typically happens when the conversation is too long for the model. Try starting a new session or switching to a model with a larger context window.`,
+```
+
+**File**: `src/server/agent.js` — `runAgentLoop`, empty choices early return
+
+---
+
+## Why Fix 1 is Primary
+
+Fix 1 is the root fix. With context stripped after each failure, the model operates on a tiny session (~10 messages) on subsequent turns. The free model handles this easily. Fix 2 is a safety net for persistent non-context failures. Fix 3 improves user-facing error messages for the residual cases that slip through.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/server/agent.js` | Strip tool history on `model_error`/`format_error` (same as checkpoint) |
+| `src/server/agent.js` | `hasConsecutiveModelErrors` function + check before each run in handoff loop |
+| `src/server/agent.js` | Include message count in empty choices response |

package/docs/findings/016-file-writing-corruption-and-stderr-loop.md

@@ -0,0 +1,119 @@
+# Finding 016: File Writing Corruption, Misleading Stderr Nudge, and Repeated-Error Loop
+
+**Date:** 2026-03-02
+**Severity:** High — agent burned 10 iterations on the wrong diagnosis; task abandoned
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `a25fd973-3e92-4902-a96d-536ef0eb3005`. Model: `nvidia/nemotron-3-nano-30b-a3b:free`. User asked to run a ZAP security scanner script.
+
+The script (`scan.sh`) failed immediately with `$ZAP_CMD: command not found`. The agent used all 10 iterations investigating PATH issues and gave up without solving the problem or producing a handoff checkpoint.
+
+---
+
+## Root Cause 1: Shell Script Written with Escaped Dollar Signs
+
+### What happened
+
+`scan.sh` was written in a prior session by the agent using `exec` with a shell command (echo or heredoc). Multi-layered escaping — JS string → JSON encoding → bash variable expansion — caused every `$` in the script to be written as `\$` in the file.
+
+In bash, `"\$VAR"` (backslash-dollar in double quotes) suppresses variable expansion and produces the literal string `$VAR`. The script ran but nothing expanded:
+
+```
+bash -x scan.sh http://testphp.vulnweb.com:
+
++ DOMAIN='$1'                              ← should be 'http://testphp.vulnweb.com'
++ OUTPUT_DIR='/path/results/$DOMAIN'       ← should be '/path/results/http://...'
++ '$ZAP_CMD' -cmd ...                      ← tries to run a command literally named '$ZAP_CMD'
+scan.sh: line 27: $ZAP_CMD: command not found
+```
+
+Secondary confirmation: the project directory contained folders literally named `$OUTPUT_DIR` and `$RESULTS_DIR`, created by a prior run of the broken script.
+
+### Fix
+
+Added `write_file` as a seed tool. It calls `fs.promises.writeFile` directly — content arrives as a JSON string and is written to disk verbatim. No shell is involved, so no escaping layer can corrupt dollar signs or backslashes.
+
+Added an optional `mode` parameter (e.g. `"755"`) to make scripts executable in the same call.
+
+Updated the system prompt with a dedicated "Writing Files" section (peer-level to "exec Safety") stating: use `write_file` for all file creation — never `exec` with `echo`, `printf`, or heredoc.
+
+**Files changed:**
+- `src/server/tools.js` — `write_file` added to `SEED_TOOLS`
+- `docs/system-prompt.md` — new "Writing Files" section; removed the buried `echo -e` bullet from exec Safety
+
+---
+
+## Root Cause 2: Stderr Nudge Misdirected the Model
+
+### What happened
+
+After every failed tool call with stderr output, the system injected:
+
+> *"Examine the stderr field in the tool result carefully — it likely describes the root cause of the failure."*
+
+The stderr was `$ZAP_CMD: command not found` — which looks like a PATH problem. The real diagnostic clue was in the **stdout** of `bash -x`: `DOMAIN='$1'` (variable not expanded). By directing the model to stderr, the nudge trained its attention away from the evidence.
+
+The model then spent iterations on: explicit PATH overrides, `command -v` checks, `sed -n l`, `cat -A`, `which bash` — all stderr-adjacent investigation — and never examined what the `-x` trace was actually showing.
+
+### Fix
+
+Changed the stderr nudge to cover both stdout and stderr, with an explicit callout to `bash -x` as a debug tool whose key output is in stdout:
+
+```
+[System: A command failed. Examine both the stdout AND stderr fields in the tool result —
+stderr names the error, but stdout (especially from debug commands like bash -x) often shows
+the root cause. Do not retry without first understanding what the full output is telling you.]
+```
+
+**File changed:** `src/server/agent.js` — `stderrErrorInIteration` nudge
+
+---
+
+## Root Cause 3: No Detection for the Same Error Repeating Across Different Commands
+
+### What happened
+
+The existing loop detector tracks `tool + args + result` triples. Because the model varied its tool calls each iteration (different PATH strings, different diagnostic commands), this never triggered. Meanwhile `$ZAP_CMD: command not found` appeared in stderr across 5+ tool calls from entirely different commands — a strong signal that the diagnosis is wrong, not the approach.
+
+Existing detectors that missed this:
+- `loopTracker` — requires identical tool + args + result; missed because args varied
+- `consecutiveFailures` — tracks back-to-back failures; partially reset when some calls succeeded
+- `maxHandoffs` / zero-progress — apply only to checkpoint-reached runs
+
+### Fix
+
+Added `stderrTracker = new Map()` in `runAgentLoop` (parallel to `loopTracker`). After each failed tool call, the first line of stderr is extracted as the key and its count incremented.
+
+When any stderr string reaches `CONSECUTIVE_FAILURE_THRESHOLD` (3), a targeted "step back" nudge is injected, quoting the repeating error, instead of the basic stderr nudge:
+
+```
+[System: The error "$ZAP_CMD: command not found" has now appeared 3 times across different
+commands. You are repeatedly diagnosing the wrong thing. Stop, step back, and reconsider
+from scratch — what is this error fundamentally telling you about the state of the system?]
+```
+
+Using only the first line of stderr (not the full message) makes the tracker robust to verbose multi-line output where later lines may contain timestamps or variable content.
+
+**File changed:** `src/server/agent.js` — `stderrTracker`, `firstStderrLine` extraction, `repeatedStderr` check replacing basic stderr nudge when threshold reached
+
+---
+
+## Why the Session Never Produced a Checkpoint
+
+The model produced a final text response on iteration 10 (it didn't time out — it gave up). This means `!done` was never true after the while loop, so the wrap-up / checkpoint path never ran. The model exhausted its budget investigating PATH and on the last iteration concluded it couldn't solve the problem.
+
+Fix 3 (repeated-error nudge) would have fired by iteration 5–6 with the specific message quoting `$ZAP_CMD: command not found`. At that point the model would have had a chance to reconsider what the error was telling it rather than continuing PATH investigation.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/server/tools.js` | Added `write_file` seed tool |
+| `docs/system-prompt.md` | Added "Writing Files" section; removed `echo -e` bullet from exec Safety |
+| `src/server/agent.js` | Added `stderrTracker`; first-line stderr key extraction; repeated-error nudge overriding basic stderr nudge; broadened stderr nudge wording to cover stdout |

package/docs/system-prompt.md

@@ -54,7 +54,15 @@ The `exec` tool runs real shell commands on the server. Use it responsibly:
 - **Use known paths.** Prefer `process.cwd()`, `$HOME`, or paths you already know over broad searches. Use `which <binary>` to locate executables.
 - **Prefer targeted reads.** Use `grep`, `head`, or `tail` instead of `cat` on files you haven't seen before. Large file output is truncated anyway — a targeted command gives you better signal.
 - **Avoid commands with unbounded runtime.** If a command could run indefinitely or scan an unknown-size tree, scope it first.
-- **Writing multi-line files**: use `printf '...'` or a heredoc (`cat <<'EOF' > file`) instead of `echo -e`. The `-e` flag is not portable — on Ubuntu `/bin/sh` it is treated as literal text, corrupting the file.
+
+## Writing Files
+
+Use the `write_file` tool to create or overwrite any file. Never use `exec` with `echo`, `printf`, or heredoc to write files.
+
+Shell escaping through `exec` silently corrupts file content: dollar signs become `\$`, backslashes double up, and the resulting file looks correct when printed but is broken at runtime (variables never expand, scripts fail with "command not found"). `write_file` bypasses all shell interpretation — content arrives as a JSON string and lands in the file exactly as written.
+
+- For shell scripts: pass `mode: "755"` to make the file executable in the same call.
+- For any other file: omit `mode` or use `"644"`.
 
 ## Execution Timeouts
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.31",
+  "version": "1.0.33",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -69,6 +69,25 @@ async function callModelWithFallback(client, config, messages, tools) {
   }
 }
 
+/**
+ * Returns true if the last two assistant messages in the session are both
+ * synthetic model_error notes, indicating a confirmed failure loop that cannot
+ * self-resolve (e.g. persistent empty choices from context overflow).
+ */
+function hasConsecutiveModelErrors(messages) {
+  const assistantTail = messages
+    .filter(m => m.role === 'assistant')
+    .slice(-2);
+  return (
+    assistantTail.length === 2 &&
+    assistantTail.every(
+      m =>
+        typeof m.content === 'string' &&
+        m.content.startsWith('[System: Previous run failed (model_error)')
+    )
+  );
+}
+
 /**
  * Runs a single agent loop up to maxIterations.
  * Returns { iteration, response, logSummary, status, runToolCalls, checkpoint }.
@@ -84,6 +103,7 @@ async function runAgentLoop(client, config, session, prepareMessages) {
   let logSummary = '';
   let status = 'ok';
   let consecutiveFailures = 0;
+  const stderrTracker = new Map();
 
   while (iteration < config.maxIterations) {
     iteration++;
@@ -112,7 +132,7 @@ async function runAgentLoop(client, config, session, prepareMessages) {
     if (!modelResult.choices || modelResult.choices.length === 0) {
       return {
         iteration,
-        response: 'Model returned an empty response.',
+        response: `Model returned an empty response (${preparedMessages.length} messages in context). This typically happens when the conversation is too long for the model. Try starting a new session or switching to a model with a larger context window.`,
         logSummary: `Model error on iteration ${iteration}: Empty choices array.`,
         status: 'model_error',
         runToolCalls,
@@ -180,6 +200,10 @@ async function runAgentLoop(client, config, session, prepareMessages) {
           consecutiveFailures++;
           if (resultObj && resultObj.stderr) {
             stderrErrorInIteration = true;
+            const firstStderrLine = resultObj.stderr.split('\n')[0].trim();
+            if (firstStderrLine) {
+              stderrTracker.set(firstStderrLine, (stderrTracker.get(firstStderrLine) || 0) + 1);
+            }
           }
         } else {
           consecutiveFailures = 0;
@@ -217,10 +241,16 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         });
       }
 
-      if (stderrErrorInIteration && !loopDetected) {
+      const repeatedStderr = [...stderrTracker.entries()].find(([, count]) => count >= CONSECUTIVE_FAILURE_THRESHOLD);
+      if (repeatedStderr && !loopDetected) {
+        session.messages.push({
+          role: 'user',
+          content: `[System: The error "${repeatedStderr[0].slice(0, 200)}" has now appeared ${repeatedStderr[1]} times across different commands. You are repeatedly diagnosing the wrong thing. Stop, step back, and reconsider from scratch — what is this error fundamentally telling you about the state of the system?]`,
+        });
+      } else if (stderrErrorInIteration && !loopDetected) {
         session.messages.push({
           role: 'user',
-          content: '[System: A command failed and produced stderr output. Examine the stderr field in the tool result carefully — it likely describes the root cause of the failure. Do not retry the same command without first addressing what stderr reports.]',
+          content: '[System: A command failed. Examine both the stdout AND stderr fields in the tool result — stderr names the error, but stdout (especially from debug commands like bash -x) often shows the root cause. Do not retry without first understanding what the full output is telling you.]',
         });
       }
 
@@ -482,6 +512,25 @@ async function _runHandleChat(config, sessionId, userMessage) {
   try {
     // Handoff loop
     while (true) {
+      // Safety check: if the last two assistant messages are both model_error
+      // synthetic notes, we are in a confirmed failure loop. Escalate immediately
+      // rather than burning more iterations on a stuck session.
+      if (hasConsecutiveModelErrors(session.messages)) {
+        finalResponse = 'The model has failed twice in a row. This is likely due to the conversation being too long for the model to process. Please start a new session or switch to a model with a larger context window.';
+        finalLogSummary = 'Consecutive model_error detected: session escalated to intervention_required without running another agent loop.';
+        finalStatus = 'intervention_required';
+        await appendLog(sessionId, {
+          iteration: 0,
+          model: config.selectedModel,
+          userInput: userMessage,
+          toolCalls: [],
+          response: finalResponse,
+          logSummary: finalLogSummary,
+          status: 'intervention_required',
+        });
+        break;
+      }
+
       const runStartIndex = session.messages.length;
       const run = await runAgentLoop(client, config, session, prepareMessages);
       allToolCalls.push(...run.runToolCalls);
@@ -505,8 +554,14 @@ async function _runHandleChat(config, sessionId, userMessage) {
         if (run.rawResponse) logEntry.rawResponse = run.rawResponse;
         await appendLog(sessionId, logEntry);
 
-        // Inject synthetic error note so the model has context on the next user turn
+        // Inject synthetic error note so the model has context on the next user turn.
+        // For failed runs, also strip the tool call history — keeping it would bloat
+        // the context and create a positive-feedback death spiral where each failure
+        // makes the next one more likely (especially on free models with small context
+        // windows). The synthetic note is sufficient context; tool results are preserved
+        // in the JSONL log and accessible via read_session_log.
         if (finalStatus === 'model_error' || finalStatus === 'format_error') {
+          session.messages.splice(runStartIndex, session.messages.length - runStartIndex);
           const errorDetail = run.errorDetail ? ` Error detail: ${JSON.stringify(run.errorDetail)}` : '';
           session.messages.push({
             role: 'assistant',

package/src/server/tools.js

@@ -347,6 +347,43 @@ const SEED_TOOLS = {
       }
     `,
   },
+  write_file: {
+    definition: {
+      type: 'function',
+      function: {
+        name: 'write_file',
+        description: 'Write content directly to a file on the filesystem, bypassing all shell escaping. Use this to create or overwrite any file — shell scripts, config files, code, etc. Content is written exactly as provided: dollar signs, backslashes, and special characters are preserved without modification. Always prefer this over exec+echo, exec+printf, or exec+heredoc for writing files. For shell scripts, pass mode: "755" to make the file executable. Example: write_file({ path: "/path/to/scan.sh", content: "#!/bin/bash\\nDOMAIN=$1\\n...", mode: "755" })',
+        parameters: {
+          type: 'object',
+          properties: {
+            path: {
+              type: 'string',
+              description: 'Absolute or relative path to the file to write. Parent directories are created automatically.',
+            },
+            content: {
+              type: 'string',
+              description: 'The content to write to the file. Written as-is — no shell interpretation occurs.',
+            },
+            mode: {
+              type: 'string',
+              description: 'Optional Unix file mode in octal string form, e.g. "755" for executable scripts, "644" for regular files. Defaults to "644".',
+            },
+          },
+          required: ['path', 'content'],
+        },
+      },
+    },
+    code: `
+      const targetPath = path.resolve(args.path);
+      await fs.promises.mkdir(path.dirname(targetPath), { recursive: true });
+      await fs.promises.writeFile(targetPath, args.content, 'utf8');
+      if (args.mode) {
+        await fs.promises.chmod(targetPath, parseInt(args.mode, 8));
+      }
+      const bytes = Buffer.byteLength(args.content, 'utf8');
+      return { status: 'ok', path: targetPath, bytes, mode: args.mode || '644' };
+    `,
+  },
   get_recent_sessions: {
     definition: {
       type: 'function',