@ducci/jarvis 1.0.32 → 1.0.33

package/docs/findings/016-file-writing-corruption-and-stderr-loop.md

@@ -0,0 +1,119 @@
+# Finding 016: File Writing Corruption, Misleading Stderr Nudge, and Repeated-Error Loop
+
+**Date:** 2026-03-02
+**Severity:** High — agent burned 10 iterations on the wrong diagnosis; task abandoned
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `a25fd973-3e92-4902-a96d-536ef0eb3005`. Model: `nvidia/nemotron-3-nano-30b-a3b:free`. User asked to run a ZAP security scanner script.
+
+The script (`scan.sh`) failed immediately with `$ZAP_CMD: command not found`. The agent used all 10 iterations investigating PATH issues and gave up without solving the problem or producing a handoff checkpoint.
+
+---
+
+## Root Cause 1: Shell Script Written with Escaped Dollar Signs
+
+### What happened
+
+`scan.sh` was written in a prior session by the agent using `exec` with a shell command (echo or heredoc). Multi-layered escaping — JS string → JSON encoding → bash variable expansion — caused every `$` in the script to be written as `\$` in the file.
+
+In bash, `"\$VAR"` (backslash-dollar in double quotes) suppresses variable expansion and produces the literal string `$VAR`. The script ran but nothing expanded:
+
+```
+bash -x scan.sh http://testphp.vulnweb.com:
+
++ DOMAIN='$1'                              ← should be 'http://testphp.vulnweb.com'
++ OUTPUT_DIR='/path/results/$DOMAIN'       ← should be '/path/results/http://...'
++ '$ZAP_CMD' -cmd ...                      ← tries to run a command literally named '$ZAP_CMD'
+scan.sh: line 27: $ZAP_CMD: command not found
+```
+
+Secondary confirmation: the project directory contained folders literally named `$OUTPUT_DIR` and `$RESULTS_DIR`, created by a prior run of the broken script.
+
+### Fix
+
+Added `write_file` as a seed tool. It calls `fs.promises.writeFile` directly — content arrives as a JSON string and is written to disk verbatim. No shell is involved, so no escaping layer can corrupt dollar signs or backslashes.
+
+Added an optional `mode` parameter (e.g. `"755"`) to make scripts executable in the same call.
+
+Updated the system prompt with a dedicated "Writing Files" section (peer-level to "exec Safety") stating: use `write_file` for all file creation — never `exec` with `echo`, `printf`, or heredoc.
+
+**Files changed:**
+- `src/server/tools.js` — `write_file` added to `SEED_TOOLS`
+- `docs/system-prompt.md` — new "Writing Files" section; removed the buried `echo -e` bullet from exec Safety
+
+---
+
+## Root Cause 2: Stderr Nudge Misdirected the Model
+
+### What happened
+
+After every failed tool call with stderr output, the system injected:
+
+> *"Examine the stderr field in the tool result carefully — it likely describes the root cause of the failure."*
+
+The stderr was `$ZAP_CMD: command not found` — which looks like a PATH problem. The real diagnostic clue was in the **stdout** of `bash -x`: `DOMAIN='$1'` (variable not expanded). By directing the model to stderr, the nudge trained its attention away from the evidence.
+
+The model then spent iterations on: explicit PATH overrides, `command -v` checks, `sed -n l`, `cat -A`, `which bash` — all stderr-adjacent investigation — and never examined what the `-x` trace was actually showing.
+
+### Fix
+
+Changed the stderr nudge to cover both stdout and stderr, with an explicit callout to `bash -x` as a debug tool whose key output is in stdout:
+
+```
+[System: A command failed. Examine both the stdout AND stderr fields in the tool result —
+stderr names the error, but stdout (especially from debug commands like bash -x) often shows
+the root cause. Do not retry without first understanding what the full output is telling you.]
+```
+
+**File changed:** `src/server/agent.js` — `stderrErrorInIteration` nudge
+
+---
+
+## Root Cause 3: No Detection for the Same Error Repeating Across Different Commands
+
+### What happened
+
+The existing loop detector tracks `tool + args + result` triples. Because the model varied its tool calls each iteration (different PATH strings, different diagnostic commands), this never triggered. Meanwhile `$ZAP_CMD: command not found` appeared in stderr across 5+ tool calls from entirely different commands — a strong signal that the diagnosis is wrong, not the approach.
+
+Existing detectors that missed this:
+- `loopTracker` — requires identical tool + args + result; missed because args varied
+- `consecutiveFailures` — tracks back-to-back failures; partially reset when some calls succeeded
+- `maxHandoffs` / zero-progress — apply only to checkpoint-reached runs
+
+### Fix
+
+Added `stderrTracker = new Map()` in `runAgentLoop` (parallel to `loopTracker`). After each failed tool call, the first line of stderr is extracted as the key and its count incremented.
+
+When any stderr string reaches `CONSECUTIVE_FAILURE_THRESHOLD` (3), a targeted "step back" nudge is injected, quoting the repeating error, instead of the basic stderr nudge:
+
+```
+[System: The error "$ZAP_CMD: command not found" has now appeared 3 times across different
+commands. You are repeatedly diagnosing the wrong thing. Stop, step back, and reconsider
+from scratch — what is this error fundamentally telling you about the state of the system?]
+```
+
+Using only the first line of stderr (not the full message) makes the tracker robust to verbose multi-line output where later lines may contain timestamps or variable content.
+
+**File changed:** `src/server/agent.js` — `stderrTracker`, `firstStderrLine` extraction, `repeatedStderr` check replacing basic stderr nudge when threshold reached
+
+---
+
+## Why the Session Never Produced a Checkpoint
+
+The model produced a final text response on iteration 10 (it didn't time out — it gave up). This means `!done` was never true after the while loop, so the wrap-up / checkpoint path never ran. The model exhausted its budget investigating PATH and on the last iteration concluded it couldn't solve the problem.
+
+Fix 3 (repeated-error nudge) would have fired by iteration 5–6 with the specific message quoting `$ZAP_CMD: command not found`. At that point the model would have had a chance to reconsider what the error was telling it rather than continuing PATH investigation.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/server/tools.js` | Added `write_file` seed tool |
+| `docs/system-prompt.md` | Added "Writing Files" section; removed `echo -e` bullet from exec Safety |
+| `src/server/agent.js` | Added `stderrTracker`; first-line stderr key extraction; repeated-error nudge overriding basic stderr nudge; broadened stderr nudge wording to cover stdout |

package/docs/system-prompt.md

@@ -54,7 +54,15 @@ The `exec` tool runs real shell commands on the server. Use it responsibly:
 - **Use known paths.** Prefer `process.cwd()`, `$HOME`, or paths you already know over broad searches. Use `which <binary>` to locate executables.
 - **Prefer targeted reads.** Use `grep`, `head`, or `tail` instead of `cat` on files you haven't seen before. Large file output is truncated anyway — a targeted command gives you better signal.
 - **Avoid commands with unbounded runtime.** If a command could run indefinitely or scan an unknown-size tree, scope it first.
-- **Writing multi-line files**: use `printf '...'` or a heredoc (`cat <<'EOF' > file`) instead of `echo -e`. The `-e` flag is not portable — on Ubuntu `/bin/sh` it is treated as literal text, corrupting the file.
+
+## Writing Files
+
+Use the `write_file` tool to create or overwrite any file. Never use `exec` with `echo`, `printf`, or heredoc to write files.
+
+Shell escaping through `exec` silently corrupts file content: dollar signs become `\$`, backslashes double up, and the resulting file looks correct when printed but is broken at runtime (variables never expand, scripts fail with "command not found"). `write_file` bypasses all shell interpretation — content arrives as a JSON string and lands in the file exactly as written.
+
+- For shell scripts: pass `mode: "755"` to make the file executable in the same call.
+- For any other file: omit `mode` or use `"644"`.
 
 ## Execution Timeouts
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.32",
+  "version": "1.0.33",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -103,6 +103,7 @@ async function runAgentLoop(client, config, session, prepareMessages) {
   let logSummary = '';
   let status = 'ok';
   let consecutiveFailures = 0;
+  const stderrTracker = new Map();
 
   while (iteration < config.maxIterations) {
     iteration++;
@@ -199,6 +200,10 @@ async function runAgentLoop(client, config, session, prepareMessages) {
           consecutiveFailures++;
           if (resultObj && resultObj.stderr) {
             stderrErrorInIteration = true;
+            const firstStderrLine = resultObj.stderr.split('\n')[0].trim();
+            if (firstStderrLine) {
+              stderrTracker.set(firstStderrLine, (stderrTracker.get(firstStderrLine) || 0) + 1);
+            }
           }
         } else {
           consecutiveFailures = 0;
@@ -236,10 +241,16 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         });
       }
 
-      if (stderrErrorInIteration && !loopDetected) {
+      const repeatedStderr = [...stderrTracker.entries()].find(([, count]) => count >= CONSECUTIVE_FAILURE_THRESHOLD);
+      if (repeatedStderr && !loopDetected) {
         session.messages.push({
           role: 'user',
-          content: '[System: A command failed and produced stderr output. Examine the stderr field in the tool result carefully — it likely describes the root cause of the failure. Do not retry the same command without first addressing what stderr reports.]',
+          content: `[System: The error "${repeatedStderr[0].slice(0, 200)}" has now appeared ${repeatedStderr[1]} times across different commands. You are repeatedly diagnosing the wrong thing. Stop, step back, and reconsider from scratch — what is this error fundamentally telling you about the state of the system?]`,
+        });
+      } else if (stderrErrorInIteration && !loopDetected) {
+        session.messages.push({
+          role: 'user',
+          content: '[System: A command failed. Examine both the stdout AND stderr fields in the tool result — stderr names the error, but stdout (especially from debug commands like bash -x) often shows the root cause. Do not retry without first understanding what the full output is telling you.]',
         });
       }
 

package/src/server/tools.js

@@ -347,6 +347,43 @@ const SEED_TOOLS = {
       }
     `,
   },
+  write_file: {
+    definition: {
+      type: 'function',
+      function: {
+        name: 'write_file',
+        description: 'Write content directly to a file on the filesystem, bypassing all shell escaping. Use this to create or overwrite any file — shell scripts, config files, code, etc. Content is written exactly as provided: dollar signs, backslashes, and special characters are preserved without modification. Always prefer this over exec+echo, exec+printf, or exec+heredoc for writing files. For shell scripts, pass mode: "755" to make the file executable. Example: write_file({ path: "/path/to/scan.sh", content: "#!/bin/bash\\nDOMAIN=$1\\n...", mode: "755" })',
+        parameters: {
+          type: 'object',
+          properties: {
+            path: {
+              type: 'string',
+              description: 'Absolute or relative path to the file to write. Parent directories are created automatically.',
+            },
+            content: {
+              type: 'string',
+              description: 'The content to write to the file. Written as-is — no shell interpretation occurs.',
+            },
+            mode: {
+              type: 'string',
+              description: 'Optional Unix file mode in octal string form, e.g. "755" for executable scripts, "644" for regular files. Defaults to "644".',
+            },
+          },
+          required: ['path', 'content'],
+        },
+      },
+    },
+    code: `
+      const targetPath = path.resolve(args.path);
+      await fs.promises.mkdir(path.dirname(targetPath), { recursive: true });
+      await fs.promises.writeFile(targetPath, args.content, 'utf8');
+      if (args.mode) {
+        await fs.promises.chmod(targetPath, parseInt(args.mode, 8));
+      }
+      const bytes = Buffer.byteLength(args.content, 'utf8');
+      return { status: 'ok', path: targetPath, bytes, mode: args.mode || '644' };
+    `,
+  },
   get_recent_sessions: {
     definition: {
       type: 'function',