@ducci/jarvis 1.0.29 → 1.0.31

package/docs/findings/013-stderr-visibility-and-truncation.md

@@ -0,0 +1,59 @@
+# Finding 013 — stderr Visibility and Output Truncation
+
+## Observed Behaviour
+
+During a multi-run ZAP security scan session, the agent repeatedly failed to diagnose and fix the root cause of scan failures. It issued `pkill`/`kill` variants dozens of times, burned through 6 iteration limits and 2 handoffs, and ultimately gave up without producing results.
+
+Post-mortem analysis of the debug session logs revealed two compounding problems.
+
+## Root Causes
+
+### 1. Head-only truncation buried errors at the end of output
+
+`MAX_TOOL_RESULT = 4000` was applied as a simple head slice: `resultStr.slice(0, 4000)`. ZAP produces verbose startup logs (JVM init, add-on loading, database migration) that easily exceed 4000 characters. Five separate ZAP exec results were truncated exactly at the limit, cutting off during database migration messages. Any errors that appeared later in the output — after the verbose preamble — were silently dropped before the model ever saw them.
+
+### 2. Model ignored stderr even when it was visible
+
+In one un-truncated result (810 chars), the critical error was plainly present:
+
+```
+g_module_open() failed for libpixbufloader-tiff.so: libtiff.so.5: cannot open shared object file: No such file or directory
+```
+
+The model's subsequent response ignored it entirely and concluded only that "no results were found in the output directory." There was no mechanism forcing the model to re-examine stderr before forming its conclusion or retrying.
+
+Similarly, all `pkill`/`kill` commands returned `exitCode: 1` with clear stderr — yet the model continued issuing variations of the same commands without diagnosing why process termination was failing.
+
+## Fixes
+
+### Fix 1 — Head + tail truncation (`agent.js`)
+
+Replace head-only truncation with a head+tail strategy:
+
+```js
+// Before
+resultStr.slice(0, MAX_TOOL_RESULT) + '\n[...truncated]'
+
+// After
+resultStr.slice(0, 2000) + `\n[...${resultStr.length - 4000} chars truncated...]\n` + resultStr.slice(-2000)
+```
+
+The first 2000 chars preserve startup context; the last 2000 chars preserve the diagnostic tail where errors typically appear. The marker in the middle shows how much was dropped. Total budget stays at 4000 chars.
+
+### Fix 2 — Stderr nudge injection (`agent.js`)
+
+After each iteration's tool-call loop, if any tool failed (`status === 'error'`) with non-empty `stderr`, inject a system message:
+
+```
+[System: A command failed and produced stderr output. Examine the stderr field in the tool result carefully — it likely describes the root cause of the failure. Do not retry the same command without first addressing what stderr reports.]
+```
+
+This creates an active forcing function — the model cannot continue to the next iteration without the nudge appearing in its context. The nudge is suppressed if loop detection already fired (to avoid contradictory instructions).
+
+## Known Gap
+
+The stderr nudge only fires when `toolFailed` is true (i.e., `exec` returned `status: 'error'`). Commands that return `exitCode: 0` but still emit meaningful errors to stderr (e.g., a shell script that succeeds but a subprocess inside it fails) will not trigger the nudge. Catching that case without generating noise from normal stderr usage (npm warnings, apt-get progress) requires more context than is available at this level. Documented as a known limitation.
+
+## Files Changed
+
+- `src/server/agent.js` — truncation strategy + stderr nudge injection

package/docs/findings/014-exec-stderr-artifact-and-malformed-tool-args.md

@@ -0,0 +1,202 @@
+# Finding 014: exec stderr Artifact and Malformed Tool Call Arguments
+
+**Date:** 2026-03-02
+**Severity:** Medium — caused spurious context noise (spurious nudges), agent confusion on malformed args, and silent loss of failedApproaches across user turns
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `d97070f7-e50f-4d9e-a38b-b68e7b27e7b7`. User asked Jarvis to set up an OWASP ZAP web security scanning project. The session ran 4 runs (40 total iterations) without completing the task. The snap-installed ZAP does not include `zap-baseline.sh` (which only ships in the ZAP Docker image); the agent never discovered that `zaproxy -cmd -quickurl` is the correct snap-native CLI equivalent.
+
+Three compounding issues were identified that degraded the agent's ability to self-correct.
+
+---
+
+## Issue 1: exec Tool Injects Node.js Error Message into `stderr` Field
+
+### What happened
+
+Commands like `which zap-cli` (not installed), `grep` returning no matches, and piped `find | grep` with no results all return exit code 1. In each case the actual process wrote nothing to stderr. But the exec tool result showed:
+
+```json
+{"status":"error","exitCode":1,"stdout":"","stderr":"Command failed: which zap-cli\n"}
+```
+
+The `"Command failed: ..."` string is not from the process — it is `e.message` from Node.js's `execAsync` error object, injected via `e.stderr || e.message`.
+
+The stderr nudge check in `agent.js` fires on any non-empty `resultObj.stderr`:
+
+```js
+if (resultObj && resultObj.stderr) {
+  stderrErrorInIteration = true;
+}
+```
+
+This triggered the nudge "Examine the stderr field carefully — it likely describes the root cause of the failure" 3–4 times per run, when there was nothing actionable to examine in stderr.
+
+### Root cause
+
+In the exec seed tool (`src/server/tools.js`, line 76):
+
+```js
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+```
+
+The `|| e.message` fallback was designed to show something when `e.stderr` is empty. But it conflates process-generated stderr with Node.js meta-messages about the process exiting non-zero.
+
+### Fix
+
+```js
+// Before:
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+
+// After:
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || "" };
+```
+
+`status: error` and `exitCode` already signal failure. The "Command failed: ..." Node.js string is not diagnostic and should not appear in the stderr field.
+
+**File**: `src/server/tools.js` — exec seed tool code (propagates via seedTools() on next server start)
+
+---
+
+## Issue 2: Malformed Tool Call JSON Arguments Silently Swallowed
+
+### What happened
+
+The model sent a tool call with malformed arguments (missing opening `{`):
+
+```json
+{"name": "exec", "arguments": "\"cmd\": \"find /snap/zaproxy/current -type f -name '*.sh' | head -10\"}"}
+```
+
+`JSON.parse` threw, the catch block silently used `toolArgs = {}`, and the exec tool failed with:
+
+```
+{"status":"error","exitCode":"ERR_INVALID_ARG_TYPE","stdout":"","stderr":"The \"command\" argument must be of type string. Received undefined"}
+```
+
+The model saw `ERR_INVALID_ARG_TYPE` / "Received undefined" — no indication that the JSON formatting was wrong. The stderr nudge also fired (Issue 1 compounding: `err.message` put in stderr).
+
+### Root cause
+
+In `src/server/agent.js`:
+
+```js
+try {
+  toolArgs = JSON.parse(toolCall.function.arguments || '{}');
+} catch {
+  toolArgs = {};
+}
+```
+
+The JSON parse error is swallowed. The tool is called with empty args. The resulting type error is cryptic and doesn't tell the model to fix its JSON.
+
+### Fix
+
+Detect the parse failure early, push an explicit error tool result, and skip execution:
+
+```js
+let toolArgs;
+let argParseError = null;
+try {
+  toolArgs = JSON.parse(toolCall.function.arguments || '{}');
+} catch (e) {
+  argParseError = e;
+}
+
+if (argParseError) {
+  const errorContent = JSON.stringify({
+    status: 'error',
+    error: `Tool arguments could not be parsed as JSON: ${argParseError.message}. Ensure arguments are a valid JSON object, e.g. {"key": "value"}.`,
+  });
+  session.messages.push({ role: 'tool', tool_call_id: toolCall.id, content: errorContent });
+  runToolCalls.push({ name: toolName, args: {}, status: 'error', result: errorContent });
+  consecutiveFailures++;
+  continue;
+}
+```
+
+The model immediately sees "Tool arguments could not be parsed as JSON" instead of an opaque `ERR_INVALID_ARG_TYPE`. It can fix its JSON and retry.
+
+**File**: `src/server/agent.js` — inner tool execution loop
+
+---
+
+## Issue 3: Accumulated failedApproaches Cleared on New User Message
+
+### What happened
+
+In multi-run sessions where multiple checkpoint handoffs accumulate `session.metadata.failedApproaches`, the next user message resets this list to `[]`:
+
+```js
+session.metadata.handoffCount = 0;
+session.metadata.failedApproaches = [];
+```
+
+This was designed to give the model a clean slate after human review. But "Ok do it" is not a review — it's a continuation. The model loses knowledge of what was already tried and can repeat the same failed strategies in the new round of runs.
+
+(Note: in this specific session run 1 ended with `ok`, so `failedApproaches` was empty at reset time anyway. But in sessions where checkpoint runs accumulate a list, the reset discards it entirely.)
+
+### Fix
+
+Embed accumulated `failedApproaches` into the incoming user message before resetting:
+
+```js
+let userMessageWithContext = userMessage;
+if (session.metadata.failedApproaches && session.metadata.failedApproaches.length > 0) {
+  userMessageWithContext += `\n\n[System: The following approaches were tried and failed in previous runs — consider them exhausted:\n${session.metadata.failedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
+}
+session.messages.push({ role: 'user', content: userMessageWithContext });
+session.metadata.handoffCount = 0;
+session.metadata.failedApproaches = [];
+```
+
+The model enters the new round with awareness of what has already been exhausted. If the user's message implies a fresh task, the model can ignore the list; if it's a continuation, it benefits from the context.
+
+**File**: `src/server/agent.js` — `_runHandleChat`, before user message push
+
+---
+
+## Issue 4: WRAP_UP_NOTE Did Not Require Verified Progress Claims
+
+### What happened
+
+Runs 2 and 3 claimed to have created project directories and files in the `progress` field of their checkpoints — but their tool calls contained no file creation commands. The model fabricated progress, causing subsequent resume messages to start from false premises.
+
+### Fix
+
+Updated the `progress` field description in `WRAP_UP_NOTE`:
+
+```
+// Before:
+"progress": "What has been fully completed so far.",
+
+// After:
+"progress": "What has been fully completed — only include items confirmed by tool output (e.g., successful exec with exit code 0, or verified by ls/cat). Do not report planned steps as completed.",
+```
+
+**File**: `src/server/agent.js` — `WRAP_UP_NOTE` constant
+
+---
+
+## Secondary Observations (Not Fixed)
+
+**zap-baseline.sh does not exist in snap ZAP**: This script is part of the ZAP Docker image. The snap installation provides `zaproxy -cmd -quickurl <url> -quickout <file>` instead, which is visible in `zaproxy -help` output. The agent saw this output but never connected it to the need for a baseline scan. This is a model knowledge gap.
+
+**Zero-progress detection correctly did not fire**: Runs 2 and 3 had genuinely different `remaining` strings (run 3 claimed partial progress). The detection works as designed; the circumvention was through model hallucination of progress, not a code bug.
+
+**failedApproaches from `ok`-status runs are not captured**: Run 1 ended with `status: ok` despite having 10 iterations of failed searches. The `failedApproaches` mechanism only captures failures from `checkpoint_reached` runs. Capturing failures from `ok`-status runs would require the model to include a `failedApproaches` field in its final response — a more significant protocol change left for a future finding.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/server/tools.js` | exec seed tool: `e.stderr \|\| ''` instead of `e.stderr \|\| e.message` |
+| `src/server/agent.js` | Malformed JSON args: inject error tool result instead of silent `{}` |
+| `src/server/agent.js` | Preserve failedApproaches in user message before resetting |
+| `src/server/agent.js` | Strengthen WRAP_UP_NOTE `progress` field description |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.29",
+  "version": "1.0.31",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -18,7 +18,7 @@ Respond with your normal JSON, but add a checkpoint field:
   "response": "Brief message to the user that the task is still in progress.",
   "logSummary": "Human-readable summary of what happened in this run.",
   "checkpoint": {
-    "progress": "What has been fully completed so far.",
+    "progress": "What has been fully completed — only include items confirmed by tool output (e.g., successful exec with exit code 0, or verified by ls/cat). Do not report planned steps as completed.",
     "remaining": "What still needs to be done to finish the task — as a plain text string, never an array or object.",
     "failedApproaches": ["Concise description of each approach that was tried and failed, e.g. 'downloading subfinder via curl from GitHub releases — connection reset'. Omit array entries for things that succeeded. Leave as empty array if nothing failed."]
   }
@@ -139,13 +139,26 @@ async function runAgentLoop(client, config, session, prepareMessages) {
       });
 
       let toolsModified = false;
+      let stderrErrorInIteration = false;
       for (const toolCall of assistantMessage.tool_calls) {
         const toolName = toolCall.function.name;
         let toolArgs;
+        let argParseError = null;
         try {
           toolArgs = JSON.parse(toolCall.function.arguments || '{}');
-        } catch {
-          toolArgs = {};
+        } catch (e) {
+          argParseError = e;
+        }
+
+        if (argParseError) {
+          const errorContent = JSON.stringify({
+            status: 'error',
+            error: `Tool arguments could not be parsed as JSON: ${argParseError.message}. Ensure arguments are a valid JSON object, e.g. {"key": "value"}.`,
+          });
+          session.messages.push({ role: 'tool', tool_call_id: toolCall.id, content: errorContent });
+          runToolCalls.push({ name: toolName, args: {}, status: 'error', result: errorContent });
+          consecutiveFailures++;
+          continue;
         }
 
         let result;
@@ -165,6 +178,9 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         const toolFailed = toolStatus === 'error' || (resultObj && resultObj.status === 'error');
         if (toolFailed) {
           consecutiveFailures++;
+          if (resultObj && resultObj.stderr) {
+            stderrErrorInIteration = true;
+          }
         } else {
           consecutiveFailures = 0;
         }
@@ -173,7 +189,7 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         runToolCalls.push({ name: toolName, args: toolArgs, status: toolStatus, result: resultStr });
 
         const sessionContent = resultStr.length > MAX_TOOL_RESULT
-          ? resultStr.slice(0, MAX_TOOL_RESULT) + '\n[...truncated]'
+          ? resultStr.slice(0, 2000) + `\n[...${resultStr.length - 4000} chars truncated...]\n` + resultStr.slice(-2000)
           : resultStr;
         session.messages.push({
           role: 'tool',
@@ -201,6 +217,13 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         });
       }
 
+      if (stderrErrorInIteration && !loopDetected) {
+        session.messages.push({
+          role: 'user',
+          content: '[System: A command failed and produced stderr output. Examine the stderr field in the tool result carefully — it likely describes the root cause of the failure. Do not retry the same command without first addressing what stderr reports.]',
+        });
+      }
+
       // Reload tools if any were created/updated this iteration
       if (toolsModified) {
         tools = await loadTools();
@@ -427,8 +450,15 @@ async function _runHandleChat(config, sessionId, userMessage) {
     session = createSession(systemPromptTemplate);
   }
 
+  // Preserve accumulated failedApproaches in conversation history before resetting
+  // so the model retains knowledge of what failed in the previous batch of handoff runs.
+  let userMessageWithContext = userMessage;
+  if (session.metadata.failedApproaches && session.metadata.failedApproaches.length > 0) {
+    userMessageWithContext += `\n\n[System: The following approaches were tried and failed in previous runs — consider them exhausted:\n${session.metadata.failedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
+  }
+
   // Append user message and reset handoff state
-  session.messages.push({ role: 'user', content: userMessage });
+  session.messages.push({ role: 'user', content: userMessageWithContext });
   session.metadata.handoffCount = 0;
   session.metadata.failedApproaches = [];
 

package/src/server/tools.js

@@ -73,7 +73,7 @@ const SEED_TOOLS = {
         });
         return { status: "ok", exitCode: 0, stdout, stderr };
       } catch (e) {
-        return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+        return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || "" };
       }
     `,
   },