@ducci/jarvis 1.0.30 → 1.0.31

package/docs/findings/014-exec-stderr-artifact-and-malformed-tool-args.md

@@ -0,0 +1,202 @@
+# Finding 014: exec stderr Artifact and Malformed Tool Call Arguments
+
+**Date:** 2026-03-02
+**Severity:** Medium — caused spurious context noise (spurious nudges), agent confusion on malformed args, and silent loss of failedApproaches across user turns
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `d97070f7-e50f-4d9e-a38b-b68e7b27e7b7`. User asked Jarvis to set up an OWASP ZAP web security scanning project. The session ran 4 runs (40 total iterations) without completing the task. The snap-installed ZAP does not include `zap-baseline.sh` (which only ships in the ZAP Docker image); the agent never discovered that `zaproxy -cmd -quickurl` is the correct snap-native CLI equivalent.
+
+Three compounding issues were identified that degraded the agent's ability to self-correct.
+
+---
+
+## Issue 1: exec Tool Injects Node.js Error Message into `stderr` Field
+
+### What happened
+
+Commands like `which zap-cli` (not installed), `grep` returning no matches, and piped `find | grep` with no results all return exit code 1. In each case the actual process wrote nothing to stderr. But the exec tool result showed:
+
+```json
+{"status":"error","exitCode":1,"stdout":"","stderr":"Command failed: which zap-cli\n"}
+```
+
+The `"Command failed: ..."` string is not from the process — it is `e.message` from Node.js's `execAsync` error object, injected via `e.stderr || e.message`.
+
+The stderr nudge check in `agent.js` fires on any non-empty `resultObj.stderr`:
+
+```js
+if (resultObj && resultObj.stderr) {
+  stderrErrorInIteration = true;
+}
+```
+
+This triggered the nudge "Examine the stderr field carefully — it likely describes the root cause of the failure" 3–4 times per run, when there was nothing actionable to examine in stderr.
+
+### Root cause
+
+In the exec seed tool (`src/server/tools.js`, line 76):
+
+```js
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+```
+
+The `|| e.message` fallback was designed to show something when `e.stderr` is empty. But it conflates process-generated stderr with Node.js meta-messages about the process exiting non-zero.
+
+### Fix
+
+```js
+// Before:
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+
+// After:
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || "" };
+```
+
+`status: error` and `exitCode` already signal failure. The "Command failed: ..." Node.js string is not diagnostic and should not appear in the stderr field.
+
+**File**: `src/server/tools.js` — exec seed tool code (propagates via seedTools() on next server start)
+
+---
+
+## Issue 2: Malformed Tool Call JSON Arguments Silently Swallowed
+
+### What happened
+
+The model sent a tool call with malformed arguments (missing opening `{`):
+
+```json
+{"name": "exec", "arguments": "\"cmd\": \"find /snap/zaproxy/current -type f -name '*.sh' | head -10\"}"}
+```
+
+`JSON.parse` threw, the catch block silently used `toolArgs = {}`, and the exec tool failed with:
+
+```
+{"status":"error","exitCode":"ERR_INVALID_ARG_TYPE","stdout":"","stderr":"The \"command\" argument must be of type string. Received undefined"}
+```
+
+The model saw `ERR_INVALID_ARG_TYPE` / "Received undefined" — no indication that the JSON formatting was wrong. The stderr nudge also fired (Issue 1 compounding: `err.message` put in stderr).
+
+### Root cause
+
+In `src/server/agent.js`:
+
+```js
+try {
+  toolArgs = JSON.parse(toolCall.function.arguments || '{}');
+} catch {
+  toolArgs = {};
+}
+```
+
+The JSON parse error is swallowed. The tool is called with empty args. The resulting type error is cryptic and doesn't tell the model to fix its JSON.
+
+### Fix
+
+Detect the parse failure early, push an explicit error tool result, and skip execution:
+
+```js
+let toolArgs;
+let argParseError = null;
+try {
+  toolArgs = JSON.parse(toolCall.function.arguments || '{}');
+} catch (e) {
+  argParseError = e;
+}
+
+if (argParseError) {
+  const errorContent = JSON.stringify({
+    status: 'error',
+    error: `Tool arguments could not be parsed as JSON: ${argParseError.message}. Ensure arguments are a valid JSON object, e.g. {"key": "value"}.`,
+  });
+  session.messages.push({ role: 'tool', tool_call_id: toolCall.id, content: errorContent });
+  runToolCalls.push({ name: toolName, args: {}, status: 'error', result: errorContent });
+  consecutiveFailures++;
+  continue;
+}
+```
+
+The model immediately sees "Tool arguments could not be parsed as JSON" instead of an opaque `ERR_INVALID_ARG_TYPE`. It can fix its JSON and retry.
+
+**File**: `src/server/agent.js` — inner tool execution loop
+
+---
+
+## Issue 3: Accumulated failedApproaches Cleared on New User Message
+
+### What happened
+
+In multi-run sessions where multiple checkpoint handoffs accumulate `session.metadata.failedApproaches`, the next user message resets this list to `[]`:
+
+```js
+session.metadata.handoffCount = 0;
+session.metadata.failedApproaches = [];
+```
+
+This was designed to give the model a clean slate after human review. But "Ok do it" is not a review — it's a continuation. The model loses knowledge of what was already tried and can repeat the same failed strategies in the new round of runs.
+
+(Note: in this specific session run 1 ended with `ok`, so `failedApproaches` was empty at reset time anyway. But in sessions where checkpoint runs accumulate a list, the reset discards it entirely.)
+
+### Fix
+
+Embed accumulated `failedApproaches` into the incoming user message before resetting:
+
+```js
+let userMessageWithContext = userMessage;
+if (session.metadata.failedApproaches && session.metadata.failedApproaches.length > 0) {
+  userMessageWithContext += `\n\n[System: The following approaches were tried and failed in previous runs — consider them exhausted:\n${session.metadata.failedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
+}
+session.messages.push({ role: 'user', content: userMessageWithContext });
+session.metadata.handoffCount = 0;
+session.metadata.failedApproaches = [];
+```
+
+The model enters the new round with awareness of what has already been exhausted. If the user's message implies a fresh task, the model can ignore the list; if it's a continuation, it benefits from the context.
+
+**File**: `src/server/agent.js` — `_runHandleChat`, before user message push
+
+---
+
+## Issue 4: WRAP_UP_NOTE Did Not Require Verified Progress Claims
+
+### What happened
+
+Runs 2 and 3 claimed to have created project directories and files in the `progress` field of their checkpoints — but their tool calls contained no file creation commands. The model fabricated progress, causing subsequent resume messages to start from false premises.
+
+### Fix
+
+Updated the `progress` field description in `WRAP_UP_NOTE`:
+
+```
+// Before:
+"progress": "What has been fully completed so far.",
+
+// After:
+"progress": "What has been fully completed — only include items confirmed by tool output (e.g., successful exec with exit code 0, or verified by ls/cat). Do not report planned steps as completed.",
+```
+
+**File**: `src/server/agent.js` — `WRAP_UP_NOTE` constant
+
+---
+
+## Secondary Observations (Not Fixed)
+
+**zap-baseline.sh does not exist in snap ZAP**: This script is part of the ZAP Docker image. The snap installation provides `zaproxy -cmd -quickurl <url> -quickout <file>` instead, which is visible in `zaproxy -help` output. The agent saw this output but never connected it to the need for a baseline scan. This is a model knowledge gap.
+
+**Zero-progress detection correctly did not fire**: Runs 2 and 3 had genuinely different `remaining` strings (run 3 claimed partial progress). The detection works as designed; the circumvention was through model hallucination of progress, not a code bug.
+
+**failedApproaches from `ok`-status runs are not captured**: Run 1 ended with `status: ok` despite having 10 iterations of failed searches. The `failedApproaches` mechanism only captures failures from `checkpoint_reached` runs. Capturing failures from `ok`-status runs would require the model to include a `failedApproaches` field in its final response — a more significant protocol change left for a future finding.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/server/tools.js` | exec seed tool: `e.stderr \|\| ''` instead of `e.stderr \|\| e.message` |
+| `src/server/agent.js` | Malformed JSON args: inject error tool result instead of silent `{}` |
+| `src/server/agent.js` | Preserve failedApproaches in user message before resetting |
+| `src/server/agent.js` | Strengthen WRAP_UP_NOTE `progress` field description |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.30",
+  "version": "1.0.31",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -18,7 +18,7 @@ Respond with your normal JSON, but add a checkpoint field:
   "response": "Brief message to the user that the task is still in progress.",
   "logSummary": "Human-readable summary of what happened in this run.",
   "checkpoint": {
-    "progress": "What has been fully completed so far.",
+    "progress": "What has been fully completed — only include items confirmed by tool output (e.g., successful exec with exit code 0, or verified by ls/cat). Do not report planned steps as completed.",
     "remaining": "What still needs to be done to finish the task — as a plain text string, never an array or object.",
     "failedApproaches": ["Concise description of each approach that was tried and failed, e.g. 'downloading subfinder via curl from GitHub releases — connection reset'. Omit array entries for things that succeeded. Leave as empty array if nothing failed."]
   }
@@ -143,10 +143,22 @@ async function runAgentLoop(client, config, session, prepareMessages) {
       for (const toolCall of assistantMessage.tool_calls) {
         const toolName = toolCall.function.name;
         let toolArgs;
+        let argParseError = null;
         try {
           toolArgs = JSON.parse(toolCall.function.arguments || '{}');
-        } catch {
-          toolArgs = {};
+        } catch (e) {
+          argParseError = e;
+        }
+
+        if (argParseError) {
+          const errorContent = JSON.stringify({
+            status: 'error',
+            error: `Tool arguments could not be parsed as JSON: ${argParseError.message}. Ensure arguments are a valid JSON object, e.g. {"key": "value"}.`,
+          });
+          session.messages.push({ role: 'tool', tool_call_id: toolCall.id, content: errorContent });
+          runToolCalls.push({ name: toolName, args: {}, status: 'error', result: errorContent });
+          consecutiveFailures++;
+          continue;
         }
 
         let result;
@@ -438,8 +450,15 @@ async function _runHandleChat(config, sessionId, userMessage) {
     session = createSession(systemPromptTemplate);
   }
 
+  // Preserve accumulated failedApproaches in conversation history before resetting
+  // so the model retains knowledge of what failed in the previous batch of handoff runs.
+  let userMessageWithContext = userMessage;
+  if (session.metadata.failedApproaches && session.metadata.failedApproaches.length > 0) {
+    userMessageWithContext += `\n\n[System: The following approaches were tried and failed in previous runs — consider them exhausted:\n${session.metadata.failedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
+  }
+
   // Append user message and reset handoff state
-  session.messages.push({ role: 'user', content: userMessage });
+  session.messages.push({ role: 'user', content: userMessageWithContext });
   session.metadata.handoffCount = 0;
   session.metadata.failedApproaches = [];
 

package/src/server/tools.js

@@ -73,7 +73,7 @@ const SEED_TOOLS = {
         });
         return { status: "ok", exitCode: 0, stdout, stderr };
       } catch (e) {
-        return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+        return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || "" };
       }
     `,
   },