@ducci/jarvis 1.0.30 → 1.0.32

package/docs/findings/014-exec-stderr-artifact-and-malformed-tool-args.md

@@ -0,0 +1,202 @@
+# Finding 014: exec stderr Artifact and Malformed Tool Call Arguments
+
+**Date:** 2026-03-02
+**Severity:** Medium — caused spurious context noise (spurious nudges), agent confusion on malformed args, and silent loss of failedApproaches across user turns
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `d97070f7-e50f-4d9e-a38b-b68e7b27e7b7`. User asked Jarvis to set up an OWASP ZAP web security scanning project. The session ran 4 runs (40 total iterations) without completing the task. The snap-installed ZAP does not include `zap-baseline.sh` (which only ships in the ZAP Docker image); the agent never discovered that `zaproxy -cmd -quickurl` is the correct snap-native CLI equivalent.
+
+Three compounding issues were identified that degraded the agent's ability to self-correct.
+
+---
+
+## Issue 1: exec Tool Injects Node.js Error Message into `stderr` Field
+
+### What happened
+
+Commands like `which zap-cli` (not installed), `grep` returning no matches, and piped `find | grep` with no results all return exit code 1. In each case the actual process wrote nothing to stderr. But the exec tool result showed:
+
+```json
+{"status":"error","exitCode":1,"stdout":"","stderr":"Command failed: which zap-cli\n"}
+```
+
+The `"Command failed: ..."` string is not from the process — it is `e.message` from Node.js's `execAsync` error object, injected via `e.stderr || e.message`.
+
+The stderr nudge check in `agent.js` fires on any non-empty `resultObj.stderr`:
+
+```js
+if (resultObj && resultObj.stderr) {
+  stderrErrorInIteration = true;
+}
+```
+
+This triggered the nudge "Examine the stderr field carefully — it likely describes the root cause of the failure" 3–4 times per run, when there was nothing actionable to examine in stderr.
+
+### Root cause
+
+In the exec seed tool (`src/server/tools.js`, line 76):
+
+```js
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+```
+
+The `|| e.message` fallback was designed to show something when `e.stderr` is empty. But it conflates process-generated stderr with Node.js meta-messages about the process exiting non-zero.
+
+### Fix
+
+```js
+// Before:
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+
+// After:
+return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || "" };
+```
+
+`status: error` and `exitCode` already signal failure. The "Command failed: ..." Node.js string is not diagnostic and should not appear in the stderr field.
+
+**File**: `src/server/tools.js` — exec seed tool code (propagates via seedTools() on next server start)
+
+---
+
+## Issue 2: Malformed Tool Call JSON Arguments Silently Swallowed
+
+### What happened
+
+The model sent a tool call with malformed arguments (missing opening `{`):
+
+```json
+{"name": "exec", "arguments": "\"cmd\": \"find /snap/zaproxy/current -type f -name '*.sh' | head -10\"}"}
+```
+
+`JSON.parse` threw, the catch block silently used `toolArgs = {}`, and the exec tool failed with:
+
+```
+{"status":"error","exitCode":"ERR_INVALID_ARG_TYPE","stdout":"","stderr":"The \"command\" argument must be of type string. Received undefined"}
+```
+
+The model saw `ERR_INVALID_ARG_TYPE` / "Received undefined" — no indication that the JSON formatting was wrong. The stderr nudge also fired (Issue 1 compounding: `err.message` put in stderr).
+
+### Root cause
+
+In `src/server/agent.js`:
+
+```js
+try {
+  toolArgs = JSON.parse(toolCall.function.arguments || '{}');
+} catch {
+  toolArgs = {};
+}
+```
+
+The JSON parse error is swallowed. The tool is called with empty args. The resulting type error is cryptic and doesn't tell the model to fix its JSON.
+
+### Fix
+
+Detect the parse failure early, push an explicit error tool result, and skip execution:
+
+```js
+let toolArgs;
+let argParseError = null;
+try {
+  toolArgs = JSON.parse(toolCall.function.arguments || '{}');
+} catch (e) {
+  argParseError = e;
+}
+
+if (argParseError) {
+  const errorContent = JSON.stringify({
+    status: 'error',
+    error: `Tool arguments could not be parsed as JSON: ${argParseError.message}. Ensure arguments are a valid JSON object, e.g. {"key": "value"}.`,
+  });
+  session.messages.push({ role: 'tool', tool_call_id: toolCall.id, content: errorContent });
+  runToolCalls.push({ name: toolName, args: {}, status: 'error', result: errorContent });
+  consecutiveFailures++;
+  continue;
+}
+```
+
+The model immediately sees "Tool arguments could not be parsed as JSON" instead of an opaque `ERR_INVALID_ARG_TYPE`. It can fix its JSON and retry.
+
+**File**: `src/server/agent.js` — inner tool execution loop
+
+---
+
+## Issue 3: Accumulated failedApproaches Cleared on New User Message
+
+### What happened
+
+In multi-run sessions where multiple checkpoint handoffs accumulate `session.metadata.failedApproaches`, the next user message resets this list to `[]`:
+
+```js
+session.metadata.handoffCount = 0;
+session.metadata.failedApproaches = [];
+```
+
+This was designed to give the model a clean slate after human review. But "Ok do it" is not a review — it's a continuation. The model loses knowledge of what was already tried and can repeat the same failed strategies in the new round of runs.
+
+(Note: in this specific session run 1 ended with `ok`, so `failedApproaches` was empty at reset time anyway. But in sessions where checkpoint runs accumulate a list, the reset discards it entirely.)
+
+### Fix
+
+Embed accumulated `failedApproaches` into the incoming user message before resetting:
+
+```js
+let userMessageWithContext = userMessage;
+if (session.metadata.failedApproaches && session.metadata.failedApproaches.length > 0) {
+  userMessageWithContext += `\n\n[System: The following approaches were tried and failed in previous runs — consider them exhausted:\n${session.metadata.failedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
+}
+session.messages.push({ role: 'user', content: userMessageWithContext });
+session.metadata.handoffCount = 0;
+session.metadata.failedApproaches = [];
+```
+
+The model enters the new round with awareness of what has already been exhausted. If the user's message implies a fresh task, the model can ignore the list; if it's a continuation, it benefits from the context.
+
+**File**: `src/server/agent.js` — `_runHandleChat`, before user message push
+
+---
+
+## Issue 4: WRAP_UP_NOTE Did Not Require Verified Progress Claims
+
+### What happened
+
+Runs 2 and 3 claimed to have created project directories and files in the `progress` field of their checkpoints — but their tool calls contained no file creation commands. The model fabricated progress, causing subsequent resume messages to start from false premises.
+
+### Fix
+
+Updated the `progress` field description in `WRAP_UP_NOTE`:
+
+```
+// Before:
+"progress": "What has been fully completed so far.",
+
+// After:
+"progress": "What has been fully completed — only include items confirmed by tool output (e.g., successful exec with exit code 0, or verified by ls/cat). Do not report planned steps as completed.",
+```
+
+**File**: `src/server/agent.js` — `WRAP_UP_NOTE` constant
+
+---
+
+## Secondary Observations (Not Fixed)
+
+**zap-baseline.sh does not exist in snap ZAP**: This script is part of the ZAP Docker image. The snap installation provides `zaproxy -cmd -quickurl <url> -quickout <file>` instead, which is visible in `zaproxy -help` output. The agent saw this output but never connected it to the need for a baseline scan. This is a model knowledge gap.
+
+**Zero-progress detection correctly did not fire**: Runs 2 and 3 had genuinely different `remaining` strings (run 3 claimed partial progress). The detection works as designed; the circumvention was through model hallucination of progress, not a code bug.
+
+**failedApproaches from `ok`-status runs are not captured**: Run 1 ended with `status: ok` despite having 10 iterations of failed searches. The `failedApproaches` mechanism only captures failures from `checkpoint_reached` runs. Capturing failures from `ok`-status runs would require the model to include a `failedApproaches` field in its final response — a more significant protocol change left for a future finding.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/server/tools.js` | exec seed tool: `e.stderr \|\| ''` instead of `e.stderr \|\| e.message` |
+| `src/server/agent.js` | Malformed JSON args: inject error tool result instead of silent `{}` |
+| `src/server/agent.js` | Preserve failedApproaches in user message before resetting |
+| `src/server/agent.js` | Strengthen WRAP_UP_NOTE `progress` field description |

package/docs/findings/015-failed-run-context-strip.md

@@ -0,0 +1,142 @@
+# Finding 015: Failed Runs Leave Tool History in Context (Context Bloat Death Spiral)
+
+**Date:** 2026-03-02
+**Severity:** High — caused 3 consecutive `model_error: Empty choices array` failures; session unusable
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `6123209d-ce5a-44d0-be12-29aac58b4cf3`. Model: `nvidia/nemotron-3-nano-30b-a3b:free`. User requested a ZAP security scanning project.
+
+| Entry | Trigger | Status | messageCount at failure | toolCalls |
+|-------|---------|--------|------------------------|-----------|
+| 1 | "hi all good?" | ok | — | 0 |
+| 2 | ZAP task (run 1) | checkpoint_reached | — | 10 |
+| 3 | handoff resume (run 2) | checkpoint_reached | — | 10 |
+| 4 | handoff resume (run 3) | model_error (empty choices, iter 7) | 22 | 26 |
+| 5 | "Why I get Model returned an empty response?" | model_error (empty choices, iter 3) | 27 | 2 |
+| 6 | "Why I get Model returned an empty response again?!!" | model_error (empty choices, iter 5) | 37 | 4 |
+
+The session ended without producing any result. The user received `'Model returned an empty response.'` three times.
+
+---
+
+## Root Cause 1: Failed runs leave tool call history in session
+
+### What happened
+
+The handoff loop strips tool call messages for `checkpoint_reached` runs:
+
+```js
+session.messages.splice(runStartIndex, session.messages.length - runStartIndex - 1);
+```
+
+Runs that end with `model_error` or `format_error` received **no strip**. Every tool call message (assistant+tool pair, nudge injections) from the failed run remained in `session.messages`, with only a synthetic error note appended afterward.
+
+Run 3 had 26 tool calls across 7 iterations — approximately 13 messages added to the session. These were preserved verbatim. Each subsequent user turn started with more context than the last.
+
+### Message count growth
+
+- Before run 3: ~8 messages (runs 1 and 2 were both checkpoint_reached and stripped correctly)
+- After entry 4 (model_error, no strip): 21 messages + synthetic note = 22
+- After entry 5 (model_error, no strip): 27 messages + synthetic note = 28
+- At entry 6: 37 messages in context
+
+The free model returns `choices: []` when the context exceeds what it can handle. Each failure added more context, making the next failure more likely: a **positive feedback death spiral**.
+
+### Fix
+
+Apply the same splice that checkpoint runs already use:
+
+```js
+if (finalStatus === 'model_error' || finalStatus === 'format_error') {
+  session.messages.splice(runStartIndex, session.messages.length - runStartIndex);
+  // then push synthetic error note as before
+}
+```
+
+The strip runs before the synthetic error note is pushed, returning the session to its pre-run state plus one concise note. The JSONL log preserves all tool results for retrospective inspection via `read_session_log`.
+
+**File**: `src/server/agent.js` — `_runHandleChat`, non-checkpoint break path
+
+---
+
+## Root Cause 2: No detection or escalation for consecutive model_errors
+
+### What happened
+
+After two consecutive `model_error: Empty choices array` entries (4 and 5), no protective mechanism fired. The system continued accepting new user messages and spawning new runs indefinitely.
+
+Existing protection mechanisms all missed this case:
+- `maxHandoffs` — only applies to `checkpoint_reached` runs
+- `consecutiveFailures` — tracks tool failures within a single run
+- Zero-progress detection — only applies to `checkpoint_reached` runs
+
+### Fix
+
+Detect the pattern structurally in `session.messages` before starting each new run: if the last two assistant messages are both synthetic `model_error` notes, the session is in a confirmed failure loop. Escalate to `intervention_required` without running another agent loop.
+
+```js
+function hasConsecutiveModelErrors(messages) {
+  const assistantTail = messages.filter(m => m.role === 'assistant').slice(-2);
+  return (
+    assistantTail.length === 2 &&
+    assistantTail.every(
+      m =>
+        typeof m.content === 'string' &&
+        m.content.startsWith('[System: Previous run failed (model_error)')
+    )
+  );
+}
+```
+
+This uses no additional state: it reads the session history directly. Old sessions are handled correctly. One failure is allowed (transient errors are real); two consecutive failures mean the session cannot self-recover.
+
+Combined with Fix 1, consecutive model_errors in this session would have played out as:
+1. Entry 4 (run 3): model_error → strip → synthetic note. Session back to 9 messages.
+2. Entry 5 (user "Why?"): run 4 starts with 10 messages. If it still fails → strip → synthetic note. Two model_error notes now in session.
+3. Entry 6 (user "Why again?!"): `hasConsecutiveModelErrors` fires → `intervention_required` returned immediately. User gets a clear message: start a new session or switch model.
+
+**File**: `src/server/agent.js` — `hasConsecutiveModelErrors` function + check at top of handoff loop
+
+---
+
+## Root Cause 3: Empty choices error message provides no actionable guidance
+
+### What happened
+
+The `choices.length === 0` path returned:
+
+```
+Model returned an empty response.
+```
+
+When the user asked "why?", the agent — with ZAP tool call context still present — continued ZAP investigation instead of explaining the API failure. The opaque error and the polluted context compounded: the model had no clear signal about what went wrong and no guidance on how to recover.
+
+### Fix
+
+Include the context size and recovery guidance in the response:
+
+```js
+response: `Model returned an empty response (${preparedMessages.length} messages in context). This typically happens when the conversation is too long for the model. Try starting a new session or switching to a model with a larger context window.`,
+```
+
+**File**: `src/server/agent.js` — `runAgentLoop`, empty choices early return
+
+---
+
+## Why Fix 1 is Primary
+
+Fix 1 is the root fix. With context stripped after each failure, the model operates on a tiny session (~10 messages) on subsequent turns. The free model handles this easily. Fix 2 is a safety net for persistent non-context failures. Fix 3 improves user-facing error messages for the residual cases that slip through.
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/server/agent.js` | Strip tool history on `model_error`/`format_error` (same as checkpoint) |
+| `src/server/agent.js` | `hasConsecutiveModelErrors` function + check before each run in handoff loop |
+| `src/server/agent.js` | Include message count in empty choices response |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.30",
+  "version": "1.0.32",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -18,7 +18,7 @@ Respond with your normal JSON, but add a checkpoint field:
   "response": "Brief message to the user that the task is still in progress.",
   "logSummary": "Human-readable summary of what happened in this run.",
   "checkpoint": {
-    "progress": "What has been fully completed so far.",
+    "progress": "What has been fully completed — only include items confirmed by tool output (e.g., successful exec with exit code 0, or verified by ls/cat). Do not report planned steps as completed.",
     "remaining": "What still needs to be done to finish the task — as a plain text string, never an array or object.",
     "failedApproaches": ["Concise description of each approach that was tried and failed, e.g. 'downloading subfinder via curl from GitHub releases — connection reset'. Omit array entries for things that succeeded. Leave as empty array if nothing failed."]
   }
@@ -69,6 +69,25 @@ async function callModelWithFallback(client, config, messages, tools) {
   }
 }
 
+/**
+ * Returns true if the last two assistant messages in the session are both
+ * synthetic model_error notes, indicating a confirmed failure loop that cannot
+ * self-resolve (e.g. persistent empty choices from context overflow).
+ */
+function hasConsecutiveModelErrors(messages) {
+  const assistantTail = messages
+    .filter(m => m.role === 'assistant')
+    .slice(-2);
+  return (
+    assistantTail.length === 2 &&
+    assistantTail.every(
+      m =>
+        typeof m.content === 'string' &&
+        m.content.startsWith('[System: Previous run failed (model_error)')
+    )
+  );
+}
+
 /**
  * Runs a single agent loop up to maxIterations.
  * Returns { iteration, response, logSummary, status, runToolCalls, checkpoint }.
@@ -112,7 +131,7 @@ async function runAgentLoop(client, config, session, prepareMessages) {
     if (!modelResult.choices || modelResult.choices.length === 0) {
       return {
         iteration,
-        response: 'Model returned an empty response.',
+        response: `Model returned an empty response (${preparedMessages.length} messages in context). This typically happens when the conversation is too long for the model. Try starting a new session or switching to a model with a larger context window.`,
         logSummary: `Model error on iteration ${iteration}: Empty choices array.`,
         status: 'model_error',
         runToolCalls,
@@ -143,10 +162,22 @@ async function runAgentLoop(client, config, session, prepareMessages) {
       for (const toolCall of assistantMessage.tool_calls) {
         const toolName = toolCall.function.name;
         let toolArgs;
+        let argParseError = null;
         try {
           toolArgs = JSON.parse(toolCall.function.arguments || '{}');
-        } catch {
-          toolArgs = {};
+        } catch (e) {
+          argParseError = e;
+        }
+
+        if (argParseError) {
+          const errorContent = JSON.stringify({
+            status: 'error',
+            error: `Tool arguments could not be parsed as JSON: ${argParseError.message}. Ensure arguments are a valid JSON object, e.g. {"key": "value"}.`,
+          });
+          session.messages.push({ role: 'tool', tool_call_id: toolCall.id, content: errorContent });
+          runToolCalls.push({ name: toolName, args: {}, status: 'error', result: errorContent });
+          consecutiveFailures++;
+          continue;
         }
 
         let result;
@@ -438,8 +469,15 @@ async function _runHandleChat(config, sessionId, userMessage) {
     session = createSession(systemPromptTemplate);
   }
 
+  // Preserve accumulated failedApproaches in conversation history before resetting
+  // so the model retains knowledge of what failed in the previous batch of handoff runs.
+  let userMessageWithContext = userMessage;
+  if (session.metadata.failedApproaches && session.metadata.failedApproaches.length > 0) {
+    userMessageWithContext += `\n\n[System: The following approaches were tried and failed in previous runs — consider them exhausted:\n${session.metadata.failedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
+  }
+
   // Append user message and reset handoff state
-  session.messages.push({ role: 'user', content: userMessage });
+  session.messages.push({ role: 'user', content: userMessageWithContext });
   session.metadata.handoffCount = 0;
   session.metadata.failedApproaches = [];
 
@@ -463,6 +501,25 @@ async function _runHandleChat(config, sessionId, userMessage) {
   try {
     // Handoff loop
     while (true) {
+      // Safety check: if the last two assistant messages are both model_error
+      // synthetic notes, we are in a confirmed failure loop. Escalate immediately
+      // rather than burning more iterations on a stuck session.
+      if (hasConsecutiveModelErrors(session.messages)) {
+        finalResponse = 'The model has failed twice in a row. This is likely due to the conversation being too long for the model to process. Please start a new session or switch to a model with a larger context window.';
+        finalLogSummary = 'Consecutive model_error detected: session escalated to intervention_required without running another agent loop.';
+        finalStatus = 'intervention_required';
+        await appendLog(sessionId, {
+          iteration: 0,
+          model: config.selectedModel,
+          userInput: userMessage,
+          toolCalls: [],
+          response: finalResponse,
+          logSummary: finalLogSummary,
+          status: 'intervention_required',
+        });
+        break;
+      }
+
       const runStartIndex = session.messages.length;
       const run = await runAgentLoop(client, config, session, prepareMessages);
       allToolCalls.push(...run.runToolCalls);
@@ -486,8 +543,14 @@ async function _runHandleChat(config, sessionId, userMessage) {
         if (run.rawResponse) logEntry.rawResponse = run.rawResponse;
         await appendLog(sessionId, logEntry);
 
-        // Inject synthetic error note so the model has context on the next user turn
+        // Inject synthetic error note so the model has context on the next user turn.
+        // For failed runs, also strip the tool call history — keeping it would bloat
+        // the context and create a positive-feedback death spiral where each failure
+        // makes the next one more likely (especially on free models with small context
+        // windows). The synthetic note is sufficient context; tool results are preserved
+        // in the JSONL log and accessible via read_session_log.
         if (finalStatus === 'model_error' || finalStatus === 'format_error') {
+          session.messages.splice(runStartIndex, session.messages.length - runStartIndex);
           const errorDetail = run.errorDetail ? ` Error detail: ${JSON.stringify(run.errorDetail)}` : '';
           session.messages.push({
             role: 'assistant',

package/src/server/tools.js

@@ -73,7 +73,7 @@ const SEED_TOOLS = {
         });
         return { status: "ok", exitCode: 0, stdout, stderr };
       } catch (e) {
-        return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || e.message };
+        return { status: "error", exitCode: e.code || 1, stdout: e.stdout || "", stderr: e.stderr || "" };
       }
     `,
   },