@ducci/jarvis 1.0.28 → 1.0.30

package/docs/findings/012-empty-nudge-loses-recovery-text.md

@@ -0,0 +1,121 @@
+# Finding 012: Empty-Content Nudge Includes Tools and Loses Recovery Text
+
+**Date:** 2026-03-02
+**Severity:** Medium — user sees generic error when model produces a partial recovery response
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `21fb43a7-2b11-4208-99fb-e6b54fddc07b`, entry 9 in session.jsonl:
+
+```
+status=format_error
+model=nvidia/nemotron-3-nano-30b-a3b:free
+iteration=3
+userInput='Ok. Read the results folder. Is there anything?'
+logSummary='Model returned non-JSON final response after recovery attempts.'
+response='The model did not produce a response. Please try again.'
+```
+
+The user received: **"The model did not produce a response. Please try again."**
+
+---
+
+## What Happened
+
+1. The agent executed two tool calls:
+   - `list_dir /root/.jarvis/projects/cybersecurity/results` → success
+   - `exec "list_dir /root/.jarvis/projects/cybersecurity/results/dviet.de"` → exit 127 (`list_dir: not found`)
+     - The model confused the `list_dir` jarvis tool with a shell command
+
+2. After the failed exec, the model returned `assistantMessage.content = null` with no `tool_calls` — it "went silent"
+
+3. Finding 011's empty-content nudge was triggered
+
+4. The nudge **also failed** — no valid JSON response was produced
+
+5. The agent fell through to `format_error` with the fallback message
+
+---
+
+## Bug Chain
+
+### Bug 1 — toolDefs included in empty nudge
+
+```js
+const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+```
+
+When the model is confused after a tool failure, it may respond to the nudge with **another tool call** instead of text. If it does:
+
+```
+nudgeResult.choices[0].message.content = null
+nudgeContent = ''
+JSON.parse('') → throws
+catch: // Give up — content stays ''
+```
+
+The model had an opportunity to call more tools instead of producing a text response — the wrong behavior for a recovery nudge.
+
+### Bug 2 — content assigned after parse
+
+```js
+const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+parsed = JSON.parse(nudgeContent);   // ← throws on non-JSON or empty
+content = nudgeContent;              // ← only reached if parse succeeded
+```
+
+If the model responds to the nudge with non-empty but non-JSON text (e.g. a plain English answer), `JSON.parse` throws and `content` is **never updated**. The non-JSON text is discarded. The `!parsed` handler then shows the fallback message instead of the model's actual text.
+
+---
+
+## Difference from Finding 011
+
+| Finding | Problem | Trigger |
+|---------|---------|---------|
+| 011 | Empty model response propagates to Telegram | Initial empty content, no recovery chain |
+| 012 | Recovery nudge discards best-effort text; model can respond with tool call | Recovery nudge called with toolDefs + content assigned after parse |
+
+Finding 012 is a refinement of the recovery path introduced in Finding 011.
+
+---
+
+## Fix
+
+### `src/server/agent.js` — empty-content nudge block
+
+**Before:**
+```js
+const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+parsed = JSON.parse(nudgeContent);
+content = nudgeContent;
+```
+
+**After:**
+```js
+// No tools: force text response, prevent model from calling another tool
+const nudgeResult = await callModelWithFallback(client, config, emptyNudge, []);
+const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+// Persist before parsing — if JSON parse throws, content still carries the
+// model's best-effort text so the !parsed handler can show it to the user
+if (nudgeContent.trim()) {
+  content = nudgeContent;
+}
+parsed = JSON.parse(nudgeContent);
+```
+
+---
+
+## Outcome
+
+| Nudge response | Before | After |
+|---|---|---|
+| Valid JSON | Clean recovery | Clean recovery (no change) |
+| Non-JSON text | Text discarded, fallback shown | Text shown to user |
+| Tool call (no content) | content='', fallback shown | Less likely; content='', fallback shown |
+| Empty again | content='', fallback shown | content='', fallback shown (no change) |
+
+The user in the observed session would have received the model's best-effort text about the results folder contents, rather than "The model did not produce a response. Please try again."

package/docs/findings/013-stderr-visibility-and-truncation.md

@@ -0,0 +1,59 @@
+# Finding 013 — stderr Visibility and Output Truncation
+
+## Observed Behaviour
+
+During a multi-run ZAP security scan session, the agent repeatedly failed to diagnose and fix the root cause of scan failures. It issued `pkill`/`kill` variants dozens of times, burned through 6 iteration limits and 2 handoffs, and ultimately gave up without producing results.
+
+Post-mortem analysis of the debug session logs revealed two compounding problems.
+
+## Root Causes
+
+### 1. Head-only truncation buried errors at the end of output
+
+`MAX_TOOL_RESULT = 4000` was applied as a simple head slice: `resultStr.slice(0, 4000)`. ZAP produces verbose startup logs (JVM init, add-on loading, database migration) that easily exceed 4000 characters. Five separate ZAP exec results were truncated exactly at the limit, cutting off during database migration messages. Any errors that appeared later in the output — after the verbose preamble — were silently dropped before the model ever saw them.
+
+### 2. Model ignored stderr even when it was visible
+
+In one un-truncated result (810 chars), the critical error was plainly present:
+
+```
+g_module_open() failed for libpixbufloader-tiff.so: libtiff.so.5: cannot open shared object file: No such file or directory
+```
+
+The model's subsequent response ignored it entirely and concluded only that "no results were found in the output directory." There was no mechanism forcing the model to re-examine stderr before forming its conclusion or retrying.
+
+Similarly, all `pkill`/`kill` commands returned `exitCode: 1` with clear stderr — yet the model continued issuing variations of the same commands without diagnosing why process termination was failing.
+
+## Fixes
+
+### Fix 1 — Head + tail truncation (`agent.js`)
+
+Replace head-only truncation with a head+tail strategy:
+
+```js
+// Before
+resultStr.slice(0, MAX_TOOL_RESULT) + '\n[...truncated]'
+
+// After
+resultStr.slice(0, 2000) + `\n[...${resultStr.length - 4000} chars truncated...]\n` + resultStr.slice(-2000)
+```
+
+The first 2000 chars preserve startup context; the last 2000 chars preserve the diagnostic tail where errors typically appear. The marker in the middle shows how much was dropped. Total budget stays at 4000 chars.
+
+### Fix 2 — Stderr nudge injection (`agent.js`)
+
+After each iteration's tool-call loop, if any tool failed (`status === 'error'`) with non-empty `stderr`, inject a system message:
+
+```
+[System: A command failed and produced stderr output. Examine the stderr field in the tool result carefully — it likely describes the root cause of the failure. Do not retry the same command without first addressing what stderr reports.]
+```
+
+This creates an active forcing function — the model cannot continue to the next iteration without the nudge appearing in its context. The nudge is suppressed if loop detection already fired (to avoid contradictory instructions).
+
+## Known Gap
+
+The stderr nudge only fires when `toolFailed` is true (i.e., `exec` returned `status: 'error'`). Commands that return `exitCode: 0` but still emit meaningful errors to stderr (e.g., a shell script that succeeds but a subprocess inside it fails) will not trigger the nudge. Catching that case without generating noise from normal stderr usage (npm warnings, apt-get progress) requires more context than is available at this level. Documented as a known limitation.
+
+## Files Changed
+
+- `src/server/agent.js` — truncation strategy + stderr nudge injection

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.28",
+  "version": "1.0.30",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -139,6 +139,7 @@ async function runAgentLoop(client, config, session, prepareMessages) {
       });
 
       let toolsModified = false;
+      let stderrErrorInIteration = false;
       for (const toolCall of assistantMessage.tool_calls) {
         const toolName = toolCall.function.name;
         let toolArgs;
@@ -165,6 +166,9 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         const toolFailed = toolStatus === 'error' || (resultObj && resultObj.status === 'error');
         if (toolFailed) {
           consecutiveFailures++;
+          if (resultObj && resultObj.stderr) {
+            stderrErrorInIteration = true;
+          }
         } else {
           consecutiveFailures = 0;
         }
@@ -173,7 +177,7 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         runToolCalls.push({ name: toolName, args: toolArgs, status: toolStatus, result: resultStr });
 
         const sessionContent = resultStr.length > MAX_TOOL_RESULT
-          ? resultStr.slice(0, MAX_TOOL_RESULT) + '\n[...truncated]'
+          ? resultStr.slice(0, 2000) + `\n[...${resultStr.length - 4000} chars truncated...]\n` + resultStr.slice(-2000)
           : resultStr;
         session.messages.push({
           role: 'tool',
@@ -201,6 +205,13 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         });
       }
 
+      if (stderrErrorInIteration && !loopDetected) {
+        session.messages.push({
+          role: 'user',
+          content: '[System: A command failed and produced stderr output. Examine the stderr field in the tool result carefully — it likely describes the root cause of the failure. Do not retry the same command without first addressing what stderr reports.]',
+        });
+      }
+
       // Reload tools if any were created/updated this iteration
       if (toolsModified) {
         tools = await loadTools();
@@ -218,17 +229,24 @@ async function runAgentLoop(client, config, session, prepareMessages) {
     if (!content.trim()) {
       // Model returned no content at all — use a targeted nudge instead of the
       // standard JSON recovery chain (designed for non-empty non-JSON responses).
+      // Send with no tools so the model cannot respond with another tool call,
+      // which would leave content empty and discard any recovery text.
       try {
         const emptyNudge = [
           ...preparedMessages,
           { role: 'user', content: 'You returned an empty response. ' + FORMAT_NUDGE },
         ];
-        const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+        const nudgeResult = await callModelWithFallback(client, config, emptyNudge, []);
         const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+        // Persist nudge text before parsing — if JSON parse throws, content still
+        // carries the model's best-effort text so the !parsed handler can show it
+        // rather than falling back to "The model did not produce a response."
+        if (nudgeContent.trim()) {
+          content = nudgeContent;
+        }
         parsed = JSON.parse(nudgeContent);
-        content = nudgeContent;
       } catch {
-        // Give up — fall through to !parsed handler below
+        // Fall through to !parsed handler; content may now carry the nudge text
       }
     } else {
       try {