@ducci/jarvis 1.0.27 → 1.0.29

package/docs/findings/011-empty-model-response.md

@@ -0,0 +1,157 @@
+# Finding 011: Empty Model Response Causes Generic Telegram Error
+
+**Date:** 2026-03-01
+**Severity:** High — user sees generic "please try again" with no actionable information
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `33a50dfe-38ea-4972-adac-498ef0525b0c`, run 16 of 17 (session.jsonl line 16):
+
+```
+status=format_error
+model=nvidia/nemotron-3-nano-30b-a3b:free
+iteration=4
+userInput='Ok. Kannst du bitte jetzt das shell script ausführen mit der domain...'
+logSummary='Model returned non-JSON final response after recovery attempts.'
+rawResponse=''
+response=''
+```
+
+The Telegram user received:
+
+```
+The agent encountered an error and could not produce a response. Please try again.
+```
+
+---
+
+## What Happened
+
+The agent executed a ZAP scan (`./scan.sh juice-shop.herokuapp.com`). The tool result was a large ZAP startup log, truncated at 4000 characters. Two subsequent tool calls failed:
+
+- `pkill -f zaproxy || true` → exit 1 (no process to kill)
+- `zaproxy -help | grep -i shutdown -A5` → failed (`libtiff.so.5` missing)
+
+On iteration 4, the model returned `assistantMessage.content = null` with no `tool_calls`. This is the "went silent" case: the model produced neither a response nor another tool call.
+
+---
+
+## Bug Chain
+
+### Step 1 — Model returns null content
+
+```js
+let content = assistantMessage.content || '';
+// content = ''
+```
+
+### Step 2 — Recovery chain falls through on empty content
+
+The existing format recovery chain was designed for *non-empty, non-JSON* responses:
+
+1. `JSON.parse('')` → throws
+2. Retry with fallback model (same messages, no nudge) → also `''`
+3. Retry with nudge "Your previous response was not valid JSON" → technically wrong for empty content; model still returns `''`
+4. Give up
+
+### Step 3 — Empty `response` propagates to Telegram
+
+```js
+response = content; // ''
+```
+
+`handleChat` returns `{ response: '', ... }`. In `telegram/index.js`:
+
+```js
+const rawResponse = typeof result.response === 'string' ? result.response : ...;
+// rawResponse = ''
+const text = rawResponse.trim() || 'The agent encountered an error...';
+// '' → fallback shown
+```
+
+The user sees the generic Telegram fallback instead of any information about what happened or what to do.
+
+---
+
+## Root Causes
+
+**Primary**: The `format_error` path set `response = content` without a fallback for the empty string case. An empty `response` triggers the Telegram handler's last-resort fallback message, giving the user no context.
+
+**Secondary**: The format recovery chain was designed for non-empty non-JSON responses. When `content` is empty, the nudge message "Your previous response was not valid JSON" is inaccurate — the model produced nothing, not invalid JSON. A targeted nudge for the empty case increases the chance of recovery.
+
+**Model-level cause**: The free model `nvidia/nemotron-3-nano-30b-a3b:free` can fail to produce any output after processing a heavily truncated tool result followed by consecutive tool failures. This is a model quality limitation that the recovery layer must account for.
+
+---
+
+## Difference from Finding 009 and 010
+
+| Finding | Model produces... | Bug manifests at... |
+|---------|-------------------|---------------------|
+| 009 | Non-string `response` field (array/object) | Telegram `.trim()` crash |
+| 010 | Non-string `checkpoint.remaining` | Zero-progress `.trim()` crash |
+| 011 | Empty/null content (no text, no tool calls) | Telegram generic fallback (no crash, but useless to user) |
+
+Finding 011 is the third in the same class: model output type does not match what the system expects.
+
+---
+
+## Fix
+
+### `src/server/agent.js` — two changes
+
+**1. Empty-content detection with targeted nudge**
+
+When `content` is empty, skip the standard recovery chain (designed for non-JSON text) and apply a targeted nudge that accurately describes the situation:
+
+```js
+if (!content.trim()) {
+  // Model returned no content at all — use a targeted nudge instead of the
+  // standard JSON recovery chain (designed for non-empty non-JSON responses).
+  try {
+    const emptyNudge = [
+      ...preparedMessages,
+      { role: 'user', content: 'You returned an empty response. ' + FORMAT_NUDGE },
+    ];
+    const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+    const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+    parsed = JSON.parse(nudgeContent);
+    content = nudgeContent;
+  } catch {
+    // Give up — fall through to !parsed handler below
+  }
+} else {
+  // Non-empty content — use the existing 3-step JSON recovery chain
+  try { parsed = JSON.parse(content); } catch {
+    // Step 1: fallback model...
+    // Step 2: nudge...
+  }
+}
+```
+
+**2. Non-empty fallback on format_error**
+
+```js
+if (!parsed) {
+  // Ensure response is never empty so the delivery layer can show something
+  // meaningful rather than its generic fallback message.
+  response = content.trim() || 'The model did not produce a response. Please try again.';
+  logSummary = 'Model returned non-JSON final response after recovery attempts.';
+  status = 'format_error';
+  return { ... };
+}
+```
+
+---
+
+## Outcome
+
+| Scenario | Before | After |
+|----------|--------|-------|
+| Model returns empty content, nudge succeeds | format_error (3 wasted API calls) | Clean recovery (1 targeted API call) |
+| Model returns empty content, nudge fails | Telegram generic fallback | "The model did not produce a response. Please try again." |
+| Model returns non-JSON text, all recovery fails | Telegram generic fallback (if text was empty) | Raw model output shown to user |
+
+**Effect on the debugging session**: instead of the generic Telegram fallback, the user would have received "The model did not produce a response. Please try again." — a clear signal that the model failed, not their message. In the best case, the new targeted nudge would have elicited a valid JSON response.

package/docs/findings/012-empty-nudge-loses-recovery-text.md

@@ -0,0 +1,121 @@
+# Finding 012: Empty-Content Nudge Includes Tools and Loses Recovery Text
+
+**Date:** 2026-03-02
+**Severity:** Medium — user sees generic error when model produces a partial recovery response
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `21fb43a7-2b11-4208-99fb-e6b54fddc07b`, entry 9 in session.jsonl:
+
+```
+status=format_error
+model=nvidia/nemotron-3-nano-30b-a3b:free
+iteration=3
+userInput='Ok. Read the results folder. Is there anything?'
+logSummary='Model returned non-JSON final response after recovery attempts.'
+response='The model did not produce a response. Please try again.'
+```
+
+The user received: **"The model did not produce a response. Please try again."**
+
+---
+
+## What Happened
+
+1. The agent executed two tool calls:
+   - `list_dir /root/.jarvis/projects/cybersecurity/results` → success
+   - `exec "list_dir /root/.jarvis/projects/cybersecurity/results/dviet.de"` → exit 127 (`list_dir: not found`)
+     - The model confused the `list_dir` jarvis tool with a shell command
+
+2. After the failed exec, the model returned `assistantMessage.content = null` with no `tool_calls` — it "went silent"
+
+3. Finding 011's empty-content nudge was triggered
+
+4. The nudge **also failed** — no valid JSON response was produced
+
+5. The agent fell through to `format_error` with the fallback message
+
+---
+
+## Bug Chain
+
+### Bug 1 — toolDefs included in empty nudge
+
+```js
+const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+```
+
+When the model is confused after a tool failure, it may respond to the nudge with **another tool call** instead of text. If it does:
+
+```
+nudgeResult.choices[0].message.content = null
+nudgeContent = ''
+JSON.parse('') → throws
+catch: // Give up — content stays ''
+```
+
+The model had an opportunity to call more tools instead of producing a text response — the wrong behavior for a recovery nudge.
+
+### Bug 2 — content assigned after parse
+
+```js
+const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+parsed = JSON.parse(nudgeContent);   // ← throws on non-JSON or empty
+content = nudgeContent;              // ← only reached if parse succeeded
+```
+
+If the model responds to the nudge with non-empty but non-JSON text (e.g. a plain English answer), `JSON.parse` throws and `content` is **never updated**. The non-JSON text is discarded. The `!parsed` handler then shows the fallback message instead of the model's actual text.
+
+---
+
+## Difference from Finding 011
+
+| Finding | Problem | Trigger |
+|---------|---------|---------|
+| 011 | Empty model response propagates to Telegram | Initial empty content, no recovery chain |
+| 012 | Recovery nudge discards best-effort text; model can respond with tool call | Recovery nudge called with toolDefs + content assigned after parse |
+
+Finding 012 is a refinement of the recovery path introduced in Finding 011.
+
+---
+
+## Fix
+
+### `src/server/agent.js` — empty-content nudge block
+
+**Before:**
+```js
+const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+parsed = JSON.parse(nudgeContent);
+content = nudgeContent;
+```
+
+**After:**
+```js
+// No tools: force text response, prevent model from calling another tool
+const nudgeResult = await callModelWithFallback(client, config, emptyNudge, []);
+const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+// Persist before parsing — if JSON parse throws, content still carries the
+// model's best-effort text so the !parsed handler can show it to the user
+if (nudgeContent.trim()) {
+  content = nudgeContent;
+}
+parsed = JSON.parse(nudgeContent);
+```
+
+---
+
+## Outcome
+
+| Nudge response | Before | After |
+|---|---|---|
+| Valid JSON | Clean recovery | Clean recovery (no change) |
+| Non-JSON text | Text discarded, fallback shown | Text shown to user |
+| Tool call (no content) | content='', fallback shown | Less likely; content='', fallback shown |
+| Empty again | content='', fallback shown | content='', fallback shown (no change) |
+
+The user in the observed session would have received the model's best-effort text about the results folder contents, rather than "The model did not produce a response. Please try again."

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.27",
+  "version": "1.0.29",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -215,32 +215,58 @@ async function runAgentLoop(client, config, session, prepareMessages) {
     let content = assistantMessage.content || '';
     let parsed = null;
 
-    try {
-      parsed = JSON.parse(content);
-    } catch {
-      // Step 1: retry with fallback model
+    if (!content.trim()) {
+      // Model returned no content at all — use a targeted nudge instead of the
+      // standard JSON recovery chain (designed for non-empty non-JSON responses).
+      // Send with no tools so the model cannot respond with another tool call,
+      // which would leave content empty and discard any recovery text.
+      try {
+        const emptyNudge = [
+          ...preparedMessages,
+          { role: 'user', content: 'You returned an empty response. ' + FORMAT_NUDGE },
+        ];
+        const nudgeResult = await callModelWithFallback(client, config, emptyNudge, []);
+        const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+        // Persist nudge text before parsing — if JSON parse throws, content still
+        // carries the model's best-effort text so the !parsed handler can show it
+        // rather than falling back to "The model did not produce a response."
+        if (nudgeContent.trim()) {
+          content = nudgeContent;
+        }
+        parsed = JSON.parse(nudgeContent);
+      } catch {
+        // Fall through to !parsed handler; content may now carry the nudge text
+      }
+    } else {
       try {
-        const fallbackResult = await callModel(client, config.fallbackModel, preparedMessages, toolDefs);
-        const fallbackContent = fallbackResult.choices[0]?.message?.content || '';
-        parsed = JSON.parse(fallbackContent);
-        content = fallbackContent;
+        parsed = JSON.parse(content);
       } catch {
-        // Step 2: nudge retry via both models
+        // Step 1: retry with fallback model
         try {
-          const nudgeMessages = [...preparedMessages, { role: 'user', content: FORMAT_NUDGE }];
-          const nudgeResult = await callModelWithFallback(client, config, nudgeMessages, toolDefs);
-          const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
-          parsed = JSON.parse(nudgeContent);
-          content = nudgeContent;
+          const fallbackResult = await callModel(client, config.fallbackModel, preparedMessages, toolDefs);
+          const fallbackContent = fallbackResult.choices[0]?.message?.content || '';
+          parsed = JSON.parse(fallbackContent);
+          content = fallbackContent;
         } catch {
-          // Give up
+          // Step 2: nudge retry via both models
+          try {
+            const nudgeMessages = [...preparedMessages, { role: 'user', content: FORMAT_NUDGE }];
+            const nudgeResult = await callModelWithFallback(client, config, nudgeMessages, toolDefs);
+            const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+            parsed = JSON.parse(nudgeContent);
+            content = nudgeContent;
+          } catch {
+            // Give up
+          }
         }
       }
     }
 
     if (!parsed) {
-      // Don't push bad content — handleChat will inject a synthetic error note
-      response = content;
+      // Don't push bad content — handleChat will inject a synthetic error note.
+      // Ensure response is never empty so the delivery layer (e.g. Telegram) can
+      // show the user something meaningful rather than its generic fallback message.
+      response = content.trim() || 'The model did not produce a response. Please try again.';
       logSummary = 'Model returned non-JSON final response after recovery attempts.';
       status = 'format_error';
       return { iteration, response, logSummary, status, runToolCalls, checkpoint: null, rawResponse: content };