@ducci/jarvis 1.0.27 → 1.0.28

package/docs/findings/011-empty-model-response.md

@@ -0,0 +1,157 @@
+# Finding 011: Empty Model Response Causes Generic Telegram Error
+
+**Date:** 2026-03-01
+**Severity:** High — user sees generic "please try again" with no actionable information
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `33a50dfe-38ea-4972-adac-498ef0525b0c`, run 16 of 17 (session.jsonl line 16):
+
+```
+status=format_error
+model=nvidia/nemotron-3-nano-30b-a3b:free
+iteration=4
+userInput='Ok. Kannst du bitte jetzt das shell script ausführen mit der domain...'
+logSummary='Model returned non-JSON final response after recovery attempts.'
+rawResponse=''
+response=''
+```
+
+The Telegram user received:
+
+```
+The agent encountered an error and could not produce a response. Please try again.
+```
+
+---
+
+## What Happened
+
+The agent executed a ZAP scan (`./scan.sh juice-shop.herokuapp.com`). The tool result was a large ZAP startup log, truncated at 4000 characters. Two subsequent tool calls failed:
+
+- `pkill -f zaproxy || true` → exit 1 (no process to kill)
+- `zaproxy -help | grep -i shutdown -A5` → failed (`libtiff.so.5` missing)
+
+On iteration 4, the model returned `assistantMessage.content = null` with no `tool_calls`. This is the "went silent" case: the model produced neither a response nor another tool call.
+
+---
+
+## Bug Chain
+
+### Step 1 — Model returns null content
+
+```js
+let content = assistantMessage.content || '';
+// content = ''
+```
+
+### Step 2 — Recovery chain falls through on empty content
+
+The existing format recovery chain was designed for *non-empty, non-JSON* responses:
+
+1. `JSON.parse('')` → throws
+2. Retry with fallback model (same messages, no nudge) → also `''`
+3. Retry with nudge "Your previous response was not valid JSON" → technically wrong for empty content; model still returns `''`
+4. Give up
+
+### Step 3 — Empty `response` propagates to Telegram
+
+```js
+response = content; // ''
+```
+
+`handleChat` returns `{ response: '', ... }`. In `telegram/index.js`:
+
+```js
+const rawResponse = typeof result.response === 'string' ? result.response : ...;
+// rawResponse = ''
+const text = rawResponse.trim() || 'The agent encountered an error...';
+// '' → fallback shown
+```
+
+The user sees the generic Telegram fallback instead of any information about what happened or what to do.
+
+---
+
+## Root Causes
+
+**Primary**: The `format_error` path set `response = content` without a fallback for the empty string case. An empty `response` triggers the Telegram handler's last-resort fallback message, giving the user no context.
+
+**Secondary**: The format recovery chain was designed for non-empty non-JSON responses. When `content` is empty, the nudge message "Your previous response was not valid JSON" is inaccurate — the model produced nothing, not invalid JSON. A targeted nudge for the empty case increases the chance of recovery.
+
+**Model-level cause**: The free model `nvidia/nemotron-3-nano-30b-a3b:free` can fail to produce any output after processing a heavily truncated tool result followed by consecutive tool failures. This is a model quality limitation that the recovery layer must account for.
+
+---
+
+## Difference from Finding 009 and 010
+
+| Finding | Model produces... | Bug manifests at... |
+|---------|-------------------|---------------------|
+| 009 | Non-string `response` field (array/object) | Telegram `.trim()` crash |
+| 010 | Non-string `checkpoint.remaining` | Zero-progress `.trim()` crash |
+| 011 | Empty/null content (no text, no tool calls) | Telegram generic fallback (no crash, but useless to user) |
+
+Finding 011 is the third in the same class: model output type does not match what the system expects.
+
+---
+
+## Fix
+
+### `src/server/agent.js` — two changes
+
+**1. Empty-content detection with targeted nudge**
+
+When `content` is empty, skip the standard recovery chain (designed for non-JSON text) and apply a targeted nudge that accurately describes the situation:
+
+```js
+if (!content.trim()) {
+  // Model returned no content at all — use a targeted nudge instead of the
+  // standard JSON recovery chain (designed for non-empty non-JSON responses).
+  try {
+    const emptyNudge = [
+      ...preparedMessages,
+      { role: 'user', content: 'You returned an empty response. ' + FORMAT_NUDGE },
+    ];
+    const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+    const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+    parsed = JSON.parse(nudgeContent);
+    content = nudgeContent;
+  } catch {
+    // Give up — fall through to !parsed handler below
+  }
+} else {
+  // Non-empty content — use the existing 3-step JSON recovery chain
+  try { parsed = JSON.parse(content); } catch {
+    // Step 1: fallback model...
+    // Step 2: nudge...
+  }
+}
+```
+
+**2. Non-empty fallback on format_error**
+
+```js
+if (!parsed) {
+  // Ensure response is never empty so the delivery layer can show something
+  // meaningful rather than its generic fallback message.
+  response = content.trim() || 'The model did not produce a response. Please try again.';
+  logSummary = 'Model returned non-JSON final response after recovery attempts.';
+  status = 'format_error';
+  return { ... };
+}
+```
+
+---
+
+## Outcome
+
+| Scenario | Before | After |
+|----------|--------|-------|
+| Model returns empty content, nudge succeeds | format_error (3 wasted API calls) | Clean recovery (1 targeted API call) |
+| Model returns empty content, nudge fails | Telegram generic fallback | "The model did not produce a response. Please try again." |
+| Model returns non-JSON text, all recovery fails | Telegram generic fallback (if text was empty) | Raw model output shown to user |
+
+**Effect on the debugging session**: instead of the generic Telegram fallback, the user would have received "The model did not produce a response. Please try again." — a clear signal that the model failed, not their message. In the best case, the new targeted nudge would have elicited a valid JSON response.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.27",
+  "version": "1.0.28",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -215,32 +215,51 @@ async function runAgentLoop(client, config, session, prepareMessages) {
     let content = assistantMessage.content || '';
     let parsed = null;
 
-    try {
-      parsed = JSON.parse(content);
-    } catch {
-      // Step 1: retry with fallback model
+    if (!content.trim()) {
+      // Model returned no content at all — use a targeted nudge instead of the
+      // standard JSON recovery chain (designed for non-empty non-JSON responses).
+      try {
+        const emptyNudge = [
+          ...preparedMessages,
+          { role: 'user', content: 'You returned an empty response. ' + FORMAT_NUDGE },
+        ];
+        const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+        const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+        parsed = JSON.parse(nudgeContent);
+        content = nudgeContent;
+      } catch {
+        // Give up — fall through to !parsed handler below
+      }
+    } else {
       try {
-        const fallbackResult = await callModel(client, config.fallbackModel, preparedMessages, toolDefs);
-        const fallbackContent = fallbackResult.choices[0]?.message?.content || '';
-        parsed = JSON.parse(fallbackContent);
-        content = fallbackContent;
+        parsed = JSON.parse(content);
       } catch {
-        // Step 2: nudge retry via both models
+        // Step 1: retry with fallback model
         try {
-          const nudgeMessages = [...preparedMessages, { role: 'user', content: FORMAT_NUDGE }];
-          const nudgeResult = await callModelWithFallback(client, config, nudgeMessages, toolDefs);
-          const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
-          parsed = JSON.parse(nudgeContent);
-          content = nudgeContent;
+          const fallbackResult = await callModel(client, config.fallbackModel, preparedMessages, toolDefs);
+          const fallbackContent = fallbackResult.choices[0]?.message?.content || '';
+          parsed = JSON.parse(fallbackContent);
+          content = fallbackContent;
         } catch {
-          // Give up
+          // Step 2: nudge retry via both models
+          try {
+            const nudgeMessages = [...preparedMessages, { role: 'user', content: FORMAT_NUDGE }];
+            const nudgeResult = await callModelWithFallback(client, config, nudgeMessages, toolDefs);
+            const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+            parsed = JSON.parse(nudgeContent);
+            content = nudgeContent;
+          } catch {
+            // Give up
+          }
         }
       }
     }
 
     if (!parsed) {
-      // Don't push bad content — handleChat will inject a synthetic error note
-      response = content;
+      // Don't push bad content — handleChat will inject a synthetic error note.
+      // Ensure response is never empty so the delivery layer (e.g. Telegram) can
+      // show the user something meaningful rather than its generic fallback message.
+      response = content.trim() || 'The model did not produce a response. Please try again.';
       logSummary = 'Model returned non-JSON final response after recovery attempts.';
       status = 'format_error';
       return { iteration, response, logSummary, status, runToolCalls, checkpoint: null, rawResponse: content };