@ducci/jarvis 1.0.26 → 1.0.28

package/docs/findings/010-checkpoint-field-type-safety.md

@@ -0,0 +1,121 @@
+# Finding 010: Non-String `checkpoint.remaining` Crashes Zero-Progress Detection
+
+**Date:** 2026-03-01
+**Severity:** High — caused "Sorry, something went wrong" in Telegram with no useful context; crashed the handoff loop mid-run
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+The session ran 13+ agent runs working on OWASP ZAP installation. Runs 8–13 were consecutive `checkpoint_reached` handoffs. On entry 14 (immediately after entry 13), the server logged:
+
+```
+status: error
+response: "An unexpected server error occurred: (run.checkpoint.remaining || "").trim is not a function"
+```
+
+The Telegram user received:
+
+```
+Sorry, something went wrong: (run.checkpoint.remaining || "").trim is not a function
+```
+
+---
+
+## Bug Chain
+
+### Step 1 — Wrap-up call returns non-string `remaining`
+
+At iteration limit, `runAgentLoop` sends the `WRAP_UP_NOTE` and parses the model's JSON response. The model returned `checkpoint.remaining` as a non-string value (array or object) instead of a plain text string. `parsedWrapUp.checkpoint` was stored and returned with no type validation.
+
+### Step 2 — Zero-progress detection crashes on `.trim()`
+
+In `_runHandleChat`, finding 007 introduced zero-progress detection:
+
+```js
+const currentRemaining = (run.checkpoint.remaining || '').trim();
+```
+
+The `|| ''` guard only catches falsy values (null, undefined). A truthy non-string (array, object) passes through the `||` and `.trim()` is called on a non-string:
+
+```
+TypeError: (run.checkpoint.remaining || "").trim is not a function
+```
+
+### Step 3 — Outer catch logs the error and re-throws
+
+The `try/catch` at the top of the handoff loop caught the TypeError, wrote an `error` status log entry, and re-threw. The Telegram handler surfaced the raw error message.
+
+---
+
+## Secondary Issues
+
+**`resumeContent` (line 520)**: `run.checkpoint.remaining || 'Continue with the task.'` — if `remaining` is a truthy non-string, it would be pushed directly into `session.messages` as the next user message content. The message API expects a string, so this would produce a malformed conversation message.
+
+**`failedApproaches` spread (lines 461–463)**: If the model returns `failedApproaches` as a non-array (string, object), `push(...value)` would spread wrong data. A string spreads individual characters; an object spreads its enumerable values.
+
+---
+
+## Root Cause
+
+Same class of bug as finding 009 (non-string `response` field). Finding 009 hardened `response` and `logSummary` extraction, but the `checkpoint` sub-object fields were not included in that hardening pass. Models — especially smaller/free models under iteration-limit pressure — sometimes return structured data (arrays, objects) in fields the system prompt specifies as plain text strings.
+
+---
+
+## Fix
+
+### `src/server/agent.js` — normalize checkpoint fields at source
+
+Added a normalization block immediately inside the `if (parsedWrapUp.checkpoint)` branch, before any checkpoint field is accessed downstream:
+
+```js
+const cp = parsedWrapUp.checkpoint;
+// remaining must be a string — used as the next run's resume prompt
+if (typeof cp.remaining !== 'string') {
+  cp.remaining = Array.isArray(cp.remaining)
+    ? cp.remaining.map(String).join('\n')
+    : cp.remaining != null ? JSON.stringify(cp.remaining) : '';
+}
+// failedApproaches must be an array of strings — spread into session metadata
+if (!Array.isArray(cp.failedApproaches)) {
+  cp.failedApproaches = [];
+} else {
+  cp.failedApproaches = cp.failedApproaches.map(item =>
+    typeof item === 'string' ? item : JSON.stringify(item)
+  );
+}
+```
+
+**Array coercion for `remaining`**: when the model returns an array (e.g., `["install Java", "create symlink"]`), elements are joined with newlines rather than JSON-stringified — producing a natural readable resume prompt rather than raw JSON syntax.
+
+**Centralized normalization**: fixing at source (right after parse) rather than at each use site means lines 469 and 520 need no change. Any future use of `checkpoint.remaining` or `checkpoint.failedApproaches` is automatically safe.
+
+### `src/server/agent.js` — update `WRAP_UP_NOTE`
+
+Added explicit type constraints to the `remaining` field description and a trailing instruction:
+
+```
+"remaining": "What still needs to be done — as a plain text string, never an array or object."
+...
+remaining must be a plain text string. failedApproaches must be a JSON array of strings.
+```
+
+---
+
+## What Was Not Changed
+
+- `agent.js` lines 469 and 520 — no changes needed; normalization at source makes them safe
+- `src/channels/telegram/index.js` — finding 007 and 009 already added `.catch(() => {})` and type guards on delivery
+- `sessions.js`, `tools.js` — no changes needed
+
+---
+
+## Outcome
+
+| Fix | Files changed |
+|-----|--------------|
+| Normalize `checkpoint.remaining` to string and `checkpoint.failedApproaches` to string array at source | `src/server/agent.js` |
+| Add explicit type constraints to WRAP_UP_NOTE | `src/server/agent.js` |
+
+**Effect**: instead of a `TypeError` crash mid-handoff-loop, the model's non-string `remaining` value is coerced to a readable string and used as the resume prompt. The session continues normally.

package/docs/findings/011-empty-model-response.md

@@ -0,0 +1,157 @@
+# Finding 011: Empty Model Response Causes Generic Telegram Error
+
+**Date:** 2026-03-01
+**Severity:** High — user sees generic "please try again" with no actionable information
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session `33a50dfe-38ea-4972-adac-498ef0525b0c`, run 16 of 17 (session.jsonl line 16):
+
+```
+status=format_error
+model=nvidia/nemotron-3-nano-30b-a3b:free
+iteration=4
+userInput='Ok. Kannst du bitte jetzt das shell script ausführen mit der domain...'
+logSummary='Model returned non-JSON final response after recovery attempts.'
+rawResponse=''
+response=''
+```
+
+The Telegram user received:
+
+```
+The agent encountered an error and could not produce a response. Please try again.
+```
+
+---
+
+## What Happened
+
+The agent executed a ZAP scan (`./scan.sh juice-shop.herokuapp.com`). The tool result was a large ZAP startup log, truncated at 4000 characters. Two subsequent tool calls failed:
+
+- `pkill -f zaproxy || true` → exit 1 (no process to kill)
+- `zaproxy -help | grep -i shutdown -A5` → failed (`libtiff.so.5` missing)
+
+On iteration 4, the model returned `assistantMessage.content = null` with no `tool_calls`. This is the "went silent" case: the model produced neither a response nor another tool call.
+
+---
+
+## Bug Chain
+
+### Step 1 — Model returns null content
+
+```js
+let content = assistantMessage.content || '';
+// content = ''
+```
+
+### Step 2 — Recovery chain falls through on empty content
+
+The existing format recovery chain was designed for *non-empty, non-JSON* responses:
+
+1. `JSON.parse('')` → throws
+2. Retry with fallback model (same messages, no nudge) → also `''`
+3. Retry with nudge "Your previous response was not valid JSON" → technically wrong for empty content; model still returns `''`
+4. Give up
+
+### Step 3 — Empty `response` propagates to Telegram
+
+```js
+response = content; // ''
+```
+
+`handleChat` returns `{ response: '', ... }`. In `telegram/index.js`:
+
+```js
+const rawResponse = typeof result.response === 'string' ? result.response : ...;
+// rawResponse = ''
+const text = rawResponse.trim() || 'The agent encountered an error...';
+// '' → fallback shown
+```
+
+The user sees the generic Telegram fallback instead of any information about what happened or what to do.
+
+---
+
+## Root Causes
+
+**Primary**: The `format_error` path set `response = content` without a fallback for the empty string case. An empty `response` triggers the Telegram handler's last-resort fallback message, giving the user no context.
+
+**Secondary**: The format recovery chain was designed for non-empty non-JSON responses. When `content` is empty, the nudge message "Your previous response was not valid JSON" is inaccurate — the model produced nothing, not invalid JSON. A targeted nudge for the empty case increases the chance of recovery.
+
+**Model-level cause**: The free model `nvidia/nemotron-3-nano-30b-a3b:free` can fail to produce any output after processing a heavily truncated tool result followed by consecutive tool failures. This is a model quality limitation that the recovery layer must account for.
+
+---
+
+## Difference from Finding 009 and 010
+
+| Finding | Model produces... | Bug manifests at... |
+|---------|-------------------|---------------------|
+| 009 | Non-string `response` field (array/object) | Telegram `.trim()` crash |
+| 010 | Non-string `checkpoint.remaining` | Zero-progress `.trim()` crash |
+| 011 | Empty/null content (no text, no tool calls) | Telegram generic fallback (no crash, but useless to user) |
+
+Finding 011 is the third in the same class: model output type does not match what the system expects.
+
+---
+
+## Fix
+
+### `src/server/agent.js` — two changes
+
+**1. Empty-content detection with targeted nudge**
+
+When `content` is empty, skip the standard recovery chain (designed for non-JSON text) and apply a targeted nudge that accurately describes the situation:
+
+```js
+if (!content.trim()) {
+  // Model returned no content at all — use a targeted nudge instead of the
+  // standard JSON recovery chain (designed for non-empty non-JSON responses).
+  try {
+    const emptyNudge = [
+      ...preparedMessages,
+      { role: 'user', content: 'You returned an empty response. ' + FORMAT_NUDGE },
+    ];
+    const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+    const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+    parsed = JSON.parse(nudgeContent);
+    content = nudgeContent;
+  } catch {
+    // Give up — fall through to !parsed handler below
+  }
+} else {
+  // Non-empty content — use the existing 3-step JSON recovery chain
+  try { parsed = JSON.parse(content); } catch {
+    // Step 1: fallback model...
+    // Step 2: nudge...
+  }
+}
+```
+
+**2. Non-empty fallback on format_error**
+
+```js
+if (!parsed) {
+  // Ensure response is never empty so the delivery layer can show something
+  // meaningful rather than its generic fallback message.
+  response = content.trim() || 'The model did not produce a response. Please try again.';
+  logSummary = 'Model returned non-JSON final response after recovery attempts.';
+  status = 'format_error';
+  return { ... };
+}
+```
+
+---
+
+## Outcome
+
+| Scenario | Before | After |
+|----------|--------|-------|
+| Model returns empty content, nudge succeeds | format_error (3 wasted API calls) | Clean recovery (1 targeted API call) |
+| Model returns empty content, nudge fails | Telegram generic fallback | "The model did not produce a response. Please try again." |
+| Model returns non-JSON text, all recovery fails | Telegram generic fallback (if text was empty) | Raw model output shown to user |
+
+**Effect on the debugging session**: instead of the generic Telegram fallback, the user would have received "The model did not produce a response. Please try again." — a clear signal that the model failed, not their message. In the best case, the new targeted nudge would have elicited a valid JSON response.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.26",
+  "version": "1.0.28",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -19,12 +19,12 @@ Respond with your normal JSON, but add a checkpoint field:
   "logSummary": "Human-readable summary of what happened in this run.",
   "checkpoint": {
     "progress": "What has been fully completed so far.",
-    "remaining": "What still needs to be done to finish the task.",
+    "remaining": "What still needs to be done to finish the task — as a plain text string, never an array or object.",
     "failedApproaches": ["Concise description of each approach that was tried and failed, e.g. 'downloading subfinder via curl from GitHub releases — connection reset'. Omit array entries for things that succeeded. Leave as empty array if nothing failed."]
   }
 }
 
-The checkpoint field will be used to automatically resume the task in the next run. failedApproaches is injected into the next run so the agent does not waste iterations repeating strategies that already failed.]`;
+The checkpoint field will be used to automatically resume the task in the next run. failedApproaches is injected into the next run so the agent does not waste iterations repeating strategies that already failed. remaining must be a plain text string. failedApproaches must be a JSON array of strings.]`;
 
 // Serializes concurrent requests for the same session. Maps sessionId to the
 // tail of the current request chain (a Promise that resolves when the last
@@ -215,32 +215,51 @@ async function runAgentLoop(client, config, session, prepareMessages) {
     let content = assistantMessage.content || '';
     let parsed = null;
 
-    try {
-      parsed = JSON.parse(content);
-    } catch {
-      // Step 1: retry with fallback model
+    if (!content.trim()) {
+      // Model returned no content at all — use a targeted nudge instead of the
+      // standard JSON recovery chain (designed for non-empty non-JSON responses).
+      try {
+        const emptyNudge = [
+          ...preparedMessages,
+          { role: 'user', content: 'You returned an empty response. ' + FORMAT_NUDGE },
+        ];
+        const nudgeResult = await callModelWithFallback(client, config, emptyNudge, toolDefs);
+        const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+        parsed = JSON.parse(nudgeContent);
+        content = nudgeContent;
+      } catch {
+        // Give up — fall through to !parsed handler below
+      }
+    } else {
       try {
-        const fallbackResult = await callModel(client, config.fallbackModel, preparedMessages, toolDefs);
-        const fallbackContent = fallbackResult.choices[0]?.message?.content || '';
-        parsed = JSON.parse(fallbackContent);
-        content = fallbackContent;
+        parsed = JSON.parse(content);
       } catch {
-        // Step 2: nudge retry via both models
+        // Step 1: retry with fallback model
         try {
-          const nudgeMessages = [...preparedMessages, { role: 'user', content: FORMAT_NUDGE }];
-          const nudgeResult = await callModelWithFallback(client, config, nudgeMessages, toolDefs);
-          const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
-          parsed = JSON.parse(nudgeContent);
-          content = nudgeContent;
+          const fallbackResult = await callModel(client, config.fallbackModel, preparedMessages, toolDefs);
+          const fallbackContent = fallbackResult.choices[0]?.message?.content || '';
+          parsed = JSON.parse(fallbackContent);
+          content = fallbackContent;
         } catch {
-          // Give up
+          // Step 2: nudge retry via both models
+          try {
+            const nudgeMessages = [...preparedMessages, { role: 'user', content: FORMAT_NUDGE }];
+            const nudgeResult = await callModelWithFallback(client, config, nudgeMessages, toolDefs);
+            const nudgeContent = nudgeResult.choices[0]?.message?.content || '';
+            parsed = JSON.parse(nudgeContent);
+            content = nudgeContent;
+          } catch {
+            // Give up
+          }
         }
       }
     }
 
     if (!parsed) {
-      // Don't push bad content — handleChat will inject a synthetic error note
-      response = content;
+      // Don't push bad content — handleChat will inject a synthetic error note.
+      // Ensure response is never empty so the delivery layer (e.g. Telegram) can
+      // show the user something meaningful rather than its generic fallback message.
+      response = content.trim() || 'The model did not produce a response. Please try again.';
       logSummary = 'Model returned non-JSON final response after recovery attempts.';
       status = 'format_error';
       return { iteration, response, logSummary, status, runToolCalls, checkpoint: null, rawResponse: content };
@@ -319,6 +338,22 @@ async function runAgentLoop(client, config, session, prepareMessages) {
         : parsedWrapUp.response != null ? JSON.stringify(parsedWrapUp.response, null, 2) : '';
       logSummary = parsedWrapUp.logSummary || '';
       if (parsedWrapUp.checkpoint) {
+        // Normalize checkpoint fields to their expected types. Models sometimes
+        // return arrays or objects in fields that must be strings — the same class
+        // of bug fixed for `response` in finding 009.
+        const cp = parsedWrapUp.checkpoint;
+        if (typeof cp.remaining !== 'string') {
+          cp.remaining = Array.isArray(cp.remaining)
+            ? cp.remaining.map(String).join('\n')
+            : cp.remaining != null ? JSON.stringify(cp.remaining) : '';
+        }
+        if (!Array.isArray(cp.failedApproaches)) {
+          cp.failedApproaches = [];
+        } else {
+          cp.failedApproaches = cp.failedApproaches.map(item =>
+            typeof item === 'string' ? item : JSON.stringify(item)
+          );
+        }
         return {
           iteration,
           response,