@ducci/jarvis 1.0.33 → 1.0.34

package/docs/findings/017-looping-intervention-and-lossy-checkpoint.md

@@ -0,0 +1,110 @@
+# Finding 017: Looping Intervention and Lossy Checkpoint
+
+**Date:** 2026-03-04
+**Severity:** High — agent burned 40 iterations (4 full runs) on a structurally impossible task without escaping; concrete facts like file paths regressed between handoff runs
+**Status:** Fixed
+
+---
+
+## Observed Session
+
+Session from a remote server. Model: `nvidia/nemotron-3-nano-30b-a3b:free`. User requested a ZAP security scanning workflow (create project dir, write README + scan.sh, run a test scan).
+
+| Entry | Trigger | Status | Iterations | Notes |
+|-------|---------|--------|------------|-------|
+| 1 | "hello" | ok | 0 | greeting |
+| 2 | ZAP task | checkpoint_reached | 10 | ZAP daemon lock blocked scan |
+| 3 | handoff resume | checkpoint_reached | 10 | same lock, same result |
+| 4 | zero-progress | intervention_required | 0 | correctly detected |
+| 5 | "Ok can you do it?" | checkpoint_reached | 10 | loop restarted, same result |
+| 6 | handoff resume | checkpoint_reached | 10 | same lock again |
+| 7 | zero-progress | intervention_required | 0 | detected again, session abandoned |
+
+Total: 40 wasted iterations. Additionally, between Entry 2 and Entry 6, the agent changed the project path from `/root/.jarvis/projects/cybersecurity` to `/root/projects/cybersecurity`, causing `list_dir` to fail at the start of that run.
+
+---
+
+## Root Cause 1: Zero-Progress Detection Resets Across User Messages
+
+### What happened
+
+`previousRemaining` is a local variable initialized to `null` on every call to `_runHandleChat`. Zero-progress detection requires two identical `checkpoint.remaining` values, but both must occur within the same invocation. When `intervention_required` fires and the user sends a new message, `_runHandleChat` is called fresh with `previousRemaining = null`. The detection resets.
+
+The result: each new user message grants the agent 2 full runs (20 iterations) before zero-progress fires again — regardless of how many times the cycle has already repeated. The intervention mechanism correctly identifies "stuck" but provides no structural escape when the user replies.
+
+### Fix
+
+1. When zero-progress fires, persist `session.metadata.lastCheckpointRemaining = currentRemaining`.
+2. In `_runHandleChat`, initialize `previousRemaining` from `session.metadata.lastCheckpointRemaining` (cleared after reading). Zero-progress now fires after just one run (10 iterations) on the next user message if the agent produces the same remaining.
+3. Inject a note into `userMessageWithContext` when `lastCheckpointRemaining` was set:
+
+```
+[System: This task previously hit zero-progress and required intervention. If the user has given new direction or clarification, follow it. Otherwise, immediately explain what specific obstacle is blocking progress — do not resume the same failing approach.]
+```
+
+This gives the agent explicit guidance to respond to what the user actually asked instead of blindly resuming.
+
+**File:** `src/server/agent.js` — `_runHandleChat`
+
+---
+
+## Root Cause 2: Checkpoint Loses Concrete Facts Between Runs
+
+### What happened
+
+The checkpoint schema had `progress`, `remaining`, and `failedApproaches` — all natural-language prose. Concrete facts the agent discovers during a run (file paths, binary locations, config values) are paraphrased or omitted when the agent writes the summary. On resume, the model reconstructs these facts from vague prose and sometimes gets them wrong.
+
+In this session, the project directory `/root/.jarvis/projects/cybersecurity` was written correctly in runs 1–3 but reconstructed as `/root/projects/cybersecurity` in run 6. The first action of that run was `list_dir` on the wrong path, which failed.
+
+### Fix
+
+1. Added a `state` field to `WRAP_UP_NOTE`'s checkpoint schema: a flat key-value JSON object for concrete facts confirmed by tool output (file paths created, binary locations found, config values discovered).
+
+2. After each handoff, merge `run.checkpoint.state` into `session.metadata.checkpointState` (later runs overwrite earlier values for the same key).
+
+3. When building `resumeContent` for the next handoff run, inject the accumulated state:
+
+```
+[System: Known facts from previous runs:
+- projectDir: /root/.jarvis/projects/cybersecurity
+- zapBinary: /snap/bin/zaproxy
+- scanScriptPath: /root/.jarvis/projects/cybersecurity/scan.sh]
+```
+
+4. When re-entering after zero-progress (via `wasZeroProgress`), also include the state in the `userMessageWithContext` injection so facts are available immediately on the first run.
+
+5. `session.metadata.checkpointState` is reset on each new user message (same lifecycle as `failedApproaches`) to avoid stale facts from previous tasks leaking into new ones.
+
+**File:** `src/server/agent.js` — `WRAP_UP_NOTE`, checkpoint normalization, `_runHandleChat`
+
+---
+
+## Interaction Between the Two Fixes
+
+The zero-progress note injected into `userMessageWithContext` now also includes the `priorCheckpointState` (captured before the metadata reset), so the agent that receives the note also has the concrete facts it needs if the user does provide new direction. This means both fixes compound: the agent is told it was stuck AND given the facts it needs to act on whatever the user says.
+
+---
+
+## What Was Not Changed
+
+- The existing exact-match loop detector (`loopTracker`) — unchanged
+- The consecutive failure detector — unchanged
+- The `maxHandoffs` cap — unchanged
+- The `hasConsecutiveModelErrors` escalation — unchanged
+- The context strip on checkpoint/intervention — unchanged
+
+---
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/server/agent.js` | `WRAP_UP_NOTE` — added `state` field to checkpoint schema |
+| `src/server/agent.js` | Checkpoint normalization — normalize `cp.state` to `{}` if missing/malformed |
+| `src/server/agent.js` | `_runHandleChat` — capture `wasZeroProgress`, `priorCheckpointRemaining`, `priorCheckpointState` before metadata reset |
+| `src/server/agent.js` | `_runHandleChat` — inject zero-progress note + prior state into `userMessageWithContext` when applicable |
+| `src/server/agent.js` | `_runHandleChat` — reset `lastCheckpointRemaining` and `checkpointState` on new user message |
+| `src/server/agent.js` | `_runHandleChat` — initialize `previousRemaining` from `priorCheckpointRemaining` |
+| `src/server/agent.js` | Zero-progress block — persist `session.metadata.lastCheckpointRemaining` |
+| `src/server/agent.js` | Handoff accumulation — merge `run.checkpoint.state` into `session.metadata.checkpointState` |
+| `src/server/agent.js` | `resumeContent` — inject accumulated `checkpointState` as known facts |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.33",
+  "version": "1.0.34",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/agent.js

@@ -20,11 +20,12 @@ Respond with your normal JSON, but add a checkpoint field:
   "checkpoint": {
     "progress": "What has been fully completed — only include items confirmed by tool output (e.g., successful exec with exit code 0, or verified by ls/cat). Do not report planned steps as completed.",
     "remaining": "What still needs to be done to finish the task — as a plain text string, never an array or object.",
-    "failedApproaches": ["Concise description of each approach that was tried and failed, e.g. 'downloading subfinder via curl from GitHub releases — connection reset'. Omit array entries for things that succeeded. Leave as empty array if nothing failed."]
+    "failedApproaches": ["Concise description of each approach that was tried and failed, e.g. 'downloading subfinder via curl from GitHub releases — connection reset'. Omit array entries for things that succeeded. Leave as empty array if nothing failed."],
+    "state": {"factKey": "factValue — concrete facts confirmed by tool output this run: file paths created, binary locations found, config values discovered. Use short stable keys, e.g. projectDir, zapBinary, scanScriptPath. Omit or use {} if nothing concrete was discovered."}
   }
 }
 
-The checkpoint field will be used to automatically resume the task in the next run. failedApproaches is injected into the next run so the agent does not waste iterations repeating strategies that already failed. remaining must be a plain text string. failedApproaches must be a JSON array of strings.]`;
+The checkpoint field will be used to automatically resume the task in the next run. failedApproaches is injected into the next run so the agent does not waste iterations repeating strategies that already failed. state is injected verbatim so the next run does not need to rediscover file paths or binary locations. remaining must be a plain text string. failedApproaches must be a JSON array of strings. state must be a flat JSON object.]`;
 
 // Serializes concurrent requests for the same session. Maps sessionId to the
 // tail of the current request chain (a Promise that resolves when the last
@@ -414,6 +415,9 @@ async function runAgentLoop(client, config, session, prepareMessages) {
             typeof item === 'string' ? item : JSON.stringify(item)
           );
         }
+        if (typeof cp.state !== 'object' || cp.state === null || Array.isArray(cp.state)) {
+          cp.state = {};
+        }
         return {
           iteration,
           response,
@@ -480,6 +484,11 @@ async function _runHandleChat(config, sessionId, userMessage) {
     session = createSession(systemPromptTemplate);
   }
 
+  // Capture persisted state BEFORE resetting metadata so we can inject it below.
+  const wasZeroProgress = !!session.metadata.lastCheckpointRemaining;
+  const priorCheckpointRemaining = session.metadata.lastCheckpointRemaining || null;
+  const priorCheckpointState = session.metadata.checkpointState || {};
+
   // Preserve accumulated failedApproaches in conversation history before resetting
   // so the model retains knowledge of what failed in the previous batch of handoff runs.
   let userMessageWithContext = userMessage;
@@ -487,10 +496,24 @@ async function _runHandleChat(config, sessionId, userMessage) {
     userMessageWithContext += `\n\n[System: The following approaches were tried and failed in previous runs — consider them exhausted:\n${session.metadata.failedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
   }
 
+  // If this message follows a zero-progress intervention, tell the agent explicitly so
+  // it responds to the user's input instead of blindly resuming the same failing approach.
+  if (wasZeroProgress) {
+    const stateLines = Object.entries(priorCheckpointState).map(([k, v]) => `- ${k}: ${v}`);
+    let note = `\n\n[System: This task previously hit zero-progress and required intervention. If the user has given new direction or clarification, follow it. Otherwise, immediately explain what specific obstacle is blocking progress — do not resume the same failing approach.`;
+    if (stateLines.length > 0) {
+      note += `\n\nKnown facts from previous run:\n${stateLines.join('\n')}`;
+    }
+    note += `]`;
+    userMessageWithContext += note;
+  }
+
   // Append user message and reset handoff state
   session.messages.push({ role: 'user', content: userMessageWithContext });
   session.metadata.handoffCount = 0;
   session.metadata.failedApproaches = [];
+  session.metadata.lastCheckpointRemaining = null;
+  session.metadata.checkpointState = {};
 
   // Resolves {{user_info}} in system prompt at runtime (never persisted)
   function prepareMessages(messages) {
@@ -506,8 +529,11 @@ async function _runHandleChat(config, sessionId, userMessage) {
   let finalResponse = '';
   let finalLogSummary = '';
   let finalStatus = 'ok';
-  // Tracks checkpoint.remaining from the previous handoff run to detect zero progress
-  let previousRemaining = null;
+  // Tracks checkpoint.remaining from the previous handoff run to detect zero progress.
+  // Initialized from persisted metadata so detection works across user messages too —
+  // if the agent was stuck before and produces the same remaining again on the next
+  // user turn, zero-progress fires after just one run instead of two.
+  let previousRemaining = priorCheckpointRemaining;
 
   try {
     // Handoff loop
@@ -590,6 +616,15 @@ async function _runHandleChat(config, sessionId, userMessage) {
         session.metadata.failedApproaches.push(...run.checkpoint.failedApproaches);
       }
 
+      // Merge concrete facts from this run's checkpoint.state into session metadata.
+      // Later runs overwrite earlier values for the same key (newer discoveries win).
+      if (run.checkpoint.state && Object.keys(run.checkpoint.state).length > 0) {
+        session.metadata.checkpointState = {
+          ...(session.metadata.checkpointState || {}),
+          ...run.checkpoint.state,
+        };
+      }
+
       // Zero-progress detection: if checkpoint.remaining is identical to the previous
       // handoff's remaining, the agent completed a full run without making any progress.
       // Stop immediately rather than burning more iterations on a stuck task.
@@ -599,6 +634,10 @@ async function _runHandleChat(config, sessionId, userMessage) {
         finalLogSummary = 'Zero progress detected: task state unchanged after a full run. Human intervention required.';
         finalStatus = 'intervention_required';
 
+        // Persist so that the next user message initializes previousRemaining from this
+        // value — zero-progress will then fire after just one run instead of two.
+        session.metadata.lastCheckpointRemaining = currentRemaining;
+
         await appendLog(sessionId, {
           iteration: 0,
           model: config.selectedModel,
@@ -642,13 +681,17 @@ async function _runHandleChat(config, sessionId, userMessage) {
 
       // Resume with checkpoint.remaining as new prompt.
       // Guard against null/undefined in case the model omitted the field.
-      // Use the full accumulated failedApproaches list across ALL handoff runs so the
-      // agent has complete memory of what has already been tried and failed.
+      // Inject the full accumulated failedApproaches and concrete state so the agent
+      // has complete memory of what failed and what was already discovered.
       let resumeContent = run.checkpoint.remaining || 'Continue with the task.';
       const allFailedApproaches = session.metadata.failedApproaches || [];
       if (allFailedApproaches.length > 0) {
         resumeContent += `\n\n[System: The following approaches were tried and failed in previous runs — do not repeat them:\n${allFailedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
       }
+      const stateToInject = session.metadata.checkpointState || {};
+      if (Object.keys(stateToInject).length > 0) {
+        resumeContent += `\n\n[System: Known facts from previous runs:\n${Object.entries(stateToInject).map(([k, v]) => `- ${k}: ${v}`).join('\n')}]`;
+      }
       session.messages.push({ role: 'user', content: resumeContent });
     }
   } catch (e) {