@ducci/jarvis 1.0.38 → 1.0.40

package/docs/findings/004-agent-reliability-improvements.md

@@ -1,162 +0,0 @@
-# Finding 004: Agent Reliability — Failure Loops, Checkpoint Memory, and Iteration Awareness
-
-**Date:** 2026-02-27
-**Severity:** High — caused observed session failure with 54 tool calls, `format_error` crash, and no useful output after 42 minutes
-**Status:** Fixed
-
----
-
-## What Happened
-
-A session was started to build a cybersecurity project installing three tools: Nuclei, Subfinder, and Naabu. The agent went through 5 handoffs (hitting `maxHandoffs`) and crashed with a `format_error` in the final iteration. Observations:
-
-- **181 exec calls** across 19 agent iterations
-- **21 perplexity_search calls**, including 11+ in the final iteration alone
-- The agent oscillated between download strategies (Docker → `go install` → tarball → direct binary) without memory of what had already failed
-- The existing loop detector never fired because each failed command had slightly different arguments (different URLs, different flags), producing different `callKey` values
-- Each handoff resumed with only `checkpoint.remaining` — no record of what approaches had already been exhausted
-- The model degraded and eventually produced a non-JSON response (`format_error`), crashing the session
-
----
-
-## Root Causes
-
-### 1. Loop detection only caught exact-match repetition
-
-The existing `loopTracker` detects when the _exact same_ tool call (name + args + result) is repeated 3 times. It does not detect _semantic_ failure loops: repeated attempts to do the same thing via slightly different commands. In the session, each download attempt used a different URL or flags, so every `callKey` was unique.
-
-### 2. Checkpoint carried no memory of failed strategies
-
-`WRAP_UP_NOTE` asked the model for `progress` and `remaining`, but not for a record of what _failed_. Each new run after a handoff started with a blank slate — the model had no way to know that curl downloads, `go install`, and tarball extraction had already been tried and failed. It repeated them.
-
-### 3. No iteration budget awareness
-
-The model had no visibility into how many iterations remained in the current run. It kept taking exploratory steps as if budget were unlimited, then was surprised by the wrap-up call. A model that knows it has 2 iterations left will consolidate and report; one that doesn't will keep digging.
-
-### 4. `perplexity_search` used without restraint
-
-The tool description had no guidance on usage limits. The model searched Perplexity 21 times in one session (11 in the final iteration), including redundant queries for the same version information. Each search consumed an iteration and added latency without improving outcomes.
-
-### 5. System prompt had no "give up" rule
-
-The system prompt told the agent to "decide whether to retry with a corrected call or explain the failure to the user" — but gave no threshold for when to stop retrying. In practice, the agent always chose to retry, regardless of how many times the same approach had failed.
-
----
-
-## Fixes
-
-### 1. Consecutive failure detection (`src/server/agent.js`)
-
-Added a `consecutiveFailures` counter that tracks back-to-back failed tool calls across all iterations within a run. A tool call counts as failed if `executeTool` throws (`toolStatus === 'error'`) _or_ the result object has `status === 'error'` (catching exec failures that are returned, not thrown). A successful call resets the counter to 0.
-
-After each iteration's tool calls, if `consecutiveFailures >= CONSECUTIVE_FAILURE_THRESHOLD` (3), a system break message is injected into the session and the counter resets:
-
-```js
-const resultObj = typeof result === 'object' && result !== null ? result : null;
-const toolFailed = toolStatus === 'error' || (resultObj && resultObj.status === 'error');
-if (toolFailed) {
-  consecutiveFailures++;
-} else {
-  consecutiveFailures = 0;
-}
-```
-
-```js
-if (consecutiveFailures >= CONSECUTIVE_FAILURE_THRESHOLD) {
-  session.messages.push({
-    role: 'user',
-    content: '[System: You have had 3 or more consecutive tool failures. Stop retrying the same approach. Either pivot to a fundamentally different strategy or provide your final response explaining what failed and why.]',
-  });
-  consecutiveFailures = 0;
-}
-```
-
-This complements the existing exact-match loop detector — together they cover both identical repetition and semantic failure loops.
-
-### 2. `failedApproaches` in checkpoint schema (`src/server/agent.js`)
-
-Updated `WRAP_UP_NOTE` to request a `failedApproaches` array alongside `progress` and `remaining`:
-
-```json
-{
-  "checkpoint": {
-    "progress": "...",
-    "remaining": "...",
-    "failedApproaches": ["downloading subfinder via curl from GitHub releases — connection reset", "..."]
-  }
-}
-```
-
-When resuming after a handoff, the agent loop now appends a system note to the resume message listing every failed approach from the previous run:
-
-```js
-let resumeContent = run.checkpoint.remaining || 'Continue with the task.';
-if (run.checkpoint.failedApproaches && run.checkpoint.failedApproaches.length > 0) {
-  resumeContent += `\n\n[System: The following approaches were tried and failed in the previous run — do not repeat them:\n${run.checkpoint.failedApproaches.map((a, i) => `${i + 1}. ${a}`).join('\n')}]`;
-}
-```
-
-The next run starts with concrete knowledge of what not to try, rather than repeating the same failed strategies.
-
-### 3. Iteration budget awareness (`src/server/agent.js`)
-
-Each model request now includes the remaining iteration count when 5 or fewer iterations are left in the run. The count is appended to the prepared messages (never stored in `session.messages`):
-
-```js
-const iterationsLeft = config.maxIterations - iteration + 1;
-const preparedMessages = iterationsLeft <= 5
-  ? [...base, { role: 'user', content: `[System: ${iterationsLeft} iteration${iterationsLeft === 1 ? '' : 's'} remaining in this run. Budget your remaining steps accordingly — if you cannot finish in time, consolidate progress and provide a checkpoint.]` }]
-  : base;
-```
-
-This gives the model enough warning to consolidate and produce a clean checkpoint rather than being cut off mid-task.
-
-### 4. `perplexity_search` description updated (`src/server/tools.js`)
-
-Added explicit usage guidance to the tool description:
-
-> Use sparingly — at most 3 searches per topic. Do not repeat the same query with minor variations; if an initial search does not yield what you need, switch to a different approach or verify locally with exec.
-
-### 5. "Failure Recovery" section added to system prompt (`docs/system-prompt.md`)
-
-Added a dedicated `## Failure Recovery` section making the give-up rule explicit:
-
-- Retry at most once with a meaningfully different approach; if it fails again, report to the user
-- Never repeat a failed strategy with minor variations
-- Use `perplexity_search` at most 3 times per topic
-- Escalate cleanly with a useful failure report rather than looping
-
----
-
-## What Was Not Changed
-
-- The existing exact-match loop detector (`loopTracker`) — it remains and now works alongside the new consecutive failure detector
-- The checkpoint/handoff system, `maxHandoffs` limit, or tool history strip logic — unchanged
-- The `format_error` recovery path (fallback model + nudge retry) — unchanged
-- No per-session Perplexity call counter was added server-side; the guidance is model-facing
-
----
-
-## Note: Homebrew as an Alternative for Binary Tool Installation
-
-During the session that exposed these issues, the agent struggled to install Nuclei, Subfinder, and Naabu from GitHub releases. All three tools from [ProjectDiscovery](https://projectdiscovery.io/) are available via Homebrew:
-
-```
-brew install nuclei
-brew install subfinder
-brew install naabu
-```
-
-Homebrew handles binary verification, PATH setup, and version management automatically — far more reliably than manual `curl` downloads or `go install`. On macOS (and Linux via Linuxbrew), this is the recommended installation method. The agent could discover this via `perplexity_search` or by checking `brew search projectdiscovery` — but only if it knows to try Homebrew _before_ attempting manual downloads.
-
-This is a guidance gap rather than a code bug: the system prompt doesn't mention package managers as a preferred strategy for binary installation. A future improvement could add: "When installing CLI tools, check for a package manager installation first (`brew install`, `apt install`, `snap install`) before attempting manual downloads."
-
----
-
-## Outcome
-
-- Failure loops are now intercepted after 3 consecutive failures instead of running to exhaustion
-- Handoff runs start with knowledge of what has already failed, enabling genuine strategy pivots
-- The model receives iteration budget warnings before hitting the wrap-up call
-- Perplexity search overuse is constrained by both tool description and system prompt guidance
-- The system prompt now has an explicit rule for when to stop retrying and report

package/docs/findings/005-installation-timeout.md

@@ -1,128 +0,0 @@
-# Finding 005: 60-Second Exec Timeout Breaks Package Installation
-
-**Date:** 2026-02-27
-**Severity:** High — any package installation via exec will timeout, regardless of which package manager is used
-**Status:** Fixed
-
----
-
-## What Happened
-
-In the session analysed in Finding 004, the agent attempted to install Nuclei, Subfinder, and Naabu using several strategies — `go install`, direct `curl` downloads, tarball extraction. All either timed out or failed with network errors.
-
-The natural follow-up question was: would switching to a proper package manager (`brew`, `apt-get`) solve the problem? The answer is no — not without fixing the timeout first.
-
----
-
-## Root Cause
-
-### The `exec` tool has a hard 60-second timeout
-
-```js
-// exec seed tool
-const { stdout, stderr } = await execAsync(args.cmd, {
-  encoding: 'utf8',
-  timeout: 60000,   // ← 60 seconds
-  maxBuffer: 2 * 1024 * 1024,
-});
-```
-
-On top of that, `executeTool` wraps every tool in its own `Promise.race` against `TOOL_TIMEOUT_MS = 60_000`. This means even if a tool's internal `execAsync` had a longer timeout, the outer race would kill it at 60 seconds anyway.
-
-Package installation routinely takes longer than 60 seconds:
-
-| Operation | Typical duration |
-|---|---|
-| `apt-get update` | 15–60s (varies with server speed, mirror load) |
-| `apt-get install nuclei` | 10–60s (binary download + extraction) |
-| `apt-get update && apt-get install nuclei` | 30–120s combined |
-| `brew install nuclei` | 20–90s |
-| `go install github.com/...` | 60–180s (compilation) |
-
-So `go install` was almost guaranteed to timeout. `apt-get` with an update step would regularly exceed 60s too. Swapping the install method without fixing the timeout would not have solved the problem.
-
-### The `npm_install` tool has the same problem
-
-`npm_install` also uses a 60-second timeout — it would fail for large packages or slow networks for the same reason.
-
-### No per-tool timeout mechanism existed
-
-All tools shared the same `TOOL_TIMEOUT_MS` constant. There was no way to declare that a specific tool legitimately needed more time.
-
----
-
-## Fix
-
-### 1. Per-tool timeout override (`src/server/tools.js`)
-
-`executeTool` now checks for a `timeout` property on the tool definition and uses it instead of the global `TOOL_TIMEOUT_MS`:
-
-```js
-const timeoutMs = tool.timeout || TOOL_TIMEOUT_MS;
-
-const timeout = new Promise((_, reject) =>
-  setTimeout(
-    () => reject(new Error(`Tool '${name}' timed out after ${timeoutMs / 1000}s`)),
-    timeoutMs
-  )
-);
-```
-
-Tools that don't declare a timeout continue to use the 60s default — no behaviour change for existing tools.
-
-### 2. `system_install` seed tool with 5-minute timeout (`src/server/tools.js`)
-
-A new built-in tool handles system binary installation:
-
-```js
-system_install: {
-  timeout: 300_000, // 5 minutes
-  ...
-}
-```
-
-Behaviour:
-- **Already installed check**: runs `which <package>` first. If the binary is already on PATH, returns immediately without installing — no wasted time.
-- **Auto-detection**: tries `brew`, `apt-get`, `snap` in order. Uses the first one found. Can be overridden with an explicit `packageManager` argument.
-- **apt-get**: always runs `apt-get update -qq` before `apt-get install -y` to avoid stale package list failures. Uses `DEBIAN_FRONTEND=noninteractive` to suppress interactive prompts.
-- **Inner timeout**: 4.5 minutes on the internal `execAsync`, leaving headroom before the outer 5-minute tool timeout fires.
-- **Structured errors**: returns `exitCode`, `stdout`, `stderr` on failure so the agent can read the actual error without guessing.
-
-### 3. System prompt updated (`docs/system-prompt.md`)
-
-Added explicit guidance in the "Tool Creation" section:
-
-> **Installing a system binary** (e.g. nuclei, jq, ffmpeg, git): use the `system_install` tool — never use exec for this. It auto-detects the available package manager (brew/apt-get/snap) and has a 5-minute timeout sized for real downloads.
-
-This mirrors the existing `npm_install` guidance and gives the model an explicit directive to reach for `system_install` over `exec`.
-
----
-
-## Why `system_install` Instead of Increasing the Global Timeout
-
-Increasing `TOOL_TIMEOUT_MS` globally would make every tool — including `exec` — hang for much longer before failing. A runaway `exec` command (e.g. `find / -name "*.js"`) that produces no output would block the event loop for 5 minutes instead of 60 seconds. The per-tool timeout keeps the safe default for general tools while letting installation tools declare a legitimate need for more time.
-
----
-
-## Notes on Package Availability
-
-Not all tools are available in every package manager. For ProjectDiscovery tools (nuclei, subfinder, naabu) specifically:
-
-- **macOS (brew)**: `brew install nuclei`, `brew install subfinder`, `brew install naabu` — all available via the official Homebrew formulae
-- **Linux (apt-get)**: ProjectDiscovery does not maintain an official apt repository. `apt-get install nuclei` will likely fail with "package not found"
-- **Linux (snap)**: `snap install nuclei` is available on Ubuntu/Debian
-
-On Linux, if `system_install` fails because the package isn't in the apt repository, the agent should try snap, or fall back to the ProjectDiscovery install script:
-```sh
-curl -sL https://github.com/projectdiscovery/nuclei/releases/latest/download/nuclei_linux_amd64.zip -o nuclei.zip
-```
-This would still use `exec` for the download, but with the failure recovery guidance from Finding 004 the agent should report the failure rather than looping.
-
----
-
-## Outcome
-
-- Package installation no longer races against a 60-second wall
-- The agent has a clear, named tool to reach for when installing system binaries
-- The system prompt actively directs the agent away from `exec` for this use case
-- Per-tool timeouts are now supported for any future tools that need them

package/docs/findings/006-malformed-tool-schema.md

@@ -1,118 +0,0 @@
-# Finding 006: Malformed Tool Schema Poisons Every Subsequent Request
-
-**Date:** 2026-02-27
-**Severity:** High — permanently breaks all model calls for the session until tools.json is manually repaired
-**Status:** Fixed
-
----
-
-## What Happened
-
-During a session installing security tools (Nuclei, Subfinder, Naabu), the agent called `save_tool` to create a custom `scan` tool. The model passed the `parameters` field as a **JSON-serialized string** instead of a JSON object:
-
-```json
-"parameters": "{\"type\":\"object\",\"properties\":{\"domain\":{\"type\":\"string\",...}}}"
-```
-
-`save_tool` stored this verbatim — no validation occurred. The malformed tool definition was written to `~/.jarvis/data/tools/tools.json` as tool index 12.
-
-On the very next model call (within the same run), Jarvis reloaded tools after `save_tool` completed (`toolsModified = true`) and sent all tool definitions to the provider. OpenRouter's provider API returned:
-
-```
-400 Provider returned error
-[{'type': 'dict_type', 'loc': ('body', 'tools', 12, 'function', 'parameters'),
-  'msg': 'Input should be a valid dictionary', 'input': '{"type":"object",...}'}]
-```
-
-Every subsequent user message — including trivial ones like "Was ist schief gegangen?" and "Wie ist deine session id" — also failed with the same 400. The malformed tool was permanently in `tools.json` and included in every model request.
-
----
-
-## Root Cause
-
-### 1. `save_tool` did not validate `parameters`
-
-The `save_tool` code stored `args.parameters` directly into `tools.json` without checking its type. The OpenAI tool-calling spec requires `parameters` to be a JSON Schema object (a dictionary). When a model passes a JSON string instead of an object — a common mistake with weaker models — the result is a permanently malformed tool definition.
-
-### 2. `getToolDefinitions` sent all tools to the provider without validation
-
-`getToolDefinitions` returned all tool definitions unconditionally. A single malformed tool poisoned every request that included the tools list — which is every request, since tool definitions are always sent.
-
-These two gaps compound each other: the first allows bad data in, the second ensures it breaks everything downstream.
-
----
-
-## Why it Persists Across All Subsequent Messages
-
-Every `handleChat` call loads tools fresh from `tools.json` via `loadTools()`. The malformed `scan` tool is always in the list. Every model call sends all tool definitions. The provider rejects every request. The session is stuck until `tools.json` is manually repaired.
-
----
-
-## Intermittent Behavior
-
-The failure is not always immediate. In the debugging session, two tool calls succeeded in later runs before the 400 fired. This is consistent with free/preview model providers (nvidia via OpenRouter) applying schema validation inconsistently across backend instances. The bug is therefore not "always broken" but **reliably broken under load** — which is harder to detect and debug than a consistent failure.
-
----
-
-## Fix
-
-Two targeted changes to `src/server/tools.js`.
-
-### 1. `save_tool` validates and auto-corrects `parameters`
-
-Before writing to `tools.json`, the `save_tool` code now:
-- If `parameters` is a string, attempts `JSON.parse()` — models commonly serialize the object when they should pass it directly. If parsing succeeds and yields an object, the corrected value is used silently.
-- If `parameters` is still not a plain object after the attempted parse, returns an error immediately with a clear message. Nothing is written to disk.
-
-```js
-let parameters = args.parameters;
-if (typeof parameters === 'string') {
-  try {
-    parameters = JSON.parse(parameters);
-  } catch {
-    return { status: 'error', error: 'parameters must be a JSON Schema object, not a string. Pass the object directly, not as a JSON-serialized string.' };
-  }
-}
-if (typeof parameters !== 'object' || parameters === null || Array.isArray(parameters)) {
-  return { status: 'error', error: 'parameters must be a JSON Schema object (e.g. { type: "object", properties: {...} }).' };
-}
-```
-
-### 2. `getToolDefinitions` filters out malformed tools
-
-`getToolDefinitions` now validates `parameters` on each tool before including it in the definitions sent to the provider. A tool with a non-object `parameters` is skipped with a `console.warn`, not thrown — this is a defence-in-depth guard, not a primary error path.
-
-```js
-export function getToolDefinitions(tools) {
-  const defs = [];
-  for (const [name, t] of Object.entries(tools)) {
-    const params = t.definition?.function?.parameters;
-    if (typeof params !== 'object' || params === null || Array.isArray(params)) {
-      console.warn(`[tools] Skipping tool '${name}': parameters is not a valid object (got ${typeof params})`);
-      continue;
-    }
-    defs.push(t.definition);
-  }
-  return defs;
-}
-```
-
-Together: Fix 1 prevents malformed tools from entering `tools.json`. Fix 2 ensures that even if a malformed tool somehow ends up in `tools.json` (e.g. from an older version, manual edit, or a bug that slips through), it is silently excluded from every model request rather than poisoning the session.
-
----
-
-## Secondary Issue: Agent Hallucinated Successful Action
-
-In the run that preceded the `save_tool` call, the agent responded with:
-
-> "The scanning script has been created at /root/.jarvis/projects/cybersecurity/scan.sh."
-
-No such file was ever created — the agent had only installed Nuclei successfully. This hallucination forced the next run to attempt recovery via `save_tool`, which is where the malformed tool was introduced. The hallucination itself is a model-quality issue with the free nvidia model, not a Jarvis bug.
-
----
-
-## Outcome
-
-- `save_tool` auto-corrects the common case (string instead of object) and rejects the rest with a clear error before writing to disk
-- A pre-existing malformed tool in `tools.json` no longer poisons model requests — it is silently skipped per call
-- Sessions are no longer permanently broken by a single bad `save_tool` call