@ducci/jarvis 1.0.22 → 1.0.23

package/docs/findings/006-malformed-tool-schema.md

@@ -0,0 +1,118 @@
+# Finding 006: Malformed Tool Schema Poisons Every Subsequent Request
+
+**Date:** 2026-02-27
+**Severity:** High — permanently breaks all model calls for the session until tools.json is manually repaired
+**Status:** Fixed
+
+---
+
+## What Happened
+
+During a session installing security tools (Nuclei, Subfinder, Naabu), the agent called `save_tool` to create a custom `scan` tool. The model passed the `parameters` field as a **JSON-serialized string** instead of a JSON object:
+
+```json
+"parameters": "{\"type\":\"object\",\"properties\":{\"domain\":{\"type\":\"string\",...}}}"
+```
+
+`save_tool` stored this verbatim — no validation occurred. The malformed tool definition was written to `~/.jarvis/data/tools/tools.json` as tool index 12.
+
+On the very next model call (within the same run), Jarvis reloaded tools after `save_tool` completed (`toolsModified = true`) and sent all tool definitions to the provider. OpenRouter's provider API returned:
+
+```
+400 Provider returned error
+[{'type': 'dict_type', 'loc': ('body', 'tools', 12, 'function', 'parameters'),
+  'msg': 'Input should be a valid dictionary', 'input': '{"type":"object",...}'}]
+```
+
+Every subsequent user message — including trivial ones like "Was ist schief gegangen?" and "Wie ist deine session id" — also failed with the same 400. The malformed tool was permanently in `tools.json` and included in every model request.
+
+---
+
+## Root Cause
+
+### 1. `save_tool` did not validate `parameters`
+
+The `save_tool` code stored `args.parameters` directly into `tools.json` without checking its type. The OpenAI tool-calling spec requires `parameters` to be a JSON Schema object (a dictionary). When a model passes a JSON string instead of an object — a common mistake with weaker models — the result is a permanently malformed tool definition.
+
+### 2. `getToolDefinitions` sent all tools to the provider without validation
+
+`getToolDefinitions` returned all tool definitions unconditionally. A single malformed tool poisoned every request that included the tools list — which is every request, since tool definitions are always sent.
+
+These two gaps compound each other: the first allows bad data in, the second ensures it breaks everything downstream.
+
+---
+
+## Why it Persists Across All Subsequent Messages
+
+Every `handleChat` call loads tools fresh from `tools.json` via `loadTools()`. The malformed `scan` tool is always in the list. Every model call sends all tool definitions. The provider rejects every request. The session is stuck until `tools.json` is manually repaired.
+
+---
+
+## Intermittent Behavior
+
+The failure is not always immediate. In the debugging session, two tool calls succeeded in later runs before the 400 fired. This is consistent with free/preview model providers (nvidia via OpenRouter) applying schema validation inconsistently across backend instances. The bug is therefore not "always broken" but **reliably broken under load** — which is harder to detect and debug than a consistent failure.
+
+---
+
+## Fix
+
+Two targeted changes to `src/server/tools.js`.
+
+### 1. `save_tool` validates and auto-corrects `parameters`
+
+Before writing to `tools.json`, the `save_tool` code now:
+- If `parameters` is a string, attempts `JSON.parse()` — models commonly serialize the object when they should pass it directly. If parsing succeeds and yields an object, the corrected value is used silently.
+- If `parameters` is still not a plain object after the attempted parse, returns an error immediately with a clear message. Nothing is written to disk.
+
+```js
+let parameters = args.parameters;
+if (typeof parameters === 'string') {
+  try {
+    parameters = JSON.parse(parameters);
+  } catch {
+    return { status: 'error', error: 'parameters must be a JSON Schema object, not a string. Pass the object directly, not as a JSON-serialized string.' };
+  }
+}
+if (typeof parameters !== 'object' || parameters === null || Array.isArray(parameters)) {
+  return { status: 'error', error: 'parameters must be a JSON Schema object (e.g. { type: "object", properties: {...} }).' };
+}
+```
+
+### 2. `getToolDefinitions` filters out malformed tools
+
+`getToolDefinitions` now validates `parameters` on each tool before including it in the definitions sent to the provider. A tool with a non-object `parameters` is skipped with a `console.warn`, not thrown — this is a defence-in-depth guard, not a primary error path.
+
+```js
+export function getToolDefinitions(tools) {
+  const defs = [];
+  for (const [name, t] of Object.entries(tools)) {
+    const params = t.definition?.function?.parameters;
+    if (typeof params !== 'object' || params === null || Array.isArray(params)) {
+      console.warn(`[tools] Skipping tool '${name}': parameters is not a valid object (got ${typeof params})`);
+      continue;
+    }
+    defs.push(t.definition);
+  }
+  return defs;
+}
+```
+
+Together: Fix 1 prevents malformed tools from entering `tools.json`. Fix 2 ensures that even if a malformed tool somehow ends up in `tools.json` (e.g. from an older version, manual edit, or a bug that slips through), it is silently excluded from every model request rather than poisoning the session.
+
+---
+
+## Secondary Issue: Agent Hallucinated Successful Action
+
+In the run that preceded the `save_tool` call, the agent responded with:
+
+> "The scanning script has been created at /root/.jarvis/projects/cybersecurity/scan.sh."
+
+No such file was ever created — the agent had only installed Nuclei successfully. This hallucination forced the next run to attempt recovery via `save_tool`, which is where the malformed tool was introduced. The hallucination itself is a model-quality issue with the free nvidia model, not a Jarvis bug.
+
+---
+
+## Outcome
+
+- `save_tool` auto-corrects the common case (string instead of object) and rejects the rest with a clear error before writing to disk
+- A pre-existing malformed tool in `tools.json` no longer poisons model requests — it is silently skipped per call
+- Sessions are no longer permanently broken by a single bad `save_tool` call

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ducci/jarvis",
-  "version": "1.0.22",
+  "version": "1.0.23",
   "description": "A fully automated agent system that lives on a server.",
   "main": "./src/index.js",
   "type": "module",

package/src/server/tools.js

@@ -149,7 +149,7 @@ const SEED_TOOLS = {
         },
       },
     },
-    code: `const toolsFile = path.join(process.env.HOME, '.jarvis/data/tools/tools.json'); const raw = await fs.promises.readFile(toolsFile, 'utf8').catch(() => '{}'); const tools = JSON.parse(raw); tools[args.name] = { definition: { type: 'function', function: { name: args.name, description: args.description, parameters: args.parameters } }, code: args.code }; await fs.promises.writeFile(toolsFile, JSON.stringify(tools, null, 2), 'utf8'); return { status: 'ok', saved: args.name };`,
+    code: `const toolsFile = path.join(process.env.HOME, '.jarvis/data/tools/tools.json'); const raw = await fs.promises.readFile(toolsFile, 'utf8').catch(() => '{}'); const tools = JSON.parse(raw); let parameters = args.parameters; if (typeof parameters === 'string') { try { parameters = JSON.parse(parameters); } catch { return { status: 'error', error: 'parameters must be a JSON Schema object, not a string. Pass the object directly, not as a JSON-serialized string.' }; } } if (typeof parameters !== 'object' || parameters === null || Array.isArray(parameters)) { return { status: 'error', error: 'parameters must be a JSON Schema object (e.g. { type: "object", properties: {...} }).' }; } tools[args.name] = { definition: { type: 'function', function: { name: args.name, description: args.description, parameters } }, code: args.code }; await fs.promises.writeFile(toolsFile, JSON.stringify(tools, null, 2), 'utf8'); return { status: 'ok', saved: args.name };`,
   },
   get_tool: {
     definition: {
@@ -424,7 +424,16 @@ export async function loadTools() {
 }
 
 export function getToolDefinitions(tools) {
-  return Object.values(tools).map(t => t.definition);
+  const defs = [];
+  for (const [name, t] of Object.entries(tools)) {
+    const params = t.definition?.function?.parameters;
+    if (typeof params !== 'object' || params === null || Array.isArray(params)) {
+      console.warn(`[tools] Skipping tool '${name}': parameters is not a valid object (got ${typeof params})`);
+      continue;
+    }
+    defs.push(t.definition);
+  }
+  return defs;
 }
 
 export async function executeTool(tools, name, toolArgs) {