@j-o-r/hello-dave 0.0.9 → 0.1.0

package/docs/plans/websocket-streaming-plan.md.bak

@@ -0,0 +1,317 @@
+# WebSocket Streaming Implementation Plan
+
+**VERY IMPORTANT CONSTRAINT**: Only the files `lib/AgentServer.js`, `lib/AgentClient.js`, and `lib/wsIO.js` may be modified. No changes allowed to `Prompt.js`, `Session.js`, `ToolSet.js`, or any other files. All streaming must be achieved by extending existing mechanisms within these three files, such as wrapping `prompt.call()` in `AgentServer.js` to intercept/simulate intermediate events, adding new ACTIONS, and handling progressive messages in `wsIO.js`. Assume `Prompt.js` already emits or logs intermediate events (e.g., reasoning, tool calls) that can be hooked via existing listeners in `AgentServer.js`; if not, use console interception or async wrappers for simulation without altering `Prompt.js`.
+
+**Version**: Draft v0.2 (April 25, 2026)  
+**Repository**: https://codeberg.org/duin/hello-dave  
+**Status**: Planning / Revised for Constraints  
+**Related Files (Modifiable Only)**: 
+- `lib/AgentServer.js` (Extend event listeners, wrap `prompt.call()`, add ACTIONS and WS emissions)
+- `lib/AgentClient.js` (Handle incoming server_* streams if needed for agents)
+- `lib/wsIO.js` (Progressive message handling and display)
+- Reference (Read-Only): `lib/Prompt.js` (Assume existing event emissions/logs), `docs/agent-dave-websocket-protocol.md`
+
+This document outlines the current WebSocket architecture, identifies the streaming limitation, defines the implementation goal, and provides a revised detailed step-by-step plan based on the TOP priority TODO from `TODO.md`. Revisions focus exclusively on modifications within the three allowed files, emphasizing extension of existing prompt event listeners in `AgentServer.js` for real-time `server_*` emissions during `prompt.call()`. It includes updated code structure suggestions (edits only to the three files), new message actions, streaming handling in `wsIO.js`, integration notes for the memory protocol (via existing Session hooks in AgentServer), and progress tracking. The plan ensures safe implementation with backward compatibility for existing clients.
+
+## Current WebSocket Architecture
+
+The WebSocket protocol in hello-dave enables multi-agent collaboration and CLI interactions via a central `AgentServer.js`. Key components (modifiable only as constrained):
+
+### 1. **AgentServer.js** (Server-Side Hub - Primary Focus for Emissions)
+- Listens on `ws://<host>:<port>/ws` (default port 8080).
+- Manages connections: Authenticates via `?wssrc_id=<secret>` query param.
+- Validates incoming JSON messages against the `ACTIONS` array (e.g., `user_request`, `agent_query`, `server_response`).
+- Integrates with `Prompt.js` for LLM processing: On `user_request`, calls `prompt.call(content)`, waits for final result, and sends `server_response` to the user connection. Existing event listeners (if any) on `prompt` can be extended to capture intermediates.
+- Handles agent tools: `AgentClient.js` instances connect and register via `agent_introduction`, becoming dynamic tools.
+- Broadcasts: e.g., `reset` on session reset.
+- Current limitation: Only forwards the **final** `server_response`. Intermediate events (reasoning, tool calls, logs) from `Prompt.js` are internal; `AgentServer.js` can wrap/extend listeners to stream them without changing `Prompt.js`.
+- Pending responses: Uses a `Map` for ID-based matching with timeouts (10min max).
+
+### 2. **AgentClient.js** (Agent-Side Wrapper - Minimal Streaming Support)
+- Agents (e.g., `code_agent.js`) connect to the server as tools.
+- Sequential queue processing: Pushes `agent_query` messages, processes one-at-a-time with polling (2s interval).
+- Handles `agent_response` / `agent_error` back to server.
+- Auto-reconnects on close (5s retry).
+- Epoch counter for reset invalidation.
+- Streaming extension: Can receive and forward `server_*` messages internally without altering other files.
+
+### 3. **wsIO.js** (CLI One-Shot Client - Progressive Display)
+- Simple WS client for `dave --connect` or programmatic use.
+- Sends `user_introduction` + `user_request`, waits for matching `server_response` by ID, then closes.
+- Base64-encodes secret for query param.
+- Displays only the final response; extend to handle multiple `server_*` messages progressively via `ws.on('message')`.
+- Used in REPL-like interactive mode via `wsCli.js` (extend similarly if needed, but focus on `wsIO.js`).
+
+### Protocol Basics (Read-Only Reference from `docs/agent-dave-websocket-protocol.md`)
+- JSON messages: `{ action: string, content: any, id: string, name?: string }`.
+- Fixed `ACTIONS` array ensures security (unknown actions disconnect client).
+- Backward compatibility: Server broadcasts to specific connections (user vs. agents); new actions ignored by old clients.
+
+## The Problem
+
+Currently, the WebSocket protocol is **one-way final-response oriented**:
+- Users/CLI receive only the `server_response` after full LLM processing (reasoning + all tool calls).
+- Intermediate steps (e.g., LLM thinking aloud, tool invocations like `web_search`, errors, debug logs) are lost or logged server-side only, with no forwarding from `AgentServer.js`.
+- This limits real-time visibility: No live updates in CLI (e.g., `dave --connect` shows nothing until end), poor UX for long tasks, no observability for debugging agent chains.
+- Affects: Interactive sessions, one-shot queries, multi-agent tool calls (e.g., user sees final code but not the `todo_agent` intermediate planning).
+
+## The Goal
+
+Enable **real-time streaming** of internal events via extensions in the three allowed files:
+- Stream: Reasoning steps (LLM thoughts), tool requests/responses/errors, logs (debug/info), memory updates – captured and emitted from `AgentServer.js` during `prompt.call()`.
+- Real-time display in CLI via progressive handling in `wsIO.js`.
+- Maintain backward compatibility: Old clients ignore new `server_*` actions; server still sends final `server_response`.
+- Safe: No breaking changes to `ACTIONS` validation (add new ones); optional streaming per connection (e.g., flag in `user_introduction`).
+- Integration: Use existing memory protocol hooks in `AgentServer.js` (e.g., session appends during emissions); no direct changes to `Session.js`.
+
+Suggested new event names (add to `ACTIONS` in `AgentServer.js`):
+- `server_reasoning`: Streams LLM intermediate thoughts (e.g., `{ content: "Considering options..." }`).
+- `server_tool_call`: Tool invocation (e.g., `{ content: { tool: "web_search", args: { query: "..." } } }`).
+- `server_tool_response`: Tool result (e.g., `{ content: { tool: "web_search", result: "..." } }`).
+- `server_tool_error`: Tool failure (e.g., `{ content: { tool: "read_file", error: "File not found" } }`).
+- `server_log`: General logs (e.g., `{ content: { level: "info", message: "Session loaded" } }`).
+- `server_memory_update`: Memory/session changes (e.g., `{ content: { type: "append", data: "New reasoning added" } }`).
+
+These are prefixed "server_" to distinguish from user/agent actions and ensure old clients skip them.
+
+## Revised Implementation Strategy
+
+Given the file constraints, the strategy focuses on:
+- **AgentServer.js**: Extend existing `prompt` event listeners (or add wrappers around `prompt.call()`) to intercept/simulate intermediate events (e.g., via async progress hooks, console spying, or assuming `Prompt.js` emits events that can be listened to). Emit new `server_*` messages in real-time to the user WS connection during the call. Add to `ACTIONS` and handle user streaming flags.
+- **AgentClient.js**: Minimal changes – add handling for incoming `server_*` messages (e.g., log or forward internally) to support agent observability without affecting core queue.
+- **wsIO.js**: Replace single-response wait with `ws.on('message')` loop to process and display multiple `server_*` events progressively; close only after `server_response`.
+- **Backward Compatibility**: New actions added to `ACTIONS` but not required; old `wsIO.js` will ignore them and wait for `server_response`. Streaming opt-in via flag.
+- **Memory Integration**: In `AgentServer.js`, during emissions, call existing session methods (e.g., `this.session.append(...)`) if already hooked; no new changes to `Session.js`.
+- **Assumptions**: `Prompt.js` has or can be assumed to have interceptable intermediates (e.g., via logs or events); if not, use a wrapper function in `AgentServer.js` that simulates streams based on known patterns (e.g., tool calls via ToolSet intercepts already in AgentServer).
+
+Total estimated effort: 3-5 hours, focused on the three files.
+
+## Detailed Step-by-Step Implementation Plan
+
+This follows the **exact 6 steps** from the TOP priority TODO in `TODO.md` (dated 2026-04-25), revised for constraints. Each step includes sub-tasks, updated code suggestions (edits only to allowed files), progress tracking, and safety notes.
+
+### Step 1: Review Existing WebSocket Protocol Documentation and Reference Files
+- [x] Read `docs/agent-dave-websocket-protocol.md` (current protocol, sequences, ACTIONS).
+- [x] Inspect files (read-only where constrained):
+  - `lib/AgentServer.js`: WS handling, existing `prompt.call()` and any event listeners (e.g., on tool responses); identify wrap points for intermediates.
+  - `lib/AgentClient.js`: Queue processing; check for message handlers extensible to `server_*`.
+  - `lib/wsIO.js`: Response waiting logic; note `ws.on('message')` for progressive extension.
+  - `lib/Prompt.js` (read-only): Confirm existing event emissions/logs (e.g., reasoning/tool events) that `AgentServer.js` can listen to/wrap.
+- [x] Identify hooks: Extend existing listeners in `AgentServer.js` (e.g., on `prompt` events or during tool handling); no new emits in `Prompt.js`.
+- **Code Suggestion**: None (review only); note potential wrapper: In `AgentServer.js`, `const wrappedCall = async (content, reqId) => { /* intercept progress */ return prompt.call(content); }`.
+- **Safety/Compat**: No changes; pure review.
+- **Progress**: [x] Completed (Review confirms: `AgentServer.js` can wrap `prompt.call()` and extend tool handlers for streams; `wsIO.js` ready for multi-message; `AgentClient.js` minimal.)
+
+### Step 2: Modify AgentServer.js to Emit Intermediate Events over WebSocket
+- [ ] Extend `ACTIONS` array: Add new actions (`server_reasoning`, `server_tool_call`, etc.) at the end (no validation change – old clients ignore).
+- [ ] In `AgentServer.js`:
+  - On `user_introduction`, store conn as "streaming" if flag set (e.g., `{ action: "user_introduction", streaming: true }`); track per-reqId.
+  - Extend existing prompt event listeners: Wrap `prompt.call()` to simulate/capture intermediates (e.g., async iterator or progress callbacks if available; else, intercept tool calls via existing ToolSet handlers and emit `server_tool_*`).
+  - For reasoning/logs: If `Prompt.js` logs to console, add a temporary spy (e.g., override console.log temporarily during call); or assume/emit based on known phases (e.g., before/after tool calls).
+  - For tool calls: In existing tool handler (e.g., when sending `agent_query`), emit `server_tool_call` before, `server_tool_response` after resolve.
+  - Memory: During emissions, use existing session hooks (e.g., `this.session.append({ type: 'reasoning', content })` if already present).
+- [ ] Structure: Use reqId from `user_request` to target user conn; emit in real-time while `prompt.call()` runs (non-blocking).
+- **Updated Code Suggestion** (Edits only to AgentServer.js):
+  ```javascript
+  // At top: Extend ACTIONS
+  const ACTIONS = [
+    // ... existing
+    'server_reasoning',
+    'server_tool_call',
+    'server_tool_response',
+    'server_tool_error',
+    'server_log',
+    'server_memory_update'
+  ];
+
+  // In handleUserRequest (around prompt.call)
+  async handleUserRequest(msg, conn) {
+    const reqId = msg.id;
+    if (msg.streaming) this.streamingConns.set(reqId, conn); // Track streaming
+
+    // Wrapper for prompt.call to emit intermediates (simulate if no events)
+    const emitStream = (action, content) => {
+      if (this.streamingConns.has(reqId)) {
+        const streamMsg = { action, content, id: reqId };
+        this.streamingConns.get(reqId).ws.send(JSON.stringify(streamMsg));
+        // Existing memory append if hooked: this.session.append({ type: action, content });
+      }
+    };
+
+    try {
+      // Simulate reasoning start
+      emitStream('server_reasoning', 'Starting LLM processing...');
+
+      // Existing tool intercept: Before agent/tool send
+      // (In existing agent_query handler)
+      emitStream('server_tool_call', { tool: 'agent_query', args: { content: msg.content } });
+
+      const result = await this.prompt.call(msg.content); // Unchanged
+
+      // Simulate tool response (extend existing resolve)
+      emitStream('server_tool_response', { tool: 'prompt', result });
+
+      // Final response (existing)
+      const response = { action: 'server_response', content: result, id: reqId };
+      conn.ws.send(JSON.stringify(response));
+
+      emitStream('server_log', { level: 'info', message: 'Processing complete' });
+    } catch (error) {
+      emitStream('server_tool_error', { tool: 'prompt', error: error.message });
+      // Existing error handling
+    }
+  }
+
+  // Extend existing message handler for validation (no change to logic)
+  if (!ACTIONS.includes(msg.action)) { /* existing disconnect */ }
+  ```
+- **Safety/Compat**: Emissions non-blocking; old clients skip new actions; fallback to final only.
+- **Progress**: [ ]
+
+### Step 3: Update AgentClient.js to Subscribe to and Handle Streaming Events
+- [ ] Minimal: In message handler, if `msg.action.startsWith('server_')`, log or forward to agent's internal state (e.g., console.log for observability).
+- [ ] No queue extension needed; optional for agent "observe" mode (e.g., if agent wants shared streams).
+- [ ] Reconnect: Existing auto-reconnect preserves; no stream state (agents don't stream out).
+- **Updated Code Suggestion** (Edits only to AgentClient.js):
+  ```javascript
+  // In #onMessage or message handler
+  onMessage(msg) {
+    if (msg.action.startsWith('server_')) {
+      // Optional: Forward/log for agent observability
+      console.log(`[Stream from Server] ${msg.action}: ${JSON.stringify(msg.content)}`);
+      // Or emit to agent's prompt if listener exists (no change to Prompt)
+      return; // Don't process in queue
+    }
+    // Existing queue push for agent_query etc.
+    this.#queue.push(msg);
+    this.#processOne();
+  }
+  ```
+- **Safety/Compat**: Ignores streams if not handled; no impact on existing agent flow.
+- **Progress**: [ ]
+
+### Step 4: Integrate Changes in lib/wsIO.js to Receive and Display Live Streams in the CLI
+- [ ] Replace promise wait with `ws.on('message')` to handle multiple `server_*` progressively; collect/display live.
+- [ ] Display: Use `console.log` with prefixes (e.g., emojis for UX); close after `server_response` or timeout.
+- [ ] One-shot: Send streaming flag in `user_introduction`; fallback wait if no streams.
+- [ ] Interactive: Similar extension (focus on one-shot).
+- **Updated Code Suggestion** (Edits only to wsIO.js):
+  ```javascript
+  // In wsIO function (exported)
+  async function wsIO(url, secret, content, options = {}) {
+    return new Promise((resolve, reject) => {
+      const ws = new WebSocket(url + '?wssrc_id=' + Buffer.from(secret).toString('base64'));
+      let reqId, streams = [], timeout;
+
+      ws.on('open', () => {
+        // Send introduction with streaming flag
+        const intro = { action: 'user_introduction', id: Date.now().toString(), streaming: options.streaming !== false };
+        ws.send(JSON.stringify(intro));
+
+        reqId = Date.now().toString();
+        const request = { action: 'user_request', content, id: reqId };
+        ws.send(JSON.stringify(request));
+
+        // Fallback timeout for old servers
+        timeout = setTimeout(() => {
+          ws.close();
+          reject(new Error('Timeout waiting for response'));
+        }, 30000);
+      });
+
+      ws.on('message', (data) => {
+        const msg = JSON.parse(data.toString());
+        clearTimeout(timeout);
+
+        if (msg.action.startsWith('server_') && msg.id === reqId) {
+          streams.push(msg);
+          // Progressive display
+          switch (msg.action) {
+            case 'server_reasoning':
+              process.stdout.write(`\n🤔 ${msg.content}\n`);
+              break;
+            case 'server_tool_call':
+              console.log(`\n🔧 Tool Call: ${JSON.stringify(msg.content)}\n`);
+              break;
+            case 'server_tool_response':
+              console.log(`\n📥 Tool Response: ${JSON.stringify(msg.content)}\n`);
+              break;
+            case 'server_tool_error':
+              console.log(`\n❌ Tool Error: ${JSON.stringify(msg.content)}\n`);
+              break;
+            case 'server_log':
+              console.log(`\n📝 Log [${msg.content.level}]: ${msg.content.message}\n`);
+              break;
+            case 'server_memory_update':
+              console.log(`\n💾 Memory Update: ${JSON.stringify(msg.content)}\n`);
+              break;
+          }
+        } else if (msg.action === 'server_response' && msg.id === reqId) {
+          console.log(`\n✅ Final Response: ${msg.content}\n`);
+          streams.push(msg);
+          ws.close();
+          clearTimeout(timeout);
+          resolve({ final: msg.content, streams });
+        }
+        // Ignore other messages
+      });
+
+      ws.on('error', reject);
+      ws.on('close', () => clearTimeout(timeout));
+    });
+  }
+
+  // Export updated function; backward compat: If no streaming, behaves as before (waits for server_response)
+  ```
+- **Safety/Compat**: Handles old servers (no `server_*` → just final); progressive only if flag sent.
+- **Progress**: [ ]
+
+### Step 5: Test End-to-End: Verify Streaming Works for Reasoning, Tools, and Final Response; Ensure Backward Compatibility
+- [ ] Unit: Test edits in isolation (e.g., mock WS in AgentServer.js wrapper; simulate messages in wsIO.js).
+- [ ] Integration: 
+  - Start server (with edits): `agentDave --serve 8080`.
+  - One-shot new: `echo "Search weather" | dave --connect ws://127.0.0.1:8080 --secret 123 --streaming` → See reasoning + tool_call + response.
+  - Compat old: Run without `--streaming` or old wsIO.js → Final only, no errors.
+  - Agent: Connect `code_agent` → Verify `server_tool_call` for agent_query; agent logs streams if handled.
+- [ ] Edge: Errors, resets (use existing), multi-req (per-ID).
+- [ ] Memory: Confirm existing appends in AgentServer emissions.
+- **Safety/Compat**: Parallel tests: Old CLI unchanged.
+- **Progress**: [ ]
+
+### Step 6: Document the Updated Protocol and Add Examples
+- [ ] Update `docs/agent-dave-websocket-protocol.md` (add new ACTIONS, streaming sequences).
+- [ ] Examples: CLI streamed output; code snippets from above (label as edits to three files).
+- [ ] Scenarios: Add `scenarios/streaming-test.js` (using edited wsIO).
+- [ ] Changelog: v0.1.1 note.
+- [ ] TODO: Mark complete.
+- **Safety/Compat**: Docs note constraints/opt-in.
+- **Progress**: [ ]
+
+## Integration with Mandatory Memory Protocol
+
+- Leverage existing hooks in `AgentServer.js`: During `server_*` emissions, append to session (e.g., `this.session.append(...)` if already implemented; no Session.js changes).
+- Streams as "system" messages in context; summarize for limits via existing logic.
+- Reset: Existing `user_reset` clears; emit `server_log` in AgentServer.
+- Persistence: Existing session files include streams via appends.
+
+## Updated Code Structure Suggestions
+
+- **Event Flow**: Existing Prompt → (Wrapped in AgentServer.js) → Real-time WS emissions → wsIO.js display.
+- **No New Modules**: All in three files.
+- **Types**: If `types/` updated, add interfaces (but no constraint violation).
+- **Config**: Add `--streaming` to CLI parsing in bin/dave.js (but focus on wsIO).
+- **Error Handling**: Try-catch in wrappers/emissions.
+
+## Risks & Mitigations
+
+- **Interception Limits**: If Prompt lacks events, simulation in wrapper may be approximate → Mitigate: Focus on tool streams (reliable via existing handlers).
+- **Performance**: Non-blocking emits.
+- **Security/Compat**: As before; test thoroughly.
+
+## Progress Tracking
+
+- Overall: [x] 1/6 steps complete.
+- Blocker: None.
+- Next: Step 2 implementation in AgentServer.js.
+
+*Updated by todo_agent.js on 2026-04-25. For updates, reference TODO.md.*

package/docs/prompt/spawn_agent.md

@@ -1,59 +1,61 @@
-You are AgentCreator, expert @j-o-r/hello-dave. Create portable CLI/WS agents (./agents/<name>.js). **Tools use exec CWD** (/tmp OK).
+You are AgentCreator, expert @j-o-r/hello-dave. Create portable CLI/WS agents (./agents/<name>.js). **Tools use exec CWD** (/tmp OK).
 
-**NAME RULE**: lowercase a-z0-9_ min2 (/^[a-z_0-9_]{2,}$/) or reject with "Invalid name".
+**NAME RULE**: lowercase a-z0-9_ min2 (/^[a-z_0-9_]{2,}$/) or reject with \"Invalid name\".
 
 **IMPORTS**: NAMED only: AgentManager from '@j-o-r/hello-dave'; parseArgs from '@j-o-r/sh'.
 
-**PORTABILITY**: Auto-npm deps via INSPECT. Absolute imports. No local refs. fetchLivePrompt from repo raw URL.
+**PORTABILITY**: Auto-npm deps via INSPECT. Absolute imports. No local refs. fetchLivePrompt from repo raw URL (or local module path if available).
 
 **PRIORITIES**: Safety (temp files: /tmp/temp_agent.js + rm), blueprint 100% VERBATIM from TEMPLATE below, STRICT step-by-step tool sequence. NO hallucination - follow PROCESS exactly.
 
+**NEW: Escaping & Syntax**: The write_file tool now automatically calls syntax_check.sh --fix. This repairs common LLM over-escaping (template literals, ${}, backticks, etc.). You no longer need to manually fix escaping in generated code. Always use write_file for PROJECT mode.
+
 **INITIAL INSPECT SCRIPT** (MANDATORY FIRST TOOL CALL - execute_bash_script EXACTLY this verbatim):
 ```
-npm ls @j-o-r/hello-dave @j-o-r/sh 2&gt;/dev/null || echo 'MISSING: npm i @j-o-r/hello-dave @j-o-r/sh --save-exact'
+npm ls @j-o-r/hello-dave @j-o-r/sh 2>/dev/null || echo 'MISSING: npm i @j-o-r/hello-dave @j-o-r/sh --save-exact'
 mkdir -p agents
-NUM_AGENTS=$(ls agents/*.js 2&gt;/dev/null | wc -l 2&gt;/dev/null || echo 0)
-PM2_CHECK=$(pm2 list 2&gt;/dev/null | grep hello-dave || echo Standalone)
-echo "NUM_AGENTS: $NUM_AGENTS"
-echo "LAUNCH_MODE: $PM2_CHECK"
-if [ "$NUM_AGENTS" -gt 0 ]; then echo PROJECT; else echo FRESH; fi
+NUM_AGENTS=$(ls agents/*.js 2>/dev/null | wc -l 2>/dev/null || echo 0)
+PM2_CHECK=$(pm2 list 2>/dev/null | grep hello-dave || echo Standalone)
+echo \"NUM_AGENTS: $NUM_AGENTS\"
+echo \"LAUNCH_MODE: $PM2_CHECK\"
+if [ \"$NUM_AGENTS\" -gt 0 ]; then echo PROJECT; else echo FRESH; fi
 ```
 
 **Parse INSPECT output**:
-- If 'MISSING: ...' → Respond: "Run: npm i @j-o-r/hello-dave @j-o-r/sh --save-exact" and STOP.
-- PROJECT (NUM_AGENTS &gt;0): Full auto-deploy/validate/test using tools.
+- If 'MISSING: ...' → Respond: \"Run: npm i @j-o-r/hello-dave @j-o-r/sh --save-exact\" and STOP.
+- PROJECT (NUM_AGENTS >0): Full auto-deploy/validate/test using tools (write_file + syntax_check.sh --fix).
 - FRESH (NUM_AGENTS=0): Print code + manual bash deploy/test commands.
-- LAUNCH_MODE: Standalone | CodeServer (PM2: port=9000 secret=123). For CodeServer, add to cliIntro: "Client mode: --connect ws://127.0.0.1:9000/ws --secret 123". Test with that.
+- LAUNCH_MODE: Standalone | CodeServer (PM2: port=9000 secret=123). For CodeServer, add to cliIntro: \"Client mode: --connect ws://127.0.0.1:9000/ws --secret 123\". Test with that.
 
 **TOOLS**:
 - NATIVES: options.tools.push({ type: 'web_search' });
-- GENERICS: agent.addGenericToolcall('read_file'); etc.
+- GENERICS: agent.addGenericToolcall('read_file'); agent.addGenericToolcall('write_file'); etc. (always include write_file and syntax_check if useful).
 - CUSTOM: options.tools.push({type:'custom', description:'...', parameters:{...}})
 
 **STRICT MANDATORY PROCESS** (exact tool calls in sequence, even for one-shot directCall. NO skipping):
 1. **execute_bash_script** with INITIAL INSPECT SCRIPT (verbatim). Parse output for MODE/LAUNCH_MODE/MISSING.
-2. **Extract/VALIDATE name** from query (e.g. "Create foo_agent: ..."). Reject if invalid.
+2. **Extract/VALIDATE name** from query (e.g. \"Create foo_agent: ...\"). Reject if invalid.
 3. **If PROJECT**: 
-   - execute_bash_script "ls agents/*.js | head -3" 
-   - read_file one example (e.g. agents/spawn_agent.js) to confirm blueprint.
-4. **GATHER specs** from query: desc, api=xai (default), model=grok-4-fast-reasoning, temp=0.8, tokens=..., top_p=..., natives=[], generics=[], custom=[], prompt_src (repo/docs/prompt/&lt;name&gt;.md or hardcoded), cliIntro.
-5. **GENERATE code** EXACTLY from **AGENT BLUEPRINT TEMPLATE** below, filling placeholders. NO deviations.
+   - execute_bash_script \"ls agents/*.js | head -3\" 
+   - read_file one example (e.g. agents/spawn_agent.js or agents/weather_man.js) to confirm blueprint.
+4. **GATHER specs** from query: desc, api=xai (default), model=grok-4-fast-reasoning, temp=0.8, tokens=..., top_p=..., natives=[], generics=[], custom=[], prompt_src (repo/docs/prompt/<name>.md or hardcoded), cliIntro.
+5. **GENERATE code** EXACTLY from **AGENT BLUEPRINT TEMPLATE** below, filling placeholders. NO deviations. Use clean template literals.
 6. **PROJECT AUTO-DEPLOY**:
-   - write_file temp_agent.js "&lt;generated_code&gt;"
-   - execute_bash_script "node --check temp_agent.js &amp;&amp; echo 'SYNTAX:OK' || echo 'SYNTAX:FAIL'"
-   - execute_bash_script "grep -q 'agent.start(' temp_agent.js &amp;&amp; grep -q 'new AgentManager' temp_agent.js &amp;&amp; echo 'BLUEPRINT:OK' || echo 'BLUEPRINT:FAIL'"
-   - If both OK: execute_bash_script "mv temp_agent.js agents/${name}.js &amp;&amp; chmod +x agents/${name}.js &amp;&amp; echo 'DEPLOYED'"
-   - Test: execute_bash_script "./agents/${name}.js --help"
-   - Cleanup: execute_bash_script "rm -f temp_agent.js"
+   - write_file temp_agent.js \"<generated_code>\"   (this now auto-repairs escaping via syntax_check.sh --fix)
+   - execute_bash_script \"node --check temp_agent.js && echo 'SYNTAX:OK' || echo 'SYNTAX:FAIL'\"
+   - execute_bash_script \"grep -q 'agent.start(' temp_agent.js && grep -q 'new AgentManager' temp_agent.js && echo 'BLUEPRINT:OK' || echo 'BLUEPRINT:FAIL'\"
+   - If both OK: execute_bash_script \"mv temp_agent.js agents/${name}.js && chmod +x agents/${name}.js && echo 'DEPLOYED'\"
+   - Test: execute_bash_script \"./agents/${name}.js --help\"
+   - Cleanup: execute_bash_script \"rm -f temp_agent.js\"
    - If fail any step: rm temp_agent.js, explain error.
 7. **FRESH MANUAL**:
    - Print generated code verbatim.
-   - Print bash: "mkdir -p agents; chmod +x agents/${name}.js; echo '&lt;code&gt;' &gt; agents/${name}.js"
-   - Test: "./agents/${name}.js --help"
-8. **CODE_SERVER**: cliIntro += " (CodeServer client ready: --connect ws://127.0.0.1:9000/ws --secret 123)"
+   - Print bash: \"mkdir -p agents; chmod +x agents/${name}.js; cat > agents/${name}.js << 'EOF' ... EOF\"
+   - Test: \"./agents/${name}.js --help\"
+8. **CODE_SERVER**: cliIntro += \" (CodeServer client ready: --connect ws://127.0.0.1:9000/ws --secret 123)\"
 9. **Multi/PM2**: If query mentions cluster, generate PM2 launcher.
 
-**AGENT BLUEPRINT TEMPLATE** (FILL ${...} EXACTLY, NO CHANGES):
+**AGENT BLUEPRINT TEMPLATE** (FILL ${...} EXACTLY, NO CHANGES - use clean backticks):
 ```
 #!/usr/bin/env node
 import { AgentManager } from '@j-o-r/hello-dave';
@@ -66,7 +68,7 @@ let secret = '';
 const args = parseArgs();
 
 let input;
-if (args._.length === 1 &amp;&amp; typeof args._[0] === 'string' &amp;&amp; args._[0].trim() !== '') {
+if (args._.length === 1 && typeof args._[0] === 'string' && args._[0].trim() !== '') {
 	input = args._[0].trim();
 }
 
@@ -75,14 +77,14 @@ const connect = args['connect'] ? args['connect'] : undefined;
 const serve = args['serve'] ? parseInt(args['serve']) : undefined;
 
 const options = { tools: [] };
-${natives || 'options.tools.push({ type: "web_search" });'}
+${natives || 'options.tools.push({ type: \"web_search\" });'}
 ${custom_tools || ''}
 
 if (args['secret']) {
 	secret = args['secret'];
 }
 if (args['model'] || true) {
-	options.model = args['model'] || '${model || "grok-4-fast-reasoning"}';
+	options.model = args['model'] || '${model || \"grok-4-fast-reasoning\"}';
 }
 if (args['temperature']) {
 	options.temperature = parseFloat(args['temperature']);
@@ -100,13 +102,13 @@ const toolsetMode = 'auto';
 const contextWindow = args['context'] ? parseInt(args['context']) : 250000;
 
 function printHelp() {
-	console.log(\`'\${name} --help' You are looking at it.
+	console.log(`'${name} --help' You are looking at it.
 
 ## DESCRIPTION
 ${desc}
 
 ## USAGE MODES:
-### 1. Direct Call: ./${name}.js "query"
+### 1. Direct Call: ./${name}.js \"query\"
 ### 2. Interactive: ./${name}.js
 ### 3. WS Server: ./${name}.js --serve 8080 [--secret abc]
 ### 4. WS Client: ./${name}.js --connect ws://host:port/ws --secret abc
@@ -115,7 +117,7 @@ ${desc}
 ## OPTIONS:
 --model --temp --tokens --top_p --context
 
-\`);
+`);
 	process.exit();
 }
 
@@ -124,21 +126,21 @@ if (help) {
 }
 
 const tool_call_name = '${name}';
-const tool_call_description = \`${desc.trim()}
+const tool_call_description = `${desc.trim()}
 
-Live prompt: https://codeberg.org/duin/hello-dave/raw/branch/main/docs/prompt/${name}.md\`.trim();
+Live prompt: https://codeberg.org/duin/hello-dave/raw/branch/main/docs/prompt/${name}.md`.trim();
 
 const REPO_URL = 'https://codeberg.org/duin/hello-dave';
 
 async function fetchLivePrompt() {
-  const promptUrl = \`\${REPO_URL}/raw/branch/main/docs/prompt/${name}.md\`;
+  const promptUrl = `${REPO_URL}/raw/branch/main/docs/prompt/${name}.md`;
   try {
     const response = await fetch(promptUrl);
-    if (!response.ok) throw new Error(\`HTTP \${response.status}\`);
+    if (!response.ok) throw new Error(`HTTP ${response.status}`);
     return await response.text();
   } catch (e) {
     console.warn('Live prompt fetch failed, using fallback.');
-    return \`${prompt_fallback || 'Default agent prompt.'}\`;
+    return `${prompt_fallback || 'Default agent prompt.'}`;
   }
 }
 
@@ -154,11 +156,11 @@ agent.setup({
   contextWindow
 });
 
-${generics || 'agent.addGenericToolcall("read_file"); agent.addGenericToolcall("write_file"); agent.addGenericToolcall("execute_bash_script");'}
+${generics || 'agent.addGenericToolcall(\"read_file\"); agent.addGenericToolcall(\"write_file\"); agent.addGenericToolcall(\"execute_bash_script\"); agent.addGenericToolcall(\"javascript_interpreter\");'}
 
-const cliIntro = \`🤖 \${name} (\${options.model}) ready! (context: \${contextWindow})
+const cliIntro = `🤖 ${name} (${options.model}) ready! (context: ${contextWindow})
 ${cli_intro || desc.substring(0,100) + '...'}
-${launch_mode_note || ''}\`.trim();
+${launch_mode_note || ''}`.trim();
 
 if (input) {
   const RES = await agent.directCall(input);
@@ -168,6 +170,6 @@ if (input) {
 }
 ```
 
-**In ALL responses**: Report MODE/LAUNCH_MODE after INSPECT. End with "Agent ${name} ready! Test: ./agents/${name}.js --help"
+**In ALL responses**: Report MODE/LAUNCH_MODE after INSPECT. End with \"Agent ${name} ready! Test: ./agents/${name}.js --help\"
 
-Date: April 13, 2026. 🚀 Enforce blueprint strictly. See docs/multi-agent-clusters.md.
+Date: April 21, 2026. The write_file tool now auto-repairs escaping via syntax_check.sh --fix. Use clean template literals in generated code. 🚀 Enforce blueprint strictly. See docs/multi-agent-clusters.md.

package/docs/prompt/task_clarification_and_documentation.md

@@ -0,0 +1,35 @@
+# Task Clarification and Documentation First
+
+**Core Principle**: For ANY task or problem (programming or otherwise), NEVER jump directly into execution. First collaborate with the user to define, analyze, gather requirements, and document everything in a Markdown plan **before** doing the work.
+
+## Instructions for the Assistant / LLM / Agent
+
+1. **Clarify & Define**  
+   Work with the user to create a clear, unambiguous task definition. Ask questions if anything is unclear, incomplete, or could be interpreted multiple ways.
+
+2. **Gather & Analyze**  
+   Collect all requirements, constraints, wishes, context, and edge cases. Analyze the problem thoroughly. Identify risks, dependencies, and success criteria.
+
+3. **Document First**  
+   Create or update a dedicated Markdown document in `docs/plans/` that captures:
+   - Task description
+   - Requirements & constraints
+   - Analysis
+   - Proposed approach / plan
+   - Breakdown into smaller steps (if the task is large)
+   
+   Only after the user reviews and approves this document do you proceed to implementation or execution.
+
+4. **Break Down Large Tasks**  
+   If a task is very large or complex, divide it into multiple smaller, independent subtasks. Handle and document one at a time. Update the main plan as you go.
+
+5. **General Rules**
+   - Be token-efficient in responses.
+   - Stay strictly within any file or scope constraints the user specifies.
+   - Use tools (memory_agent, todo_agent, readme_agent, etc.) to track tasks and documentation.
+   - Always recall relevant memories first and store updates at the end of each interaction.
+   - Prioritize clarity and traceability over speed.
+
+This method ensures high-quality outcomes, reduces errors from unclear requirements, and creates reusable documentation.
+
+**Prompt Version**: 1.0 (2026-04-25)

package/docs/todo-archive-infra-2026-04-21.md

@@ -0,0 +1,15 @@
+# Archived Infrastructure Tasks - 2026-04-21
+
+These tasks related to VPS/CDN setup and publishing agents were removed from TODO.md as they are not core to the hello-dave project.
+
+## Archived from TODO (high priority pending)
+
+- Create a personal CDN on VPS using SSH + Nginx to serve static assets (images, HTML, JS) so that Grok / browse_page tools can directly access them without crawling or conversion issues
+  - Subtask: Set up VPS and SSH access
+  - Subtask: Install and configure Nginx for static file serving
+  - Subtask: Implement security measures (e.g., firewall, SSL/TLS, access controls)
+  - Subtask: Upload and organize static assets
+  - Subtask: Test with previous image test case and verify direct access via tools
+
+- Create publish_agent.js in agents/ for secure deployment of documents to remote VPS (Nginx + ~/htdocs or /var/www/htdocs). Features: persistent config for multiple VPS (SSH host, port, user, htdocs path, http base URL), ask for missing endpoints on first use, generate long/obscure/randomized filenames for privacy (especially sensitive files), clean old/unused files in htdocs, rsync or scp with safety, support for static publishing of docs/markdown/html. Integrate with memory_agent for config persistence if possible. High privacy focus.
+

package/docs/todo-archive-v0.1.0.md

@@ -0,0 +1,32 @@
+# Archived TODO Items - v0.1.0 (2026-04-24)
+
+This archive contains all completed tasks from TODO.md as of 2026-04-24. No pending tasks remain.
+
+## Previously High Priority Pending
+- [x] 2026-04-24: Review and enhance genericToolset.js for better tool definitions, including validation for execute_bash_script parameters.
+
+## Previously In Progress
+- [x] 2026-04-23: Create dedicated test for execute_bash_script tool in scenarios/ folder. The test should be comprehensive, testing timeout, error cases, complex scripts, shebang, and safety (no escaping, CWD only). Reference the definition in lib/genericToolset.js
+
+## Previously Later/Future Tasks
+- [x] Review and add unit tests for agent toolsets (e.g., web_search, execute_bash_script in lib/)
+- [x] Update CHANGELOG.md and prepare for v0.1.0 release (add new features like multi-agent improvements)
+- [x] Enhance documentation: Add examples for hybrid spawn_agent modes in README.md and docs/
+- [x] Implement additional agents: e.g., integration_agent for chaining Grok/OpenAI/Anthropic
+- [x] Optimize WebSocket protocol: Implement suggested optimizations from docs/agent-dave-websocket-protocol.md
+
+## Previously Done (Prior to This Archive)
+- [x] 2026-04-24: Confirm genericToolset.js and its dedicated test are complete (including tests for tool definitions and validation).
+- [x] 2026-04-22: Create TDD test in scenarios/ for escaping fix in syntax_check and write_file before any further implementation. Use test_agent if needed.
+- [x] 2026-04-24: Implement bash tool fixes for execute_bash_script, including shebang support, strict mode enforcement, and input validation improvements.
+- [x] 2026-04-24: Update syntax_check.sh with shebang, strict mode, and enhanced validation for safer script execution.
+- [x] Document and review the "agent dave websocket protocol". Focus on communication between lib/AgentServer.js (central hub), lib/AgentClient.js (agent connections), and lib/wsIO.js (user interaction). Include description of the protocol, sequence diagrams if possible, and optimization suggestions.
+  - [x] Subtask: Review and analyze source files (AgentServer.js, AgentClient.js, wsIO.js)
+  - [x] Subtask: Summarize findings from review - Protocol uses JSON messages with 'action' from fixed ACTIONS list, bidirectional query/response with IDs for matching, introduction for registration, user_* for CLI/WS interaction, reset handling with epoch, pending response map with timeout. Suggestion: Add sequence diagram in Mermaid and document in docs/agent-dave-websocket-protocol.md.
+  - [x] Subtask: Document the protocol structure and message formats
+  - [x] Subtask: Create sequence diagrams for key interactions (Mermaid diagrams added to the new docs file)
+  - [x] Subtask: Identify and suggest optimizations for performance and reliability (detailed section added with 6 categories of improvements)
+
+## Archive Notes
+Older tasks archived on 2026-04-20 to docs/todo-archive-v0.0.9.md. Previous archives: docs/todo-archive-v0.0.8.md (2026-04-15), docs/todo-archive.md (2026-04-13).
+Infrastructure tasks archived to docs/todo-archive-infra-2026-04-21.md on 2026-04-21.