@j-o-r/hello-dave 0.0.8 → 0.0.9

package/TODO.md

@@ -1,5 +1,11 @@
 ## TODO (high priority pending)
-(none)
+- [ ] Create a personal CDN on VPS using SSH + Nginx to serve static assets (images, HTML, JS) so that Grok / browse_page tools can directly access them without crawling or conversion issues
+  - [ ] Subtask: Set up VPS and SSH access
+  - [ ] Subtask: Install and configure Nginx for static file serving
+  - [ ] Subtask: Implement security measures (e.g., firewall, SSL/TLS, access controls)
+  - [ ] Subtask: Upload and organize static assets
+  - [ ] Subtask: Test with previous image test case and verify direct access via tools
+- [ ] Create publish_agent.js in agents/ for secure deployment of documents to remote VPS (Nginx + ~/htdocs or /var/www/htdocs). Features: persistent config for multiple VPS (SSH host, port, user, htdocs path, http base URL), ask for missing endpoints on first use, generate long/obscure/randomized filenames for privacy (especially sensitive files), clean old/unused files in htdocs, rsync or scp with safety, support for static publishing of docs/markdown/html. Integrate with memory_agent for config persistence if possible. High privacy focus.
 
 ## In Progress
 (none)
@@ -8,6 +14,11 @@
 (none)
 
 ## Done
-(none - archived to docs/todo-archive-v0.0.8.md on 2026-04-15)
+- [x] Document and review the "agent dave websocket protocol". Focus on communication between lib/AgentServer.js (central hub), lib/AgentClient.js (agent connections), and lib/wsIO.js (user interaction). Include description of the protocol, sequence diagrams if possible, and optimization suggestions.
+  - [x] Subtask: Review and analyze source files (AgentServer.js, AgentClient.js, wsIO.js)
+  - [x] Subtask: Summarize findings from review - Protocol uses JSON messages with 'action' from fixed ACTIONS list, bidirectional query/response with IDs for matching, introduction for registration, user_* for CLI/WS interaction, reset handling with epoch, pending response map with timeout. Suggestion: Add sequence diagram in Mermaid and document in docs/agent-dave-websocket-protocol.md.
+  - [x] Subtask: Document the protocol structure and message formats
+  - [x] Subtask: Create sequence diagrams for key interactions (Mermaid diagrams added to the new docs file)
+  - [x] Subtask: Identify and suggest optimizations for performance and reliability (detailed section added with 6 categories of improvements)
 
-Older tasks archived on 2026-04-13 to docs/todo-archive.md.
+Older tasks archived on 2026-04-20 to docs/todo-archive-v0.0.9.md. Previous archives: docs/todo-archive-v0.0.8.md (2026-04-15), docs/todo-archive.md (2026-04-13).

package/agents/code_agent.js

@@ -24,7 +24,7 @@ if (args['secret']) {
 // Set properties only if provided via command line (except model which has default)
 if (args['model'] || true) { // model gets default value
 	// @ts-ignore
-	options.model = args['model'] || 'grok-4-1-fast-reasoning';
+	options.model = args['model'] || 'grok-4.20-reasoning';
 }
 // if (args['temperature']) {
 	options.temperature = 0.2;
@@ -35,14 +35,14 @@ if (args['tokens']) {
 if (args['top_p']) {
 	options.top_p = parseFloat(args['top_p']);
 }
-const reasoning = args['reasoning'] ? args['reasoning'] : 'medium';
-if (reasoning) {
-	options.reasoning = {
-		// @ts-ignore
-		effort:reasoning,
-		summary: 'auto'
-	}
-}
+// const reasoning = args['reasoning'] ? args['reasoning'] : 'medium';
+// if (reasoning) {
+// 	options.reasoning = {
+// 		// @ts-ignore
+// 		effort:reasoning,
+// 		summary: 'auto'
+// 	}
+// }
 const toolsetMode = 'auto';
 const contextWindow = args['context'] ? parseInt(args['context']) : 2565000;
 

package/bin/dave.js

@@ -118,9 +118,9 @@ if (args.clear) {
 		};
 		options.tools.push({ type: 'web_search' });
 		options.tools.push({ type: 'x_search' });
-		options.model = args.model || 'grok-4-1-fast-reasoning';
+		options.model = args.model || 'grok-4.20-reasoning';
 		options.temperature = 0.2;
-		options.reasoning = { effort: 'medium', summary: 'auto' };
+		// options.reasoning = { effort: 'medium', summary: 'auto' };
 
 		const prompt = `
 Respond briefly and directly, using minimal words. Reason step-by-step first. Focus solely on core point; avoid elaboration or follow-ups. If unclear, ask clarifying questions before proceeding.

package/docs/agent-dave-websocket-protocol.md

@@ -0,0 +1,180 @@
+# Agent Dave WebSocket Protocol
+
+**Version**: 0.0.8 (as of April 2026)  
+**Repository**: https://codeberg.org/duin/hello-dave  
+**Core Files**:
+- `lib/AgentServer.js` — Central hub, registers agents as dynamic tools, handles user interactions and sessions.
+- `lib/AgentClient.js` — Agent-side wrapper (Prompt + ToolSet), queue-based message processor, auto-reconnect, epoch-based reset handling.
+- `lib/wsIO.js` — One-shot WS client for CLI tools (`dave --connect`, `wsio()` helper).
+
+## Overview
+
+The protocol enables **multi-agent collaboration** over WebSocket. 
+
+- A central **AgentServer** listens on `ws://127.0.0.1:<port>/ws`.
+- **AgentClient** instances connect, introduce themselves, and get registered as **dynamic tools** in the server's `ToolSet`.
+- The server exposes these agents to the main Prompt, allowing the LLM to call them like any other tool (`agent_query` / `agent_response`).
+- **User/CLI interaction** uses `user_*` actions (via `wsIO.js` or `dave` CLI).
+- All messages are **JSON** objects with an `action` field validated against a fixed list.
+- **Authentication**: `?wssrc_id=<secret>` query param (plain in AgentServer, base64 in wsIO.js).
+- **Reset handling**: Broadcast `reset`, clients use an `#epoch` counter to invalidate in-flight work.
+- **Response matching**: Uses `id` (timestamp-based) + pending map on server.
+
+This design turns specialized agents (code, todo, memory, readme, etc.) into callable tools while supporting direct CLI, interactive REPL, and one-shot queries.
+
+## Message Format
+
+Every message is a JSON object:
+
+```json
+{
+  "action": "one_of_the_actions_below",
+  "content": "string or object (varies)",
+  "id": "unique_timestamp_string_or_number",
+  "name"?: "agent_name_on_introduction"
+}
+```
+
+### Supported Actions (ACTIONS array in AgentServer.js)
+
+**Agent ↔ Server (Tool Calls)**
+- `agent_introduction` — Agent registers itself (sends `name` + `description`).
+- `agent_query` — Server calls an agent tool with natural language `content` (query).
+- `agent_response` — Agent returns result.
+- `agent_error` — Agent reports failure.
+
+**User/CLI ↔ Server**
+- `user_introduction` — CLI connects (sent by wsIO.js).
+- `user_request` — User query to main prompt.
+- `server_response` — Final answer from main prompt.
+- `server_error` — Error from main prompt.
+- `user_reset` — Reset current session.
+- `user_sessionlist` — Request list of saved sessions.
+- `user_loadsession` — Load a previous session.
+- `user_info` — Get server/prompt/session info.
+
+**Server → Clients (broadcasts/responses)**
+- `reset` — Broadcast to all agents on reset.
+- `server_reset`, `server_sessionlist`, `server_sessionloaded`, `server_info` — Responses to user actions.
+
+Unknown actions cause immediate client disconnection.
+
+## Key Interactions & Sequence Diagrams
+
+### 1. Agent Registration & Tool Call
+
+```mermaid
+sequenceDiagram
+    participant AgentClient
+    participant AgentServer
+    participant Prompt/ToolSet
+
+    AgentClient->>AgentServer: WS connect + ?wssrc_id=secret
+    AgentClient->>AgentServer: {action: "agent_introduction", name: "code", content: "Code execution agent..."}
+    AgentServer->>ToolSet: add("code", description, schema, handler)
+    Note over ToolSet,AgentServer: Tool now available to LLM
+
+    Prompt->>ToolSet: LLM decides to call "code" with query
+    ToolSet->>AgentServer: handler(query) → Promise
+    AgentServer->>AgentClient: {action: "agent_query", id: "1745...", content: "Write a bash script..."}
+    AgentClient->>AgentClient: #queue.push(), #processOne()
+    AgentClient->>Prompt: prompt.call(query)
+    AgentClient->>AgentServer: {action: "agent_response", id: "1745...", content: "```bash ...```"}
+    AgentServer->>Prompt: resolve(promise)
+    Prompt->>LLM: Continue with tool result
+```
+
+### 2. User One-Shot Query (via wsIO.js or dave --connect)
+
+```mermaid
+sequenceDiagram
+    participant CLI (wsIO)
+    participant AgentServer
+    participant Prompt
+
+    CLI->>AgentServer: WS connect + wssrc_id
+    CLI->>AgentServer: {action: "user_introduction", id: X}
+    CLI->>AgentServer: {action: "user_request", id: Y, content: "Hello Dave"}
+    AgentServer->>Prompt: prompt.call("Hello Dave")
+    Prompt->>LLM: Full reasoning + tool calls (may call registered agents)
+    Prompt-->>AgentServer: Final content
+    AgentServer->>CLI: {action: "server_response", id: Y, content: "..."}
+    CLI->>CLI: resolve promise, close WS
+```
+
+### 3. Reset Handling
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant AgentServer
+    participant AgentClient
+
+    User->>AgentServer: {action: "user_reset"}
+    AgentServer->>Prompt: prompt.reset()
+    AgentServer-->>All Agents: broadcast {action: "reset"}
+    AgentClient->>AgentClient: #epoch++, #queue=[], prompt.reset()
+```
+
+## Implementation Details
+
+- **Server-side pending responses**: `pendingResponses` Map keyed by `${conn.id}:${msg.id}` with timeout via `setInterval` (30s intervals, max 20 → 10min).
+- **Client-side queue**: Strict sequential processing (`#processing` guard, 2s polling interval by default). Prevents race conditions in tool calls.
+- **Auto-reconnect**: AgentClient retries every 5s on close.
+- **Session Management**: Integrated with `Session.js` for `user_sessionlist` / `user_loadsession`.
+- **wsIO.js specifics**: Sends both `user_introduction` + action in one connection, matches response by `id`, then closes. Uses base64 secret.
+
+## Optimizations & Suggestions
+
+**Current Strengths**
+- Simple JSON, no complex framing.
+- Epoch prevents stale responses after reset.
+- Dynamic tool registration — agents can be added at runtime.
+
+**Potential Improvements**
+1. **Performance**:
+   - Replace polling interval in AgentClient with event-driven queue (e.g. `setImmediate` or microtask after each message).
+   - Use a proper request-response correlation library or WebSocket subprotocol.
+   - Add backpressure handling for high-volume tool calls.
+
+2. **Reliability**:
+   - Heartbeat/ping-pong to detect dead connections faster than timeout.
+   - Exponential backoff on reconnect.
+   - Persistent message IDs (UUID instead of timestamp).
+   - Schema validation for all incoming messages (e.g. Zod or JSON Schema).
+
+3. **Observability**:
+   - Add structured logging (instead of console.log).
+   - Metrics: active clients, avg response time, error rate.
+   - OpenTelemetry or Prometheus integration.
+
+4. **Security**:
+   - Current secret is weak (query param). Consider JWT or mTLS for production.
+   - Rate limiting per connection.
+   - Validate `content` size.
+
+5. **Extensibility**:
+   - Support binary messages for large data (e.g. images, files).
+   - Add `agent_stream` for streaming responses from agents.
+   - Formalize as a subprotocol (`Sec-WebSocket-Protocol: agent-dave-v1`).
+
+6. **Documentation**:
+   - Generate Mermaid diagrams automatically from code comments.
+   - Add protocol test suite in `scenarios/`.
+
+**Next Steps** (from TODO):
+- Complete sequence diagrams (expand the Mermaid examples above).
+- Identify specific performance bottlenecks via profiling.
+- Create formal spec with error codes and versioning.
+
+## References
+
+- `lib/AgentServer.js`
+- `lib/AgentClient.js`
+- `lib/wsIO.js`
+- `agents/spawn_agent.js` (uses this protocol for hybrid/server/client modes)
+- `docs/multi-agent-clusters.md`
+- `docs/prompt/spawn_agent.md`
+
+*Last updated: April 20, 2026*  
+*This document was generated as part of the TODO task "Document and review the 'agent dave websocket protocol'."*

package/lib/genericToolset.js

@@ -44,8 +44,82 @@ const getJSError = (errorStr) => {
 	}
 	return result;
 };
+// /**
+// * Try to prevent double escaping by LLMS models
+// * When a string stays a string it is probably double escaped
+// * @parameter {string} s
+// * @returns {string}
+// */
+// const guessOverEscaping = (s) => {
+// 	let test;
+// 	if (typeof s === 'string') {
+// 		try {
+// 			test = JSON.parse(s);
+// 		} catch (_e) { }
+// 		if (typeof test === 'string') {
+// 			return test;
+// 		}
+// 	}
+// 	return s;
+// }
 
 /**
+ * Try to prevent double escaping by LLMs.
+ * Handles common patterns like \"path\", \\\"path\\\", JSON strings,
+ * and multiple layers while being much safer on complex bash/JS scripts.
+ */
+const guessOverEscaping = (s) => {
+	if (typeof s !== 'string') return s;
+
+	let current = s.trim();
+	const seen = new Set();
+	let iterations = 0;
+	const MAX_ITER = 6; // safety limit
+
+	while (iterations < MAX_ITER && !seen.has(current)) {
+		seen.add(current);
+		iterations++;
+
+		// 1. Most reliable: JSON.parse (undoes proper JSON string escaping)
+		try {
+			const parsed = JSON.parse(current);
+			if (typeof parsed === 'string') {
+				current = parsed;
+				continue;
+			}
+		} catch (e) { }
+
+		// 2. Specifically strip the pattern you saw: \"...\"  (including the backslashes)
+		if (current.startsWith('\\"') && current.endsWith('\\"')) {
+			current = current.slice(2, -2);
+			continue;
+		}
+
+		// 3. Strip normal outer quotes, but protect JSON objects/arrays
+		if (current.startsWith('"') && current.endsWith('"') && current.length > 2) {
+			const inner = current.slice(1, -1);
+			const trimmedInner = inner.trim();
+			if (!trimmedInner.startsWith('{') && !trimmedInner.startsWith('[')) {
+				current = inner;
+				continue;
+			}
+		}
+
+		// 4. Conservative global unescape of \" → "
+		//    Only applied to short strings without newlines (i.e. not full scripts)
+		if (!current.includes('\n') && (current.match(/\\"/g) || []).length >= 1) {
+			const lessEscaped = current.replace(/\\"/g, '"');
+			if (lessEscaped !== current) {
+				current = lessEscaped;
+				continue;
+			}
+		}
+
+		break; // no more productive changes
+	}
+
+	return current;
+};/**
  * @module lib/genericToolset
  * Secure utility tools for code execution, file I/O, system ops. Pre-populated ToolSet.
  * 
@@ -71,9 +145,10 @@ tools.add(
 	async (params) => {
 		let response = '';
 		try {
+			const script = guessOverEscaping(params.script);
 			const delim = `JS_STDIN_${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 6)}`;
 			response = await SH`cat <<'${delim}' | node --input-type=module -
-${params.script}
+${script}
 ${delim}
 `.run();
 		} catch (e) {
@@ -110,14 +185,15 @@ tools.add(
 	},
 	async (params) => {
 		const timeoutSec = Number(params.timeout ?? 300);
+		const bash_script = guessOverEscaping(params.bash_script);
 		if (isNaN(timeoutSec) || timeoutSec < 0) throw new Error('Invalid timeout');
 		const timeout = timeoutSec * 1000;
 		// return await SH`bash`.options({ timeout: timeoutMs }).run(params.bash_script);
-    const delim = `END_SCRIPT_${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 8)}`;
+		const delim = `END_SCRIPT_${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 8)}`;
 		return await SH`bash <<'${delim}'
-${params.bash_script}
+${bash_script}
 ${delim}
-`.options({timeout}).run();
+`.options({ timeout }).run();
 
 	}
 );
@@ -135,12 +211,15 @@ tools.add(
 		required: ['to', 'subject', 'body']
 	},
 	async (params) => {
+		const to = guessOverEscaping(params.to);
+		const subject = guessOverEscaping(params.subject);
+		const body = guessOverEscaping(params.body);
 		const delim = `END_EMAIL_${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 8)}`;
 		return await SH`msmtp ${params.to} <<'${delim}'
-To: ${params.to}
-Subject: ${params.subject}
+To: ${to}
+Subject: ${subject}
 
-${params.body}
+${body}
 ${delim}
 `.run();
 	}
@@ -156,7 +235,10 @@ tools.add(
 		},
 		required: ['url']
 	},
-	async (params) => await SH`xdg-open ${[params.url]}`.run()
+	async (params) => {
+		let url = guessOverEscaping(params.url?.trim());
+		return await SH`xdg-open ${[url]}`.run();
+	}
 );
 
 tools.add(
@@ -176,7 +258,10 @@ tools.add(
 		required: ['url', 'script']
 	},
 	async (params) => {
-		const { url, script } = params;
+		let { url, script } = params;
+		url = guessOverEscaping(url);
+		script = guessOverEscaping(script);
+
 		const timeoutSec = Number(params.timeout ?? 30);
 		if (isNaN(timeoutSec) || timeoutSec <= 0) throw new Error('Invalid timeout');
 		const timeoutMs = timeoutSec * 1000;
@@ -203,7 +288,8 @@ tools.add(
 	},
 	async (params) => {
 		if (typeof params.query === 'string' && params.query.trim()) {
-			return await SH`${searchSessionsSh} "${bashEscape(params.query)}"`.run();
+			const q = guessOverEscaping(params.query);
+			return await SH`${searchSessionsSh} "${bashEscape(q)}"`.run();
 		}
 		return await SH`${listSessionsSh}`.run();
 	}
@@ -220,7 +306,7 @@ tools.add(
 		required: ['file']
 	},
 	async (params) => {
-		const file = params.file?.trim();
+		let file = guessOverEscaping(params.file?.trim());
 		if (typeof file !== 'string' || !file || file.startsWith('/') || file.includes('..') || file.includes('\\\\')) {
 			throw new Error('Relative CWD path only (no /, .., \\\\).');
 		}
@@ -242,8 +328,8 @@ tools.add(
 		required: ['file', 'content']
 	},
 	async (params) => {
-		const file = params.file?.trim();
-		const content = params.content ?? '';
+		let file = guessOverEscaping(params.file?.trim());
+		let content = guessOverEscaping(params.content ?? '');
 		if (typeof file !== 'string' || !file || file.startsWith('/') || file.includes('..') || file.includes('\\\\')) {
 			throw new Error('Relative CWD path only.');
 		}
@@ -253,29 +339,29 @@ tools.add(
 		const ext = path.extname(file);
 		let finalContent = content, validationMsg = '';
 
-		const temp1 = path.join(dir, `temp_${Date.now()}_${Math.random().toString(36).slice(2,6)}${ext || '.tmp'}`);
+		const temp1 = path.join(dir, `temp_${Date.now()}_${Math.random().toString(36).slice(2, 6)}${ext || '.tmp'}`);
 		await fs.writeFile(temp1, content, 'utf8');
 
 		try {
 			await SH`${syntaxCheckSh} ${[temp1]}`.run();
 			validationMsg = ' ✓ Syntax OK';
 		} catch (e) {
-			if (!['.js','.mjs'].includes(ext)) {
-				await fs.unlink(temp1).catch(()=>{}); 
+			if (!['.js', '.mjs'].includes(ext)) {
+				await fs.unlink(temp1).catch(() => { });
 				throw new Error(`Syntax error: ${e.message}`);
 			}
 			let fixed = content.replace(/\\\\`/g, '\\`').replace(/\\\`/g, '`').replace(/\\`/g, '`').replace(/\\\$/g, '$').replace(/\\\${/g, '${');
-			const temp2 = path.join(dir, `temp_fix_${Date.now()}_${Math.random().toString(36).slice(2,6)}.js`);
+			const temp2 = path.join(dir, `temp_fix_${Date.now()}_${Math.random().toString(36).slice(2, 6)}.js`);
 			await fs.writeFile(temp2, fixed, 'utf8');
 			try {
 				await SH`${syntaxCheckSh} ${[temp2]}`.run();
 				finalContent = fixed;
 				await fs.rename(temp2, resolved);
-				await fs.unlink(temp1).catch(()=>{}); 
+				await fs.unlink(temp1).catch(() => { });
 				validationMsg = ' ✓ Fixed JS';
 			} catch {
-				await fs.unlink(temp1).catch(()=>{}); 
-				await fs.unlink(temp2).catch(()=>{}); 
+				await fs.unlink(temp1).catch(() => { });
+				await fs.unlink(temp2).catch(() => { });
 				throw new Error(`Syntax error (fix failed): ${e.message}`);
 			}
 		}
@@ -302,7 +388,7 @@ tools.add(
 		required: ['file']
 	},
 	async (params) => {
-		const file = params.file?.trim();
+		const file = guessOverEscaping(params.file?.trim());
 		if (typeof file !== 'string' || !file) throw new Error('Relative path required.');
 		const resolved = path.resolve(process.cwd(), file);
 		if (!resolved.startsWith(process.cwd())) throw new Error('Escapes CWD.');

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@j-o-r/hello-dave",
   "type": "module",
-  "version": "0.0.8",
+  "version": "0.0.9",
   "description": "ESM toolkit for building AI agents with unified access to Grok (XAI), OpenAI, and Anthropic endpoints",
   "main": "./lib/index.js",
   "types": "./types/index.d.ts",