@j-o-r/hello-dave 0.1.1 → 0.1.5

package/docs/creating-toolsets.md

@@ -0,0 +1,336 @@
+# Creating ToolSets
+
+This document describes the current project pattern for creating `ToolSet` wrappers around API modules and other helper functions.
+
+The main goal is:
+
+> A tool response must be self-describing enough that the model can write a normal assistant message containing the important follow-up references before old raw `function_call` / `function_response` history is pruned.
+
+This keeps prompts compact while preserving durable IDs, URLs, paths, statuses, and other handles in normal conversation history.
+
+## Why this pattern exists
+
+`Prompt.pruneResolvedFunctionCallsExceptLast()` removes old resolved tool I/O from prompt history. This prevents repeated tool transcripts, large JSON outputs, command logs, buffers, and stale function-call messages from polluting future requests.
+
+However, pruning is only safe if the important facts from the tool result are preserved elsewhere. The intended flow is:
+
+```txt
+assistant: function_call create_video_openai(...)
+tool: function_response { "videoId": "vid_123", ... }
+assistant: Created the video. The videoId is `vid_123`.
+--- later: old resolved function call/response may be pruned ---
+```
+
+Do not rely on old raw tool outputs as long-term memory. Toolsets must help the assistant preserve important references in ordinary assistant text.
+
+## Required tool response style
+
+Every tool should return a compact, structured JSON string, not a bare string or raw wrapper object.
+
+Use a local helper:
+
+```js
+/**
+ * Serialize a tool response as pretty JSON for function-call output.
+ *
+ * @param {Record<string, unknown>} payload - JSON-serializable response payload.
+ * @returns {string} Pretty JSON response.
+ */
+function json(payload) {
+  return JSON.stringify(payload, null, 2);
+}
+```
+
+A good tool response includes:
+
+- `tool`: the function-call name;
+- `success`: boolean when meaningful;
+- exact durable references:
+  - `id`, `*_id`, `*Id`;
+  - `url`, `*_url`, `*Url`;
+  - `image_url`, `audio_url`, `video_url`;
+  - `file_id`, `task_id`, `request_id`, `trace_id`;
+  - `local_path`, `localPath`, `local_paths`;
+  - pagination cursors;
+- operation status, e.g. `status`, `finish_reason`;
+- source references, e.g. `sourceVideoId`, `source_image_url`, `source_audio_url`;
+- new output references, clearly distinguished from source references;
+- a short `message` or `note` telling the assistant what to preserve in its normal response.
+
+Example:
+
+```js
+return json({
+  tool: 'create_video_openai',
+  success: true,
+  videoId: result.id,
+  id: result.id,
+  status: result.status,
+  prompt: params.prompt,
+  note: 'Assistant: tell the user the exact videoId. Use videoId for retrieve, download, edit, extend, remix, or delete follow-up calls.'
+});
+```
+
+## Tool descriptions must guide the assistant
+
+Tool descriptions are part of the model-facing contract. They should state:
+
+1. what the tool does;
+2. which input references are preferred;
+3. which output references are important;
+4. what the assistant should repeat in its normal response.
+
+Example:
+
+```js
+tools.add(
+  'create_video_openai',
+  'Create an OpenAI video from a text prompt. Prefer image_url for visual references when possible. Returns videoId, status, and optional local_path. After success, include the exact videoId in your normal assistant response because it is needed for retrieve/download/edit/extend/remix/delete follow-up calls.',
+  schema,
+  method
+);
+```
+
+Do not assume the model will infer which ID matters. Tell it explicitly.
+
+## Prefer URL-style references where possible
+
+For media tools, prefer public URLs or data URLs in the tool schema when the underlying API supports them.
+
+Common preferred fields:
+
+- `image_url` instead of provider-specific image objects when possible;
+- `audio_url` instead of local binary blobs when possible;
+- `video_url` instead of opaque media response objects when possible;
+- `public_url` for CDN-published assets.
+
+If the wrapper needs a provider-specific shape, map the simple tool input internally.
+
+Example:
+
+```js
+/**
+ * Build provider options from tool parameters.
+ *
+ * @param {Record<string, unknown>} params - Tool parameters.
+ * @returns {Record<string, unknown>} Provider-specific options.
+ */
+function buildCreateOptions(params) {
+  const options = { ...params };
+  delete options.prompt;
+  delete options.image_url;
+
+  if (params.image_url) {
+    options.input_reference = { image_url: params.image_url };
+  }
+
+  return options;
+}
+```
+
+## Keep outputs compact
+
+Tool responses are sent back to the model. Avoid unnecessary transcript bloat.
+
+Do not include:
+
+- raw binary buffers;
+- base64 media unless it is explicitly needed as a durable reference;
+- full HTTP response objects;
+- full provider debug payloads;
+- huge command logs when a concise summary is enough;
+- duplicate aliases beyond useful durable forms.
+
+Do include:
+
+- exact saved file paths;
+- exact public URLs;
+- exact IDs and task handles;
+- byte counts/content types for downloads;
+- concise errors;
+- enough context for the assistant to explain the result.
+
+Example download response:
+
+```js
+return json({
+  tool: 'download_video_openai',
+  success: true,
+  videoId: params.videoId,
+  local_path: result.local_path,
+  localPath: result.local_path,
+  contentType: result.contentType,
+  bytes: result.bytes,
+  note: 'Assistant: tell the user the exact local_path and videoId.'
+});
+```
+
+## Distinguish source references from output references
+
+Edit, remix, extend, inpaint, upscale, and transform operations often take one existing resource and create a new one. The response must make that relationship explicit.
+
+Good:
+
+```js
+return json({
+  tool: 'extend_video_openai',
+  success: true,
+  sourceVideoId: params.videoId,
+  videoId: result.id,
+  id: result.id,
+  status: result.status,
+  note: 'Assistant: tell the user that sourceVideoId is the input video and videoId is the new extended video.'
+});
+```
+
+Bad:
+
+```js
+return result.id;
+```
+
+Bad responses lose context after pruning and can cause the assistant to reuse the wrong ID.
+
+## Error responses should also be structured
+
+If a tool catches and returns errors, return a structured payload.
+
+```js
+return json({
+  tool: 'publish_file',
+  success: false,
+  error: error.message,
+  localPath: params.localPath,
+  note: 'Assistant: explain the failure and preserve the localPath if the user may retry.'
+});
+```
+
+`ToolSet.execute()` already catches thrown errors and converts them to an error string. For tools where retry context matters, catching locally and returning JSON can be more helpful.
+
+## Recommended file structure
+
+A typical toolset file:
+
+```js
+/**
+ * @file lib/API/provider/ExampleToolset.js
+ * @module provider/exampleToolset
+ * @description ToolSet wrapper for provider example API functions.
+ */
+
+import ToolSet from '../../ToolSet.js';
+import * as example from './example.js';
+
+const tools = new ToolSet('auto');
+
+/**
+ * Serialize a tool response as pretty JSON for function-call output.
+ *
+ * @param {Record<string, unknown>} payload - JSON-serializable response payload.
+ * @returns {string} Pretty JSON response.
+ */
+function json(payload) {
+  return JSON.stringify(payload, null, 2);
+}
+
+/**
+ * Build a compact self-describing response for example tools.
+ *
+ * @param {string} tool - Tool name.
+ * @param {Record<string, unknown>} params - Original tool parameters.
+ * @param {Record<string, unknown>} result - Wrapper result.
+ * @returns {string} Tool response JSON.
+ */
+function exampleToolResponse(tool, params, result) {
+  return json({
+    tool,
+    success: true,
+    id: result.id,
+    status: result.status,
+    source_url: params.url,
+    note: 'Assistant: include exact id, status, and source_url in your normal response if useful for follow-up.'
+  });
+}
+
+tools.add(
+  'example_create',
+  'Create an example resource. Returns id and status. After success, include the exact id in your normal assistant response for follow-up calls.',
+  {
+    type: 'object',
+    properties: {
+      prompt: {
+        type: 'string',
+        description: 'Prompt for creating the resource.'
+      },
+      image_url: {
+        type: 'string',
+        description: 'Preferred image reference URL when available.'
+      }
+    },
+    required: ['prompt']
+  },
+  async (params) => {
+    const result = await example.create(params);
+    return exampleToolResponse('example_create', params, result);
+  }
+);
+
+export default tools;
+```
+
+## JSDoc requirements
+
+Use ESM JavaScript and add JSDoc comments for helpers and important functions.
+
+At minimum, document:
+
+- response helper functions;
+- provider option builders;
+- functions that strip raw/binary data;
+- functions that summarize wrapper results;
+- typedefs for repeated response shapes when useful.
+
+Prefer `Record<string, unknown>` for generic JSON-like helper parameters in JSDoc.
+
+## Validation checklist
+
+Before finishing a toolset change, run:
+
+```bash
+node --check lib/path/to/Toolset.js
+```
+
+Then verify import and registered tool names:
+
+```bash
+node --input-type=module -e "import tools from './lib/path/to/Toolset.js'; console.log(tools.length); console.log(tools.list().map(t => t.name).join('\n'));"
+```
+
+Check that:
+
+- every tool name matches `ToolSet` naming rules;
+- every tool has a clear description;
+- schemas have required fields where needed;
+- descriptions prefer URL inputs when possible;
+- responses are JSON strings from `json(...)`;
+- responses preserve durable IDs/URLs/paths/statuses;
+- raw buffers/base64/debug objects are not returned unnecessarily;
+- source IDs and new output IDs are clearly distinguished;
+- response `note` fields tell the assistant what exact references to include in normal text.
+
+## Existing examples
+
+Good examples of this pattern include:
+
+- `lib/API/openai.com/videoToolset.js`
+- `lib/genericToolset.js`
+- `lib/cdnToolset.js`
+- `lib/API/stability.ai/ImageToolset.js`
+- `lib/API/stability.ai/MusicToolset.js`
+- `lib/API/mureka/MusicToolset.js`
+- `lib/API/minimax/ImageToolset.js`
+- `lib/API/minimax/VideoToolset.js`
+- `lib/API/minimax/MusicToolset.js`
+- `lib/API/x.ai/imageToolset.js`
+
+Use these files as references when creating new provider toolsets.

package/docs/docs-organization.md

@@ -0,0 +1,48 @@
+# Documentation Organization Guidelines
+
+**Purpose**  
+The `docs/` folder serves two critical roles:
+1. Human-readable reference.
+2. Machine context for agents (`code_agent`, `agent_creator`, etc.).
+
+## Framework Documentation Source
+
+For `@j-o-r/hello-dave` framework behavior, the authoritative documentation lives only in the `@j-o-r/hello-dave` package docs folder.
+
+Valid framework-doc locations are:
+
+1. **Direct `@j-o-r/hello-dave` checkout**
+   - When CWD is the checked-out `@j-o-r/hello-dave` repository, use CWD-relative docs such as `docs/creating-agents.md`.
+
+2. **Installed package usage**
+   - In all other projects, use the installed package docs, preferably from `node_modules/@j-o-r/hello-dave/docs`.
+   - A global install may be used when no local install is available.
+
+Do not treat another project's `docs/` folder as the source of truth for `hello-dave` framework API behavior.
+
+## Project-local Documentation
+
+Other projects may keep their own `docs/` folders for project-specific conventions, product context, plans, and operational notes.
+
+Those docs can guide project work, but they do not replace the `@j-o-r/hello-dave` framework docs for agent-definition patterns, API option shapes, launcher behavior, or toolset conventions.
+
+`agent_creator` is dedicated to `@j-o-r/hello-dave` agent definitions. It should read framework guidance from the package docs only: CWD-relative docs when inside the direct checkout, otherwise installed package docs under `node_modules` or a global install.
+
+## Folder Structure
+
+- Root `docs/` — high-level current framework documents (`creating-agents.md`, `project-overview.md`, `bin-dave.md`, `docs-organization.md`, etc.).
+- `howtos/`, `plans/`, `prompts/`, `_notes/`, `_legacy/`.
+
+Follow kebab-case, start every file with `# Title`, use consistent Markdown, and cross-link heavily.
+
+## Key Rules
+
+- Accuracy first.
+- Read relevant framework docs before significant framework or agent-definition work.
+- Use direct-checkout docs only for this repository; use installed package docs in all other projects.
+- Update docs after changes that affect patterns or architecture.
+- Treat documentation work as first-class.
+
+This document lives at `docs/docs-organization.md`.
+
+**Current date context**: June 2026.

package/docs/project-overview.md

@@ -6,62 +6,97 @@ The root contains essential project configuration, documentation, and entry poin
 - **TODO.md**: Outstanding tasks and development notes.
 - **package.json**: NPM package metadata, dependencies, and scripts.
 - **LICENSE**: Project license (Apache 2.0).
-- **jsconfig.json** / **tsc.json**: TypeScript/JavaScript configuration for editor and compiler support.
-- **.gitignore** / **.npmignore**: Git and NPM ignore rules.
-- **workspace.js**: Development or workspace management script.
-- Directories: `bin/`, `docs/`, `agents/`, `lib/`, `scenarios/`, `types/`, `utils/`, `release/`, `node_modules/`, `.cache/` (for sessions).
+- **jsconfig.json** / **tsc.json**: TypeScript/JavaScript configuration.
+- **.gitignore** / **.npmignore**.
+- **workspace.js**.
+- Directories: `bin/`, `docs/`, `agents/`, `lib/`, `scenarios/`, `types/`, `utils/`, `release/`, `node_modules/`, `.cache/`.
 
 ## bin/
-Executable CLI scripts (npm bin entries) for spawning and interacting with specialized \"Dave\" agents. These are user-facing tools for quick tasks:
-- `spawn_agent.js`: Generates and runs custom agent scripts.
-- `ask_agent.js`, `chat2_agent.js`: General interactive chat/query.
-- `code_agent.js`: Code-related agent.
-- `prompt_agent.js`, `docs_agent.js`: Prompt engineering and docs management.
-- `readme_agent.js`, `todo_agent.js`: Project maintenance (README/TODO).
-- `npm_agent.js`: NPM module inspection/publishing.
-- Others: `format_agent.js`, `clear_agent.js`, `CodeServer`.
+- `bin/dave.js` (the `dave` binary).
+- Early bare-name delegation to `AgentLauncher` for `*_agent.js` files.
+- Limited built-in commands for cache/session management and discovery.
 
 ## lib/
-Core library code (ESM modules) **named exports only** from `lib/index.js` (NO default export):
-- **index.js**: Main export barrel (`{ AgentManager, Prompt, ToolSet, ... }`).
-- **AgentManager.js**: Central orchestrator for agents—handles setup, sessions, tools, networking (WS server/client), and `start()` for CLI/server modes.
-- **Prompt.js**: Conversation manager (EventEmitter)—tracks messages, tools, truncation, token counting, provider adapters.
-- **ToolSet.js**: Defines/extends tools (JSON Schema + handlers) like bash execution, web search, JS interpreter.
-- **AgentServer.js**, **AgentClient.js**: WebSocket server/client for agent networking.
-- **Session.js**: Persistent chat history/logs (NDJSON in `.cache/`).
-- **Cli.js**: CLI input/output handling.
-- **genericToolset.js**: Pre-configured toolset with common tools.
-- **fafs.js**, **promptHelpers.js**: Utilities (env, file access, prompt ops).
-- **API/**: Provider-specific wrappers (subdirs):
-  - `anthropic.com/`: Claude API.
-  - `brave.com/`: Search API.
-  - `openai.com/`: GPT API.
-  - `x.ai/`: Grok/xAI API (responses, files, collections).
-
-## Agent Workflow
-An **Agent** (via `AgentManager`) is configured with:
-1. **System prompt** (via `Prompt.js`): Defines personality/task.
-2. **API provider** (from `lib/API/`): xAI/Grok, OpenAI/GPT, Anthropic/Claude.
-3. **ToolSet** (`ToolSet.js`): Custom/local functions (e.g., `execute_bash_script`, `web_search`); extensible.
-4. **Session**: Conversation history.
-
-Modes: Standalone CLI, one-shot query, WS **server** (expose as tool), **client** (attach to server), hybrid. `bin/spawn_agent.js` generates custom agents leveraging this.
-
-## docs/
-Project documentation (Markdown) ✅ **Complete** (reviewed Mar 24, 2026):
-- `project-overview.md`: This file.
-- `prompt-class.md`: Deep dive on `Prompt.js`.
-- `toolset.md`: `ToolSet.js` details + 9 generics.
-- `agent-manager.md`: **AgentManager** workflows, options, Mermaid diagrams.
-- `xai-responses.md`: xAI Responses API integration/examples.
-- `xai-collections.md`: xAI Collections (raw; Later priority).
+Core library (named exports from `lib/index.js`):
+
+**Current primary components**:
+- `Agent.js` — Unified minimal Agent class (prompt + api + call_name + call_description). Designed for clean handoff.
+- `AgentLauncher.js` — Discovery, loading, execution, and in-process handoff (fresh context only).
+- `Prompt.js`, `ToolSet.js`, `Session.js`, `Cli.js`
+- `agentLoader.js`
+- `genericToolset.js`, `handOffToolset.js`, `cdnToolset.js`
+- `API/` (providers + toolsets)
+- `fafs.js`, `promptHelpers.js`
+
+**Session model (current and future)**:
+- Interactive CLI provides rich in-memory session continuation with Alt shortcuts (reset, load session, info, etc.).
+- One-shot calls (`dave <agent> "message"`) are independent processes.
+- **Primary direction (no background services)**: Session continuation for one-shots is achieved via explicit CLI flags (`--list-sessions`, `--load-session`, `--reset-session`, `--session-info`, `--session-help`) plus a lightweight per-agent "current session pointer" file. Every invocation is a fresh short-lived process. See `docs/plans/no-service-session-continuation.md`.
+- Previous exploration of file sockets for live attachment has been deprioritized in favor of the zero-service flag-based approach.
+
+**Legacy / de-emphasized** (still present but not the primary path):
+- `AgentServer.js`, `AgentClient.js`, `Swarm.js`, `wsIO.js`, `wsCli.js` — from the previous WebSocket swarm / CodeServer model (abandoned).
+
+## Agent Workflow (Current)
+
+Agents are built with the `Agent` class and executed via `AgentLauncher`:
+- `dave <name>` → interactive CLI or one-shot.
+- Handoffs via `hand_over` / `load_agent` tools (from `API.toolset.generic.handoff`).
+- Fresh handoff: receiving agent gets only its own system prompt + short task-focused context (no history copied).
+- Self-reset supported for deliberate fresh starts.
+
+Old WS server/client/hybrid modes are no longer the focus.
+
+**Session continuation strategy**:
+- Interactive: natural in-memory continuation via the live `Cli`.
+- One-shot / scripting: explicit flags + pointer file to preload previous session history from the existing NDJSON storage. No daemons or listeners.
+
+## Creating Agents
+
+**Canonical guide**: **[docs/creating-agents.md](creating-agents.md)**
+
+Core requirements:
+- `Agent.deriveCallName(import.meta.url)`
+- `new Agent({ ... call_name, call_description ... })`
+- `export default agent;`
+- Optional sibling `.prompt.md` + `Agent.loadPrompt(import.meta.url)`
+- Register tools + `toolset.borrow(API.toolset.generic.handoff)` for collaboration
+
+## Documentation
+
+Follow **[docs/docs-organization.md](docs-organization.md)**.
+
+**Current / active documents** (root `docs/` and subdirs):
+- `creating-agents.md` (the standard)
+- `bin-dave.md`
+- `project-overview.md` (this file)
+- `toolset.md`, `generic-toolset.md`
+- `docs-organization.md`
+- `howtos/`, `plans/`, `prompts/`
+- `plans/no-service-session-continuation.md` (primary session strategy for one-shots)
+- `plans/agent-file-socket-communication.md` (earlier exploration, superseded for the no-service requirement)
+
+**Legacy documents** (abandoned WS/swarm/CodeServer model):
+- Located in `docs/_legacy/`
+- `multi-agent-clusters.md`
+- `codeserver-pattern.md`
+- `agent-networking.md`
+- `agent-dave-websocket-protocol.md`
+
+These are kept only for historical reference. New work must not rely on the patterns they describe. All legacy files start with a clear "Legacy / Historical" notice and redirect to current docs.
 
 ## Other Directories
-- **agents/**: Usage demos (e.g., `daisy_agent.js` music agent).
-- **scenarios/**: Tests/demos (e.g., `grok_agent.js`, `wsAgent.js`).
-- **types/**: TypeScript `.d.ts` files.
-- **utils/**: Miscellaneous utilities.
-- **release/**: NPM pack artifacts.
+- **agents/**: Discoverable agents (all follow the modern `Agent` pattern).
+- **scenarios/**, **types/**, **utils/**.
 
 ## Summary
-`hello-dave` (@j-o-r/hello-dave NPM pkg) is a lightweight ESM Node.js framework for **tool-using AI agents** with unified LLM access (Grok/Claude/GPT), WS networking, and CLI tools. **Import via named exports: `import { AgentManager } from '@j-o-r/hello-dave';`**. Focus: Rapid prototyping of domain agents (code, docs, music) via `AgentManager` + `Prompt` + `ToolSet` + APIs. Sessions in `.cache/`. Node >=20.
+
+`@j-o-r/hello-dave` is a lightweight ESM toolkit for tool-using AI agents with unified access to Grok / OpenAI / Anthropic.
+
+**Current focus**: Clean `Agent` + `AgentLauncher` + rich toolsets + in-process handoff + practical session continuation for one-shots **without any background services**.
+
+The architecture was deliberately simplified in 2026 by removing the complex persistent WebSocket cluster layer.
+
+**Import**: `import { Agent, AgentLauncher, API, ... } from '@j-o-r/hello-dave';`
+
+*Last updated: June 2026 (session continuation strategy clarified to no-service flag-based model)*