open-agents-ai 0.17.0 → 0.19.0

package/README.md

@@ -29,13 +29,24 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 
 ## Features
 
-- **26 autonomous tools** — file I/O, shell, grep, web search/fetch, memory, sub-agents, background tasks, image/OCR, git, diagnostics
+- **35 autonomous tools** — file I/O, shell, grep, web search/fetch, memory, sub-agents, background tasks, image/OCR, git, diagnostics, vision, desktop automation, structured files, code sandbox
+- **Moondream vision** — see and interact with the desktop via Moondream VLM (caption, query, detect, point-and-click)
+- **Desktop automation** — vision-guided clicking: describe a UI element in natural language, the agent finds and clicks it
+- **Auto-install desktop deps** — screenshot, mouse, OCR, and image tools auto-install missing system packages (scrot, xdotool, tesseract, imagemagick) on first use
 - **Parallel tool execution** — read-only tools run concurrently via `Promise.allSettled`
 - **Sub-agent delegation** — spawn independent agents for parallel workstreams
 - **Ralph Loop** — iterative task execution that keeps retrying until completion criteria are met
 - **Dream Mode** — creative idle exploration modeled after real sleep architecture (NREM→REM cycles)
 - **Live Listen** — bidirectional voice communication with real-time Whisper transcription
 - **Neural TTS** — hear what the agent is doing via GLaDOS or Overwatch ONNX voices
+- **Cost tracking** — real-time token cost estimation for 15+ cloud providers
+- **Work evaluation** — LLM-as-judge scoring with task-type-specific rubrics
+- **Session metrics** — track turns, tool calls, tokens, files modified, tasks completed per session
+- **Structured file generation** — create CSV, TSV, JSON, Markdown tables, and Excel-compatible files
+- **Code sandbox** — isolated code execution in subprocess or Docker (JS, Python, Bash, TypeScript)
+- **Structured file reading** — parse CSV, TSV, JSON, Markdown tables with binary format detection
+- **Multi-provider web search** — DuckDuckGo (free), Tavily (structured), Jina AI (markdown) with auto-detection
+- **Task templates** — specialized system prompts and tool recommendations for code, document, analysis, plan tasks
 - **Auto-expanding context** — detects RAM/VRAM and creates an optimized model variant on first run
 - **Mid-task steering** — type while the agent works to add context without interrupting
 - **Smart compaction** — long conversations compressed preserving files, commands, errors, decisions
@@ -139,6 +150,105 @@ The `transcribe-cli` dependency auto-installs in the background on first use.
 
 **File transcription**: Drag-and-drop audio/video files (`.mp3`, `.wav`, `.mp4`, `.mkv`, etc.) onto the terminal to transcribe them. Results are saved to `.oa/transcripts/`.
 
+## Vision & Desktop Automation (Moondream)
+
+Open Agents can see your screen, understand UI elements, and interact with desktop applications through natural language — powered by the Moondream vision language model running entirely locally.
+
+### Desktop Awareness
+
+The agent can take a screenshot and describe what's on screen:
+
+```
+You: what's on my desktop right now?
+
+Agent: [Turn 1] desktop_describe()
+       → "A Linux desktop showing three terminal windows with code editors,
+          a file manager in the background, and a taskbar at the bottom
+          with Firefox, Files, and Terminal icons."
+```
+
+Ask specific questions about the screen:
+
+```
+Agent: [Turn 1] desktop_describe(question="What application is in focus?")
+       → "The focused application is a terminal running vim with a Python file open."
+```
+
+### Vision Analysis
+
+Analyze any image with four actions:
+
+```
+Agent: vision(image="screenshot.png", action="caption")
+       → "A terminal window displaying code with syntax highlighting"
+
+Agent: vision(image="ui.png", action="query", prompt="How many buttons are visible?")
+       → "There are 4 buttons visible: Save, Cancel, Help, and Close"
+
+Agent: vision(image="ui.png", action="detect", prompt="button")
+       → Detected 4 "button" in ui.png:
+         1. bbox: [0.10, 0.85, 0.25, 0.95]
+         2. bbox: [0.30, 0.85, 0.45, 0.95]
+         ...
+
+Agent: vision(image="ui.png", action="point", prompt="close button")
+       → Found 1 "close button" at (0.95, 0.02) — pixel (1824, 22)
+```
+
+### Point-and-Click
+
+Describe what to click in plain English — the agent screenshots, finds the element with Moondream, and clicks it:
+
+```
+Agent: desktop_click(target="the Save button")
+       → Clicked "Save button" at (480, 920)
+
+Agent: desktop_click(target="File menu", button="left")
+       → Clicked "File menu" at (45, 12)
+
+Agent: desktop_click(target="terminal icon", click_type="double")
+       → Clicked "terminal icon" at (1850, 540)
+```
+
+Supports left/right/middle click, single/double click, multi-match selection by index, dry-run mode for verification, and configurable delay for UI transitions.
+
+### Setup
+
+Moondream runs locally — no API keys, no cloud, your screen data never leaves your machine:
+
+```bash
+# Create a Python venv and install Moondream Station
+python3 -m venv .moondream-venv
+.moondream-venv/bin/pip install moondream-station pydantic uvicorn fastapi packaging
+
+# Start the vision server (downloads model on first run, ~1.7GB)
+.moondream-venv/bin/python packages/execution/scripts/start-moondream.py
+```
+
+The vision tools auto-detect a running Moondream Station on `localhost:2020`. For cloud inference, set `MOONDREAM_API_KEY` instead.
+
+**System dependencies (auto-installed on first use):**
+
+Desktop tools automatically install missing system packages when first needed. No manual setup required — just use the tool and it handles the rest:
+
+| Tool | Linux Package | What It Does |
+|------|--------------|-------------|
+| `scrot` | `apt install scrot` | Screenshot capture |
+| `xdotool` | `apt install xdotool` | Mouse/keyboard automation |
+| `tesseract` | `apt install tesseract-ocr` | OCR text extraction |
+| `identify` | `apt install imagemagick` | Image dimensions/conversion |
+
+Supports `apt` (Debian/Ubuntu), `dnf` (Fedora), `pacman` (Arch), and `brew` (macOS). You can also pre-install everything at once:
+
+```bash
+./scripts/setup-desktop.sh          # Install all desktop deps
+./scripts/setup-desktop.sh --check-only  # Just check what's missing
+```
+
+**Vision backend:**
+- Moondream Station (local) — runs entirely on your machine, no API keys needed
+- Moondream Cloud API — set `MOONDREAM_API_KEY` for cloud inference
+
 ## Interactive TUI
 
 Launch without arguments to enter the interactive REPL:
@@ -164,6 +274,10 @@ The TUI features an animated multilingual phrase carousel, live metrics bar with
 | `/tools` | List available tools |
 | `/skills` | List/search available skills |
 | `/update` | Check for and install updates (seamless reload) |
+| `/cost` | Show token cost breakdown for the current session |
+| `/evaluate` | Score the last completed task with LLM-as-judge |
+| `/stats` | Show session metrics (turns, tools, tokens, files) |
+| `/task-type <type>` | Set task type for specialized prompts (code, document, analysis, plan) |
 | `/config` | Show current configuration |
 | `/clear` | Clear the screen |
 | `/exit` | Quit |
@@ -181,33 +295,53 @@ While the agent is working (shown by the `+` prompt), type to add context:
   ⎿  Edit: src/auth.ts
 ```
 
-## Tools (26)
+## Tools (35)
 
 | Tool | Description |
 |------|-------------|
+| **File Operations** | |
 | `file_read` | Read file contents with line numbers (offset/limit) |
 | `file_write` | Create or overwrite files |
 | `file_edit` | Precise string replacement in files |
-| `shell` | Execute any shell command |
+| `batch_edit` | Multiple edits across files in one call |
+| `list_directory` | List directory contents |
+| **Search & Navigation** | |
 | `grep_search` | Search file contents with regex (ripgrep) |
 | `find_files` | Find files by glob pattern |
-| `list_directory` | List directory contents |
-| `web_search` | Search the web via DuckDuckGo |
-| `web_fetch` | Fetch and extract text from web pages |
-| `memory_read` | Read from persistent memory store |
-| `memory_write` | Store patterns for future sessions |
-| `batch_edit` | Multiple edits across files in one call |
 | `codebase_map` | High-level project structure overview |
-| `diagnostic` | Lint/typecheck/test/build validation pipeline |
-| `git_info` | Structured git status, log, diff, branch info |
+| **Shell & Execution** | |
+| `shell` | Execute any shell command |
+| `code_sandbox` | Isolated code execution (JS, Python, Bash, TS) in subprocess or Docker |
 | `background_run` | Run shell command in background |
 | `task_status` | Check background task status |
 | `task_output` | Read background task output |
 | `task_stop` | Stop a background task |
-| `sub_agent` | Delegate to an independent agent |
+| **Web** | |
+| `web_search` | Search the web (DuckDuckGo, Tavily, Jina AI — auto-detected) |
+| `web_fetch` | Fetch and extract text from web pages |
+| **Structured Data** | |
+| `structured_file` | Generate CSV, TSV, JSON, Markdown tables, Excel-compatible files |
+| `read_structured_file` | Parse CSV, TSV, JSON, Markdown tables with binary detection |
+| **Vision & Desktop** | |
+| `vision` | Moondream VLM — caption, query, detect, point on any image |
+| `desktop_click` | Vision-guided clicking: describe a UI element, agent finds and clicks it |
+| `desktop_describe` | Screenshot + Moondream caption/query for desktop awareness |
 | `image_read` | Read images (base64 + OCR) |
 | `screenshot` | Capture screen/window |
-| `ocr` | Extract text from images |
+| `ocr` | Extract text from images (Tesseract) |
+| **Memory & Knowledge** | |
+| `memory_read` | Read from persistent memory store |
+| `memory_write` | Store patterns for future sessions |
+| **Git & Diagnostics** | |
+| `diagnostic` | Lint/typecheck/test/build validation pipeline |
+| `git_info` | Structured git status, log, diff, branch info |
+| **Agents & Skills** | |
+| `sub_agent` | Delegate to an independent agent |
+| `create_tool` | Create reusable custom tools at runtime |
+| `manage_tools` | List, inspect, delete custom tools |
+| `skill_list` | Discover available AIWG skills |
+| `skill_execute` | Run an AIWG skill |
+| **AIWG SDLC** | |
 | `aiwg_setup` | Deploy AIWG SDLC framework |
 | `aiwg_health` | Analyze SDLC health |
 | `aiwg_workflow` | Execute AIWG workflows |
@@ -237,6 +371,88 @@ On startup and `/model` switch, Open Agents detects your RAM/VRAM and creates an
 
 Auto-downloads the ONNX voice model (~50MB) on first use. Install `espeak-ng` for best quality (`apt install espeak-ng` / `brew install espeak-ng`).
 
+## Cost Tracking & Session Metrics
+
+Real-time token cost estimation for cloud providers. The status bar shows running cost when using a paid endpoint.
+
+```
+/cost              # Show cost breakdown by model/provider
+/stats             # Session metrics: turns, tool calls, tokens, files modified
+/evaluate          # Score the last completed task (LLM-as-judge, 5 rubric dimensions)
+```
+
+Cost tracking supports 15+ providers including Groq, Together AI, OpenRouter, Fireworks AI, DeepInfra, Mistral, Cerebras, and more. Pricing is per-million tokens with separate input/output rates.
+
+Work evaluation uses five task-type-specific rubrics (code, document, analysis, plan, general) scoring correctness, completeness, efficiency, code quality, and communication on a 1-5 scale.
+
+## Code Sandbox
+
+Execute code snippets in isolated environments without affecting your project:
+
+```
+Agent: code_sandbox(language="python", code="import math; print(math.factorial(20))")
+       → 2432902008176640000
+
+Agent: code_sandbox(language="javascript", code="console.log([...new Set([1,2,2,3])].length)")
+       → 3
+```
+
+Supports JavaScript, TypeScript, Python, and Bash. Two execution modes:
+- **Subprocess** (default) — runs in a child process with timeout and output limits
+- **Docker** — runs in an isolated container when `docker` is available
+
+## Structured Data Tools
+
+### Generate structured files
+
+Create CSV, TSV, JSON, Markdown tables, and Excel-compatible files from data:
+
+```
+Agent: structured_file(format="csv", path="results.csv", columns=["name","score"],
+         data=[{"name":"Alice","score":95},{"name":"Bob","score":87}])
+       → Created results.csv (2 rows, 2 columns)
+```
+
+### Read structured files
+
+Parse existing data files with automatic format detection:
+
+```
+Agent: read_structured_file(path="data.csv")
+       → CSV: 150 rows, 5 columns [showing first 100]
+
+Agent: read_structured_file(path="report.md")
+       → Markdown: 3 table(s) extracted
+```
+
+Detects binary formats (XLSX, PDF, DOCX) and suggests conversion tools.
+
+## Multi-Provider Web Search
+
+Web search automatically selects the best available provider:
+
+| Provider | Trigger | Features |
+|----------|---------|----------|
+| **DuckDuckGo** | Default (no key needed) | Free, privacy-focused |
+| **Tavily** | `TAVILY_API_KEY` set | Structured results + AI-generated answer |
+| **Jina AI** | `JINA_API_KEY` set | Markdown-formatted results |
+
+```bash
+export TAVILY_API_KEY=tvly-...   # Enable Tavily (optional)
+export JINA_API_KEY=jina_...     # Enable Jina AI (optional)
+```
+
+## Task Templates
+
+Set a task type to get specialized system prompts, recommended tools, and output guidance:
+
+```
+/task-type code       # Code generation/fix — emphasizes tests, diffs, file edits
+/task-type document   # Documentation — emphasizes clarity, structure, completeness
+/task-type analysis   # Analysis tasks — emphasizes data, metrics, evidence
+/task-type plan       # Planning — emphasizes steps, dependencies, risks
+```
+
 ## Configuration
 
 Config priority: CLI flags > env vars > `~/.open-agents/config.json` > defaults.