open-agents-ai 0.34.2 → 0.34.3

package/README.md

@@ -29,7 +29,7 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 
 ## Features
 
-- **35+ autonomous tools** — file I/O, shell, grep, web search/fetch/crawl, memory, sub-agents, background tasks, image/OCR, git, diagnostics, vision, desktop automation, structured files, code sandbox
+- **47 autonomous tools** — file I/O, shell, grep, web search/fetch/crawl, memory (read/write/search), sub-agents, background tasks, image/OCR/PDF, git, diagnostics, vision, desktop automation, structured files, code sandbox, transcription, skills
 - **Moondream vision** — see and interact with the desktop via Moondream VLM (caption, query, detect, point-and-click)
 - **Desktop automation** — vision-guided clicking: describe a UI element in natural language, the agent finds and clicks it
 - **Auto-install desktop deps** — screenshot, mouse, OCR, and image tools auto-install missing system packages (scrot, xdotool, tesseract, imagemagick) on first use
@@ -39,6 +39,7 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 - **Dream Mode** — creative idle exploration modeled after real sleep architecture (NREM→REM cycles)
 - **Live Listen** — bidirectional voice communication with real-time Whisper transcription
 - **Neural TTS** — hear what the agent is doing via GLaDOS or Overwatch ONNX voices
+- **Human expert speed ratio** — real-time `Exp: Nx` gauge comparing agent speed to a leading human expert, calibrated across 47 tool baselines
 - **Cost tracking** — real-time token cost estimation for 15+ cloud providers
 - **Work evaluation** — LLM-as-judge scoring with task-type-specific rubrics
 - **Session metrics** — track turns, tool calls, tokens, files modified, tasks completed per session
@@ -371,7 +372,7 @@ Launch without arguments to enter the interactive REPL:
 oa
 ```
 
-The TUI features an animated multilingual phrase carousel, live metrics bar with pastel-colored labels (token in/out, context window usage), rotating tips, syntax-highlighted tool output, and dynamic terminal-width cropping.
+The TUI features an animated multilingual phrase carousel, live metrics bar with pastel-colored labels (token in/out, context window usage, human expert speed ratio, cost), rotating tips, syntax-highlighted tool output, and dynamic terminal-width cropping.
 
 ### Slash Commands
 
@@ -433,7 +434,7 @@ While the agent is working (shown by the `+` prompt), type to add context:
   ⎿  Edit: src/auth.ts
 ```
 
-## Tools (37)
+## Tools (47)
 
 | Tool | Description |
 |------|-------------|
@@ -447,9 +448,9 @@ While the agent is working (shown by the `+` prompt), type to add context:
 | **Search & Navigation** | |
 | `grep_search` | Search file contents with regex (ripgrep with grep fallback) |
 | `find_files` | Find files by glob pattern (excludes node_modules/.git) |
-| `codebase_map` | High-level project structure overview with directory tree |
+| `codebase_map` | High-level project structure overview with directory tree and language breakdown |
 | **Shell & Execution** | |
-| `shell` | Execute any shell command (non-interactive, CI=true) |
+| `shell` | Execute any shell command (non-interactive, CI=true, sudo support) |
 | `code_sandbox` | Isolated code execution (JS, Python, Bash, TS) in subprocess or Docker |
 | `background_run` | Run shell command in background, returns task ID |
 | `task_status` | Check background task status |
@@ -461,26 +462,34 @@ While the agent is working (shown by the `+` prompt), type to add context:
 | `web_crawl` | Multi-page web scraping with Crawlee/Playwright for deep documentation |
 | **Structured Data** | |
 | `structured_file` | Generate CSV, TSV, JSON, Markdown tables, Excel-compatible files |
-| `read_structured_file` | Parse CSV, TSV, JSON, Markdown tables with binary detection |
+| `structured_read` | Parse CSV, TSV, JSON, Markdown tables with binary format detection |
 | **Vision & Desktop** | |
 | `vision` | Moondream VLM — caption, query, detect, point on any image |
 | `desktop_click` | Vision-guided clicking: describe a UI element, agent finds and clicks it |
 | `desktop_describe` | Screenshot + Moondream caption/query for desktop awareness |
-| `image_read` | Read images (base64 + OCR) |
+| `image_read` | Read images (base64 + OCR metadata) |
 | `screenshot` | Capture screen/window/active window |
 | `ocr` | Extract text from images (Tesseract with multi-variant preprocessing) |
+| `ocr_image_advanced` | Advanced multi-variant OCR pipeline with preprocessing, multi-PSM, and confidence scoring |
 | `ocr_pdf` | Add searchable text layer to scanned/image PDFs |
-| `pdf_to_text` | Extract text from PDF using pdftotext (Poppler) |
-| `transcribe_file` | Transcribe audio/video to text (Whisper) |
+| `pdf_to_text` | Extract text from PDF using pdftotext (Poppler) with OCR fallback |
+| **Transcription** | |
+| `transcribe_file` | Transcribe local audio/video files to text (Whisper) |
+| `transcribe_url` | Download and transcribe audio/video from URLs |
 | **Memory & Knowledge** | |
-| `memory_read` | Read from persistent memory store by topic |
+| `memory_read` | Read from persistent memory store by topic and key |
 | `memory_write` | Store facts/patterns in persistent memory with provenance tracking |
+| `memory_search` | Semantic search across all memory entries by query |
 | `memex_retrieve` | Recover full tool output archived during context compaction by hash ID |
 | **Git & Diagnostics** | |
 | `diagnostic` | Lint/typecheck/test/build validation pipeline in one call |
 | `git_info` | Structured git status, log, diff, branch, staged/unstaged files |
-| **Agents & Skills** | |
-| `create_tool` | Create reusable custom tools from workflow at runtime |
+| **Agents & Delegation** | |
+| `sub_agent` | Delegate subtasks to independent agent instances (foreground or background) |
+| `explore_tools` | Meta-tool: discover and unlock additional tools on demand (for small models) |
+| `task_complete` | Signal task completion with summary |
+| **Custom Tools & Skills** | |
+| `create_tool` | Create reusable custom tools from workflow patterns at runtime |
 | `manage_tools` | List, inspect, delete custom tools |
 | `skill_list` | Discover available AIWG skills |
 | `skill_execute` | Run an AIWG skill |
@@ -510,13 +519,13 @@ Open Agents classifies models into three tiers and adapts its behavior according
 
 | Tier | Parameters | Base Tools | System Prompt | Compaction |
 |------|-----------|------------|---------------|------------|
-| **Large** (≥30B) | 70B, 122B | All 37 tools | Full (344 lines) | 40K threshold |
+| **Large** (≥30B) | 70B, 122B | All 47 tools | Full (344 lines) | 40K threshold |
 | **Medium** (8-29B) | 9B, 27B | 15 core tools | Condensed (100 lines) | 24K threshold |
 | **Small** (≤7B) | 4B, 1.5B | 6 base tools + explore_tools | Minimal (15 lines) | 12K threshold |
 
 ### Tool Nesting for Small Models
 
-Small models use an **explore_tools** meta-tool pattern inspired by hierarchical API retrieval research (ToolLLM, arXiv:2307.16789). Instead of presenting all 37 tools (which overwhelms small context windows), only 6 core tools are loaded initially:
+Small models use an **explore_tools** meta-tool pattern inspired by hierarchical API retrieval research (ToolLLM, arXiv:2307.16789). Instead of presenting all 47 tools (which overwhelms small context windows), only 6 core tools are loaded initially:
 
 - `file_read`, `file_write`, `file_edit`, `shell`, `task_complete`, `explore_tools`
 
@@ -549,6 +558,45 @@ All context-dependent values scale automatically with the actual context window
 
 Auto-downloads the ONNX voice model (~50MB) on first use. Install `espeak-ng` for best quality (`apt install espeak-ng` / `brew install espeak-ng`).
 
+## Human Expert Speed Ratio
+
+The status bar displays a real-time `Exp: Nx` gauge estimating how fast the agent is working relative to a leading human expert performing equivalent tasks.
+
+```
+In: 12,345 | Out: 4,567 | Ctx: 18,000/131,072 86% | Exp: 4.2x | Cost: $0.34
+                                                       ^^^^^^^^
+                                                    Agent is 4.2x faster
+                                                    than a human expert
+```
+
+### How It Works
+
+Each tool call maps to a calibrated expert baseline time — the estimated seconds a top-tier human developer would take to perform the equivalent operation manually:
+
+| Operation | Expert Time | Agent Equivalent |
+|-----------|-------------|-----------------|
+| Read a file | 12s | `file_read` |
+| Write a new file | 90s | `file_write` |
+| Make a precise edit | 25s | `file_edit` |
+| Grep search + scan results | 15s | `grep_search` |
+| Run a shell command | 20s | `shell` |
+| Web search + evaluate | 60s | `web_search` |
+| Survey codebase structure | 180s | `codebase_map` |
+
+Additional overhead per action:
+- **+5s context-switch** per tool call (expert switching between tools)
+- **+15s planning** per reasoning turn (expert thinking about next step)
+
+The ratio accumulates across all tasks in the session:
+
+```
+speedRatio = totalHumanExpertTime / totalAgentWallClockTime
+```
+
+Color coding: green (2x+ faster), yellow (1-2x, comparable), red (<1x, slower than expert).
+
+All 47 tools have calibrated baselines ranging from 3s (`task_stop`) to 180s (`codebase_map`). Unknown tools default to 20s.
+
 ## Cost Tracking & Session Metrics
 
 Real-time token cost estimation for cloud providers. The status bar shows running cost when using a paid endpoint.
@@ -717,7 +765,7 @@ The agent auto-detects the provider, normalizes the URL (strips `/v1/chat/comple
 
 ## Evaluation Suite
 
-33 evaluation tasks test the agent's autonomous capabilities across coding, web research, SDLC analysis, tool creation, and multi-file reasoning:
+40 evaluation tasks test the agent's autonomous capabilities across coding, web research, SDLC analysis, tool creation, multi-file reasoning, and memory systems:
 
 ```bash
 node eval/run-agentic.mjs                          # Run all tasks
@@ -754,15 +802,23 @@ node eval/run-agentic.mjs --model qwen2.5-coder:32b  # Different model
 | 31 | Web extractor bug fixes (3 bugs) | Multi-Bug Fix |
 | 32 | CSV pipeline across 3 files | Multi-File Tracking |
 | 33 | FSM bug fixes + factory implementation | State Machine |
+| 34 | Search pre-populated memories | Memory Search |
+| 35 | Analyze code, write to memory, cross-reference | Memory Cross-Reference |
+| 36 | Discover explore_tools, unlock grep_search | Explore Tools |
+| 37 | Analyze code patterns, store and recall from memory | Memory Store & Recall |
+| 38 | Read configs, write to multiple memory topics | Memory Multi-Topic |
+| 39 | Search pre-loaded memories across 3 topics | Memory Pre-Loaded Search |
+| 40 | Combined explore_tools + memory analysis pipeline | Explore + Memory |
 
-Tasks 31-33 are designed for small model (≤9B) evaluation using `file_edit` patterns instead of `file_write` to avoid JSON truncation issues with smaller models.
+Tasks 31-33 are designed for small model (≤9B) evaluation using `file_edit` patterns. Tasks 34-40 test the memory system (read/write/search) and tool discovery.
 
 ### Benchmark Results
 
 ```
-Qwen3.5-122B: 100% pass rate (30/30 tasks)
+Qwen3.5-122B: 100% pass rate (37/37 tasks, including memory tasks 34-40)
 Qwen3.5-27B:  100% pass rate (30/30 tasks)
 Qwen3.5-9B:   100% pass rate (tasks 31-33, file_edit-optimized)
+              71% pass rate (5/7 memory tasks 34-40)
 ```
 
 The eval runner includes model-tier-aware features: automatic tool set filtering, HTTP 500 recovery with file_edit hints, loop detection with tool banning, and tier-based output truncation.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "open-agents-ai",
-  "version": "0.34.2",
+  "version": "0.34.3",
   "description": "AI coding agent powered by open-source models (Ollama/vLLM) — interactive TUI with agentic tool-calling loop",
   "type": "module",
   "main": "./dist/index.js",