open-agents-ai 0.34.1 → 0.34.3

package/README.md

@@ -29,7 +29,7 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 
 ## Features
 
-- **35+ autonomous tools** — file I/O, shell, grep, web search/fetch/crawl, memory, sub-agents, background tasks, image/OCR, git, diagnostics, vision, desktop automation, structured files, code sandbox
+- **47 autonomous tools** — file I/O, shell, grep, web search/fetch/crawl, memory (read/write/search), sub-agents, background tasks, image/OCR/PDF, git, diagnostics, vision, desktop automation, structured files, code sandbox, transcription, skills
 - **Moondream vision** — see and interact with the desktop via Moondream VLM (caption, query, detect, point-and-click)
 - **Desktop automation** — vision-guided clicking: describe a UI element in natural language, the agent finds and clicks it
 - **Auto-install desktop deps** — screenshot, mouse, OCR, and image tools auto-install missing system packages (scrot, xdotool, tesseract, imagemagick) on first use
@@ -39,6 +39,7 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 - **Dream Mode** — creative idle exploration modeled after real sleep architecture (NREM→REM cycles)
 - **Live Listen** — bidirectional voice communication with real-time Whisper transcription
 - **Neural TTS** — hear what the agent is doing via GLaDOS or Overwatch ONNX voices
+- **Human expert speed ratio** — real-time `Exp: Nx` gauge comparing agent speed to a leading human expert, calibrated across 47 tool baselines
 - **Cost tracking** — real-time token cost estimation for 15+ cloud providers
 - **Work evaluation** — LLM-as-judge scoring with task-type-specific rubrics
 - **Session metrics** — track turns, tool calls, tokens, files modified, tasks completed per session
@@ -371,7 +372,7 @@ Launch without arguments to enter the interactive REPL:
 oa
 ```
 
-The TUI features an animated multilingual phrase carousel, live metrics bar with pastel-colored labels (token in/out, context window usage), rotating tips, syntax-highlighted tool output, and dynamic terminal-width cropping.
+The TUI features an animated multilingual phrase carousel, live metrics bar with pastel-colored labels (token in/out, context window usage, human expert speed ratio, cost), rotating tips, syntax-highlighted tool output, and dynamic terminal-width cropping.
 
 ### Slash Commands
 
@@ -433,7 +434,7 @@ While the agent is working (shown by the `+` prompt), type to add context:
   ⎿  Edit: src/auth.ts
 ```
 
-## Tools (37)
+## Tools (47)
 
 | Tool | Description |
 |------|-------------|
@@ -447,9 +448,9 @@ While the agent is working (shown by the `+` prompt), type to add context:
 | **Search & Navigation** | |
 | `grep_search` | Search file contents with regex (ripgrep with grep fallback) |
 | `find_files` | Find files by glob pattern (excludes node_modules/.git) |
-| `codebase_map` | High-level project structure overview with directory tree |
+| `codebase_map` | High-level project structure overview with directory tree and language breakdown |
 | **Shell & Execution** | |
-| `shell` | Execute any shell command (non-interactive, CI=true) |
+| `shell` | Execute any shell command (non-interactive, CI=true, sudo support) |
 | `code_sandbox` | Isolated code execution (JS, Python, Bash, TS) in subprocess or Docker |
 | `background_run` | Run shell command in background, returns task ID |
 | `task_status` | Check background task status |
@@ -461,26 +462,34 @@ While the agent is working (shown by the `+` prompt), type to add context:
 | `web_crawl` | Multi-page web scraping with Crawlee/Playwright for deep documentation |
 | **Structured Data** | |
 | `structured_file` | Generate CSV, TSV, JSON, Markdown tables, Excel-compatible files |
-| `read_structured_file` | Parse CSV, TSV, JSON, Markdown tables with binary detection |
+| `structured_read` | Parse CSV, TSV, JSON, Markdown tables with binary format detection |
 | **Vision & Desktop** | |
 | `vision` | Moondream VLM — caption, query, detect, point on any image |
 | `desktop_click` | Vision-guided clicking: describe a UI element, agent finds and clicks it |
 | `desktop_describe` | Screenshot + Moondream caption/query for desktop awareness |
-| `image_read` | Read images (base64 + OCR) |
+| `image_read` | Read images (base64 + OCR metadata) |
 | `screenshot` | Capture screen/window/active window |
 | `ocr` | Extract text from images (Tesseract with multi-variant preprocessing) |
+| `ocr_image_advanced` | Advanced multi-variant OCR pipeline with preprocessing, multi-PSM, and confidence scoring |
 | `ocr_pdf` | Add searchable text layer to scanned/image PDFs |
-| `pdf_to_text` | Extract text from PDF using pdftotext (Poppler) |
-| `transcribe_file` | Transcribe audio/video to text (Whisper) |
+| `pdf_to_text` | Extract text from PDF using pdftotext (Poppler) with OCR fallback |
+| **Transcription** | |
+| `transcribe_file` | Transcribe local audio/video files to text (Whisper) |
+| `transcribe_url` | Download and transcribe audio/video from URLs |
 | **Memory & Knowledge** | |
-| `memory_read` | Read from persistent memory store by topic |
+| `memory_read` | Read from persistent memory store by topic and key |
 | `memory_write` | Store facts/patterns in persistent memory with provenance tracking |
+| `memory_search` | Semantic search across all memory entries by query |
 | `memex_retrieve` | Recover full tool output archived during context compaction by hash ID |
 | **Git & Diagnostics** | |
 | `diagnostic` | Lint/typecheck/test/build validation pipeline in one call |
 | `git_info` | Structured git status, log, diff, branch, staged/unstaged files |
-| **Agents & Skills** | |
-| `create_tool` | Create reusable custom tools from workflow at runtime |
+| **Agents & Delegation** | |
+| `sub_agent` | Delegate subtasks to independent agent instances (foreground or background) |
+| `explore_tools` | Meta-tool: discover and unlock additional tools on demand (for small models) |
+| `task_complete` | Signal task completion with summary |
+| **Custom Tools & Skills** | |
+| `create_tool` | Create reusable custom tools from workflow patterns at runtime |
 | `manage_tools` | List, inspect, delete custom tools |
 | `skill_list` | Discover available AIWG skills |
 | `skill_execute` | Run an AIWG skill |
@@ -510,13 +519,13 @@ Open Agents classifies models into three tiers and adapts its behavior according
 
 | Tier | Parameters | Base Tools | System Prompt | Compaction |
 |------|-----------|------------|---------------|------------|
-| **Large** (≥30B) | 70B, 122B | All 37 tools | Full (344 lines) | 40K threshold |
+| **Large** (≥30B) | 70B, 122B | All 47 tools | Full (344 lines) | 40K threshold |
 | **Medium** (8-29B) | 9B, 27B | 15 core tools | Condensed (100 lines) | 24K threshold |
 | **Small** (≤7B) | 4B, 1.5B | 6 base tools + explore_tools | Minimal (15 lines) | 12K threshold |
 
 ### Tool Nesting for Small Models
 
-Small models use an **explore_tools** meta-tool pattern inspired by hierarchical API retrieval research (ToolLLM, arXiv:2307.16789). Instead of presenting all 37 tools (which overwhelms small context windows), only 6 core tools are loaded initially:
+Small models use an **explore_tools** meta-tool pattern inspired by hierarchical API retrieval research (ToolLLM, arXiv:2307.16789). Instead of presenting all 47 tools (which overwhelms small context windows), only 6 core tools are loaded initially:
 
 - `file_read`, `file_write`, `file_edit`, `shell`, `task_complete`, `explore_tools`
 
@@ -549,6 +558,45 @@ All context-dependent values scale automatically with the actual context window
 
 Auto-downloads the ONNX voice model (~50MB) on first use. Install `espeak-ng` for best quality (`apt install espeak-ng` / `brew install espeak-ng`).
 
+## Human Expert Speed Ratio
+
+The status bar displays a real-time `Exp: Nx` gauge estimating how fast the agent is working relative to a leading human expert performing equivalent tasks.
+
+```
+In: 12,345 | Out: 4,567 | Ctx: 18,000/131,072 86% | Exp: 4.2x | Cost: $0.34
+                                                       ^^^^^^^^
+                                                    Agent is 4.2x faster
+                                                    than a human expert
+```
+
+### How It Works
+
+Each tool call maps to a calibrated expert baseline time — the estimated seconds a top-tier human developer would take to perform the equivalent operation manually:
+
+| Operation | Expert Time | Agent Equivalent |
+|-----------|-------------|-----------------|
+| Read a file | 12s | `file_read` |
+| Write a new file | 90s | `file_write` |
+| Make a precise edit | 25s | `file_edit` |
+| Grep search + scan results | 15s | `grep_search` |
+| Run a shell command | 20s | `shell` |
+| Web search + evaluate | 60s | `web_search` |
+| Survey codebase structure | 180s | `codebase_map` |
+
+Additional overhead per action:
+- **+5s context-switch** per tool call (expert switching between tools)
+- **+15s planning** per reasoning turn (expert thinking about next step)
+
+The ratio accumulates across all tasks in the session:
+
+```
+speedRatio = totalHumanExpertTime / totalAgentWallClockTime
+```
+
+Color coding: green (2x+ faster), yellow (1-2x, comparable), red (<1x, slower than expert).
+
+All 47 tools have calibrated baselines ranging from 3s (`task_stop`) to 180s (`codebase_map`). Unknown tools default to 20s.
+
 ## Cost Tracking & Session Metrics
 
 Real-time token cost estimation for cloud providers. The status bar shows running cost when using a paid endpoint.
@@ -717,7 +765,7 @@ The agent auto-detects the provider, normalizes the URL (strips `/v1/chat/comple
 
 ## Evaluation Suite
 
-33 evaluation tasks test the agent's autonomous capabilities across coding, web research, SDLC analysis, tool creation, and multi-file reasoning:
+40 evaluation tasks test the agent's autonomous capabilities across coding, web research, SDLC analysis, tool creation, multi-file reasoning, and memory systems:
 
 ```bash
 node eval/run-agentic.mjs                          # Run all tasks
@@ -754,15 +802,23 @@ node eval/run-agentic.mjs --model qwen2.5-coder:32b  # Different model
 | 31 | Web extractor bug fixes (3 bugs) | Multi-Bug Fix |
 | 32 | CSV pipeline across 3 files | Multi-File Tracking |
 | 33 | FSM bug fixes + factory implementation | State Machine |
+| 34 | Search pre-populated memories | Memory Search |
+| 35 | Analyze code, write to memory, cross-reference | Memory Cross-Reference |
+| 36 | Discover explore_tools, unlock grep_search | Explore Tools |
+| 37 | Analyze code patterns, store and recall from memory | Memory Store & Recall |
+| 38 | Read configs, write to multiple memory topics | Memory Multi-Topic |
+| 39 | Search pre-loaded memories across 3 topics | Memory Pre-Loaded Search |
+| 40 | Combined explore_tools + memory analysis pipeline | Explore + Memory |
 
-Tasks 31-33 are designed for small model (≤9B) evaluation using `file_edit` patterns instead of `file_write` to avoid JSON truncation issues with smaller models.
+Tasks 31-33 are designed for small model (≤9B) evaluation using `file_edit` patterns. Tasks 34-40 test the memory system (read/write/search) and tool discovery.
 
 ### Benchmark Results
 
 ```
-Qwen3.5-122B: 100% pass rate (30/30 tasks)
+Qwen3.5-122B: 100% pass rate (37/37 tasks, including memory tasks 34-40)
 Qwen3.5-27B:  100% pass rate (30/30 tasks)
 Qwen3.5-9B:   100% pass rate (tasks 31-33, file_edit-optimized)
+              71% pass rate (5/7 memory tasks 34-40)
 ```
 
 The eval runner includes model-tier-aware features: automatic tool set filtering, HTTP 500 recovery with file_edit hints, loop detection with tool banning, and tier-based output truncation.

package/dist/index.js

@@ -21970,12 +21970,113 @@ var init_braille_spinner = __esm({
 });
 
 // packages/cli/dist/tui/status-bar.js
-var StatusBar;
+var EXPERT_TOOL_BASELINES, CONTEXT_SWITCH_OVERHEAD, TURN_PLANNING_OVERHEAD, DEFAULT_TOOL_BASELINE, HumanSpeedTracker, StatusBar;
 var init_status_bar = __esm({
   "packages/cli/dist/tui/status-bar.js"() {
     "use strict";
     init_render();
     init_braille_spinner();
+    EXPERT_TOOL_BASELINES = {
+      file_read: 12,
+      structured_read: 15,
+      file_write: 90,
+      structured_file: 90,
+      file_edit: 25,
+      file_patch: 90,
+      batch_edit: 90,
+      grep_search: 15,
+      glob_find: 8,
+      shell: 20,
+      web_fetch: 45,
+      web_search: 60,
+      web_crawl: 90,
+      memory_read: 5,
+      memory_write: 15,
+      memory_search: 10,
+      list_directory: 5,
+      codebase_map: 180,
+      git_info: 8,
+      diagnostic: 10,
+      explore_tools: 10,
+      image_read: 8,
+      screenshot: 15,
+      ocr: 20,
+      ocr_pdf: 30,
+      pdf_to_text: 20,
+      ocr_image_advanced: 30,
+      transcribe_file: 60,
+      transcribe_url: 60,
+      vision: 15,
+      desktop_click: 5,
+      desktop_describe: 10,
+      code_sandbox: 45,
+      skill_list: 5,
+      skill_execute: 30,
+      create_tool: 120,
+      manage_tools: 15,
+      aiwg_setup: 30,
+      aiwg_health: 10,
+      aiwg_workflow: 30,
+      background_run: 30,
+      task_status: 5,
+      task_output: 5,
+      task_stop: 3,
+      task_complete: 0
+    };
+    CONTEXT_SWITCH_OVERHEAD = 5;
+    TURN_PLANNING_OVERHEAD = 15;
+    DEFAULT_TOOL_BASELINE = 20;
+    HumanSpeedTracker = class {
+      /** Accumulated estimated human-expert time in seconds */
+      humanTimeS = 0;
+      /** Accumulated agent wall-clock time in milliseconds */
+      agentTimeMs = 0;
+      /** Timestamp when current task started (0 = no active task) */
+      taskStartMs = 0;
+      /** Number of tool calls in current session */
+      toolCalls = 0;
+      /** Number of turns in current session */
+      turns = 0;
+      /** Record a tool call — adds the expert baseline time */
+      recordToolCall(toolName) {
+        const baseline = EXPERT_TOOL_BASELINES[toolName] ?? DEFAULT_TOOL_BASELINE;
+        this.humanTimeS += baseline + CONTEXT_SWITCH_OVERHEAD;
+        this.toolCalls++;
+      }
+      /** Record a turn (assistant reasoning cycle) */
+      recordTurn() {
+        this.humanTimeS += TURN_PLANNING_OVERHEAD;
+        this.turns++;
+      }
+      /** Mark the start of a task (for wall-clock tracking) */
+      taskStart() {
+        this.taskStartMs = Date.now();
+      }
+      /** Mark the end of a task — accumulates elapsed wall-clock time */
+      taskEnd() {
+        if (this.taskStartMs > 0) {
+          this.agentTimeMs += Date.now() - this.taskStartMs;
+          this.taskStartMs = 0;
+        }
+      }
+      /**
+       * Get the current speed ratio (human expert time / agent time).
+       * Returns 0 if no work has been done yet.
+       * During an active task, includes in-flight elapsed time.
+       */
+      getSpeedRatio() {
+        if (this.humanTimeS === 0)
+          return 0;
+        const totalAgentMs = this.agentTimeMs + (this.taskStartMs > 0 ? Date.now() - this.taskStartMs : 0);
+        if (totalAgentMs <= 0)
+          return 0;
+        return this.humanTimeS * 1e3 / totalAgentMs;
+      }
+      /** Whether any work has been recorded */
+      get hasData() {
+        return this.toolCalls > 0;
+      }
+    };
     StatusBar = class {
       metrics = {
         promptTokens: 0,
@@ -22114,6 +22215,24 @@ var init_status_bar = __esm({
       setContextWindowSize(size) {
         this.metrics.contextWindowSize = size;
       }
+      /** Human expert speed ratio tracker */
+      _speedTracker = new HumanSpeedTracker();
+      /** Record a tool call for speed ratio tracking */
+      recordSpeedToolCall(toolName) {
+        this._speedTracker.recordToolCall(toolName);
+      }
+      /** Record a turn for speed ratio tracking */
+      recordSpeedTurn() {
+        this._speedTracker.recordTurn();
+      }
+      /** Mark task start for speed ratio wall-clock tracking */
+      recordSpeedTaskStart() {
+        this._speedTracker.taskStart();
+      }
+      /** Mark task end for speed ratio wall-clock tracking */
+      recordSpeedTaskEnd() {
+        this._speedTracker.taskEnd();
+      }
       /** Model capabilities — shown as emoji indicators on the status bar */
       _caps = {
         vision: false,
@@ -22320,6 +22439,15 @@ var init_status_bar = __esm({
           const costStr = m.estimatedCost < 0.01 ? `$${m.estimatedCost.toFixed(4)}` : m.estimatedCost < 1 ? `$${m.estimatedCost.toFixed(3)}` : `$${m.estimatedCost.toFixed(2)}`;
           costLabel = pipe + pastel2(222, "Cost: ") + c2.bold(costStr);
         }
+        let speedLabel = "";
+        if (this._speedTracker.hasData) {
+          const ratio = this._speedTracker.getSpeedRatio();
+          if (ratio > 0) {
+            const ratioStr = ratio >= 10 ? `${Math.round(ratio)}x` : `${ratio.toFixed(1)}x`;
+            const ratioColor = ratio >= 2 ? c2.green : ratio >= 1 ? c2.yellow : c2.red;
+            speedLabel = pipe + pastel2(218, "Exp: ") + ratioColor(c2.bold(ratioStr));
+          }
+        }
         let recordingLabel = "";
         if (this._recording) {
           const dot = this._recBlink ? pastel2(210, "\u25CF") : " ";
@@ -22334,7 +22462,7 @@ var init_status_bar = __esm({
         if (this._caps.thinking)
           capParts.push("\u{1F9E0}");
         const capsLabel = capParts.length > 0 ? pipe + pastel2(183, capParts.join(" ")) : "";
-        return ` ${tokInLabel}${pipe}${tokOutLabel}${pipe}${ctxLabel}${costLabel}${capsLabel}${recordingLabel}`;
+        return ` ${tokInLabel}${pipe}${tokOutLabel}${pipe}${ctxLabel}${speedLabel}${costLabel}${capsLabel}${recordingLabel}`;
       }
       // -------------------------------------------------------------------------
       // Private
@@ -22939,6 +23067,7 @@ ${entry.fullContent}`
           }
         }
         lastToolCall = { name: event.toolName ?? "unknown", args: event.toolArgs ?? {} };
+        statusBar?.recordSpeedToolCall(event.toolName ?? "unknown");
         statusBar?.setActiveTool(event.toolName ?? null);
         contentWrite(() => {
           if (voice?.enabled) {
@@ -22967,6 +23096,7 @@ ${entry.fullContent}`
         });
         break;
       case "model_response":
+        statusBar?.recordSpeedTurn();
         if (config.verbose && !stream?.enabled && event.content) {
           contentWrite(() => renderAssistantText(event.content));
         }
@@ -24002,6 +24132,7 @@ NEW TASK: ${fullInput}`;
     }
     try {
       statusBar.setProcessing(true);
+      statusBar.recordSpeedTaskStart();
       const task = startTask(taskInput, currentConfig, repoRoot, voiceEngine, {
         enabled: streamEnabled,
         renderer: streamRenderer
@@ -24040,6 +24171,7 @@ NEW TASK: ${fullInput}`;
       }
     } finally {
       statusBar.setProcessing(false);
+      statusBar.recordSpeedTaskEnd();
       if (activeTask) {
         sessionFilesTouched = Array.from(activeTask.filesTouched);
         sessionToolCallCount = activeTask.toolCallCount;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "open-agents-ai",
-  "version": "0.34.1",
+  "version": "0.34.3",
   "description": "AI coding agent powered by open-source models (Ollama/vLLM) — interactive TUI with agentic tool-calling loop",
   "type": "module",
   "main": "./dist/index.js",