open-agents-ai 0.35.2 → 0.36.0

package/README.md

@@ -29,7 +29,7 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 
 ## Features
 
-- **47 autonomous tools** — file I/O, shell, grep, web search/fetch/crawl, memory (read/write/search), sub-agents, background tasks, image/OCR/PDF, git, diagnostics, vision, desktop automation, structured files, code sandbox, transcription, skills
+- **48 autonomous tools** — file I/O, shell, grep, web search/fetch/crawl, memory (read/write/search), sub-agents, background tasks, image/OCR/PDF, git, diagnostics, vision, desktop automation, browser automation, structured files, code sandbox, transcription, skills
 - **Moondream vision** — see and interact with the desktop via Moondream VLM (caption, query, detect, point-and-click)
 - **Desktop automation** — vision-guided clicking: describe a UI element in natural language, the agent finds and clicks it
 - **Auto-install desktop deps** — screenshot, mouse, OCR, and image tools auto-install missing system packages (scrot, xdotool, tesseract, imagemagick) on first use
@@ -48,6 +48,7 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 - **Code sandbox** — isolated code execution in subprocess or Docker (JS, Python, Bash, TypeScript)
 - **Structured file reading** — parse CSV, TSV, JSON, Markdown tables with binary format detection
 - **Multi-provider web search** — DuckDuckGo (free), Tavily (structured), Jina AI (markdown) with auto-detection
+- **Browser automation** — headless Chrome control via Selenium: navigate, click, type, screenshot, read DOM — auto-starts on first use with self-bootstrapping Python venv
 - **Web crawling** — multi-page web scraping with Crawlee/Playwright for deep documentation extraction
 - **Task templates** — specialized system prompts and tool recommendations for code, document, analysis, plan tasks
 - **Auto-expanding context** — detects RAM/VRAM and creates an optimized model variant on first run
@@ -348,6 +349,35 @@ Agent: desktop_click(target="terminal icon", click_type="double")
 
 Supports left/right/middle click, single/double click, multi-match selection by index, dry-run mode for verification, and configurable delay for UI transitions.
 
+### Browser Automation
+
+Headless Chrome automation via Selenium — no display server required. The scrape service auto-starts on first use, creates its own Python venv, and installs all dependencies:
+
+```
+You: go to github.com and screenshot the page
+
+Agent: [Turn 1] browser_action(action="navigate", url="https://github.com")
+       → Navigated to https://github.com
+       [Turn 2] browser_action(action="screenshot")
+       → Screenshot captured (1920x1080)
+```
+
+Available actions:
+
+| Action | Description |
+|--------|-------------|
+| `navigate` | Go to a URL |
+| `click` | Click element by CSS selector |
+| `click_xy` | Click at viewport coordinates |
+| `type` | Type text into a form element |
+| `screenshot` | Capture the current page |
+| `dom` | Read the page DOM (up to 50K chars) |
+| `scroll` / `scroll_up` / `scroll_down` | Scroll the page |
+| `back` / `forward` | Browser history navigation |
+| `close` | End the browser session |
+
+The service runs on `localhost:8130` and uses headless Chrome/Chromium. Requires Python 3.9+ and Chrome or Chromium installed on the system.
+
 ### Setup
 
 Moondream runs locally — no API keys, no cloud, your screen data never leaves your machine:
@@ -483,6 +513,7 @@ While the agent is working (shown by the `+` prompt), type to add context:
 | `web_search` | Search the web (DuckDuckGo, Tavily, Jina AI — auto-detected) |
 | `web_fetch` | Fetch and extract text from web pages (HTML stripping) |
 | `web_crawl` | Multi-page web scraping with Crawlee/Playwright for deep documentation |
+| `browser_action` | Headless Chrome automation: navigate, click, type, screenshot, read DOM, scroll, history |
 | **Structured Data** | |
 | `structured_file` | Generate CSV, TSV, JSON, Markdown tables, Excel-compatible files |
 | `structured_read` | Parse CSV, TSV, JSON, Markdown tables with binary format detection |