open-agents-ai 0.35.2 → 0.37.0

package/README.md

@@ -29,7 +29,7 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 
 ## Features
 
-- **47 autonomous tools** — file I/O, shell, grep, web search/fetch/crawl, memory (read/write/search), sub-agents, background tasks, image/OCR/PDF, git, diagnostics, vision, desktop automation, structured files, code sandbox, transcription, skills
+- **51 autonomous tools** — file I/O, shell, grep, web search/fetch/crawl, memory (read/write/search), sub-agents, background tasks, image/OCR/PDF, git, diagnostics, vision, desktop automation, browser automation, temporal agency (scheduler/reminders/agenda), structured files, code sandbox, transcription, skills
 - **Moondream vision** — see and interact with the desktop via Moondream VLM (caption, query, detect, point-and-click)
 - **Desktop automation** — vision-guided clicking: describe a UI element in natural language, the agent finds and clicks it
 - **Auto-install desktop deps** — screenshot, mouse, OCR, and image tools auto-install missing system packages (scrot, xdotool, tesseract, imagemagick) on first use
@@ -48,6 +48,8 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 - **Code sandbox** — isolated code execution in subprocess or Docker (JS, Python, Bash, TypeScript)
 - **Structured file reading** — parse CSV, TSV, JSON, Markdown tables with binary format detection
 - **Multi-provider web search** — DuckDuckGo (free), Tavily (structured), Jina AI (markdown) with auto-detection
+- **Browser automation** — headless Chrome control via Selenium: navigate, click, type, screenshot, read DOM — auto-starts on first use with self-bootstrapping Python venv
+- **Temporal agency** — schedule future tasks via OS cron, set cross-session reminders, flag attention items — startup injection surfaces due items automatically
 - **Web crawling** — multi-page web scraping with Crawlee/Playwright for deep documentation extraction
 - **Task templates** — specialized system prompts and tool recommendations for code, document, analysis, plan tasks
 - **Auto-expanding context** — detects RAM/VRAM and creates an optimized model variant on first run
@@ -348,6 +350,84 @@ Agent: desktop_click(target="terminal icon", click_type="double")
 
 Supports left/right/middle click, single/double click, multi-match selection by index, dry-run mode for verification, and configurable delay for UI transitions.
 
+### Browser Automation
+
+Headless Chrome automation via Selenium — no display server required. The scrape service auto-starts on first use, creates its own Python venv, and installs all dependencies:
+
+```
+You: go to github.com and screenshot the page
+
+Agent: [Turn 1] browser_action(action="navigate", url="https://github.com")
+       → Navigated to https://github.com
+       [Turn 2] browser_action(action="screenshot")
+       → Screenshot captured (1920x1080)
+```
+
+Available actions:
+
+| Action | Description |
+|--------|-------------|
+| `navigate` | Go to a URL |
+| `click` | Click element by CSS selector |
+| `click_xy` | Click at viewport coordinates |
+| `type` | Type text into a form element |
+| `screenshot` | Capture the current page |
+| `dom` | Read the page DOM (up to 50K chars) |
+| `scroll` / `scroll_up` / `scroll_down` | Scroll the page |
+| `back` / `forward` | Browser history navigation |
+| `close` | End the browser session |
+
+The service runs on `localhost:8130` and uses headless Chrome/Chromium. Requires Python 3.9+ and Chrome or Chromium installed on the system.
+
+### Temporal Agency — Scheduling, Reminders & Attention
+
+The agent has persistent temporal awareness across sessions. Three tools work together to let the agent schedule future work, leave notes for its future self, and track items that need attention.
+
+**Scheduler** — Create OS-level cron jobs that auto-launch the agent:
+
+```
+Agent: scheduler(action="create", task="run npm audit and fix vulnerabilities", schedule="weekly")
+       → Scheduled task created: sched-a1b2c3d4
+         Schedule: weekly on day 1 at 9:00
+
+Agent: scheduler(action="create", task="check API health", schedule="every 30 minutes")
+       → Scheduled task created: sched-e5f6a7b8
+```
+
+Schedule formats: presets (`daily`, `hourly`, `every 5 minutes`, `weekly`), natural language (`in 30m`, `at 14:30`), or raw cron (`0 */2 * * *`).
+
+**Reminder** — Cross-session messages-in-a-bottle:
+
+```
+Agent: reminder(action="set", message="Verify auth migration tokens after deploy", priority="high", due="tomorrow")
+       → Reminder set: rem-c4d5e6f7 (due: tomorrow morning)
+
+# Next startup:
+⚠ 1 urgent item(s) need attention
+  Reminder: Verify auth migration tokens after deploy
+```
+
+Reminders support priority levels (`low`/`normal`/`high`/`critical`), due dates, tags, context, snoozing, and auto-surface at startup.
+
+**Agenda** — Unified temporal dashboard:
+
+```
+Agent: agenda()
+       → AGENT AGENDA
+         ──────────────────────────────────────────────
+         REMINDERS DUE (2):
+           [!!] [rem-a1b2] Verify auth migration tokens
+           [*]  [rem-c3d4] Update API docs
+
+         ATTENTION ITEMS (1):
+           [!!] [attn-e5f6] (followup) PR #42 needs re-review
+
+         SCHEDULED TASKS (1 active):
+           [sched-g7h8] weekly on day 1 at 9:00: run npm audit
+```
+
+Research basis: Reflexion (Shinn et al. 2023) — persistent self-reflection improves task success 20-30%. Generative Agents (Park et al. 2023) — time-weighted memory retrieval. MemGPT (Packer et al. 2023) — hierarchical memory across agent lifetimes.
+
 ### Setup
 
 Moondream runs locally — no API keys, no cloud, your screen data never leaves your machine:
@@ -483,6 +563,7 @@ While the agent is working (shown by the `+` prompt), type to add context:
 | `web_search` | Search the web (DuckDuckGo, Tavily, Jina AI — auto-detected) |
 | `web_fetch` | Fetch and extract text from web pages (HTML stripping) |
 | `web_crawl` | Multi-page web scraping with Crawlee/Playwright for deep documentation |
+| `browser_action` | Headless Chrome automation: navigate, click, type, screenshot, read DOM, scroll, history |
 | **Structured Data** | |
 | `structured_file` | Generate CSV, TSV, JSON, Markdown tables, Excel-compatible files |
 | `structured_read` | Parse CSV, TSV, JSON, Markdown tables with binary format detection |
@@ -516,6 +597,10 @@ While the agent is working (shown by the `+` prompt), type to add context:
 | `manage_tools` | List, inspect, delete custom tools |
 | `skill_list` | Discover available AIWG skills |
 | `skill_execute` | Run an AIWG skill |
+| **Temporal Agency** | |
+| `scheduler` | Schedule tasks for automatic future execution via OS cron (presets, natural language, raw cron) |
+| `reminder` | Set cross-session reminders with priority, due dates, tags — surfaces at startup |
+| `agenda` | Unified view of reminders, schedules, and attention items with startup brief |
 | **AIWG SDLC** | |
 | `aiwg_setup` | Deploy AIWG SDLC framework |
 | `aiwg_health` | Analyze project SDLC health and readiness |