open-agents-ai 0.187.133 → 0.187.135

package/README.md

@@ -40,7 +40,7 @@ An autonomous multi-turn tool-calling agent that reads your code, makes changes,
 - [Model-Tier Awareness](#model-tier-awareness)
 - [Live Code Knowledge Graph](#live-code-knowledge-graph)
 - [Auto-Expanding Context Window](#auto-expanding-context-window)
-- [Tools (67+)](#tools-67)
+- [Tools (64+)](#tools-64)
 - [Ralph Loop — Iteration-First Design](#ralph-loop--iteration-first-design)
 - [Task Control](#task-control)
 - [COHERE Cognitive Framework](#cohere-cognitive-framework)
@@ -833,7 +833,7 @@ Small models (4B-7B) receive 10+ optimizations that larger models don't need, ea
 
 ### Tool Nesting for Small Models
 
-Small models use an **explore_tools** meta-tool pattern inspired by hierarchical API retrieval research ([ToolLLM](https://arxiv.org/abs/2307.16789)). Instead of presenting all 67 tools (which overwhelms small context windows), only core tools are loaded initially. The agent calls `explore_tools()` to discover additional capabilities, then activates specific tools as needed. This reduces tool schema tokens by ~80% while preserving access to the full toolset.
+Small models use an **explore_tools** meta-tool pattern inspired by hierarchical API retrieval research ([ToolLLM](https://arxiv.org/abs/2307.16789)). Instead of presenting all 64+ tools (which overwhelms small context windows), only core tools are loaded initially. The agent calls `explore_tools()` to discover additional capabilities, then activates specific tools as needed. This reduces tool schema tokens by ~80% while preserving access to the full toolset.
 
 ### Dynamic Context Limits
 
@@ -923,7 +923,7 @@ On startup and `/model` switch, Open Agents detects your RAM/VRAM and creates an
 
 
 
-## Tools (61)
+## Tools (64)
 
 <div align="right"><a href="#top">back to top</a></div>
 
@@ -1007,6 +1007,10 @@ On startup and `/model` switch, Open Agents detects your RAM/VRAM and creates an
 | `identity_kernel` | Persistent identity state — hydrate, observe events, propose updates with justification, publish snapshot, reconcile contradictions. Persists in `.oa/identity/` |
 | `reflect` | Immune-system reflection — diagnostic (find flaws), epistemic (identify missing evidence), constitutional (review self-updates). Returns pass/revise/block verdict |
 | `explore` | ARCHE strategy-space exploration — generate diverse strategies, archive successful variants with tags/confidence, compare competing approaches, retrieve past strategies |
+| **Hardware Access** | |
+| `camera_capture` | Access system cameras — list devices, capture JPEG frames, query capabilities. Uses ffmpeg + v4l2. Supports USB, CSI, and 360 cameras (QooCam, RealSense). Captured images can be piped to vision tools |
+| `audio_capture` | Record from microphone — list input devices, record WAV/MP3 (configurable duration/rate/channels), check real-time mic level (RMS dBFS). Uses arecord + ffmpeg backends |
+| `audio_playback` | Speaker control and TTS — play audio files (WAV/MP3/OGG), text-to-speech via espeak-ng (multi-language), get/set system volume. Uses aplay/ffplay/amixer backends |
 
 Read-only tools execute concurrently when called in the same turn. Mutating tools run sequentially.
 
@@ -1031,7 +1035,26 @@ The agent has 4 web tools. Pick the right one:
 
 **Structured extraction**: Pass `extract_schema='{"price": "number", "name": "string"}'` to `web_crawl` for best-effort regex-based field extraction from page content.
 
+### Hardware Tool Guide
 
+The agent can access physical hardware — cameras, microphones, and speakers — through three dedicated tools:
+
+| Need | Tool | Example |
+|------|------|---------|
+| See the environment | `camera_capture` action=capture | Grab a JPEG frame from any USB/CSI camera |
+| List cameras | `camera_capture` action=list | Discover `/dev/video*` devices |
+| Record audio | `audio_capture` action=record duration=10 | Record 10s WAV from default mic |
+| Check if mic works | `audio_capture` action=level | RMS level in dBFS |
+| Speak aloud | `audio_playback` action=speak text="Hello" | TTS via espeak-ng |
+| Play a sound file | `audio_playback` action=play file=alert.wav | Play WAV/MP3/OGG |
+| Check volume | `audio_playback` action=volume | Get current volume % |
+| Set volume | `audio_playback` action=volume volume=50 | Set to 50% |
+
+**Prerequisites**: `ffmpeg`, `arecord`, `aplay`, `amixer` (ALSA utils), `espeak-ng`. Install: `sudo apt install ffmpeg alsa-utils espeak-ng`
+
+**Camera support**: USB cameras (UVC), Intel RealSense (via UVC), 360 cameras (QooCam, Ricoh Theta — raw fisheye via v4l2loopback + ffmpeg crop). The captured frame is returned as base64 JPEG that can be fed directly to the `vision` tool for analysis.
+
+**Audio workflow**: Record → transcribe → analyze: `audio_capture action=record` → `transcribe_file` → process transcript. The tools handle device enumeration and graceful degradation when hardware is unavailable.
 
 
 ## Ralph Loop — Iteration-First Design