robot_lab 0.0.9 → 0.0.12

data/docs/api/errors.md

@@ -0,0 +1,185 @@
+# Error Reference
+
+All RobotLab exceptions inherit from `RobotLab::Error`, which inherits from `StandardError`. Rescue `RobotLab::Error` to catch any framework exception in one clause, or rescue specific subclasses for targeted handling.
+
+## Hierarchy
+
+```
+StandardError
+└── RobotLab::Error
+    ├── RobotLab::ConfigurationError
+    │   └── RobotLab::DependencyError
+    ├── RobotLab::InferenceError
+    │   └── RobotLab::ToolLoopError
+    ├── RobotLab::ToolNotFoundError
+    ├── RobotLab::MCPError
+    ├── RobotLab::BusError
+    ├── RobotLab::RactorBoundaryError
+    └── RobotLab::ToolError
+```
+
+Additionally, `DelegationFuture` defines its own scoped error:
+
+```
+StandardError
+└── RobotLab::Error
+    └── RobotLab::DelegationFuture::DelegationTimeout
+```
+
+---
+
+## RobotLab::Error
+
+Base class for all RobotLab errors. Rescue this to catch any framework exception.
+
+```ruby
+begin
+  robot.run("hello")
+rescue RobotLab::Error => e
+  puts "RobotLab error: #{e.message}"
+end
+```
+
+---
+
+## RobotLab::ConfigurationError
+
+Raised when configuration is invalid or missing required values.
+
+```ruby
+# Example: missing API key, invalid template path
+rescue RobotLab::ConfigurationError => e
+  puts "Bad config: #{e.message}"
+end
+```
+
+---
+
+## RobotLab::DependencyError < ConfigurationError
+
+Raised when a required optional gem dependency is not installed.
+
+```ruby
+# Triggered by: Convergence, HistoryCompressor when 'classifier' gem is absent
+rescue RobotLab::DependencyError => e
+  puts e.message  # "Add gem 'classifier', '~> 2.3' to your Gemfile"
+end
+```
+
+---
+
+## RobotLab::InferenceError
+
+Raised when LLM inference fails (API errors, timeouts, rate limits).
+
+```ruby
+rescue RobotLab::InferenceError => e
+  puts "LLM call failed: #{e.message}"
+end
+```
+
+---
+
+## RobotLab::ToolLoopError < InferenceError
+
+Raised when a robot's tool call count exceeds `max_tool_rounds:`. The chat history will contain a dangling `tool_use` block with no matching `tool_result`; call `robot.clear_messages` before reusing the robot.
+
+```ruby
+robot = RobotLab.build(
+  name: "runner",
+  system_prompt: "Execute every step.",
+  local_tools: [StepTool],
+  max_tool_rounds: 10
+)
+
+begin
+  robot.run("Run all steps.")
+rescue RobotLab::ToolLoopError => e
+  puts e.message  # "Tool call limit of 10 exceeded"
+  robot.clear_messages  # required before reuse
+end
+```
+
+---
+
+## RobotLab::ToolNotFoundError
+
+Raised when a tool name is referenced but cannot be found in the `ToolManifest`.
+
+---
+
+## RobotLab::MCPError
+
+Raised when MCP server communication fails (connection refused, timeout, protocol error).
+
+```ruby
+rescue RobotLab::MCPError => e
+  puts "MCP failed: #{e.message}"
+end
+```
+
+---
+
+## RobotLab::BusError
+
+Raised when message bus communication fails (no bus configured, channel not found).
+
+```ruby
+rescue RobotLab::BusError => e
+  puts "Bus error: #{e.message}"
+end
+```
+
+---
+
+## RobotLab::RactorBoundaryError
+
+Raised when a value cannot be made Ractor-shareable before crossing a Ractor boundary (e.g., a live `IO` object, a `Proc`, or an object with mutable state).
+
+```ruby
+begin
+  RobotLab::RactorBoundary.freeze_deep({ io: StringIO.new })
+rescue RobotLab::RactorBoundaryError => e
+  puts e.message  # "Cannot make value Ractor-shareable: ..."
+end
+```
+
+Raised proactively by `RactorWorkerPool#submit` and `RactorMemoryProxy#set` before any Ractor is involved.
+
+---
+
+## RobotLab::ToolError
+
+Raised when a tool fails during execution inside a Ractor worker (the pool unwraps `RactorJobError` and re-raises as `ToolError`).
+
+```ruby
+begin
+  pool.submit("MyTool", { input: "bad" })
+rescue RobotLab::ToolError => e
+  puts e.message  # "Tool 'MyTool' failed in Ractor: ..."
+end
+```
+
+---
+
+## RobotLab::DelegationFuture::DelegationTimeout
+
+Raised by `DelegationFuture#value(timeout: N)` when the delegated task does not complete within `N` seconds.
+
+```ruby
+future = manager.delegate(to: analyst, task: "...", async: true)
+
+begin
+  result = future.value(timeout: 10)
+rescue RobotLab::DelegationFuture::DelegationTimeout => e
+  puts e.message  # "Delegation to 'analyst' timed out after 10s"
+end
+```
+
+---
+
+## Related
+
+- [Building Robots](../guides/building-robots.md) — Tool loop circuit breaker, delegation
+- [Ractor Parallelism](../guides/ractor-parallelism.md) — Ractor boundary and tool errors
+- [MCP Integration](../guides/mcp-integration.md) — MCP connection errors

data/docs/guides/building-robots.md

@@ -736,6 +736,131 @@ bot = RobotLab.build(name: "latecomer", system_prompt: "Hello.")
 bot.with_bus(existing_bus)  # now connected and can send/receive messages
 ```
 
+## Context Window Compression
+
+Long-running robots accumulate conversation history that can grow to fill the context window. `compress_history` prunes old turns using TF-IDF cosine similarity against the most recent context, keeping turns that are still relevant and discarding or summarizing those that aren't.
+
+```ruby
+# Default settings: protect 3 most-recent turn pairs, drop anything below 0.2
+robot.compress_history
+
+# Tune all thresholds
+robot.compress_history(
+  recent_turns:   5,    # number of recent user+assistant pairs to always keep
+  keep_threshold: 0.6,  # turns with score >= this are kept verbatim (default 0.6)
+  drop_threshold: 0.2   # turns with score < this are dropped (default 0.2)
+)
+```
+
+Medium-relevance turns (between thresholds) are dropped by default. Pass a `summarizer:` callable to replace them with a one-sentence summary instead:
+
+```ruby
+summarizer = RobotLab.build(name: "summarizer", system_prompt: "Summarize concisely in one sentence.")
+
+robot.compress_history(
+  summarizer: ->(text) { summarizer.run("Summarize: #{text}").reply }
+)
+```
+
+**What is always preserved regardless of score:**
+- System messages
+- Tool call/result message pairs (dropping half would corrupt the conversation)
+- The most recent `recent_turns` user+assistant pairs
+
+Requires the `classifier` gem (`~> 2.3`):
+
+```ruby
+gem "classifier", "~> 2.3"
+```
+
+## Convergence Detection
+
+`RobotLab::Convergence` uses TF-IDF cosine similarity to detect when two independent agents have reached the same conclusion. The primary use case is a network router that skips an expensive reconciler robot when two verifiers already agree.
+
+```ruby
+# Check the similarity score directly (returns Float 0.0..1.0)
+score = RobotLab::Convergence.similarity(result_a.reply, result_b.reply)
+
+# Boolean convergence check (default threshold: 0.85)
+RobotLab::Convergence.detected?(result_a.reply, result_b.reply)
+
+# Custom threshold
+RobotLab::Convergence.detected?(text_a, text_b, threshold: 0.75)
+```
+
+Wire it into a network router for the reconciler fast-path:
+
+```ruby
+verifier_a = RobotLab.build(name: "verifier_a", system_prompt: "Verify the answer.")
+verifier_b = RobotLab.build(name: "verifier_b", system_prompt: "Independently verify the answer.")
+reconciler = RobotLab.build(name: "reconciler", system_prompt: "Reconcile conflicting answers.")
+
+router = lambda do |args|
+  a = args.context[:verifier_a]&.reply.to_s
+  b = args.context[:verifier_b]&.reply.to_s
+
+  # Skip reconciler when verifiers agree
+  RobotLab::Convergence.detected?(a, b) ? nil : ["reconciler"]
+end
+
+network = RobotLab.create_network(name: "verify", router: router) do
+  # ...
+end
+```
+
+Requires the `classifier` gem (`~> 2.3`).
+
+## Structured Delegation
+
+A robot can delegate a task to another robot using `delegate(to:, task:)`. The result is a `RobotResult` annotated with `delegated_by`, `duration`, and token counts.
+
+### Synchronous Delegation
+
+The default: blocks the calling robot until the delegatee finishes.
+
+```ruby
+analyst = RobotLab.build(name: "analyst", system_prompt: "Analyze data.")
+manager = RobotLab.build(name: "manager", system_prompt: "Coordinate work.")
+
+result = manager.delegate(to: analyst, task: "Summarize the Q3 report.")
+puts result.reply
+puts "Completed in %.2fs using %d tokens" % [result.duration, result.output_tokens]
+puts result.delegated_by  # => "manager"
+```
+
+### Asynchronous Delegation (Fan-out)
+
+Pass `async: true` to get a `DelegationFuture` back immediately. Call `.value` to block for the result when you need it.
+
+```ruby
+writer  = RobotLab.build(name: "writer",  system_prompt: "Write clearly.")
+analyst = RobotLab.build(name: "analyst", system_prompt: "Analyze data.")
+manager = RobotLab.build(name: "manager", system_prompt: "Coordinate.")
+
+# Fan out — both start immediately
+f1 = manager.delegate(to: analyst, task: "Analyze Q3 numbers", async: true)
+f2 = manager.delegate(to: writer,  task: "Draft the intro paragraph", async: true)
+
+# ... do other work here ...
+
+# Collect results (blocks if not yet done)
+analysis = f1.value
+draft    = f2.value
+
+# With a timeout (raises DelegationFuture::DelegationTimeout)
+result = f1.value(timeout: 30)
+```
+
+`DelegationFuture` API:
+
+| Method | Description |
+|--------|-------------|
+| `resolved?` | Non-blocking poll — true if completed or errored |
+| `value(timeout: nil)` | Block until done; re-raises any error from the delegatee |
+| `wait` | Alias for `value` |
+| `robot_name` | Name of the robot that was delegated to |
+| `delegated_by` | Name of the robot that created this future |
+
 ## Configuration
 
 RobotLab uses `MywayConfig` for configuration. Access the config object directly -- there is no `RobotLab.configure` block:

data/docs/guides/creating-networks.md

@@ -124,6 +124,7 @@ end
 | `memory` | Task-specific memory |
 | `config` | Per-task `RunConfig` (merged on top of network's config) |
 | `depends_on` | `:none`, `[:task1]`, or `:optional` |
+| `poller_group` | Bus delivery group label (`:default`, `:slow`, etc.) |
 
 ## Conditional Routing
 
@@ -164,6 +165,26 @@ network = RobotLab.create_network(name: "support") do
 end
 ```
 
+## Poller Groups
+
+Each network maintains a shared `BusPoller` that serializes TypedBus deliveries on a per-robot basis: if a robot is already processing a message, new deliveries are queued and drained after the current one completes. This prevents re-entrancy without blocking other robots.
+
+Named **poller groups** let you label tasks so slow robots are identifiable in logs and monitoring without needing separate infrastructure:
+
+```ruby
+network = RobotLab.create_network(name: "mixed_speed") do
+  # Fast robots on the default group
+  task :fetcher,   fetcher_robot,   depends_on: :none
+  task :summarize, summarizer,      depends_on: [:fetcher]
+
+  # Slow robots with expensive LLM calls — label them :slow
+  task :analyst,   analyst_robot,   depends_on: [:fetcher],  poller_group: :slow
+  task :writer,    writer_robot,    depends_on: [:analyst],  poller_group: :slow
+end
+```
+
+Group labels are informational — there is no separate queue per group. In Async execution, robots naturally yield during LLM HTTP calls, so fast and slow robots interleave without explicit isolation.
+
 ## Running Networks
 
 ### Basic Run

data/docs/guides/index.md

@@ -38,6 +38,14 @@ If you're new to RobotLab, start here:
 
     Share data between robots with the memory system
 
+-   [:octicons-pulse-24: **Observability & Safety**](observability.md)
+
+    Token tracking, circuit breakers, and learning accumulation
+
+-   [:material-cpu-64-bit: **Ractor Parallelism**](ractor-parallelism.md)
+
+    True CPU parallelism for tools and robot pipelines via Ruby Ractors
+
 </div>
 
 ## Framework Integration
@@ -61,3 +69,5 @@ If you're new to RobotLab, start here:
 | [Streaming](streaming.md) | Real-time responses | 5 min |
 | [Memory](memory.md) | Shared data store | 5 min |
 | [Rails Integration](rails-integration.md) | Rails application setup | 15 min |
+| [Observability & Safety](observability.md) | Token tracking, circuit breaker, learning loop | 10 min |
+| [Ractor Parallelism](ractor-parallelism.md) | CPU-parallel tools and robot pipelines | 15 min |

data/docs/guides/knowledge.md

@@ -0,0 +1,182 @@
+# Knowledge & Retrieval
+
+Facilities for searching and retrieving knowledge from a robot's history and from external documents:
+
+- **Chat History Search** — semantic search over accumulated conversation turns
+- **Embedding-Based Document Store** — lightweight RAG: store arbitrary text, search by meaning
+
+---
+
+## Chat History Search
+
+### The Problem
+
+Long-running robots accumulate many conversation turns. When you need to recall what was discussed earlier on a specific topic, re-sending the full history wastes tokens. `search_history` gives you a focused slice of the most relevant past messages without touching the LLM.
+
+### robot.search_history
+
+```ruby
+results = robot.search_history(query, limit: 5)
+```
+
+Scores every message in the robot's conversation history against `query` using stemmed term-frequency cosine similarity (via the `classifier` gem). Returns up to `limit` `HistoryResult` objects sorted by score descending.
+
+```ruby
+results = robot.search_history("quarterly revenue", limit: 3)
+
+results.each do |r|
+  puts "[#{r.role}] score=#{r.score.round(3)} idx=#{r.index}"
+  puts "  #{r.text}"
+end
+```
+
+### HistoryResult Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `text` | String | The message text |
+| `role` | Symbol | `:user`, `:assistant`, or `:system` |
+| `score` | Float (0.0–1.0) | Cosine similarity with the query |
+| `index` | Integer | Position in `@chat.messages` |
+
+### Typical Scores
+
+| Relationship | Typical Score |
+|---|---|
+| Direct answer to the query | 0.50 – 0.80 |
+| Same topic, different phrasing | 0.20 – 0.50 |
+| Unrelated | < 0.10 |
+
+### Short Messages
+
+Messages shorter than 20 characters are skipped — they produce no meaningful term vector.
+
+### Full Example
+
+```ruby
+robot = RobotLab.build(name: "analyst", system_prompt: "You are a financial analyst.")
+
+# … after several robot.run() calls …
+
+hits = robot.search_history("customer acquisition cost")
+hits.each { |r| puts "#{r.role} (#{r.score.round(2)}): #{r.text}" }
+```
+
+### RAG Pattern — Retrieve Then Generate
+
+Use `search_history` to inject only the relevant past context into the next call:
+
+```ruby
+hits    = robot.search_history(user_query, limit: 3)
+context = hits.map(&:text).join("\n")
+
+robot.run("Recall context:\n#{context}\n\nNew question: #{user_query}")
+```
+
+### Optional Dependency
+
+`search_history` requires the `classifier` gem:
+
+```ruby
+gem "classifier", "~> 2.3"
+```
+
+Without it, calling `search_history` raises `RobotLab::DependencyError` with an install hint.
+
+---
+
+## Embedding-Based Document Store
+
+### The Problem
+
+Sometimes the knowledge you need isn't in the conversation history — it's in a README, a product spec, a changelog. `store_document` / `search_documents` embed arbitrary text with `fastembed` and retrieve the most relevant chunk at query time.
+
+### memory.store_document / memory.search_documents
+
+```ruby
+memory.store_document(:readme,    File.read("README.md"))
+memory.store_document(:changelog, File.read("CHANGELOG.md"))
+
+hits = memory.search_documents("how to configure redis", limit: 3)
+hits.each { |h| puts "#{h[:key]} (#{h[:score].round(3)}): #{h[:text][0..80]}" }
+```
+
+Each result hash contains:
+
+| Key | Type | Description |
+|-----|------|-------------|
+| `:key` | Symbol | The key the document was stored under |
+| `:text` | String | The full stored text |
+| `:score` | Float (0.0–1.0) | Cosine similarity with the query |
+
+### Standalone DocumentStore
+
+The `Memory` methods delegate to `RobotLab::DocumentStore`, which can also be used directly:
+
+```ruby
+store = RobotLab::DocumentStore.new
+store.store(:doc_a, "Ruby on Rails is a full-stack web framework.")
+store.store(:doc_b, "Postgres is an advanced relational database.")
+
+results = store.search("relational database SQL", limit: 2)
+puts results.first[:key]  # => :doc_b
+```
+
+Management methods:
+
+```ruby
+store.size          # => 2
+store.keys          # => [:doc_a, :doc_b]
+store.empty?        # => false
+store.delete(:doc_a)
+store.clear
+```
+
+### Embedding Model
+
+Default: `BAAI/bge-small-en-v1.5` (~23 MB, downloaded on first use, cached in `~/.cache/fastembed/`).
+
+Documents are embedded with a `"passage: "` prefix and queries with `"query: "` prefix — the standard retrieval convention for BGE models.
+
+Custom model:
+
+```ruby
+store = RobotLab::DocumentStore.new(model_name: "BAAI/bge-base-en-v1.5")
+```
+
+### RAG Pattern
+
+```ruby
+# 1. Index your knowledge base at startup
+memory.store_document(:readme,    File.read("README.md"))
+memory.store_document(:changelog, File.read("CHANGELOG.md"))
+memory.store_document(:api_docs,  File.read("docs/api.md"))
+
+# 2. At query time, retrieve the most relevant chunks
+hits    = memory.search_documents(user_query, limit: 3)
+context = hits.map { |h| h[:text] }.join("\n\n")
+
+# 3. Pass context to your robot
+result = robot.run("Use the following context:\n#{context}\n\nQuestion: #{user_query}")
+```
+
+### Memory API Summary
+
+| Method | Description |
+|--------|-------------|
+| `memory.store_document(key, text)` | Embed and store a document |
+| `memory.search_documents(query, limit: 5)` | Search by semantic similarity |
+| `memory.document_keys` | List stored keys |
+| `memory.delete_document(key)` | Remove a document |
+
+### Dependency
+
+`fastembed` is a core RobotLab dependency — no optional gem required. The ONNX model is downloaded on first use.
+
+---
+
+## See Also
+
+- [Observability Guide](observability.md)
+- [Example 25 — Chat History Search](../../examples/25_history_search.rb)
+- [Example 26 — Embedding-Based Document Store](../../examples/26_document_store.rb)

data/docs/guides/mcp-integration.md

@@ -310,6 +310,112 @@ client.list_resources       # => Array of resource definitions
 client.disconnect
 ```
 
+## Connection Multiplexing
+
+When a robot connects to several local (stdio) MCP servers, each client normally blocks independently while waiting for a response. `MCP::ConnectionPoller` replaces this with a single `IO.select` call across all registered stdout file descriptors, dispatching each response to the pending request for that client.
+
+This is primarily useful in networks where many robots each have multiple stdio MCP servers. Async-based transports (SSE, WebSocket, StreamableHTTP) are unaffected — they already use the Async fiber scheduler.
+
+```ruby
+# Create a shared poller
+poller = RobotLab::MCP::ConnectionPoller.new.start
+
+# Pass the poller when building clients
+client1 = RobotLab::MCP::Client.new(
+  { name: "filesystem", transport: { type: "stdio", command: "mcp-server-fs" } },
+  poller: poller
+)
+client2 = RobotLab::MCP::Client.new(
+  { name: "github", transport: { type: "stdio", command: "mcp-server-github" } },
+  poller: poller
+)
+
+client1.connect   # registers with poller
+client2.connect   # registers with poller
+
+# Both clients share the IO.select loop
+client1.list_tools
+client2.list_tools
+
+poller.stop
+```
+
+Without a shared poller each client uses its own blocking `Timeout.timeout` call. With a poller, responses from any registered server wake the poller's select loop, which dispatches to the right waiting thread via a `Thread::Queue`.
+
+!!! note
+    Only stdio clients are registered with the poller. SSE, WebSocket, and StreamableHTTP clients passed a `poller:` argument ignore it silently.
+
+## Server Discovery
+
+When a robot has many MCP servers configured, connecting to all of them upfront is wasteful — most servers will be irrelevant for any given user message. **Server Discovery** uses TF cosine similarity to select only the semantically relevant servers before the first `ensure_mcp_clients` call.
+
+### Enabling Discovery
+
+Add `description:` to each server config and set `mcp_discovery: true` on the robot:
+
+```ruby
+robot = RobotLab.build(
+  name: "assistant",
+  system_prompt: "You are a helpful assistant.",
+  mcp_discovery: true,
+  mcp: [
+    {
+      name: "filesystem",
+      description: "Read, write, and search local files and directories",
+      transport: { type: "stdio", command: "mcp-server-filesystem" }
+    },
+    {
+      name: "github",
+      description: "GitHub repos, issues, pull requests, code search",
+      transport: { type: "stdio", command: "mcp-server-github" }
+    },
+    {
+      name: "brew",
+      description: "Install, update, and manage macOS packages via Homebrew",
+      transport: { type: "stdio", command: "mcp-server-brew" }
+    }
+  ]
+)
+
+# Discovery connects only :brew for this message — filesystem and github are skipped
+robot.run("install imagemagick")
+```
+
+### How It Works
+
+`MCP::ServerDiscovery.select(query, from:, threshold:)` computes TF cosine similarity between the user's query and each server's topic text (`name + description`). Servers scoring at or above `DEFAULT_THRESHOLD` (0.05) are returned; the rest are excluded.
+
+The threshold is intentionally low — server descriptions are short, so raw cosine scores are naturally small even for on-topic queries.
+
+Discovery only applies on the **first** `run()` call (before `@mcp_initialized`). Once a set of servers is connected they remain connected for the robot's lifetime, preserving tool continuity across a conversation.
+
+### Fallback Behaviour
+
+All servers are returned unchanged when any of the following apply:
+
+| Condition | Reason |
+|-----------|--------|
+| No server has a `description` field | Nothing to score against |
+| `classifier` gem unavailable | Raises `DependencyError`, caught internally |
+| Query is blank or nil | Nothing to compare |
+| No server scores ≥ threshold | Rather fall back than leave the robot with no tools |
+
+### Using the API Directly
+
+```ruby
+servers = [
+  { name: "filesystem", description: "Read and write files", transport: { ... } },
+  { name: "github",     description: "GitHub repos and PRs",  transport: { ... } }
+]
+
+relevant = RobotLab::MCP::ServerDiscovery.select(
+  "list open pull requests",
+  from: servers,
+  threshold: 0.05   # optional, default
+)
+# => only the :github entry
+```
+
 ## Connection Resilience
 
 ### Eager Connection

data/docs/guides/memory.md

@@ -190,6 +190,8 @@ results = memory.get(:sentiment, :entities, :keywords, wait: 60)
 # => { sentiment: {...}, entities: [...], keywords: [...] }
 ```
 
+Each blocking wait is backed by an `IO.pipe` pair (`Waiter` class). Calling `signal` writes one byte per waiting caller, so all threads blocked on `IO.select` wake immediately. This design works cleanly with Ruby's Async fiber scheduler — no mutex contention or spurious wakeups.
+
 ### Subscriptions
 
 Subscribe to key changes with asynchronous callbacks: