robot_lab 0.1.0 → 0.2.6

data/docs/guides/index.md

@@ -40,21 +40,15 @@ If you're new to RobotLab, start here:
 
 -   [:octicons-pulse-24: **Observability & Safety**](observability.md)
 
-    Token tracking, circuit breakers, and learning accumulation
+    Token tracking, circuit breakers, doom loop detection, auto compaction, and learning accumulation
 
--   [:material-cpu-64-bit: **Ractor Parallelism**](ractor-parallelism.md)
+-   [:octicons-zap-24: **Hook System**](hooks.md)
 
-    True CPU parallelism for tools and robot pipelines via Ruby Ractors
+    Lifecycle hooks for instrumentation, response caching, extensions, and cross-cutting concerns
 
-</div>
-
-## Framework Integration
-
-<div class="grid cards" markdown>
+-   [:octicons-search-24: **Knowledge & Retrieval**](knowledge.md)
 
--   [:material-language-ruby:{ .lg } **Rails Integration**](rails-integration.md)
-
-    Use RobotLab in Ruby on Rails applications
+    Chat history search and embedding-based document store for RAG workflows
 
 </div>
 
@@ -68,6 +62,17 @@ If you're new to RobotLab, start here:
 | [MCP Integration](mcp-integration.md) | External tool servers | 10 min |
 | [Streaming](streaming.md) | Real-time responses | 5 min |
 | [Memory](memory.md) | Shared data store | 5 min |
-| [Rails Integration](rails-integration.md) | Rails application setup | 15 min |
-| [Observability & Safety](observability.md) | Token tracking, circuit breaker, learning loop | 10 min |
-| [Ractor Parallelism](ractor-parallelism.md) | CPU-parallel tools and robot pipelines | 15 min |
+| [Observability & Safety](observability.md) | Token tracking, circuit breaker, doom loop detection, auto compaction, learning loop | 10 min |
+| [Hook System](hooks.md) | Lifecycle hooks, extensions, response caching, instrumentation | 15 min |
+| [Knowledge & Retrieval](knowledge.md) | Chat history search and embedding-based document store (RAG) | 10 min |
+
+## Extension Gems
+
+Additional capabilities are available as separate gems:
+
+| Gem | Description | Docs |
+|-----|-------------|------|
+| [robot_lab-rails](https://github.com/MadBomber/robot_lab-rails) | Rails generators, background jobs, Turbo Stream broadcasting | [Rails Integration guide](https://github.com/MadBomber/robot_lab-rails/blob/main/docs/guides/rails-integration.md) |
+| [robot_lab-ractor](https://github.com/MadBomber/robot_lab-ractor) | True CPU parallelism for tools and robot pipelines via Ruby Ractors | [Ractor Parallelism guide](https://github.com/MadBomber/robot_lab-ractor/blob/main/docs/guides/ractor-parallelism.md) |
+| [robot_lab-durable](https://github.com/MadBomber/robot_lab-durable) | Cross-session knowledge persistence with YAML-backed storage | [robot_lab-durable README](https://github.com/MadBomber/robot_lab-durable) |
+| [robot_lab-document_store](https://github.com/MadBomber/robot_lab-document_store) | Embedding-based semantic document search via fastembed | [robot_lab-document_store README](https://github.com/MadBomber/robot_lab-document_store) |

data/docs/guides/knowledge.md

@@ -171,12 +171,18 @@ result = robot.run("Use the following context:\n#{context}\n\nQuestion: #{user_q
 
 ### Dependency
 
-`fastembed` is a core RobotLab dependency — no optional gem required. The ONNX model is downloaded on first use.
+The embedding-based document store requires the [`robot_lab-document_store`](https://github.com/MadBomber/robot_lab-document_store) extension gem. Add it to your Gemfile:
+
+```ruby
+gem "robot_lab-document_store"
+```
+
+This gem bundles `fastembed` for ONNX-based embeddings. The `BAAI/bge-small-en-v1.5` model (~23 MB) is downloaded on first use and cached in `~/.cache/fastembed/`. Without `robot_lab-document_store` loaded, calling `memory.store_document` or `memory.search_documents` raises `RobotLab::DependencyError`.
 
 ---
 
 ## See Also
 
 - [Observability Guide](observability.md)
-- [Example 25 — Chat History Search](../../examples/25_history_search.rb)
-- [Example 26 — Embedding-Based Document Store](../../examples/26_document_store.rb)
+- [examples/25_history_search.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/25_history_search.rb) — Chat History Search
+- [examples/26_document_store.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/26_document_store.rb) — Embedding-Based Document Store

data/docs/guides/observability.md

@@ -4,10 +4,13 @@ Facilities that help you monitor, control, improve, and scale robot behaviour:
 
 - **Token & Cost Tracking** — measure LLM usage per run and cumulatively
 - **Tool Loop Circuit Breaker** — guard against runaway tool call loops
+- **Doom Loop Detection** — catch cyclic or repetitive tool-call patterns before they spiral
+- **Automatic Context Compaction** — prevent context overflow with configurable auto-compression
 - **Learning Accumulation** — build up cross-run observations that guide future runs
 - **Context Window Compression** — prune irrelevant history to stay within token budgets
 - **Convergence Detection** — detect when independent agents reach the same conclusion
 - **Structured Delegation** — synchronous inter-robot calls with duration and token metadata
+- **Live Narration** — an opt-in console feed of what a robot is doing as it happens
 
 ---
 
@@ -89,6 +92,38 @@ result = fresh.run("Explain memoization.")
 puts result.input_tokens  # smallest possible — no prior history
 ```
 
+### Budgets (Token & Cost)
+
+Where token/cost tracking above is purely observational, `token_budget:` and `cost_budget:` make it enforceable — a `Robot` refuses to keep spending once a configured limit is reached:
+
+```ruby
+robot = RobotLab.build(
+  name: "capped",
+  system_prompt: "You are a concise assistant.",
+  token_budget: 10_000,  # cumulative input + output tokens
+  cost_budget: 0.50      # cumulative $ across all runs
+)
+```
+
+Enforcement happens in two layers, backed by a thread-safe `RobotLab::Budget::Ledger`:
+
+- **Before the call** — `run()` reserves whatever remains of each configured dimension. If a *prior* call already exhausted the budget, the reservation raises `RobotLab::BudgetExceeded` immediately, refusing the call outright before spending anything on it.
+- **After the call** — actual usage (tokens from the result, cost from the response, when the provider reports pricing) replaces the reservation. If *this* call's actual usage pushes cumulative usage over budget, `RobotLab::InferenceError` is raised (unavoidable for the call that causes the overage, since totals aren't known until the response comes back) — the same error `token_budget` alone has always raised.
+
+```ruby
+begin
+  robot.run("Another expensive task")
+rescue RobotLab::BudgetExceeded => e
+  puts e.message  # "budget exceeded for cost: 0.51 > 0.5" — refused before spending
+rescue RobotLab::InferenceError => e
+  puts e.message  # "Cost budget exceeded: $0.5231 used, budget is $0.5" — this call pushed it over
+end
+```
+
+A dimension with no configured limit (e.g. `cost_budget` when only `token_budget` is set) is treated as unlimited and never raises. Both fields are also available on `RunConfig` and cascade through the same global → network → robot hierarchy as other infrastructure fields (see [RunConfig](../getting-started/configuration.md#runconfig-shared-operational-defaults)).
+
+This is a native alternative to the hand-rolled `BudgetHook` pattern in the [Hooks guide](hooks.md#cost-enforcement) for the common case of a per-robot token or dollar ceiling; reach for a hook instead when you need cross-robot session totals or custom accounting.
+
 ---
 
 ## Tool Loop Circuit Breaker
@@ -162,6 +197,120 @@ puts result.reply  # "The result is 42."
 
 ---
 
+## Doom Loop Detection
+
+### The Problem
+
+`max_tool_rounds` stops a robot that loops forever, but it fires on quantity alone. A subtler failure is when a robot cycles through the same tool call sequence repeatedly — calling tool A, then B, then C, then A again — without hitting the round limit. This is a doom loop: the robot is working but not making progress.
+
+### doom_loop_threshold
+
+```ruby
+robot = RobotLab.build(
+  name: "runner",
+  system_prompt: "Execute all steps.",
+  local_tools: [StepTool],
+  doom_loop_threshold: 3
+)
+```
+
+Set `doom_loop_threshold:` to the number of repetitions after which the detector fires. It catches two patterns:
+
+- **Consecutive repetition** — `[A, A, A]` (same tool called N times in a row)
+- **Cyclic repetition** — `[A, B, C, A, B, C, A, B, C]` (same sequence repeated N times)
+
+When a doom loop is detected, a warning message is embedded directly in the tool result, prompting the LLM to try a fundamentally different approach. This avoids corrupting the Anthropic message format (no injected user messages between `tool_use`/`tool_result` pairs).
+
+`doom_loop_threshold` can also be supplied via `RunConfig`:
+
+```ruby
+config = RobotLab::RunConfig.new(doom_loop_threshold: 3)
+robot  = RobotLab.build(name: "runner", system_prompt: "...", config: config)
+```
+
+### Complementary to max_tool_rounds
+
+Use both together for comprehensive loop protection:
+
+```ruby
+robot = RobotLab.build(
+  name: "executor",
+  system_prompt: "Execute every step.",
+  local_tools: [StepTool],
+  max_tool_rounds:    20,   # hard ceiling on total tool calls
+  doom_loop_threshold: 3    # catches repetitive patterns early
+)
+```
+
+---
+
+## Automatic Context Compaction
+
+### The Problem
+
+Long-running robots accumulate conversation history. Eventually, the cumulative token count approaches the model's context window limit, causing API errors or degraded performance. Manually calling `compress_history` at the right moment requires application-level bookkeeping.
+
+### auto_compact
+
+Set `auto_compact:` to have the robot compress its history automatically before each `run()`:
+
+```ruby
+# Compact when estimated token usage exceeds 80% of the model's context window
+robot = RobotLab.build(
+  name: "analyst",
+  system_prompt: "You are a research analyst.",
+  auto_compact: :context_window
+)
+```
+
+### Tuning the Threshold
+
+`compact_threshold:` sets the fraction of the model's context window that triggers compaction. Defaults to `0.80` (80%):
+
+```ruby
+robot = RobotLab.build(
+  name: "analyst",
+  system_prompt: "You are a research analyst.",
+  auto_compact:      :context_window,
+  compact_threshold: 0.70   # compact earlier, at 70%
+)
+```
+
+### Application-Owned Compaction
+
+Pass a `Proc` to take full control — the proc decides both when and how to compact:
+
+```ruby
+robot = RobotLab.build(
+  name: "analyst",
+  system_prompt: "You are a research analyst.",
+  auto_compact: ->(r) {
+    r.compress_history(recent_turns: 5) if r.chat.messages.size > 40
+  }
+)
+```
+
+The proc receives the robot instance and is called before every `run()` when messages are non-empty.
+
+### Options
+
+| Value | Behaviour |
+|-------|-----------|
+| `nil` / `:none` | No automatic compaction (default) |
+| `:context_window` | Compact when estimated token usage exceeds `compact_threshold` fraction of model's context window |
+| `Proc` | Called with the robot; application decides when and how to compact |
+
+Via `RunConfig`:
+
+```ruby
+config = RobotLab::RunConfig.new(auto_compact: :context_window, compact_threshold: 0.75)
+robot  = RobotLab.build(name: "analyst", system_prompt: "...", config: config)
+```
+
+Requires the `classifier` gem (`~> 2.3`) when using `:context_window`. Without it, a `RobotLab::DependencyError` is caught and logged rather than raised, so the robot continues running uncompressed.
+
+---
+
 ## Learning Accumulation
 
 ### The Problem
@@ -238,6 +387,21 @@ end
 
 After all three runs, `reviewer.learnings` contains up to three insights (fewer if any are subsets of others).
 
+### Durable Learning (learn: Constructor Shorthand)
+
+The `robot_lab-durable` gem adds automatic end-of-session learning promotion. Enable it with `learn: true` in the constructor:
+
+```ruby
+reviewer = RobotLab.build(
+  name:         "reviewer",
+  system_prompt: "You are a Ruby code reviewer.",
+  learn:        true,
+  learn_domain: "ruby_review"   # optional namespace for the durable store
+)
+```
+
+At the end of each session, the robot reflects on its observations and promotes durable insights to a YAML-backed store that persists across process restarts. On the next run, those stored insights are automatically reloaded as learnings.
+
 ### Memory Persistence
 
 Learnings are stored in `memory[:learnings]`. They survive a robot rebuild when the same `Memory` object is passed to the new robot:
@@ -475,12 +639,47 @@ puts "#{analysis.robot_name} (#{analysis.duration.round(2)}s): #{analysis.reply}
 
 ---
 
+## Live Narration (`RobotLab::Narrator`)
+
+`RobotLab::Narrator` is an opt-in [Hook](hooks.md) that narrates what a robot is doing as it happens, to `$stderr` (or any `IO`), so a run is never silent between events. It complements `RobotLab::Audit` (the `robot_lab-audit` gem, which records a persistent history for post-mortem analysis) with a live, human-facing console feed — the two are independent hooks and can both be registered at once.
+
+Enable it globally (applies to every robot run, including networks):
+
+```ruby
+RobotLab::Narrator.enable!                       # narrate to $stderr
+RobotLab::Narrator.enable!(output: $stdout)       # or any IO
+```
+
+Or register it like any hook for a narrower scope:
+
+```ruby
+robot.on(RobotLab::Narrator)
+network.on(RobotLab::Narrator)
+```
+
+Once registered, a run prints a line per event:
+
+```
+  · math_bot: thinking…
+  · → calculate operation="add"
+  · math_bot: thinking…
+```
+
+- Before each LLM call: `"<robot name>: thinking…"`
+- Before each tool call: `"→ <tool name> <first arg>=<value>"` — only the *first* argument is shown, truncated to 80 characters
+- After a tool call: `"  ✗ <error message>"` — printed only when the tool raised; silent on success
+
+Narrator uses `IO#puts` rather than `Kernel#warn`, since `warn` is silenced whenever Ruby warnings are disabled (`$VERBOSE` is `nil`, the common case under `bundle exec`). All three hooks rescue internally, so a narration failure never breaks the underlying run.
+
+---
+
 ## See Also
 
-- [Robot API](../api/core/robot.md#token--cost-tracking)
-- [Example 19 — Token & Cost Tracking](../../examples/19_token_tracking.rb)
-- [Example 20 — Tool Loop Circuit Breaker](../../examples/20_circuit_breaker.rb)
-- [Example 21 — Learning Accumulation Loop](../../examples/21_learning_loop.rb)
-- [Example 22 — Context Window Compression](../../examples/22_context_compression.rb)
-- [Example 23 — Convergence Detection](../../examples/23_convergence.rb)
-- [Example 24 — Structured Delegation](../../examples/24_structured_delegation.rb)
+- [Robot API](../api/core/robot.md#token-cost-tracking)
+- [examples/19_token_tracking.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/19_token_tracking.rb) — Token & Cost Tracking
+- [examples/20_circuit_breaker.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/20_circuit_breaker.rb) — Tool Loop Circuit Breaker
+- [examples/21_learning_loop.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/21_learning_loop.rb) — Learning Accumulation Loop
+- [examples/22_context_compression.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/22_context_compression.rb) — Context Window Compression
+- [examples/23_convergence.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/23_convergence.rb) — Convergence Detection
+- [examples/24_structured_delegation.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/24_structured_delegation.rb) — Structured Delegation
+- [RunConfig reference](../getting-started/configuration.md#runconfig-shared-operational-defaults)

data/docs/guides/using-tools.md

@@ -133,6 +133,15 @@ robot = RobotLab.build(
 )
 ```
 
+`local_tools:` and `tools:` are different mechanisms — don't confuse them. `local_tools:` attaches tool **instances or classes**; `tools:` (see [Runtime Tool Filtering](#runtime-tool-filtering) below) is a **name allowlist** that filters which of the already-attached tools are sent for a given turn. Passing an instance or class to `tools:` raises `ArgumentError` immediately, naming the offending class and pointing you at `local_tools:` instead:
+
+```ruby
+RobotLab.build(name: "bot", tools: [GetWeather.new])
+# => ArgumentError: `tools:` expects tool names (String/Symbol) to allow, but
+#    received GetWeather. To attach tool instances or classes, pass them as
+#    `local_tools:` (e.g. RobotLab.build(local_tools: [MyTool.new])).
+```
+
 ### Via Template Front Matter
 
 Declare tool class names in the template's YAML front matter. RobotLab resolves each string to a Ruby constant via `Object.const_get` and instantiates it:
@@ -163,6 +172,66 @@ robot = RobotLab.build(name: "assistant", system_prompt: "...")
 robot.with_tools(GetWeather, CalculatorTool)
 ```
 
+## Runtime Tool Filtering
+
+`tools:` (and `mcp:`) also work as a **per-run** override, passed to `run()` itself, on top of the build-time/network/global hierarchy described in [Hierarchical MCP and Tools](../getting-started/configuration.md#hierarchical-mcp-and-tools):
+
+```ruby
+robot.run("What's the weather?", tools: :inherit)        # every attached tool
+robot.run("Just chat, no tools needed.", tools: :none)    # zero tools this turn
+robot.run("Only use the calculator.", tools: %w[calculator])  # allowlist by name
+```
+
+| Value | Meaning |
+|-------|---------|
+| `:inherit` | No filter — use every tool attached at the robot/network/global level |
+| `:none`, `[]`, `nil` | Send **zero** tools this turn |
+| `["name", ...]` | Only these tool names, by exact match |
+
+An explicit `:none`/`[]` is useful for a relevance filter that decided no tool is useful for the current message — it now genuinely sends zero tools for that turn (previously an empty allowlist was silently treated as "all tools," which could overflow small-context local models with the full tool set). Each turn's resolved tool set fully **replaces** the chat's tools rather than accumulating, so a later `:none` turn correctly clears whatever a prior turn attached.
+
+> **Watch the default.** Both `Robot.new`'s and `run()`'s `tools:`/`mcp:` parameters default to `:none`, not `:inherit`. If you build a robot with `local_tools:` and then call `robot.run(message)` with no `tools:` override at all, the current runtime default takes the explicit-`:none` path above — sending no tools for that turn. Pass `tools: :inherit` explicitly (at build time, per network task, or per `run()` call — wherever fits your call site) anywhere you need the robot's attached tools available.
+
+### Tool Capping and Per-Turn Filtering
+
+Most LLM providers reject a tool array longer than 128 entries and fail the whole turn. RobotLab clamps the fully-resolved tool list to a ceiling right before handing it to the chat provider — the definitive choke point regardless of how the tools were configured, filtered, or MCP-connected:
+
+```ruby
+robot = RobotLab.build(
+  name: "power_user",
+  system_prompt: "...",
+  local_tools: many_tools,       # say, 150 tools
+  config: RobotLab::RunConfig.new(max_tools: 50)  # override the default ceiling
+)
+```
+
+- Default ceiling: **128** tools per turn (`RobotLab::Robot::DEFAULT_MAX_TOOLS`)
+- Override with `max_tools:` on `RunConfig` (or the `max_tools:` cascade field — see [Available Fields](../getting-started/configuration.md#available-fields)); `nil` or `<= 0` disables the cap
+- When a turn's resolved tools exceed the cap, RobotLab logs a warning naming how many were dropped and sends the first `max_tools` entries
+
+## Skill Scripts and Sandboxing
+
+A skill bundle (a directory with a `SKILL.md` plus `scripts/`, discovered via `AgentSkill`) can expose its scripts as tools (`ScriptTool`). Because those scripts run as real OS processes, each `SKILL.md` can declare the capabilities its scripts need directly in front matter, alongside `name`/`description`:
+
+```markdown title="skills/deploy-checker/SKILL.md"
+---
+name: deploy-checker
+description: Verifies a deployment's health before promoting it.
+fs_read: ["./data", "/etc/hosts"]
+fs_write: ["./out"]
+network: true
+timeout: 30
+trust: external   # or "core" for trusted, always-unconfined skills
+---
+```
+
+Sandboxing itself is **opt-in and off by default** — see the [`sandbox:` config section](../getting-started/configuration.md#skill-script-sandboxing-sandbox-section). When disabled, scripts run exactly as they always have, unconfined. When enabled:
+
+- The global `sandbox:` config is a **ceiling** (`fs_read`, `fs_write`, `network`, `timeout`); each skill's front matter is its **declared** request. The script actually runs under the **intersection** of the two — a path outside the ceiling's roots is dropped even if the skill declares it, `network` requires both sides to allow it, and `timeout` is the smaller of the two.
+- On macOS, confinement is enforced with a generated `sandbox-exec` (Seatbelt) profile: deny-by-default, with narrow allowances for the interpreter to boot, the granted read/write paths, and (optionally) the network. Notably, `$HOME` is never implicitly readable — SSH keys and cloud credentials stay out of reach unless a path under `$HOME` is explicitly granted.
+- Off macOS, or for any skill declaring `trust: core`, sandboxing is a passthrough — confinement is currently macOS-only and is always skipped for trusted "core" skills regardless of platform.
+- A script that runs past its `timeout` is killed (its whole process group) and reported back to the LLM as a timed-out error rather than hanging the turn.
+
 ## Parameter Types
 
 Define parameters on `RubyLLM::Tool` subclasses using `param`:

data/docs/index.md

@@ -50,7 +50,7 @@ Each robot is backed by a persistent LLM chat, configured with keyword arguments
 
     Mix reusable prompt behaviors into any robot. Skills expand depth-first with automatic cycle detection and config cascading.
 
-    [:octicons-arrow-right-24: Skills Guide](guides/building-robots.md#skills)
+    [:octicons-arrow-right-24: Skills Guide](guides/building-robots.md#composable-skills)
 
 -   :material-tools:{ .lg .middle } **Extensible Tools**
 
@@ -116,16 +116,43 @@ Each robot is backed by a persistent LLM chat, configured with keyword arguments
 
     [:octicons-arrow-right-24: Configuration](getting-started/configuration.md)
 
--   :material-train-car-container:{ .lg .middle } **Rails Integration**
+-   :material-shield-check:{ .lg .middle } **Observability & Safety**
 
     ---
 
-    Generators, background jobs, and Turbo Stream token broadcasting for real-time streaming to the browser.
+    Token tracking, `max_tool_rounds` circuit breaker, `doom_loop_threshold` pattern detection, and `auto_compact` context window management keep robots reliable in production.
 
-    [:octicons-arrow-right-24: Rails Guide](guides/rails-integration.md)
+    [:octicons-arrow-right-24: Observability Guide](guides/observability.md)
+
+-   :material-brain:{ .lg .middle } **Runtime Skill Matching**
+
+    ---
+
+    `AgentSkillMatching` selects the most relevant `AgentSkill` files from a catalog directory via semantic similarity before each `run()`, injecting only what the robot needs.
+
+    [:octicons-arrow-right-24: Using Tools](guides/using-tools.md)
+
+-   :material-book-open-outline:{ .lg .middle } **Knowledge & Retrieval**
+
+    ---
+
+    `robot.search_history` for semantic search over conversation turns, and `memory.store_document` / `memory.search_documents` for embedding-based RAG.
+
+    [:octicons-arrow-right-24: Knowledge Guide](guides/knowledge.md)
 
 </div>
 
+## Extension Gems
+
+These optional gems extend RobotLab with additional capabilities:
+
+| Gem | What it adds |
+|-----|-------------|
+| [robot_lab-rails](https://github.com/MadBomber/robot_lab-rails) | Rails Engine, generators, `RobotLab::Job` base class, Turbo Stream broadcasting |
+| [robot_lab-ractor](https://github.com/MadBomber/robot_lab-ractor) | CPU parallelism for `ractor_safe` tools and robot networks via Ruby Ractors |
+| [robot_lab-durable](https://github.com/MadBomber/robot_lab-durable) | Cross-session knowledge persistence — robots accumulate and recall learned facts |
+| [robot_lab-document_store](https://github.com/MadBomber/robot_lab-document_store) | Embedding-based semantic document search powered by fastembed |
+
 ## Quick Example
 
 ```ruby