robot_lab 0.3.0 → 0.5.0

data/docs/guides/index.md

@@ -1,78 +0,0 @@
-# Guides
-
-Practical guides for building applications with RobotLab.
-
-## Getting Started
-
-If you're new to RobotLab, start here:
-
-<div class="grid cards" markdown>
-
--   [:octicons-cpu-24: **Building Robots**](building-robots.md)
-
-    Create specialized AI agents with personalities and tools
-
--   [:octicons-git-branch-24: **Creating Networks**](creating-networks.md)
-
-    Orchestrate multiple robots for complex workflows
-
-</div>
-
-## Core Features
-
-<div class="grid cards" markdown>
-
--   [:octicons-tools-24: **Using Tools**](using-tools.md)
-
-    Give robots custom capabilities to interact with external systems
-
--   [:octicons-server-24: **MCP Integration**](mcp-integration.md)
-
-    Connect to Model Context Protocol servers
-
--   [:octicons-broadcast-24: **Streaming Responses**](streaming.md)
-
-    Real-time streaming of LLM responses
-
--   [:octicons-cpu-24: **Memory System**](memory.md)
-
-    Share data between robots with the memory system
-
--   [:octicons-pulse-24: **Observability & Safety**](observability.md)
-
-    Token tracking, circuit breakers, doom loop detection, auto compaction, and learning accumulation
-
--   [:octicons-zap-24: **Hook System**](hooks.md)
-
-    Lifecycle hooks for instrumentation, response caching, extensions, and cross-cutting concerns
-
--   [:octicons-search-24: **Knowledge & Retrieval**](knowledge.md)
-
-    Chat history search and embedding-based document store for RAG workflows
-
-</div>
-
-## Guide Index
-
-| Guide | Description | Time |
-|-------|-------------|------|
-| [Building Robots](building-robots.md) | Create and configure robots | 10 min |
-| [Creating Networks](creating-networks.md) | Multi-robot orchestration | 15 min |
-| [Using Tools](using-tools.md) | Add custom capabilities | 10 min |
-| [MCP Integration](mcp-integration.md) | External tool servers | 10 min |
-| [Streaming](streaming.md) | Real-time responses | 5 min |
-| [Memory](memory.md) | Shared data store | 5 min |
-| [Observability & Safety](observability.md) | Token tracking, circuit breaker, doom loop detection, auto compaction, learning loop | 10 min |
-| [Hook System](hooks.md) | Lifecycle hooks, extensions, response caching, instrumentation | 15 min |
-| [Knowledge & Retrieval](knowledge.md) | Chat history search and embedding-based document store (RAG) | 10 min |
-
-## Extension Gems
-
-Additional capabilities are available as separate gems:
-
-| Gem | Description | Docs |
-|-----|-------------|------|
-| [robot_lab-rails](https://github.com/MadBomber/robot_lab-rails) | Rails generators, background jobs, Turbo Stream broadcasting | [Rails Integration guide](https://github.com/MadBomber/robot_lab-rails/blob/main/docs/guides/rails-integration.md) |
-| [robot_lab-ractor](https://github.com/MadBomber/robot_lab-ractor) | True CPU parallelism for tools and robot pipelines via Ruby Ractors | [Ractor Parallelism guide](https://github.com/MadBomber/robot_lab-ractor/blob/main/docs/guides/ractor-parallelism.md) |
-| [robot_lab-durable](https://github.com/MadBomber/robot_lab-durable) | Cross-session knowledge persistence with YAML-backed storage | [robot_lab-durable README](https://github.com/MadBomber/robot_lab-durable) |
-| [robot_lab-document_store](https://github.com/MadBomber/robot_lab-document_store) | Embedding-based semantic document search via fastembed | [robot_lab-document_store README](https://github.com/MadBomber/robot_lab-document_store) |

data/docs/guides/knowledge.md

@@ -1,219 +0,0 @@
-# Knowledge & Retrieval
-
-Facilities for searching and retrieving knowledge from a robot's history and from external documents:
-
-- **Chat History Search** — semantic search over accumulated conversation turns
-- **Embedding-Based Document Store** — lightweight RAG: store arbitrary text, search by meaning
-
----
-
-## Chat History Search
-
-### The Problem
-
-Long-running robots accumulate many conversation turns. When you need to recall what was discussed earlier on a specific topic, re-sending the full history wastes tokens. `search_history` gives you a focused slice of the most relevant past messages without touching the LLM.
-
-### robot.search_history
-
-```ruby
-results = robot.search_history(query, limit: 5)
-```
-
-Scores every message in the robot's conversation history against `query` using stemmed term-frequency cosine similarity (via the `classifier` gem). Returns up to `limit` `HistoryResult` objects sorted by score descending.
-
-> [!NOTE]
-> **Every** message in `@chat.messages` is scored — there is no role filter.
-> Results routinely include the `:system` prompt and raw `:tool` result payloads
-> (often JSON) alongside `:user` and `:assistant` turns. Filter by role yourself
-> if you only want the conversation:
->
-> ```ruby
-> hits = robot.search_history(query, limit: 20)
->            .select { |r| %i[user assistant].include?(r.role) }
->            .first(3)
-> ```
-
-```ruby
-results = robot.search_history("quarterly revenue", limit: 3)
-
-results.each do |r|
-  puts "[#{r.role}] score=#{r.score.round(3)} idx=#{r.index}"
-  puts "  #{r.text}"
-end
-```
-
-### HistoryResult Fields
-
-| Field | Type | Description |
-|-------|------|-------------|
-| `text` | String | The message text |
-| `role` | Symbol | The message's role — `:user`, `:assistant`, `:system`, or `:tool` |
-| `score` | Float (0.0–1.0) | Cosine similarity with the query |
-| `index` | Integer | Position in `@chat.messages` |
-
-### Typical Scores
-
-| Relationship | Typical Score |
-|---|---|
-| Direct answer to the query | 0.50 – 0.80 |
-| Same topic, different phrasing | 0.20 – 0.50 |
-| Unrelated | < 0.10 |
-
-### Short Messages
-
-Messages shorter than 20 characters are skipped — they produce no meaningful term vector.
-
-### Full Example
-
-```ruby
-robot = RobotLab.build(name: "analyst", system_prompt: "You are a financial analyst.")
-
-# … after several robot.run() calls …
-
-hits = robot.search_history("customer acquisition cost")
-hits.each { |r| puts "#{r.role} (#{r.score.round(2)}): #{r.text}" }
-```
-
-### RAG Pattern — Retrieve Then Generate
-
-Use `search_history` to inject only the relevant past context into the next call:
-
-```ruby
-hits    = robot.search_history(user_query, limit: 3)
-context = hits.map(&:text).join("\n")
-
-robot.run("Recall context:\n#{context}\n\nNew question: #{user_query}")
-```
-
-### Optional Dependency
-
-`search_history` requires the `classifier` gem:
-
-```ruby
-gem "classifier", "~> 2.3"
-```
-
-Without it, calling `search_history` raises `RobotLab::DependencyError` with an install hint.
-
----
-
-## Embedding-Based Document Store
-
-### The Problem
-
-Sometimes the knowledge you need isn't in the conversation history — it's in a README, a product spec, a changelog. `store_document` / `search_documents` embed arbitrary text and retrieve the most relevant chunk at query time. Embeddings come from `fastembed` when it is installed, and from a built-in TF-IDF fallback when it is not (see [Embedding Model](#embedding-model)).
-
-### memory.store_document / memory.search_documents
-
-```ruby
-memory.store_document(:readme,    File.read("README.md"))
-memory.store_document(:changelog, File.read("CHANGELOG.md"))
-
-hits = memory.search_documents("how to configure redis", limit: 3)
-hits.each { |h| puts "#{h[:key]} (#{h[:score].round(3)}): #{h[:text][0..80]}" }
-```
-
-Each result hash contains:
-
-| Key | Type | Description |
-|-----|------|-------------|
-| `:key` | Symbol | The key the document was stored under |
-| `:text` | String | The full stored text |
-| `:score` | Float (0.0–1.0) | Cosine similarity with the query |
-
-### Standalone DocumentStore
-
-The `Memory` methods delegate to `RobotLab::DocumentStore`, which can also be used directly:
-
-```ruby
-store = RobotLab::DocumentStore.new
-store.store(:doc_a, "Ruby on Rails is a full-stack web framework.")
-store.store(:doc_b, "Postgres is an advanced relational database.")
-
-results = store.search("relational database SQL", limit: 2)
-puts results.first[:key]  # => :doc_b
-```
-
-Management methods:
-
-```ruby
-store.size          # => 2
-store.keys          # => [:doc_a, :doc_b]
-store.empty?        # => false
-store.delete(:doc_a)
-store.clear
-```
-
-### Embedding Model
-
-Default: `BAAI/bge-small-en-v1.5` (~23 MB, downloaded on first use, cached in `~/.cache/fastembed/`).
-
-Documents and queries are embedded asymmetrically: stored text goes through `Fastembed::TextEmbedding#passage_embed`, query text through `#query_embed`. RobotLab does **not** prepend any literal `"passage: "` / `"query: "` string of its own — it hands the raw text to the two methods and lets fastembed handle whatever instruction prefixing the model expects.
-
-Custom model:
-
-```ruby
-store = RobotLab::DocumentStore.new(model_name: "BAAI/bge-base-en-v1.5")
-```
-
-### TF-IDF Fallback
-
-`DocumentStore` decides once, at load time, whether `fastembed` is available
-(`RobotLab::DocumentStore::FASTEMBED_AVAILABLE`). When it is **not**, the store
-still works — it silently falls back to a stemmed, stop-word-filtered,
-L2-normalised term-frequency embedder and compares those sparse vectors by
-cosine similarity. Nothing is downloaded and everything runs offline, which
-makes it convenient for development and CI.
-
-> [!WARNING]
-> The fallback is **lexical, not semantic**. It matches shared word stems only,
-> so a query and a document that mean the same thing in different words score
-> `0.0`. There is no error, no warning, and no change to the API — only
-> noticeably worse results. Check `RobotLab::DocumentStore::FASTEMBED_AVAILABLE`
-> if you need to know which path you are on, and note that `model_name:` is
-> ignored entirely when it is `false`.
-
-### RAG Pattern
-
-```ruby
-# 1. Index your knowledge base at startup
-memory.store_document(:readme,    File.read("README.md"))
-memory.store_document(:changelog, File.read("CHANGELOG.md"))
-memory.store_document(:api_docs,  File.read("docs/api.md"))
-
-# 2. At query time, retrieve the most relevant chunks
-hits    = memory.search_documents(user_query, limit: 3)
-context = hits.map { |h| h[:text] }.join("\n\n")
-
-# 3. Pass context to your robot
-result = robot.run("Use the following context:\n#{context}\n\nQuestion: #{user_query}")
-```
-
-### Memory API Summary
-
-| Method | Description |
-|--------|-------------|
-| `memory.store_document(key, text)` | Embed and store a document |
-| `memory.search_documents(query, limit: 5)` | Search by semantic similarity |
-| `memory.document_keys` | List stored keys |
-| `memory.delete_document(key)` | Remove a document |
-
-### Dependency
-
-The embedding-based document store requires the [`robot_lab-document_store`](https://github.com/MadBomber/robot_lab-document_store) extension gem. Add it to your Gemfile:
-
-```ruby
-gem "robot_lab-document_store"
-```
-
-The gem uses `fastembed` for ONNX-based embeddings when it is installed: the `BAAI/bge-small-en-v1.5` model (~23 MB) is downloaded on first use and cached in `~/.cache/fastembed/`. If `fastembed` cannot be loaded, `DocumentStore` still works via the lexical [TF-IDF fallback](#tf-idf-fallback).
-
-Without `robot_lab-document_store` itself loaded, calling `memory.store_document`, `memory.search_documents`, `memory.document_keys`, or `memory.delete_document` raises `RobotLab::DependencyError`.
-
----
-
-## See Also
-
-- [Observability Guide](observability.md)
-- [examples/25_history_search.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/25_history_search.rb) — Chat History Search
-- [examples/26_document_store.rb](https://github.com/MadBomber/robot_lab/blob/main/examples/26_document_store.rb) — Embedding-Based Document Store