robot_lab-document_store 0.1.0 → 0.1.1

data/docs/getting_started.md

@@ -0,0 +1,106 @@
+# Getting Started
+
+## Prerequisites
+
+- Ruby 3.1+
+- **fastembed** (recommended) — requires a platform that can run ONNX Runtime (x86_64 and ARM64 macOS/Linux). On first use the ~23 MB `BAAI/bge-small-en-v1.5` model file is downloaded and cached in `~/.cache/fastembed`.
+- Without fastembed the store still works using the built-in TF-IDF fallback (see [Fallback Mode](#fallback-mode) below).
+
+## Installation
+
+Add to your `Gemfile`:
+
+```ruby
+gem "robot_lab-document_store"
+```
+
+Then:
+
+```sh
+bundle install
+```
+
+Or install directly:
+
+```sh
+gem install robot_lab-document_store
+```
+
+### Installing fastembed
+
+fastembed is an optional dependency. To get semantic search quality, install it:
+
+```ruby
+# Gemfile
+gem "fastembed"
+```
+
+```sh
+bundle install
+```
+
+The ONNX model is downloaded on the first embed call and cached locally. Subsequent starts reuse the cache — no download needed.
+
+## First Run
+
+```ruby
+require "robot_lab/document_store"
+
+store = RobotLab::DocumentStore.new
+
+# Store documents — embedding happens here (model downloads if needed)
+store.store(:ruby_intro,    "Ruby is a dynamic, open source programming language.")
+store.store(:python_intro,  "Python is widely used in data science and AI.")
+store.store(:js_intro,      "JavaScript runs in the browser and powers Node.js.")
+
+# Search — returns Array of { key:, text:, score: } hashes
+results = store.search("Which language is used for machine learning?", limit: 2)
+
+results.each do |r|
+  puts "#{r[:key]}  (#{r[:score].round(3)})"
+end
+# => python_intro  (0.871)
+# => ruby_intro    (0.634)
+```
+
+!!! note "Model download on first run"
+    The first call to `store` or `search` triggers a ~23 MB ONNX model download.
+    All subsequent runs reuse the cached model. Set `FASTEMBED_CACHE_PATH` to control
+    the cache location.
+
+## Fallback Mode
+
+When `fastembed` is not installed, `DocumentStore` automatically switches to a
+TF-IDF word-frequency embedder. No configuration needed — the switch is silent.
+
+```text
+fastembed installed?
+  YES → dense vector embeddings via BAAI/bge-small-en-v1.5
+   NO → sparse TF-IDF bag-of-words with Porter-style stemming
+```
+
+The fallback is lower quality — it relies on lexical overlap rather than semantic
+understanding — but it works offline with no model downloads, making it well-suited
+for development, CI, and test environments.
+
+```ruby
+# Works identically regardless of whether fastembed is installed
+store = RobotLab::DocumentStore.new
+store.store(:doc, "Ruby programming language")
+store.search("Ruby development", limit: 1)
+```
+
+## Running the Example
+
+The gem ships with a self-contained example that demonstrates all core features:
+
+```sh
+bundle exec ruby examples/01_basic_usage.rb
+```
+
+This loads five engineering runbook documents, runs several semantic queries,
+demonstrates deletion, and shows the RobotLab Memory integration.
+
+!!! note
+    The example requires `robot_lab` core for the Memory integration section.
+    The standalone `DocumentStore` section runs without it.

data/docs/how_it_works.md

@@ -0,0 +1,141 @@
+# How It Works
+
+## Architecture Overview
+
+![Embedding Pipeline](assets/architecture.svg)
+
+Two operations drive the store: **`store`** (embed and save) and **`search`** (embed and rank). Both paths share the same embedding backend — fastembed when available, TF-IDF otherwise.
+
+---
+
+## Embedding Backend Selection
+
+On load, `DocumentStore` attempts to `require 'fastembed'`. The result is captured in `FASTEMBED_AVAILABLE` and drives every subsequent embed call. There is no runtime switching — the backend is fixed for the lifetime of the object.
+
+```
+require 'fastembed'
+  → success  : FASTEMBED_AVAILABLE = true   → dense vector path
+  → LoadError: FASTEMBED_AVAILABLE = false  → TF-IDF fallback path
+```
+
+---
+
+## Fastembed Path — Dense Vectors
+
+When fastembed is available, the store uses [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) by default — a 33M parameter English bi-encoder that produces 384-dimensional float vectors.
+
+### Asymmetric Embedding
+
+The model uses **separate encoders** for passages (stored documents) and queries (search terms). This is deliberate: a passage encoder is optimised to capture what a document *contains*, while a query encoder is optimised to capture what a user *wants*. Using the same encoder for both degrades recall on semantically paraphrased queries.
+
+| Call | Encoder used | Purpose |
+|------|-------------|---------|
+| `store(key, text)` | `passage_embed` | Captures document content |
+| `search(query)` | `query_embed` | Captures search intent |
+
+### Cosine Similarity
+
+Given a query vector **q** and a stored passage vector **p**, similarity is:
+
+```
+similarity = dot(q, p) / (‖q‖ · ‖p‖)
+```
+
+This is the cosine of the angle between the vectors. A value of `1.0` means identical
+direction (maximum similarity); `0.0` means orthogonal (no relationship). The
+BGE model normalises its output to unit length, so the division reduces to a simple
+dot product — but `DocumentStore` performs explicit normalisation anyway to remain
+correct regardless of the model used.
+
+**Defensive guards** return `0.0` for nil vectors, empty vectors, or length mismatches.
+These protect against partial or corrupted embed results without raising exceptions.
+
+---
+
+## TF-IDF Fallback Path — Sparse Vectors
+
+When fastembed is unavailable, each document is converted to a sparse
+`Hash{String => Float}` where keys are stemmed terms and values are L2-normalised
+term frequencies.
+
+### Processing Pipeline
+
+```
+raw text
+  → downcase
+  → tokenise /[a-z]+/   (ASCII only — Unicode letters are dropped)
+  → remove STOP_WORDS   (38 common English words: a, an, the, is, are, …)
+  → Porter-style stem   (strips: -ies, -ness, -ment, -tion, -ing, -ed, -er, -ly, -s)
+  → count term frequencies
+  → L2-normalise counts  (divide each count by the Euclidean norm of the count vector)
+```
+
+The result is a unit-length sparse vector. Similarity between two sparse vectors
+uses the same cosine formula, but computed efficiently by iterating only over the
+keys present in the smaller vector.
+
+### Stemmer
+
+The stemmer applies suffix rules in priority order and stops at the first match:
+
+| Rule | Example |
+|------|---------|
+| `-ies` → `-y` | `activities` → `activiti` |
+| `-ness` → `` | `darkness` → `dark` |
+| `-ment` → `` | `development` → `develop` |
+| `-tion` → `` | `configuration` → `configura` |
+| `-ing` → `` | `running` → `runn` |
+| `-ed` → `` | `deployed` → `deploy` |
+| `-er` → `` | `server` → `serv` |
+| `-ly` → `` | `quickly` → `quick` |
+| `-s` → `` | `robots` → `robot` |
+
+This is intentionally simple — not a full Porter stemmer. Its purpose is to improve
+lexical recall during development and testing, not to rival production semantic search.
+
+### Limitations of the Fallback
+
+- **No semantic understanding** — synonyms, paraphrases, and cross-lingual queries will not match
+- **ASCII only** — non-ASCII characters (accented letters, CJK, emoji) are silently dropped
+- **Order-insensitive** — "connection pool exhausted" and "exhausted pool connection" score identically
+
+For any production use case, install fastembed.
+
+---
+
+## Thread Safety
+
+All public methods acquire an internal `Mutex` before touching `@documents`.
+Embedding (which can be slow — tens of milliseconds for the first call) happens
+**outside** the lock to avoid blocking concurrent readers.
+
+```ruby
+def store(key, text)
+  key    = key.to_sym
+  vector = passage_vector(text)          # ← compute outside the lock
+  @mutex.synchronize do
+    @documents[key] = { text:, vector: } # ← write inside the lock
+  end
+  self
+end
+```
+
+This means multiple threads can embed documents concurrently. The lock only
+serialises the final hash write and all reads.
+
+---
+
+## RobotLab Extension Registration
+
+The file bottom contains:
+
+```ruby
+if defined?(RobotLab) && RobotLab.respond_to?(:register_extension)
+  RobotLab.register_extension(:document_store, RobotLab::DocumentStore)
+end
+```
+
+This registers the class with robot_lab core when both gems are loaded together,
+enabling `Memory#store_document`, `Memory#search_documents`, and `Memory#document_keys`
+on any `RobotLab::Memory` instance. The guard makes the file safe to `require`
+standalone without robot_lab present.

data/docs/index.md

@@ -2,57 +2,40 @@
 
 Embedding-based semantic document search for the [RobotLab](https://github.com/MadBomber/robot_lab) LLM agent framework.
 
-> [!CAUTION]
-> This gem is under active development. APIs may change without notice.
+!!! warning "Under active development"
+    APIs may change without notice until v1.0.
 
-## What it provides
-
-`RobotLab::DocumentStore` is a thread-safe, in-memory vector store backed by [fastembed](https://github.com/Anush008/fastembed-ruby) embeddings and cosine similarity search. It supports:
-
-- **`store(key, text)`** — embed and store a document under a symbol key
-- **`search(query, limit:)`** — return the top-N most similar documents by cosine similarity
-- **`delete(key)`** / **`clear`** — remove individual entries or wipe the store
-- **Asymmetric embedding** — passage embeddings for storage, query embeddings for retrieval
-
-## Installation
-
-Add to your Gemfile:
-
-```ruby
-gem "robot_lab-document_store"
-```
-
-## Quick Example
+`RobotLab::DocumentStore` is a thread-safe, in-memory vector store. Store arbitrary text documents, then retrieve the most relevant ones by natural-language query — no keyword overlap required.
 
 ```ruby
 require "robot_lab/document_store"
 
 store = RobotLab::DocumentStore.new
 
-store.store(:alpha, "Ruby is a dynamic, open source programming language.")
-store.store(:beta,  "Python is widely used in data science and machine learning.")
-store.store(:gamma, "JavaScript runs in the browser and on Node.js servers.")
+store.store(:sidekiq_guide,   File.read("docs/sidekiq.md"))
+store.store(:postgres_guide,  File.read("docs/postgres.md"))
+store.store(:incident_report, File.read("docs/outage_2024.md"))
 
-results = store.search("What language is popular for AI?", limit: 2)
-results.each do |r|
-  puts "#{r[:key]} (score: #{"%.3f" % r[:score]})"
-end
-# => beta (score: 0.872)
-# => alpha (score: 0.641)
+hits = store.search("Jobs keep piling up when Stripe is down", limit: 2)
+hits.each { |r| puts "#{r[:key]}  score=#{r[:score].round(3)}" }
+# => sidekiq_guide   score=0.847
+# => incident_report score=0.612
 ```
 
-## Custom Model
-
-```ruby
-store = RobotLab::DocumentStore.new(
-  model_name: "BAAI/bge-small-en-v1.5"
-)
-```
+## Features
 
-The default model is `"BAAI/bge-base-en-v1.5"`.
+| Feature | Detail |
+|---------|--------|
+| **Semantic search** | Cosine similarity over dense vector embeddings |
+| **Asymmetric embedding** | Separate passage/query embeddings for higher recall |
+| **Thread-safe** | Internal `Mutex` — safe for Puma, Sidekiq, Ractor workers |
+| **Zero-config fallback** | TF-IDF word-frequency search when fastembed is unavailable |
+| **Lazy model init** | ONNX model downloads on first use, cached locally |
+| **RobotLab integration** | Drop-in via `Memory#store_document` / `Memory#search_documents` |
 
-## Links
+## Navigation
 
-- [RobotLab Core](https://github.com/MadBomber/robot_lab)
-- [fastembed-ruby](https://github.com/Anush008/fastembed-ruby)
-- [RubyGems](https://rubygems.org/gems/robot_lab-document_store)
+- [Getting Started](getting_started.md) — install, first run, fallback mode
+- [API Reference](api_reference.md) — every public method documented
+- [How It Works](how_it_works.md) — embeddings, cosine similarity, TF-IDF fallback
+- [RAG Patterns](rag_patterns.md) — retrieval-augmented generation recipes

data/docs/pluggable_backends_design.md

@@ -0,0 +1,66 @@
+# Pluggable Backend Architecture — Design Discussion
+
+**Date:** 2026-05-14
+**Status:** Parked — resume when time allows
+
+## The Vision
+
+`DocumentStore` should become an **abstract interface** that defines the storage contract but does not implement any backend itself. The gem ships with two concrete backends; applications supply their own.
+
+### Backends shipping with the gem
+
+| Class | Description |
+|---|---|
+| `DocumentStore::Memory` | In-memory store — essentially the current implementation (fastembed/TF-IDF). No persistence. |
+| `DocumentStore::FileSystem` | File-backed store — YAML persistence, similar to `robot_lab-durable`'s current `Store` class. |
+
+### Backends supplied by applications
+
+Applications implement their own adapters (same interface) for:
+- `DocumentStore::Redis`
+- `DocumentStore::Database`
+- Any other backend
+
+All backend code lives in the application, not in this gem.
+
+## Motivation
+
+`robot_lab-durable` currently maintains its own `Store` class (YAML file-backed with file locking). Once `DocumentStore::FileSystem` exists, durable can drop its custom storage layer and delegate to it — keeping only its durable-specific concerns: `Entry` (confidence scoring), `Reflector` (session reflection), `Learning` (Robot mixin), and the LLM tools.
+
+## Interface Contract
+
+All backends must implement:
+
+```ruby
+store(key, text)       # embed and persist a document under key
+search(query, limit:)  # return Array<Hash> sorted by score descending
+                       #   each Hash: { key:, text:, score: }
+delete(key)            # remove document by key; return self
+clear                  # remove all documents; return self
+size                   # Integer
+keys                   # Array<Symbol>
+empty?                 # Boolean
+```
+
+## Open Questions
+
+1. **Search semantics differ across backends.**
+   `Memory` uses embedding-based cosine similarity. `FileSystem` would use keyword matching (tokenize + stem, like durable's current approach). Should the interface treat these as equivalent, or should backends declare their search capability? Options:
+   - Accept the difference — callers get whatever the backend can do.
+   - Add a `#search_strategy` or `#semantic?` predicate to the base class.
+
+2. **Structured vs raw text storage.**
+   `DocumentStore` today stores raw text by key. `robot_lab-durable` stores structured `Entry` objects (confidence, category, domain, use_count). Two options:
+   - `FileSystem` stores raw text; durable serializes/deserializes `Entry` fields into the text before storing.
+   - `FileSystem` supports structured metadata alongside text (a `meta:` hash), which durable populates.
+
+3. **Breaking change.** Refactoring `DocumentStore` from a concrete class to an abstract base is a breaking change — this is v0.2.0 territory.
+
+## Rough Implementation Plan
+
+1. Extract the current `DocumentStore` implementation into `DocumentStore::Memory`.
+2. Define `DocumentStore` as an abstract base class with `NotImplementedError` stubs for each interface method.
+3. Implement `DocumentStore::FileSystem` — port durable's `Store` (YAML, file locking, keyword search).
+4. Update `robot_lab-durable` gemspec to add `robot_lab-document_store` as a dependency.
+5. Replace `RobotLab::Durable::Store` with `DocumentStore::FileSystem` in durable's internals.
+6. Bump `robot_lab-document_store` to v0.2.0; bump `robot_lab-durable` to v0.2.0.

data/docs/rag_patterns.md

@@ -0,0 +1,198 @@
+# RAG Patterns
+
+Retrieval-Augmented Generation (RAG) is the practice of retrieving relevant
+documents at query time and injecting them into an LLM prompt as context. This
+gives the model access to private or up-to-date information without fine-tuning.
+
+`DocumentStore` is the retrieval layer. The LLM call is your responsibility —
+typically via `RobotLab` robots or the `ruby_llm` gem.
+
+---
+
+## Pattern 1 — Standalone Retrieval
+
+The simplest pattern: build the store at startup, query it per-request.
+
+```ruby
+require "robot_lab/document_store"
+
+# ── Build once at startup ────────────────────────────────────────────────────
+store = RobotLab::DocumentStore.new
+
+Dir["docs/**/*.md"].each do |path|
+  key = File.basename(path, ".md").to_sym
+  store.store(key, File.read(path))
+end
+
+puts "Loaded #{store.size} documents"
+
+# ── Query per-request ────────────────────────────────────────────────────────
+query = "How do I investigate a slow Postgres query?"
+hits  = store.search(query, limit: 3)
+
+context = hits.map.with_index(1) do |h, i|
+  "## Document #{i}: #{h[:key]}\n\n#{h[:text]}"
+end.join("\n\n---\n\n")
+
+prompt = <<~PROMPT
+  Answer the following question using only the provided documents.
+
+  #{context}
+
+  ---
+
+  Question: #{query}
+PROMPT
+
+# Pass `prompt` to your LLM of choice
+```
+
+---
+
+## Pattern 2 — RobotLab Memory Integration
+
+When `robot_lab` core is loaded alongside this gem, `RobotLab::Memory` gains
+three document-store methods automatically via `register_extension`.
+
+```ruby
+require "robot_lab"
+require "robot_lab/document_store"
+
+memory = RobotLab::Memory.new
+
+# Store documents
+memory.store_document(:runbook,   File.read("ops/runbook.md"))
+memory.store_document(:postmortem, File.read("ops/postmortem.md"))
+
+# List keys
+memory.document_keys  # => [:runbook, :postmortem]
+
+# Search
+hits = memory.search_documents("database outage lock contention", limit: 2)
+hits.each { |h| puts "#{h[:key]}  #{h[:score].round(3)}" }
+
+# Remove
+memory.delete_document(:postmortem)
+```
+
+!!! note
+    `Memory#store_document` / `#search_documents` / `#delete_document` /
+    `#document_keys` are only available when `robot_lab-document_store` is loaded
+    before calling these methods.
+
+---
+
+## Pattern 3 — RobotLab Robot with RAG Context
+
+Inject retrieved context directly into a robot's prompt:
+
+```ruby
+require "robot_lab"
+require "robot_lab/document_store"
+
+# ── Prepare store ────────────────────────────────────────────────────────────
+store = RobotLab::DocumentStore.new
+store.store(:api_guide,   File.read("docs/api.md"))
+store.store(:error_codes, File.read("docs/errors.md"))
+store.store(:changelog,   File.read("CHANGELOG.md"))
+
+# ── Build robot ──────────────────────────────────────────────────────────────
+robot = RobotLab.build(
+  name: "support_agent",
+  system_prompt: "You are a helpful support agent. Answer questions using only the context provided."
+)
+
+# ── Per-request RAG ──────────────────────────────────────────────────────────
+def answer(robot, store, question)
+  hits    = store.search(question, limit: 3)
+  context = hits.map { |h| h[:text] }.join("\n\n---\n\n")
+
+  robot.run(<<~PROMPT)
+    Context documents:
+
+    #{context}
+
+    ---
+
+    User question: #{question}
+  PROMPT
+end
+
+result = answer(robot, store, "What changed in the last release?")
+puts result.last_text_content
+```
+
+---
+
+## Pattern 4 — Chunked Documents
+
+For long documents, split into chunks before storing. Smaller chunks improve
+retrieval precision because a single chunk covers a narrower topic.
+
+```ruby
+# Simple paragraph chunker
+def chunk(text, max_words: 150)
+  paragraphs = text.split(/\n{2,}/).map(&:strip).reject(&:empty?)
+  chunks     = []
+  buffer     = []
+  word_count = 0
+
+  paragraphs.each do |para|
+    words = para.split.size
+    if word_count + words > max_words && buffer.any?
+      chunks << buffer.join("\n\n")
+      buffer     = []
+      word_count = 0
+    end
+    buffer     << para
+    word_count += words
+  end
+  chunks << buffer.join("\n\n") if buffer.any?
+  chunks
+end
+
+# Store with indexed chunk keys
+store = RobotLab::DocumentStore.new
+
+chunk("docs/runbook.md").each_with_index do |text, i|
+  store.store(:"runbook_#{i}", text)
+end
+```
+
+---
+
+## Pattern 5 — Hybrid Key Filtering
+
+Use `search` results alongside `keys` to build filtered views or verify coverage:
+
+```ruby
+# Find which documents were never retrieved (potential gaps in coverage)
+all_keys     = store.keys.to_set
+retrieved    = questions.flat_map { |q| store.search(q, limit: 3).map { |r| r[:key] } }.to_set
+never_hit    = all_keys - retrieved
+
+puts "Documents never retrieved: #{never_hit.to_a}"
+```
+
+---
+
+## Tips
+
+**Chunk size matters.** Chunks of 100–250 words typically give the best
+recall/precision trade-off. Very long documents dilute the embedding signal;
+very short chunks lose context.
+
+**Limit and threshold.** Retrieve more than you need (`limit: 5`) then drop
+results below a quality threshold (e.g., `score >= 0.4`) before building the
+context string. This avoids injecting unrelated documents.
+
+```ruby
+hits = store.search(query, limit: 5).select { |r| r[:score] >= 0.4 }
+```
+
+**Re-embed on document update.** `store` replaces an existing key — calling it
+again with updated text re-embeds and replaces the stored vector automatically.
+
+**Persistent corpus.** `DocumentStore` is in-memory only. For a persistent
+corpus, re-load documents from disk at startup. For production use cases that
+need persistence, consider a dedicated vector database (pgvector, Qdrant, Weaviate).

data/examples/{26_document_store.rb → 01_basic_usage.rb}

@@ -1,23 +1,27 @@
 #!/usr/bin/env ruby
 # frozen_string_literal: true
 
-# Example 26: Embedding-Based Document Store
+# Embedding-Based Document Store Demo
 #
-# Demonstrates Memory#store_document and Memory#search_documents — a
-# lightweight RAG store backed by fastembed (BAAI/bge-small-en-v1.5).
+# Demonstrates RobotLab::DocumentStore standalone and via Memory#store_document /
+# Memory#search_documents — a lightweight RAG store backed by fastembed
+# (BAAI/bge-small-en-v1.5).
 #
 # Documents are multi-paragraph engineering guides stored as Markdown files in:
-#   examples/26_document_store/
+#   examples/01_basic_usage/
 #
 # Usage:
-#   ruby examples/26_document_store.rb
+#   ruby examples/01_basic_usage.rb
+#   ./examples/01_basic_usage.rb
 #   (Downloads the ~23 MB ONNX model on first run; cached afterwards.)
 
-require "robot_lab"
-require "robot_lab/document_store"
+$LOAD_PATH.unshift File.expand_path('../lib', __dir__)
+
+require 'robot_lab'
+require 'robot_lab/document_store'
 
 puts "=" * 60
-puts "Example 26: Embedding-Based Document Store"
+puts "Embedding-Based Document Store Demo"
 puts "=" * 60
 puts
 puts "Note: First run downloads the fastembed model (~23 MB, cached)."
@@ -26,7 +30,7 @@ puts
 # ---------------------------------------------------------------------------
 # Load documents from the companion directory
 # ---------------------------------------------------------------------------
-DOC_DIR = File.join(__dir__, "26_document_store")
+DOC_DIR = File.join(__dir__, '01_basic_usage')
 
 DOCUMENTS = Dir[File.join(DOC_DIR, "*.md")].sort.each_with_object({}) do |path, h|
   key = File.basename(path, ".md").to_sym

data/lib/robot_lab/document_store/version.rb

@@ -2,6 +2,6 @@
 
 module RobotLab
   class DocumentStore
-    VERSION = "0.1.0"
+    VERSION = '0.1.1'
   end
 end