robot_lab 0.3.0 → 0.3.1

data/examples/25_history_search.rb

@@ -1,133 +0,0 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-# Example 25: Chat History Search
-#
-# Demonstrates robot.search_history(query, limit:) — semantic search over a
-# robot's accumulated conversation turns using stemmed term-frequency cosine
-# similarity (classifier gem).
-#
-# The conversation fixture (30 turns across 5 topics) lives in:
-#   examples/25_history_search/conversation.jsonl
-#
-# Usage:
-#   ruby examples/25_history_search.rb
-
-require "json"
-require_relative "common"
-
-CONVERSATION_TURNS = File.readlines(
-  File.join(__dir__, "25_history_search", "conversation.jsonl"), chomp: true
-).map { |line| JSON.parse(line, symbolize_names: true) }.freeze
-
-banner "Chat History Search"
-
-# ---------------------------------------------------------------------------
-# Build a robot and load the conversation fixture
-#
-# Robot#replace_messages is the supported way to install a saved conversation
-# — checkpoint restore, replaying a transcript, or seeding a fixture like this
-# one. It resets the chat and re-adds each message, so there is no need to
-# reach into @chat and overwrite @messages by hand.
-# ---------------------------------------------------------------------------
-robot = RobotLab.build(**llm_opts, name: "tech_lead", system_prompt: "You are a senior engineering advisor.")
-
-messages = CONVERSATION_TURNS.map do |turn|
-  RubyLLM::Message.new(role: turn[:role].to_sym, content: turn[:content])
-end
-robot.replace_messages(messages)
-
-messages = robot.messages
-
-total_words = messages.sum { |m| m.content.to_s.split.size }
-puts "Conversation loaded: #{messages.size} messages, ~#{total_words} words"
-puts "Topics: database migration, API performance, deployment pipeline, background jobs, onboarding"
-puts
-
-# ---------------------------------------------------------------------------
-# Helper: print search results
-# ---------------------------------------------------------------------------
-def show_results(results)
-  results.each do |r|
-    preview = r.text.length > 100 ? "#{r.text[0..97]}..." : r.text
-    puts "  [#{r.role}] score=#{format("%.3f", r.score)} idx=#{r.index}"
-    puts "    #{preview}"
-  end
-end
-
-# ---------------------------------------------------------------------------
-# Searches across four distinct topics
-#
-# Note on TF cosine artifacts (expected behavior, not bugs):
-#
-# 'deploy rollback production incident': the rollback playbook (idx=17) ranks
-# 3rd rather than 1st. The short question "Should we do blue-green deploys?"
-# (idx=18) beats it because "deploy" appears there and TF vectors over-weight
-# single high-frequency query terms in short messages.
-#
-# 'caching Redis invalidation TTL': the Docker GHA cache step (idx=15) is a
-# false positive at rank 3 — "cache" appears in that message. The Redis hits
-# at ranks 1 and 2 are correct.
-#
-# These are genuine limitations of keyword-based cosine similarity. The results
-# shown here are authentic, not cherry-picked.
-# ---------------------------------------------------------------------------
-{
-  "database migration schema change postgres"    => 3,
-  "slow API endpoint N+1 query performance"      => 3,
-  "deploy rollback production incident"          => 3,
-  "Sidekiq retry Stripe failed jobs dead queue"  => 3,
-  "caching Redis invalidation TTL"               => 3,
-}.each do |query, limit|
-  puts "── Search: '#{query}'"
-  show_results robot.search_history(query, limit: limit)
-  puts
-end
-
-# ---------------------------------------------------------------------------
-# RAG pattern — retrieve the most relevant turns, then inject as context
-# ---------------------------------------------------------------------------
-section "RAG Pattern: Retrieve Context, Then Call LLM"
-puts "(showing retrieved context — no actual LLM call)"
-puts
-
-rag_query = "Sidekiq jobs exhausting retries during Stripe outage"
-rag_hits  = robot.search_history(rag_query, limit: 3)
-rag_ctx   = rag_hits.map(&:text).join("\n\n")
-
-puts "Query:    \"#{rag_query}\""
-puts "Retrieved #{rag_hits.size} turn(s) — #{rag_ctx.split.size} words"
-puts "Scores:   #{rag_hits.map { |h| format("%.3f", h.score) }.join("  ")}"
-puts "Token savings vs. full history: ~#{total_words} words → #{rag_ctx.split.size} words"
-puts
-puts "Top retrieved turn:"
-puts "  \"#{rag_hits.first.text[0..110]}...\""
-puts
-puts "LLM call would be:"
-puts '  robot.run("Prior context:\n#{context}\n\nQuestion: #{rag_query}")'
-puts
-
-# ---------------------------------------------------------------------------
-# When to use search_history
-# ---------------------------------------------------------------------------
-section "When to Use search_history"
-puts <<~'TEXT'
-
-  Without search_history:
-    robot.run(question)
-    — full accumulated history sent to the LLM on every call
-    — costs grow linearly with conversation length
-
-  With search_history:
-    hits    = robot.search_history(question, limit: 3)
-    context = hits.map(&:text).join("\n\n")
-    robot.run("Prior context:\n#{context}\n\nQuestion: #{question}")
-    — only the N most relevant turns are injected
-    — token cost stays flat regardless of history length
-    — pairs well with compress_history
-
-  Optional dependency: gem "classifier", "~> 2.3"
-
-TEXT
-
-puts "Done."

data/examples/26_document_store/api_versioning_adr.md

@@ -1,52 +0,0 @@
-# Architecture Decision Record #047 — API Versioning Strategy
-
-**Status:** Accepted (2024-11-12)
-**Deciders:** Platform team, Mobile team, Partner integrations team
-
-## Context
-
-The v1 API has accumulated 23 breaking changes held back by an informal freeze
-while three external partners built integrations. The mobile apps ship on a
-4-week release cycle and cannot deploy hotfixes to force users to upgrade. We
-need a versioning strategy that allows the backend to evolve without coordinated
-lockstep releases across all consumers.
-
-## Decision
-
-We adopt URI-based versioning (/api/v2/, /api/v3/) rather than header-based
-(Accept: application/vnd.company.v2+json) for the following reasons:
-
-- URI versioning is visible in logs, dashboards, and browser dev tools.
-- Proxy and CDN rules can target specific version prefixes.
-- Internal clients are all first-party and can be updated in lockstep.
-
-Header-based versioning is reserved for minor non-breaking variants (e.g.,
-adding optional fields) using the Prefer header.
-
-## Support Lifecycle
-
-Each major version is supported for 18 months from GA. Deprecation notices are
-added to response headers (Sunset: date) 6 months before EOL. The deprecation
-dashboard tracks call volume per version per consumer; we do not retire a
-version with > 100 calls/day without direct partner outreach.
-
-## Backwards Compatibility Rules
-
-Within a version, we **may**:
-- Add new fields to responses.
-- Add new optional request parameters.
-- Add new endpoints.
-- Add new enum values (consumers must ignore unknown values).
-
-We **must not**:
-- Remove or rename fields.
-- Change field types.
-- Change HTTP status codes for existing success cases.
-- Remove endpoints.
-
-## Migration Tooling
-
-A version compatibility shim layer translates v1 requests to v2 internal
-representations and back-translates responses. This allows v1 to remain
-operational without duplicating business logic. The shim is tested with a
-contract test suite against recorded v1 response fixtures.

data/examples/26_document_store/incident_postmortem.md

@@ -1,46 +0,0 @@
-# Incident Postmortem — INC-2024-089
-
-**Date:** 2024-10-03
-**Duration:** 47 minutes
-**Severity:** P1
-**Affected:** API gateway, order processing, checkout flows
-
-## Timeline
-
-| Time  | Event |
-|-------|-------|
-| 14:23 | Automated alert fires: p99 API latency exceeds 5 seconds |
-| 14:25 | On-call engineer pages in; confirms checkout error rate at 34% |
-| 14:31 | Identified spike in slow queries on orders table in Datadog APM |
-| 14:38 | Root cause confirmed: migration added non-concurrent index at peak traffic |
-| 14:44 | DBA kills the migration process; index creation aborted |
-| 14:48 | Query latency returns to baseline; error rate drops to 0.2% |
-| 15:10 | Full recovery confirmed; incident closed |
-
-## Root Cause
-
-An engineer ran a schema migration that created an index on orders.status
-without the CONCURRENTLY keyword. Postgres acquired an AccessExclusiveLock on
-the orders table for the duration of the index build (11 minutes). All queries
-touching the orders table queued behind the lock, exhausting the PgBouncer
-connection pool within 3 minutes.
-
-## Contributing Factors
-
-1. Migration review checklist did not include "concurrent index" verification.
-2. The migration was run manually during business hours, not via the deploy pipeline.
-3. No automated linting (strong_migrations) was enforced in CI.
-
-## Remediation (Completed)
-
-- `strong_migrations` gem added to Gemfile; CI fails on unsafe migration patterns.
-- Runbook updated: all migrations that touch tables > 1M rows require DBA review.
-- Index creation added to the concurrent-operations checklist.
-- PgBouncer max_client_conn increased from 150 to 300 as a buffer.
-
-## Lessons Learned
-
-Lock acquisition during index creation is silent in application logs — the first
-visible symptom is connection pool exhaustion, not a database error.
-Instrumenting pg_locks with an alert on long-held AccessExclusiveLocks would
-have cut detection time from 8 minutes to under 1 minute.

data/examples/26_document_store/postgres_runbook.md

@@ -1,49 +0,0 @@
-# PostgreSQL Operations Runbook — v3.1
-
-## Slow Query Investigation
-
-When a query exceeds 1 second, start with pg_stat_statements:
-
-    SELECT query, mean_exec_time, calls, total_exec_time
-    FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 20;
-
-Use EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) on the top offenders.
-Look for Sequential Scans on large tables (> 50k rows) and Hash Joins on
-unindexed foreign keys. Missing index candidates appear as "rows removed by
-filter" values that are an order of magnitude larger than the rows returned.
-
-## Connection Pool Exhaustion
-
-PgBouncer pools connections at the transaction level. When all connections are
-in use, new queries queue until pool_size is reached, at which point clients
-receive "too many clients" errors. Mitigate by:
-1. Reducing max_connections per Rails process via database.yml pool setting.
-2. Increasing server_pool_size in pgbouncer.ini incrementally.
-3. Identifying and killing idle-in-transaction connections:
-
-       SELECT pid, state, query, now() - query_start AS duration
-       FROM pg_stat_activity WHERE state = 'idle in transaction'
-       AND query_start < now() - interval '30 seconds';
-
-## Table Bloat and Vacuum
-
-High update/delete workloads generate table bloat. Check with:
-
-    SELECT relname, n_dead_tup, n_live_tup,
-           round(n_dead_tup::numeric / nullif(n_live_tup, 0) * 100, 1) AS dead_pct
-    FROM pg_stat_user_tables ORDER BY dead_pct DESC;
-
-If dead_pct exceeds 20% on a hot table, trigger VACUUM ANALYZE manually. For
-severe bloat, schedule an off-hours VACUUM FULL (acquires exclusive lock).
-Autovacuum scale factor defaults to 0.2; reduce to 0.05 on high-churn tables.
-
-## Replication Lag
-
-Monitor standby lag with:
-
-    SELECT client_addr, write_lag, flush_lag, replay_lag
-    FROM pg_stat_replication;
-
-Lag above 30 seconds indicates the replica is falling behind writes. Common
-causes: long-running VACUUM on primary holding WAL files, network saturation
-between primary and replica, or index builds on the replica.

data/examples/26_document_store/redis_caching_guide.md

@@ -1,48 +0,0 @@
-# Redis Caching Patterns — Implementation Guide
-
-## Cache Key Design
-
-Keys must encode every dimension that affects the cached value. For a
-user-scoped collection: `orders:user_USER_ID:page_PAGE:v2`. Always include a
-version suffix (v2) so a code deploy can invalidate globally by bumping the
-version, without a manual cache flush. Avoid encoding mutable data (e.g.,
-user.plan) directly in the key; use separate keys and join at read time,
-or accept stale reads.
-
-## TTL Strategy
-
-Set TTLs based on acceptable staleness, not on intuition:
-
-- User session data: 24h (refreshed on activity)
-- API response cache (authenticated): 5 minutes
-- API response cache (public, CDN-backed): 60 seconds
-- Computed aggregates (dashboards): 15 minutes with background refresh
-- Feature flags: 30 seconds (fast propagation of flag changes)
-
-Always set a TTL. Unbounded keys are a production outage waiting to happen
-when a runaway process fills the Redis instance.
-
-## Cache Invalidation
-
-Explicit invalidation is more reliable than TTL-only for write-heavy data. Use
-after_commit callbacks to delete or update cache entries when records change.
-For collections, track the latest updated_at timestamp as the cache key
-component (Russian doll caching). When multiple cache entries must be
-invalidated atomically, use a Redis pipeline or Lua script.
-
-## Redis Memory Pressure
-
-When Redis hits maxmemory, it evicts keys according to the eviction policy. Use
-`allkeys-lru` for pure cache workloads. Monitor `evicted_keys` in Redis INFO; a
-non-zero and growing value means your cache is too small for the working set.
-Separate cache and session data into different Redis instances (or databases)
-so session eviction cannot be triggered by cache pressure.
-
-## Stampede Protection
-
-Under high read concurrency, a cache miss causes multiple processes to
-simultaneously recompute the same expensive value — the cache stampede.
-Mitigate with probabilistic early expiration: recompute when TTL drops below a
-random fraction of the original TTL. Alternatively, use a distributed lock
-(Redlock or a simple SET NX PX lock key) to allow only one process to recompute
-while others wait briefly on the stale value.

data/examples/26_document_store/sidekiq_guide.md

@@ -1,51 +0,0 @@
-# Background Job Processing with Sidekiq — Engineering Guide
-
-## Job Design Principles
-
-Every Sidekiq job must be idempotent: running it twice with the same arguments
-must produce the same outcome. This is non-negotiable because Sidekiq retries
-failed jobs and at-least-once delivery is guaranteed, not exactly-once. Achieve
-idempotency by checking preconditions (has this invoice already been generated?),
-using database unique constraints on job output records, and passing Stripe
-idempotency keys.
-
-## Retry Configuration
-
-The default retry count is 25, which provides backoff up to ~21 days. For
-time-sensitive jobs (send_welcome_email) reduce to 3. For financial jobs
-(charge_subscription) raise to 15 to survive multi-hour outages.
-
-Configure per-job: `sidekiq_options retry: 10`
-
-Customize backoff with sidekiq_retry_in:
-
-    sidekiq_retry_in { |count| (count ** 4) + 15 + rand(30) * count }
-
-This gives approximately: 15s, 1m, 5m, 17m, 34m for the first 5 retries.
-
-## Circuit Breaker Pattern
-
-When a downstream service (Stripe, SendGrid) is degraded, jobs fail rapidly and
-fill the retry queue, creating a thundering-herd effect when the service
-recovers. Use a circuit breaker backed by Redis:
-
-- Set `stripe:circuit_open` in Redis when 3 consecutive failures occur.
-- In a job middleware, check the flag; if open, re-enqueue with 5-minute delay.
-- Auto-clear the flag after 10 minutes using Redis TTL.
-
-This converts retry churn into scheduled bursts.
-
-## Dead Queue Management
-
-Jobs reach the dead queue after exhausting all retries. Never bulk-retry
-blindly. Group dead jobs by error class, inspect a sample for root cause,
-fix the underlying issue, then use a Rake task to re-enqueue in batches of 50
-with a 1-second inter-batch sleep to avoid overwhelming the recovered service.
-Log each re-enqueue with original args and failure reason.
-
-## Queue Priority and Latency Budgets
-
-Define at least three queues: critical (< 1s SLA: auth, payments), default
-(< 30s: email, webhooks), and bulk (< 1h: exports, reports). Run dedicated
-Sidekiq processes per queue tier. Never mix critical and bulk work in the same
-process — a spike of bulk jobs will starve critical work if they share a queue.

data/examples/26_document_store.rb

@@ -1,140 +0,0 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-# Example 26: Embedding-Based Document Store
-#
-# Demonstrates Memory#store_document and Memory#search_documents — a
-# lightweight RAG store backed by fastembed (BAAI/bge-small-en-v1.5).
-#
-# Documents are multi-paragraph engineering guides stored as Markdown files in:
-#   examples/26_document_store/
-#
-# Usage:
-#   ruby examples/26_document_store.rb
-#   (Downloads the ~23 MB ONNX model on first run; cached afterwards.)
-
-require_relative "common"
-require "robot_lab/document_store"
-
-banner "Embedding-Based Document Store"
-puts "Note: First run downloads the fastembed model (~23 MB, cached)."
-puts
-
-# ---------------------------------------------------------------------------
-# Load documents from the companion directory
-# ---------------------------------------------------------------------------
-DOC_DIR = File.join(__dir__, "26_document_store")
-
-DOCUMENTS = Dir[File.join(DOC_DIR, "*.md")].sort.each_with_object({}) do |path, h|
-  key = File.basename(path, ".md").to_sym
-  h[key] = File.read(path)
-end.freeze
-
-# ---------------------------------------------------------------------------
-# Store into a standalone DocumentStore
-# ---------------------------------------------------------------------------
-store = RobotLab::DocumentStore.new
-
-print "Storing #{DOCUMENTS.size} documents... "
-DOCUMENTS.each { |key, text| store.store(key, text) }
-puts "done"
-puts
-DOCUMENTS.each { |key, text| puts "  #{key.to_s.ljust(24)} #{text.split.size} words" }
-puts
-
-# ---------------------------------------------------------------------------
-# Queries — each phrased differently from the document content
-# ---------------------------------------------------------------------------
-QUERIES = [
-  {
-    label: "Database query performance",
-    query: "Why is my Postgres query slow and how do I investigate it?",
-    want:  :postgres_runbook
-  },
-  {
-    label: "Background job failures during outage",
-    query: "Jobs keep failing when Stripe is down. How do I stop them piling up?",
-    want:  :sidekiq_guide
-  },
-  {
-    label: "API breaking changes policy",
-    query: "Can I rename a response field in the API without breaking clients?",
-    want:  :api_versioning_adr
-  },
-  {
-    label: "Cache expiry and memory pressure",
-    query: "Redis is evicting keys unexpectedly and the cache hit rate has dropped.",
-    want:  :redis_caching_guide
-  },
-  {
-    label: "Production outage from table lock",
-    query: "We had an outage caused by a database lock during a migration. What happened?",
-    want:  :incident_postmortem
-  },
-  {
-    label: "Semantic gap — no shared keywords",
-    query: "Connection pool is full and new requests are being rejected.",
-    want:  :postgres_runbook
-  },
-].freeze
-
-QUERIES.each do |q|
-  results = store.search(q[:query], limit: 3)
-  top     = results.first
-  verdict = top[:key] == q[:want] ? "✓ correct" : "✗ expected #{q[:want]}"
-
-  puts "── #{q[:label]}"
-  puts "   Query:      \"#{q[:query]}\""
-  puts "   Top result: #{top[:key]} (#{format("%.3f", top[:score])}) — #{verdict}"
-  puts "   Ranking:    " + results.map { |r| "#{r[:key]} #{format("%.3f", r[:score])}" }.join(" | ")
-  puts
-end
-
-# ---------------------------------------------------------------------------
-# Delete and verify
-# ---------------------------------------------------------------------------
-section "Delete :redis_caching_guide, Re-run Cache Query"
-store.delete(:redis_caching_guide)
-results = store.search("Redis evicting keys unexpectedly", limit: 2)
-puts "   Remaining keys: #{store.keys.inspect}"
-puts "   Top result after deletion: #{results.first[:key]}"
-puts
-
-# ---------------------------------------------------------------------------
-# Memory integration
-# ---------------------------------------------------------------------------
-section "Memory Integration"
-memory = RobotLab::Memory.new(enable_cache: false)
-
-DOCUMENTS.each { |key, text| memory.store_document(key, text) }
-puts "   Stored #{memory.document_keys.size} documents via memory.store_document"
-
-hits = memory.search_documents("slow query bloat vacuum autovacuum", limit: 2)
-puts "   Search 'slow query bloat vacuum autovacuum':"
-hits.each { |h| puts "     #{h[:key]} (#{format("%.3f", h[:score])})" }
-
-memory.delete_document(:postgres_runbook)
-puts "   After delete, keys: #{memory.document_keys.inspect}"
-puts
-
-# ---------------------------------------------------------------------------
-# RAG pattern
-# ---------------------------------------------------------------------------
-section "RAG Pattern: Retrieve Relevant Docs, Then Generate with LLM"
-
-rag_query = "Our Sidekiq jobs exhaust retries and land in the dead queue after a Stripe outage."
-
-hits    = store.search(rag_query, limit: 2)
-context = hits.map { |h| h[:text] }.join("\n\n---\n\n")
-
-puts "User question:"
-puts "  \"#{rag_query}\""
-puts
-puts "Retrieved #{hits.size} document(s) — #{context.split.size} words of context:"
-hits.each { |h| puts "  #{h[:key]} (score #{format("%.3f", h[:score])})" }
-puts
-puts "LLM call would be:"
-puts '  robot.run("Use the following docs:\n#{context}\n\nQuestion: #{rag_query}")'
-puts
-
-puts "Done."

data/examples/27_incident_response/README.md

@@ -1,65 +0,0 @@
-# Example 27: Production Incident War Room
-
-A simulated payment-service outage, used to demonstrate RobotLab's Phase 5
-infrastructure features: **reactive memory**, **BusPoller serialized
-delivery**, and **poller groups**.
-
-## What it does
-
-A network runs four robots against one incident ("elevated HTTP 500s, p99
-latency spiked to 8s"):
-
-- **Three SRE scouts** (`db_scout`, `net_scout`, `app_scout`) investigate the
-  database, network, and application layers in parallel. Each makes one LLM
-  call, writes its two-sentence finding to shared **reactive memory**
-  (`:db_finding`, `:net_finding`, `:app_finding`), and broadcasts a status
-  line to the war room over TypedBus.
-- **The war room** receives those bus messages through **BusPoller**, which
-  serializes delivery — if two scouts finish at the same instant, their
-  updates queue and are processed one at a time, in arrival order, with no
-  re-entrancy and nothing dropped.
-- **The incident commander** depends on all three scouts and blocks on
-  `memory.get(:db_finding, :net_finding, :app_finding, wait: 60)` — an
-  IO.pipe-backed waiter woken by `IO.select`, so there is no busy-wait and it
-  cooperates with Async. When all findings land it makes one synthesis call
-  and writes a 3-5 bullet action plan.
-
-Also on display:
-
-- **Poller groups** — the scouts run in `poller_group: :investigation`, the
-  commander in `:command`, declared per task on the network.
-- **Memory subscriptions** — a `memory.subscribe` callback prints a line as
-  each finding is written.
-- **Timeout degradation** — `Memory#get(wait:)` raises
-  `RobotLab::AwaitTimeout` rather than returning a sentinel; the commander
-  rescues it and re-reads without `wait:`, so a slow scout degrades the
-  report instead of killing the run.
-
-## How to run
-
-From the gem root (an LM Studio server must be running — `lms server start`;
-provider/model come from `examples/common.rb`):
-
-```bash
-bundle exec ruby examples/27_incident_response/incident_response.rb
-```
-
-Note that `examples/run_all.rb` does **not** include this demo — it only runs
-the top-level `NN_*.rb` files, so run it explicitly as above.
-
-The run makes four LLM calls (three scout diagnoses plus the commander's
-synthesis), so expect a few minutes on the default local model;
-`LLM_PROFILE=small` shortens it.
-
-## Output
-
-- Console: the network graph, each memory write as it happens, the war-room
-  updates in BusPoller delivery order, the scouts' findings, and the final
-  action plan.
-- `output/incident_report.md` — the commander's incident action plan.
-
-## Files
-
-- `incident_response.rb` — the whole demo: `SREScout`, `WarRoom`, and
-  `IncidentCommander` robot classes plus network wiring and the run itself
-- `output/` — where the incident report is written