scout-ai 1.2.3 → 2.0.0

data/doc/LLM.md

@@ -1,340 +0,0 @@
-# LLM
-
-This package provides a compact layer to talk to multiple LLM backends while keeping:
-
-- **configuration reproducible** (named endpoints)
-- **prompts reproducible** (chat files)
-- **tool calling first-class** (Workflow tasks, KnowledgeBase databases, MCP tools)
-
-The core entry points are:
-
-- `LLM.ask(question, options={}, &block)` — ask a model (optionally with tools)
-- `LLM.chat(...)` — compile/expand a chat file or message array into final messages
-- `LLM.options(messages)` — extract per-chat options (`endpoint`, `model`, `format`, …)
-- `LLM.print(messages)` — serialize messages back to chat-file format
-- `LLM.embed(text, options={})` — embeddings (not the focus of this document)
-
-For stateful multi-turn conversations, see `doc/Agent.md`.
-
-For the full chat-file role reference, see `doc/Chat.md`.
-
----
-
-## 1. Endpoints: configuring inference points (`~/.scout/etc/AI/<endpoint>`)
-
-Scout-AI uses **named endpoints** so colleagues can agree on short names like `nano` or `deep`.
-
-At runtime, `LLM.ask(..., endpoint: :nano)` will load:
-
-```text
-~/.scout/etc/AI/nano
-```
-
-This file is YAML.
-
-Minimum useful keys:
-
-- `backend`: which backend to use (`responses`, `openai`, `anthropic`, `ollama`, `vllm`, `openwebui`, `bedrock`, `relay`)
-- `url`: server URL (only for backends that need one, e.g. `ollama`, `openwebui`)
-- `model`: backend-specific model id
-
-Any **additional keys are passed to the backend** and may be interpreted there.
-
-Examples:
-
-`~/.scout/etc/AI/nano`
-
-```yaml
-backend: responses
-model: gpt-5-nano
-```
-
-`~/.scout/etc/AI/deep`
-
-```yaml
-backend: responses
-model: gpt-5
-reasoning_effort: high
-text_verbosity: high
-```
-
-`~/.scout/etc/AI/ollama`
-
-```yaml
-backend: ollama
-url: http://localhost:11434
-model: llama3.1
-```
-
-How they are used:
-
-```ruby
-LLM.ask "Say hi", endpoint: :nano
-LLM.ask "Think harder", endpoint: :deep
-```
-
-From the CLI:
-
-```bash
-scout-ai llm ask -e nano "Say hi"
-```
-
----
-
-## 2. `LLM.ask`: the core call
-
-### 2.1 Input types
-
-The `question` argument can be:
-
-- a String (free text)
-- a **path to a chat file** (if the string looks like a filename and exists)
-- an Array of `{role:, content:}` hashes
-- a `Chat` (which is an annotated array)
-
-Internally `LLM.ask` always compiles input via `LLM.chat` first.
-
-### 2.2 Option resolution order
-
-Options come from multiple places:
-
-1) options extracted from the compiled messages (`LLM.options(messages)`)
-2) options passed explicitly to `LLM.ask(question, options)`
-3) endpoint YAML (`~/.scout/etc/AI/<endpoint>`) is merged in (as defaults)
-
-### 2.3 Caching (`persist`)
-
-`LLM.ask` caches responses by default via `Persist.persist`.
-
-- default: `persist: true`
-- set `persist: false` to disable caching
-
-The cache key includes the endpoint name, messages, and most options.
-
-### 2.4 Return value
-
-- default: returns the assistant content string
-- if `return_messages: true`: returns the full message trace (including tool calls)
-
----
-
-## 3. Chat compilation (`LLM.chat`) and why it matters
-
-`LLM.chat` is what makes chat files powerful.
-
-It expands special roles **before** calling the backend:
-
-- imports: `import`, `continue`, `last`
-- pruning: `clear`, `skip`
-- workflow execution: `task`, `inline_task`, `exec_task`, and then `job`, `inline_job`
-- file inclusion: `file`, `directory`, `image`, `pdf`
-
-See `doc/Chat.md` for exact behavior and role syntax.
-
----
-
-## 4. Backends (and what is “passed through”)
-
-Backends are selected by `backend:` (from endpoint YAML, env/config, or explicit options).
-
-Currently supported backends (see `lib/scout/llm/backends/`):
-
-- `responses` — OpenAI Responses API style (default). Supports:
-  - JSON Schema responses (`format:` as schema)
-  - multimodal (`image:`, `pdf:` roles)
-  - session continuation (`previous_response_id`)
-  - optional web search tool (`websearch`)
-- `openai` — OpenAI Chat Completions style
-- `anthropic` — Anthropic Messages API style
-- `ollama` — local Ollama server
-- `vllm` — vLLM OpenAI-compatible server
-- `openwebui` — OpenWebUI-compatible chat/completions
-- `bedrock` — AWS Bedrock runtime
-- `relay` — simple SCP/poll relay
-
-### 4.1 Common backend options
-
-Many backends accept the same high-level keys:
-
-- `model`, `url`, `key`
-- `request_timeout`
-- `log_errors`
-- `tools`
-
-And then backend-specific keys are forwarded (for example: reasoning/text options for `responses`).
-
-### 4.2 Responses session continuation (`previous_response_id`)
-
-If you use the Responses backend and provide a `previous_response_id`, Scout-AI will:
-
-- keep only the messages after the most recent `previous_response_id` marker (so you do not resend the entire conversation)
-- pass `previous_response_id` to the backend
-
-If you set `previous_response: false`, session continuation is disabled.
-
----
-
-## 5. Tools and function calling
-
-Scout-AI represents tools as a **hash**:
-
-```ruby
-tools = {
-  "tool_name" => [handler, tool_definition]
-}
-```
-
-Where:
-
-- `handler` can be:
-  - a `Workflow`
-  - a `KnowledgeBase`
-  - a `Proc` (custom tool implementation)
-- `tool_definition` is an OpenAI-style function schema (adapted per backend)
-
-### 5.1 Providing tools
-
-You can provide tools in three main ways:
-
-1) programmatically: `LLM.ask(..., tools: {...})`
-2) via chat roles: `tool:`, `kb:`, `association:`, `mcp:`
-3) via an Agent: `LLM::Agent` automatically exports a workflow/KB as tools
-
-### 5.2 Workflow tools
-
-```ruby
-workflow_tools = LLM.workflow_tools(Baking)
-LLM.ask "Bake muffins", tools: workflow_tools
-```
-
-Or in a chat file:
-
-```text
-tool: Baking bake_muffin_tray
-```
-
-### 5.3 KnowledgeBase tools
-
-```ruby
-kb_tools = LLM.knowledge_base_tool_definition(kb)
-LLM.ask "Query relationships", tools: kb_tools
-```
-
-Or in a chat file:
-
-```text
-association: brothers test/data/person/brothers undirected=true
-```
-
-### 5.4 Tool execution and output limits
-
-Tool calls are executed by `LLM.process_calls`.
-
-To protect the context window, tool outputs are capped (config key `:max_content_length` under `:llm_tools`).
-
-If a tool returns more than the max length, the content is replaced by an exception-like JSON string.
-
----
-
-## 6. Convenience helpers
-
-### 6.1 `LLM.workflow_ask`
-
-```ruby
-LLM.workflow_ask(Baking, "Bake muffins", endpoint: :nano)
-```
-
-### 6.2 `LLM.knowledge_base_ask`
-
-```ruby
-LLM.knowledge_base_ask(kb, "Who is X's brother in law?", endpoint: :nano)
-```
-
----
-
-## 7. CLI usage
-
-### 7.1 `scout-ai llm ask`
-
-The main “stateless” CLI.
-
-```bash
-scout-ai llm ask -e nano "Say hi"
-```
-
-For convenience prompt does not need quotes.
-
-```bash
-scout-ai llm ask -e nano Say hi
-```
-
-Useful flags:
-
-- `-e/--endpoint <name>`
-- `-m/--model <model_id>`
-- `-b/--backend <backend>`
-- `-c/--chat <file>` — read/append to a chat file
-- `-i/--imports <a,b,c>` — import other chat files
-- `-t/--template <name_or_path>` — templates with optional `???` placeholder
-- `-f/--file <path>` — inline a file into the prompt
-- `-d/--dry_run` — print the compiled conversation (no model call)
-
-The configuration files under etc/config can be used to setup default
-endpoints or models.
-
-```config
-# ~/.scout/etc/config
-endpoint nano ask
-```
-
-### 7.2 `scout-ai agent ask`
-
-The “agent-aware” CLI.
-
-```bash
-scout-ai agent ask Baking Bake a tray of muffins
-```
-
-It loads the agent (workflow + knowledge base + start_chat) and runs a stateful conversation.
-
-See `doc/Agent.md`.
-
----
-
-## 8. Programmatic use (control loops)
-
-For loops, branching, delegation, and structured iteration, use `LLM::Agent`.
-
-See `doc/Agent.md` for the full API and examples.
-
----
-
-## 9. Minimal end-to-end example
-
-1) Create an endpoint file:
-
-```yaml
-# ~/.scout/etc/AI/nano
-backend: responses
-model: gpt-5-nano
-```
-
-2) Ask from the CLI:
-
-```bash
-scout-ai llm ask -e nano Say hi
-```
-
-3) Use a chat file with a workflow tool:
-
-```text
-user:
-
-tool: Baking bake_muffin_tray
-
-Bake muffins using the tool.
-```
-
-```bash
-scout-ai llm ask -c baking.chat -e nano
-```

data/doc/RAG.md

@@ -1,129 +0,0 @@
-# RAG (Retrieval-Augmented Generation) module
-
-This document explains how to use the RAG helper provided in Scout (lib/scout/llm/rag.rb).
-
-Audience: AI agents and developers integrating retrieval-augmented flows into other applications.
-
-Overview
---------
-LLM::RAG provides a thin helper to build a nearest-neighbor index over embedding vectors using the hnswlib library. It expects an array of fixed-size numeric vectors (Float arrays) and returns an HNSW index that can be queried with another vector to find the nearest neighbors.
-
-The RAG.index method is intentionally small and focused:
-
-- It requires the `hnswlib` Ruby gem at runtime (loaded inside the method).
-- It uses L2 (Euclidean) distance by default.
-- It sets the index dimension to the length of the first vector and initializes the HNSW index with the number of elements supplied.
-- Each vector is added in order; the integer ID stored in the index is the zero-based position in the input array.
-
-Prerequisites
--------------
-- Ruby environment with the Scout gem code available.
-- The `hnswlib` Ruby gem installed (the method requires it dynamically):
-
-  gem install hnswlib
-
-- An embedding function that produces fixed-length numeric vectors. Scout exposes LLM.embed(...) which delegates to configured backends (OpenAI, Ollama, etc.). Ensure your embedding backend is configured and working.
-
-Basic usage
------------
-The common RAG flow is:
-
-1. Prepare a corpus (array of documents or chunks).
-2. Compute embeddings for each document.
-3. Build an HNSW index from those embeddings using LLM::RAG.index.
-4. For a query, compute its embedding and run a nearest-neighbor search on the index.
-5. Map matched neighbor indices back to the original documents.
-
-Example (Ruby)
----------------
-This example shows a minimal end-to-end flow using Scout's LLM.embed helper to compute embeddings and LLM::RAG to build and query an index.
-
-```ruby
-# `documents` is an array of strings (documents/chunks).
-documents = [
-  "How to make espresso at home",
-  "Machine learning: an introduction",
-  "Ruby concurrency primitives and patterns",
-  "Cooking guide: baking sourdough"
-]
-
-# 1) Compute embeddings for each document.
-#    Use whatever embed model/backend you have configured. Pass model: if needed.
-embeddings = documents.map do |doc|
-  # returns an Array<Float> of fixed length
-  LLM.embed(doc, model: 'mxbai-embed-large')
-end
-
-# 2) Build the HNSW index
-index = LLM::RAG.index(embeddings)
-
-# 3) For a query, compute its embedding
-query = "best way to brew espresso"
-query_vec = LLM.embed(query, model: 'mxbai-embed-large')
-
-# 4) Run nearest-neighbor search
-#    search_knn returns two arrays: node indices and distances/scores
-k = 3
-nodes, scores = index.search_knn(query_vec, k)
-
-# 5) Map indices back to original documents
-results = nodes.map { |i| documents[i] }
-
-puts "Top #{k} results:" 
-results.each_with_index do |doc, idx|
-  puts "#{idx + 1}. #{doc} (score=#{scores[idx]})"
-end
-```
-
-Notes and best practices
-------------------------
-- Vector dimensionality: All vectors passed to LLM::RAG.index must have identical length. The code inspects `data.first.length` to determine the index dimension.
-- Index IDs: The HNSW index stores integer IDs equal to the input array index. Keep a mapping from those indices to your document IDs/metadata (for instance, an array of document IDs parallel to the embeddings array).
-- Persistence: The RAG helper code only constructs and populates the index in memory. The underlying `hnswlib` gem typically offers persistence APIs (save/load). To persist or reload an index, consult the `hnswlib` gem documentation for the correct methods and usage patterns.
-- Memory and performance: HNSW indexes keep data in memory and can be large for many vectors. Choose your chunking strategy and max dataset size accordingly.
-- Distance metric: The current implementation uses the `'l2'` (Euclidean) space. If your application needs cosine similarity, either normalize vectors before indexing (common practice) or check whether the hnswlib Ruby binding supports a cosine space and adapt accordingly.
-
-Example: utility wrapper
-------------------------
-Here is a small utility that wraps the typical pattern and returns the top-k documents and scores for a query.
-
-```ruby
-# documents: Array of items (strings or objects). If objects, provide a `to_embedding_source` or pass a block to extract text.
-# embed_opts: options forwarded to LLM.embed (e.g. model: ...)
-def build_rag_index(documents, embed_opts = {})
-  # compute embeddings in order
-  embeddings = documents.map { |d| LLM.embed(d, embed_opts) }
-  index = LLM::RAG.index(embeddings)
-  [index, embeddings]
-end
-
-def rag_query(index, documents, query, k = 5, embed_opts = {})
-  qvec = LLM.embed(query, embed_opts)
-  nodes, scores = index.search_knn(qvec, k)
-  results = nodes.map { |i| { doc: documents[i], score: scores[nodes.index(i)] } }
-  results
-end
-
-# Usage:
-# index, embs = build_rag_index(documents, model: 'mxbai-embed-large')
-# top = rag_query(index, documents, 'how to make coffee', 3, model: 'mxbai-embed-large')
-```
-
-Troubleshooting
----------------
-- "NoMethodError" or "uninitialized constant Hnswlib": ensure the `hnswlib` gem is installed and available to your Ruby runtime.
-- Inconsistent dimensions: If you see errors related to dimension mismatch, confirm every embedding vector has the same length and is numeric.
-- Mapping errors: Remember the index IDs correspond to the zero-based position in the `data` array passed to LLM::RAG.index. Keep a parallel array or map to metadata (IDs, titles, etc.).
-
-Further integration
--------------------
-- Use chunking for long documents: split long documents into smaller passages, embed each passage, and keep a mapping from passage index to parent document.
-- Use result reranking: after retrieval, you can rerank retrieved documents with more expensive cross-encoders or scoring functions.
-- Combine with generative models: feed retrieved passages into an LLM prompt to produce answers grounded in retrieved content.
-
-References
-----------
-- lib/scout/llm/rag.rb (implementation)
-- hnswlib Ruby gem (install and persistence documentation)
-- Scout LLM embedding helpers (lib/scout/llm/embed.rb)
-

data/scout_commands/documenter

@@ -1,148 +0,0 @@
-#!/usr/bin/env ruby
-
-require 'scout'
-
-$0 = "scout-ai #{$previous_commands.any? ? $previous_commands*" " + " " : "" }#{ File.basename(__FILE__) }" if $previous_commands
-
-options = SOPT.setup <<EOF
-Scout documenter tool
-
-$ #{$0} [<options>] <topic>
-
-Generates technical, example-driven documentation for a given Scout topic by
-analyzing Ruby source and corresponding test files. For a specified topic, it
-locates main and subtopic files, invokes an LLM agent to synthesize markdown
-documentation using real test behavior and examples, then outputs comprehensive
-topic and subtopic documentation files. 
-
--h--help Print this help
-EOF
-if options[:help]
-  if defined? scout_usage
-    scout_usage 
-  else
-    puts SOPT.doc
-  end
-  exit 0
-end
-
-require 'scout-ai'
-
-topic = ARGV.first
-
-raise MissingParameterException if topic.nil?
-
-src_files = Scout.lib.scout.glob("#{topic}*/**/*") + Scout.lib.scout.glob("#{topic}.rb")
-src_files.collect! do |file|
-  file.sub(Dir.pwd, '.')
-end
-
-bin_files = Scout.scout_commands.glob("**/*").select{|file| file.include?(topic) && ! file.directory? }
-
-main = src_files.select{|f| f.split("/").length == 4}.first
-subtopics = src_files.select{|f| f.split("/").length == 5}.collect{|f| File.basename(f).sub(".rb",'') }.uniq
-
-def source_to_test(file)
-  file.sub(%r{\A./lib/}, './test/').sub(%r{([^/]+)\.rb\z}, 'test_\1.rb')
-end
-
-documenter = LLM::Agent.new
-
-documenter.start_chat.system <<-EOF
-You are a world-class Ruby documentation author. For each (source, test) file
-pair given, produce technically precise module- and file-level documentation,
-incorporating specific code usage and behavior from the test file as worked
-examples, code idioms, and edge-case handling. 
-
-Never insert your own example code: always use live content from the tests as examples. 
-
-Integrate documentation and test-derived examples smoothly.
-
-You will be given first the main topic documentation for the main file and
-test_file, then you will be asked to produce documentation for a subtopic.
-
-Finally you will be ask to aggregate all the documentation portions into 
-a final topic documentation file
-
-User markdown
-
-Avoid initial and final comments like: Certainly! I'll do this and that
-EOF
-
-documenter.start_chat.file main
-documenter.start_chat.file source_to_test(main)
-
-documenter.start_chat.user <<-EOF
-This is the basic topic file. Write the markdown documentation for it.
-EOF
-
-docs = {}
-subtopics.each do |subtopic|
-  src = src_files.select{|f| f.include? subtopic}.select{|f| f.end_with?(".rb") }
-  test = src.collect{|f| source_to_test(f) }.select{|f| Open.exists? f  }.select{|f| f.end_with?(".rb")}
-
-  documenter.start
-  (src + test + bin_files).each do |file|
-    documenter.file file
-  end
-
-  documenter.start_chat.user <<-EOF
-Write documentation for topic #{topic} subtopic #{subtopic}
-  EOF
-  docs[subtopic] = documenter.respond
-end
-
-documenter.start
-
-docs.each do |subtopic, documentation|
-  documenter.user <<-EOF
-Please construct a comprehensive documentation on topic #{topic}. For each
-subtopic reproduce all the most important from the original documentation
-files. The subtopic documentation files will not be available anymore, so
-don't leave anything imporant out
-
-<file subtopic=#{subtopic}>
-#{documentation}
-<file>
-  EOF
-end
-
-main_documentation = documenter.chat
-
-documenter.start_chat.user <<-EOF
-This is the revise documentation for the topic:
-
----
-
-#{main_documentation}
-
----
-EOF
-
-revised_subtopics = {}
-docs.each do |subtopic, documentation|
-  documenter.start
-
-  src = src_files.select{|f| f.include? subtopic}.select{|f| f.end_with?(".rb") }
-  test = src.collect{|f| source_to_test(f) }.select{|f| Open.exists? f  }.select{|f| f.end_with?(".rb")}
-
-  documenter.start
-  (src + test + bin_files).each do |file|
-    documenter.file file
-  end
-
-  documenter.user <<-EOF
-Please revise the subtopic documentation in light of the revised main_documentation
-
-<file subtopic=#{subtopic}>
-#{documentation}
-<file>
-  EOF
-  revised_subtopics[subtopic] = documenter.respond
-end
-
-
-Open.write Scout.doc.lib.scout[topic + '.md'].find(:current), main_documentation
-revised_subtopics.each do |subtopic,documentation|
-  Open.write Scout.doc.lib.scout[topic][subtopic + '.md'].find(:current), documentation
-end