scout-ai 1.2.3 → 2.0.0

data/doc/user/Cookbook.md

@@ -0,0 +1,333 @@
+# Cookbook
+
+This page collects small, ready-to-use examples for common Scout-AI tasks. It
+is intended for workflow authors who want quick, practical patterns.
+
+**You should read this if:** you want copy-paste examples for specific tasks.
+
+---
+
+## Quick one-shot question
+
+```bash
+scout-ai llm ask "What is the capital of France?"
+```
+
+```ruby
+agent = LLM.agent(endpoint: :openai)
+agent.start
+agent.user "What is the capital of France?"
+puts agent.chat
+```
+
+---
+
+## Simple assistant with a system prompt
+
+```ruby
+agent = LLM.agent(endpoint: :openai)
+agent.start_chat.system "You are a helpful assistant. Answer concisely."
+agent.start
+agent.user "Explain recursion in one sentence."
+puts agent.chat
+```
+
+---
+
+## Chat file with endpoint and model
+
+```text
+endpoint: anthropic
+model: claude-sonnet-4-20250514
+
+system:
+
+You are a code reviewer.
+
+user:
+
+Review this function for bugs:
+
+def add(a, b)
+  a + b
+end
+```
+
+Save as `review.chat` and run:
+
+```bash
+scout-ai llm ask -c review.chat
+```
+
+---
+
+## Agent with a workflow tool
+
+```ruby
+agent = LLM.agent(endpoint: :openai)
+agent.workflow do
+  task :search => :string do |query|
+    "Results for '#{query}': ..."
+  end
+end
+agent.start
+agent.user "Search for Ruby tutorials."
+puts agent.chat
+```
+
+---
+
+## Agent from a directory
+
+Directory layout:
+
+```
+Agent/
+  Greeter/
+    start_chat
+```
+
+`start_chat`:
+
+```text
+system:
+
+You are a friendly greeter. Always greet by name.
+
+user:
+
+Hello!
+```
+
+Use it:
+
+```ruby
+agent = LLM.load_agent('Greeter')
+agent.start
+agent.user "Hi, I'm Alice."
+puts agent.chat
+```
+
+From CLI:
+
+```bash
+scout-ai agent ask Greeter "Hi, I'm Alice."
+```
+
+---
+
+## Structured JSON output
+
+```ruby
+agent = LLM.agent(endpoint: :openai)
+agent.start
+agent.user 'Return a JSON object: {"name": "Ruby", "type": "language", "year": 1995}'
+result = agent.json
+puts result["name"]   # => "Ruby"
+```
+
+With a schema:
+
+```ruby
+schema = {
+  name: 'evaluation',
+  type: 'object',
+  properties: {
+    score: { type: :integer },
+    feedback: { type: :string }
+  },
+  required: [:score, :feedback]
+}
+
+agent.json_format(schema)
+```
+
+---
+
+## Importing a file into the conversation
+
+```text
+system:
+
+You are a document analyzer.
+
+file: report.pdf
+
+user:
+
+Summarize the key findings.
+```
+
+---
+
+## Multi-turn conversation
+
+```ruby
+agent = LLM.agent(endpoint: :openai)
+agent.start_chat.system "You are a patient tutor."
+agent.start
+
+agent.user "What is a variable?"
+puts agent.chat
+
+agent.user "How is it different from a constant?"
+puts agent.chat   # remembers the previous turn
+```
+
+---
+
+## Delegation: orchestrator with specialist
+
+```ruby
+# Load a specialist
+searcher = LLM.load_agent('Searcher')
+
+# Set up the orchestrator
+orchestrator = LLM.load_agent('Manager')
+orchestrator.socialize  # model can call ask(agent: 'Searcher', prompt: ...)
+
+orchestrator.start
+orchestrator.user "Research the latest advances in protein folding."
+orchestrator.chat
+```
+
+---
+
+## Multi-agent pipeline in a workflow
+
+```ruby
+module AnalysisPipeline 
+  extend Workflow
+  include AgentWorkflow
+
+  chat_task :plan do |objective|
+    agent = self.agent('Planner', chat: chat)
+    agent.start
+    agent.user objective
+    agent.chat
+  end
+
+  chat_task :execute do |objective, plan|
+    agent = self.agent('Executor', chat: chat)
+    agent.start
+    agent.user "Objective: #{objective}\nPlan: #{plan}"
+    agent.chat
+  end
+
+  chat_task :review do |result|
+    agent = self.agent('Critic', chat: chat)
+    agent.start
+    agent.user "Review: #{result}"
+    agent.chat
+  end
+end
+```
+
+---
+
+## Python task in an agent
+
+```text
+MyAgent/
+├── start_chat
+└── python/
+    └── analyze.py
+```
+
+```python
+# python/analyze.py
+import scout
+
+def word_count(text: str) -> dict:
+    """
+    Count words in text.
+
+    Args:
+        text: Input text.
+
+    Returns:
+        Dictionary with word counts.
+    """
+    words = text.lower().split()
+    counts = {}
+    for w in words:
+        counts[w] = counts.get(w, 0) + 1
+    return counts
+
+scout.task(word_count)
+```
+
+```ruby
+agent = LLM::Agent.load_agent('MyAgent', endpoint: :openai)
+agent.start
+agent.user "Count the words in: the quick brown fox"
+puts agent.chat
+```
+
+---
+
+## Error handling with retry
+
+```ruby
+agent = LLM.agent(endpoint: :openai)
+agent.process_exception = Proc.new do |e|
+  if e.message =~ /rate limit/i
+    sleep 10
+    true  # retry
+  else
+    false # re-raise
+  end
+end
+```
+
+---
+
+## Clearing context between phases
+
+```text
+system:
+
+You are a project manager.
+
+user:
+
+Phase 1: Analyze requirements.
+
+clear:
+
+user:
+
+Phase 2: Design the system. (Phase 1 context is cleared)
+```
+
+---
+
+## Knowledge base tool
+
+```text
+system:
+
+You are a bioinformatics assistant.
+
+kb: gene_db [genes proteins diseases]
+
+user:
+
+What proteins are associated with BRCA1?
+```
+
+---
+
+## MCP tool
+
+```text
+system:
+
+You have access to an external search service.
+
+mcp: https://api.example.com/mcp/ [search]
+
+user:
+
+Search for recent papers on climate change.
+```

data/doc/user/CoreConcepts.md

@@ -0,0 +1,181 @@
+# Core Concepts
+
+This page gives you a conceptual map of Scout-AI's main abstractions. It is
+intended for workflow authors — both human developers and coding agents — who
+want to understand what Scout-AI offers before diving into specifics.
+
+**You should read this if:** you have installed Scout-AI and want a high-level
+orientation before building anything.
+
+---
+
+## The big picture
+
+Scout-AI sits between your application code and an LLM provider. It gives you
+four building blocks:
+
+| Concept | What it is | What problem it solves |
+|---------|-----------|----------------------|
+| **Chat** | A conversation format (plain text on disk, Array of hashes in memory) | Reproducibility: every conversation is inspectable, editable, and versionable |
+| **Agent** | A stateful wrapper around a Chat with persistent defaults and tools | Persistence: your agent keeps its system prompt, tools, and options across conversations |
+| **Tools** | Callable functions the LLM can invoke during inference | Grounding: the model can query real data and run real code instead of hallucinating |
+| **Inference Endpoint** | A named configuration for a provider + model + credentials | Portability: switch between OpenAI, Anthropic, Ollama, etc. without changing application code |
+
+These compose. An **Agent** has a **Chat** and a set of **Tools**, and it sends
+the chat to an **Inference Endpoint** to get a response.
+
+---
+
+## Chat: the conversation format
+
+A Chat is simultaneously two things:
+
+1. **On disk:** a plain-text file where each line block starts with a role
+   name and colon (`system:`, `user:`, `assistant:`, etc.).
+2. **In memory:** an Array of message hashes, each with `role:` and `content:`
+   keys.
+
+This dual nature is the foundation of Scout-AI's reproducibility. You can
+write a conversation by hand in a text editor, run it through the CLI, inspect
+the output, and feed it back as input.
+
+```text
+system:
+
+You are a helpful assistant.
+
+user:
+
+Hello!
+
+assistant:
+
+Hi there! How can I help?
+```
+
+**Key idea:** Every conversation — whether a one-shot CLI question, an agent
+session, or the output of a workflow job — is serialized to the same
+plain-text format. This means you can always inspect, edit, and reproduce
+what happened.
+
+→ See [WritingChats.md](WritingChats.md) for the full format.
+
+---
+
+## Agent: the stateful wrapper
+
+An Agent bundles a Chat with persistent configuration:
+
+- **A start chat** — the system prompt, tool declarations, file imports, and
+  other seed messages that prefix every new conversation.
+- **Tools** — workflows, knowledge bases, or MCP tools the agent can call.
+- **Options** — endpoint, model, format, and other defaults.
+
+Agents are **named directories**. You create an agent by making a directory
+under your Scout Agent path and adding a `start_chat` file:
+
+```
+Agent/
+  Researcher/
+    start_chat        # system prompt + tool declarations
+    workflow.rb       # optional: Scout workflow providing tools
+```
+
+Once defined, an agent is invoked by name:
+
+```bash
+scout-ai agent ask Researcher "Find papers about protein folding."
+```
+
+Agents can also **delegate** to other agents — an orchestrator agent can hand
+off sub-tasks to specialists, each with its own tools and persona.
+
+→ See [BuildingAgents.md](BuildingAgents.md) for agent lifecycle and DSL.
+→ See [Delegation.md](Delegation.md) for multi-agent delegation.
+
+---
+
+## Tools: grounding the model
+
+Tools let the LLM call functions during inference. Scout-AI supports several
+kinds:
+
+| Tool source | How you declare it | What it gives the model |
+|-------------|-------------------|----------------------|
+| **Scout Workflow** | `tool:` or `introduce:` in chat, or auto-wired from agent workflow | Tasks with typed inputs/outputs become callable functions |
+| **Knowledge Base** | `kb:` in chat | Database lookups (gene→protein, drug→disease, etc.) |
+| **MCP Server** | `mcp:` in chat | Any MCP-compatible external tool |
+
+The model sees a tool definition (name, description, JSON Schema parameters),
+decides when to call it, and receives the result as a tool-return message that
+is appended to the conversation.
+
+→ See [ToolCalling.md](ToolCalling.md) for the full tool system.
+
+---
+
+## Inference endpoint: provider abstraction
+
+An **endpoint** is a named bundle of provider + model + credentials. You
+configure endpoints once and reference them by name everywhere:
+
+```bash
+# Use the 'anthropic' endpoint for this conversation
+scout-ai llm ask -e anthropic "Hello"
+```
+
+Scout-AI supports OpenAI, Anthropic, Ollama, vLLM, and other OpenAI-compatible
+providers. Endpoints are provider-agnostic from the application's perspective —
+you write your agent once and switch models by changing the endpoint name.
+
+→ See [RunningInference.md](RunningInference.md) for endpoint configuration
+and CLI usage.
+
+---
+
+## How they compose
+
+Here is the typical flow when an agent answers a question:
+
+```
+User question
+    │
+    ▼
+Agent receives question
+    │
+    ├── Appends to current_chat (a Chat object)
+    ├── Sends chat to inference endpoint
+    │       │
+    │       ├── Model responds with text → done
+    │       └── Model calls a tool → execute tool, append result, re-send
+    │
+    └── Returns final answer
+```
+
+The tool-calling loop is automatic: if the model calls a tool, Scout-AI
+executes it, append the result, and re-sends the conversation until the model
+responds with plain text.
+
+---
+
+## When to use what
+
+| If you want to... | Use... |
+|-------------------|--------|
+| Ask a one-off question | `scout-ai llm ask` |
+| Run a saved conversation | `scout-ai llm ask -c file.chat` |
+| Create a reusable persona with tools | An Agent (directory with `start_chat`) |
+| Give the model data to query | Knowledge Base tools (`kb:`) |
+| Give the model code to run | Workflow tools (`tool:` / `introduce:`) |
+| Use external tools | MCP (`mcp:`) |
+| Build multi-agent systems | Delegation (`socialize` / `delegate`) |
+| Build reproducible pipelines | AgentWorkflow (Scout Workflow + Agent) |
+
+---
+
+## Next steps
+
+- [WritingChats.md](WritingChats.md) — master the chat-file format.
+- [BuildingAgents.md](BuildingAgents.md) — create your first agent.
+- [ToolCalling.md](ToolCalling.md) — wire up tools.
+- [Delegation.md](Delegation.md) — build multi-agent systems.

data/doc/user/Delegation.md

@@ -0,0 +1,191 @@
+# Delegation
+
+This page explains how one Scout-AI agent can delegate work to other agents.
+It is intended for workflow authors building multi-agent systems.
+
+**You should read this if:** you want an orchestrator agent to hand off
+sub-tasks to specialist agents.
+
+---
+
+## What delegation is
+
+Delegation lets one agent (the **caller**) invoke another agent (the
+**specialist**) during inference. Each specialist has its own persona, tools,
+and conversation history.
+
+Scout-AI provides two delegation mechanisms:
+
+| Mechanism | How it works | When to use |
+|-----------|-------------|-------------|
+| **`socialize` (the `ask` tool)** | Exposes a single generic `ask` tool. The model chooses which agent to call and what to ask. | When the model should decide when and whom to delegate to |
+| **`delegate` (named hand-off tools)** | Pre-registers a `hand_off_to_<name>` tool for a specific agent. | When you want explicit, named delegation to specific agents |
+
+Both mechanisms share a common concept: **inheritance modes** that control how
+much context flows from caller to specialist.
+
+---
+
+## Inheritance modes
+
+When a specialist is invoked, you control how much of the caller's context it
+receives:
+
+| Mode | What the specialist gets | Use case |
+|------|------------------------|----------|
+| **`none`** | Only its own system prompt | Fully isolated sub-agent |
+| **`tools`** *(default)* | Its own prompt + the caller's tools (but not conversation history) | Same capabilities, private history |
+| **`conversation`** | Its own prompt + the caller's entire current conversation | Deep collaboration with shared context |
+
+The specialist always gets **its own** system prompt first. Inherited context
+is appended after.
+
+> **Important:** The inheritance mode only applies when a conversation is first
+> created. Subsequent turns in the same conversation reuse the accumulated
+> history regardless of the mode.
+
+---
+
+## The `ask` tool (socialize)
+
+`socialize` registers a single tool called `ask`. When the LLM calls it, it
+provides an agent name and a prompt. Scout-AI loads the specialist, sends the
+prompt, and returns the text answer.
+
+### Enabling delegation
+
+```ruby
+agent = LLM.load_agent('Orchestrator')
+agent.socialize
+```
+
+Now the model can call the `ask` tool during inference.
+
+### What the model sees
+
+The model sees a tool with these parameters:
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `agent` | Yes | Name of the specialist agent |
+| `prompt` | Yes | Plain-text prompt for the specialist |
+| `conversation` | No | Named conversation. Omit for one-shot; reuse to continue |
+| `inherit` | No (default `tools`) | Context policy for new conversations |
+
+### Conversation persistence
+
+If the model provides a `conversation` name, the specialist keeps that
+conversation across multiple calls:
+
+```
+Call 1: ask(agent="Worker", prompt="Do X", conversation="task_1")
+Call 2: ask(agent="Worker", prompt="Now do Y", conversation="task_1")
+# Worker remembers the full conversation from task_1
+```
+
+Without a `conversation` name, each call is one-shot (the specialist answers
+and the conversation is not reused).
+
+Each delegated call leaves provenance evidence in the parent chat: the
+specialist's token usage is recorded next to the tool answer, so token costs
+can be traced afterwards with `scout-ai llm prov --evidence`. See the
+[developer provenance guide](../developer/Provenance.md) for details.
+
+---
+
+## Named hand-off tools (delegate)
+
+`delegate` creates a specific tool for a pre-loaded agent:
+
+```ruby
+worker = LLM.load_agent('Worker')
+agent.delegate(worker, :worker, "Delegate work to the Worker agent")
+```
+
+This creates a tool called `hand_off_to_worker`. The model calls it with a
+`message` parameter.
+
+### Custom delegation blocks
+
+You can customize what happens when the tool is called:
+
+```ruby
+agent.delegate(worker, :worker, "Delegate to Worker") do |_name, params|
+  worker.start if params[:new_conversation]
+  worker.user params[:message]
+  worker.chat
+end
+```
+
+---
+
+## `socialize` vs `delegate`
+
+| Aspect | `socialize` | `delegate` |
+|--------|------------|------------|
+| Agent selection | Model chooses at call time | Hard-coded at registration |
+| Tool count | One `ask` tool for all agents | One tool per agent |
+| Conversation management | Named conversations via `conversation` param | Single conversation, resettable |
+| Flexibility | High (model decides) | Controlled (you decide) |
+
+**Rule of thumb:** Use `socialize` when the model should decide delegation
+dynamically. Use `delegate` when you want explicit control over which agents
+are available.
+
+---
+
+## Security: safe delegation
+
+When the model delegates, it provides a prompt string. Scout-AI uses a safe
+method to send this prompt to the specialist — it adds it as a plain user
+message, not as chat-file syntax. This prevents the model from injecting
+control directives (like `tool:` or `system:`) into the specialist's
+conversation.
+
+This is important: it means delegation is safe even if the model produces
+untrusted output.
+
+---
+
+## Building a multi-agent system
+
+A typical pattern:
+
+1. **Create specialist agents** — each in its own directory with a `start_chat`
+   and optional workflow.
+
+2. **Create an orchestrator agent** — its `start_chat` describes the task and
+   the available specialists.
+
+3. **Enable delegation** — call `socialize` or `delegate` on the orchestrator.
+
+```ruby
+# Orchestrator
+orchestrator = LLM.load_agent('Manager')
+orchestrator.socialize
+orchestrator.start
+orchestrator.user "Plan and execute a data analysis pipeline."
+orchestrator.chat
+```
+
+The model can now delegate sub-tasks to any specialist by name.
+
+---
+
+## Common mistakes
+
+- **Forgetting to call `socialize` or `delegate`**: Without one of these, the
+  model has no tool to delegate with.
+- **Expecting specialists to share the orchestrator's tools by default**: Only
+  if `inherit` is `tools` or `conversation`. With `none`, specialists are
+  isolated.
+- **Using `conversation: 'current'`**: This is a legacy value. Use explicit
+  conversation names or omit the parameter.
+
+---
+
+## Next steps
+
+- [MultiAgentWorkflows.md](MultiAgentWorkflows.md) — orchestration patterns.
+- [BuildingAgents.md](BuildingAgents.md) — creating agents.
+- [ManagingContext.md](ManagingContext.md) — how delegation affects context.