scout-ai 1.0.0 → 1.0.1

data/doc/Agent.md

@@ -0,0 +1,279 @@
+# Agent
+
+Agent is a thin orchestrator around LLM, Chat, Workflow and KnowledgeBase that maintains a conversation state, injects tools automatically, and streamlines structured interactions (JSON lists/dictionaries) with helper methods.
+
+Core ideas:
+- Keep a live chat (conversation) object and forward the Chat DSL (user/system/…).
+- Export Workflow tasks and KnowledgeBase queries as tool definitions so the model can call them functionally.
+- Centralize backend/model/endpoint defaults for repeated asks.
+- Provide convenience helpers to iterate over structured results (lists/dictionaries).
+
+Sections:
+- Quick start
+- Construction and state
+- Tool wiring (Workflow and KnowledgeBase)
+- Running interactions
+- Iterate helpers
+- Loading an agent from a directory
+- API reference
+- CLI: scout agent …
+
+---
+
+## Quick start
+
+Build a live conversation and print it
+
+```ruby
+a = LLM::Agent.new
+a.start_chat.system 'you are a robot'
+a.user "hi"
+puts a.print   # via Chat#print, forwarded through Agent
+```
+
+Run a Workflow tool via tool calling
+
+```ruby
+m = Module.new do
+  extend Workflow
+  self.name = "Registration"
+
+  desc "Register a person"
+  input :name, :string, "Last, first name"
+  input :age, :integer, "Age"
+  input :gender, :select, "Gender", nil, :select_options => %w(male female)
+  task :person => :yaml do
+    inputs.to_hash
+  end
+end
+
+agent = LLM::Agent.new workflow: m, backend: 'ollama', model: 'llama3'
+agent.ask "Register Eduard Smith, a 25 yo male, using a tool call to the tool provided"
+```
+
+Query a KnowledgeBase (tool exported automatically)
+
+```ruby
+TmpFile.with_dir do |dir|
+  kb = KnowledgeBase.new dir
+  kb.format = {"Person" => "Alias"}
+  kb.register :brothers, datafile_test(:person).brothers, undirected: true
+  kb.register :marriages, datafile_test(:person).marriages, undirected: true, source: "=>Alias", target: "=>Alias"
+  kb.register :parents, datafile_test(:person).parents
+
+  agent = LLM::Agent.new knowledge_base: kb
+  puts agent.ask "Who is Miki's brother in law?"
+end
+```
+
+---
+
+## Construction and state
+
+- LLM::Agent.new(workflow: nil, knowledge_base: nil, start_chat: nil, **kwargs)
+  - workflow: a Workflow module or a String (loaded via Workflow.require_workflow).
+  - knowledge_base: a KnowledgeBase instance (optional).
+  - start_chat: initial messages (Chat or Array) to seed new chat branches.
+  - **kwargs: stored as @other_options and merged into calls to LLM.ask (e.g., backend:, model:, endpoint:, log_errors:, etc.).
+
+Conversation lifecycle:
+- start_chat → returns a base Chat (Chat.setup []) lazily allocated once.
+- start(chat=nil)
+  - With chat: adopt it (if not already a Chat, annotate and set).
+  - Without: branch the start_chat (non-destructive copy).
+- current_chat → the active chat instance (created on demand).
+
+Forwarding:
+- method_missing forwards any unknown method to current_chat, so you can call:
+  - agent.user "text", agent.system "policy", agent.tool "WF" "task", agent.format :json, etc.
+
+---
+
+## Tool wiring (Workflow and KnowledgeBase)
+
+When you call Agent#ask:
+- If workflow or knowledge_base is present, Agent builds a tools array:
+  - Workflow: LLM.workflow_tools(workflow) produces one tool per exported task (OpenAI/Responses-compatible function schemas).
+  - KnowledgeBase: LLM.knowledge_base_tool_definition(kb) produces a “children” tool that takes {database, entities}.
+- Agent invokes LLM.ask(messages, tools: ..., **@other_options, **user_options) with a block that handles tool calls:
+  - 'children' → returns kb.children(database, entities)
+  - any other tool name → runs workflow.job(name, jobname?, parameters).run or .exec (exec if workflow.exec_exports includes the task).
+
+Notes:
+- Tool outputs are serialized back to the model as tool results. If your tool returns a Ruby object, it is JSON-encoded automatically inside the tool response.
+- For KnowledgeBase integrations, Agent also enriches the system prompt with markdown descriptions of each registered database (system_prompt).
+
+---
+
+## Running interactions
+
+- ask(messages_or_chat, model=nil, options={}) → String (assistant content) or messages (when return_messages: true)
+  - messages_or_chat can be: Array of messages, a Chat, or a simple string (Agent will pass through LLM.chat parsing).
+  - model parameter can override the default; typically you set backend/model/endpoint in the Agent constructor.
+  - If workflow/knowledge_base is configured, tools are injected and tool calls are handled automatically.
+
+- respond(...) → ask(current_chat, ...)
+- chat(...) → ask with return_messages: true, then append the assistant reply to current_chat and return the reply content.
+- json(...) → sets current_chat.format :json, runs ask, parses JSON, returns the object (if object == {"content": ...}, returns that inner content).
+- json_format(format, ...) → sets current_chat.format to a JSON schema Hash and parses accordingly.
+
+Formatting helpers:
+- format_message and prompt are internal helpers for building a system + user prompt (used by some agents). Not required for normal use; Agent relies on LLM.chat to parse and LLM.ask to execute.
+
+---
+
+## Iterate helpers
+
+For models that support JSON schema outputs (e.g., OpenAI Responses), Agent provides sugar to iterate over structured results:
+
+- iterate(prompt=nil) { |item| … }
+  - Sets endpoint :responses (so the Responses backend is used).
+  - If prompt present, appends as user message.
+  - Requests a JSON object with an array property "content".
+  - Resets format back to :text afterwards.
+  - Yields each item in the content list.
+
+- iterate_dictionary(prompt=nil) { |k,v| … }
+  - Similar, but requests a JSON object with string values (arbitrary properties).
+  - Yields each key/value pair.
+
+Example:
+```ruby
+agent = LLM::Agent.new
+agent.iterate("List three steps to bake bread") { |s| puts "- #{s}" }
+
+agent.iterate_dictionary("Give capital cities for FR, ES, IT") do |country, capital|
+  puts "#{country}: #{capital}"
+end
+```
+
+---
+
+## Loading an agent from a directory
+
+- LLM::Agent.load_from_path(path)
+  - Expects a directory containing:
+    - workflow.rb — a Workflow definition (optional),
+    - knowledge_base — a KnowledgeBase directory (optional),
+    - start_chat — a chat file (optional).
+  - Returns a configured Agent with those components.
+
+Paths are resolved via the Path subsystem; files like workflow.rb can be located relative to the given directory.
+
+---
+
+## API reference
+
+Constructor and state:
+- Agent.new(workflow: nil|String|Module, knowledge_base: nil, start_chat: nil, **kwargs)
+- start_chat → Chat
+- start(chat=nil) → Chat (branch or adopt provided)
+- current_chat → Chat
+
+Chat DSL forwarding (method_missing):
+- All Chat methods available: user, system, assistant, file, directory, tool, task, inline_task, job, inline_job, association, format, option, endpoint, model, image, save/write/print, etc.
+
+Asking and replies:
+- ask(messages, model=nil, options={}) → String (or messages if return_messages: true)
+- respond(...) → ask(current_chat, ...)
+- chat(...) → append answer to current_chat, return answer String
+- json(...), json_format(format, ...) → parse JSON outputs
+
+Structured iteration:
+- iterate(prompt=nil) { |item| ... } — endpoint :responses, expects {content: [String]}
+- iterate_dictionary(prompt=nil) { |k,v| ... } — endpoint :responses, expects arbitrary object of string values
+
+System prompt (internal):
+- system_prompt / prompt — build a combined system message injecting KB database descriptions if knowledge_base present.
+
+Utilities:
+- self.load_from_path(path) → Agent
+
+---
+
+## CLI: scout agent commands
+
+The scout command resolves subcommands by scanning “scout_commands/**” paths using the Path subsystem, so packages and workflows can add their own. If you target a directory instead of a script, a listing of subcommands is shown.
+
+Two commands are provided by scout-ai:
+
+- Agent ask
+  - scout agent ask [options] [agent_name] [question]
+    - Options:
+      - -l|--log <level> — set log severity.
+      - -t|--template <file_or_key> — use a prompt template; positional question replaces '???' if present.
+      - -c|--chat <chat_file> — load/extend a conversation file; appends new messages to it.
+      - -m|--model, -e|--endpoint — backend/model selection (merged with per-endpoint config at Scout.etc.AI).
+      - -f|--file <path> — include file content at the start (or substitute where “...” appears in the question).
+      - -wt|--workflow_tasks <names> — limit exported workflow tasks for this agent call.
+    - Resolution:
+      - agent_name is resolved via Scout.workflows[agent_name] (a workflow) or Scout.chats[agent_name] (an agent directory with workflow.rb/knowledge_base/start_chat). The Path subsystem handles discovery across packages.
+    - Behavior:
+      - If --chat is given, the conversation is expanded (LLM.chat) and the new model output is appended (Chat.print).
+      - Supports inline file Q&A mode (not typical for agent ask).
+
+- Agent KnowledgeBase passthrough
+  - scout agent kb <agent_name> <kb subcommand...>
+    - Loads the agent’s knowledge base (agent_dir/knowledge_base) and forwards to “scout kb …” with --knowledge_base prefilled (and current Log level).
+    - Useful to manage the KB tied to an agent from the CLI.
+
+Command resolution:
+- The bin/scout dispatcher walks nested directories (e.g., “agent/kb”) and lists available scripts when a directory is targeted.
+
+---
+
+## Examples
+
+Minimal conversation with an Agent (tests)
+```ruby
+a = LLM::Agent.new
+a.start_chat.system 'you are a robot'
+a.user "hi"
+puts a.print
+```
+
+Register and run a simple workflow tool call
+```ruby
+m = Module.new do
+  extend Workflow
+  self.name = "Registration"
+  input :name, :string
+  input :age, :integer
+  input :gender, :select, nil, :select_options => %w(male female)
+  task :person => :yaml do
+    inputs.to_hash
+  end
+end
+
+puts LLM.workflow_ask(m, "Register Eduard Smith, a 25 yo male, using a tool call",
+                      backend: 'ollama', model: 'llama3')
+# Or equivalently through an Agent:
+agent = LLM::Agent.new workflow: m, backend: 'ollama', model: 'llama3'
+puts agent.ask "Register Eduard Smith, a 25 yo male, using a tool call to the tool provided"
+```
+
+Knowledge base reasoning with an Agent (tests pattern)
+```ruby
+TmpFile.with_dir do |dir|
+  kb = KnowledgeBase.new dir
+  kb.format = {"Person" => "Alias"}
+  kb.register :brothers, datafile_test(:person).brothers, undirected: true
+  kb.register :marriages, datafile_test(:person).marriages, undirected: true, source: "=>Alias", target: "=>Alias"
+  kb.register :parents, datafile_test(:person).parents
+
+  agent = LLM::Agent.new knowledge_base: kb
+  puts agent.ask "Who is Miki's brother in law?"
+end
+```
+
+Iterate structured results
+```ruby
+agent = LLM::Agent.new
+agent.iterate("List three steps to bake bread") do |step|
+  puts "- #{step}"
+end
+```
+
+---
+
+Agent gives you a stateful, tool‑aware façade over LLM.ask and Chat, so you can build conversational applications that call Workflows and explore KnowledgeBases with minimal ceremony—both from Ruby APIs and via the scout command-line.

data/doc/Chat.md

@@ -0,0 +1,258 @@
+# Chat
+
+Chat is a lightweight builder around an Array of messages that lets you construct, persist, and run LLM conversations declaratively. It integrates tightly with the LLM pipeline (LLM.chat/LLM.ask), Workflows (tool calls), and KnowledgeBase traversal.
+
+A Chat is “just” an Array annotated with Chat behavior (via Annotation). Each element is a Hash like {role: 'user', content: '…'}.
+
+Key capabilities:
+- Build conversations programmatically (user/system/assistant/…).
+- Declare tools and jobs inline (Workflow tasks, saved Step results).
+- Inline files and directories into messages (as tagged content).
+- Set per-turn options (format, endpoint, model), request JSON structures.
+- Run a conversation (ask), append responses (chat), and save/write to files.
+
+Sections:
+- Data model and setup
+- Adding messages
+- Files, directories and tagging
+- Declaring tools, tasks and jobs
+- Options, endpoint and formats
+- Running a chat: ask/chat/json/json_format
+- Persistence helpers and branching
+- Interop with LLM.chat
+- CLI: using Chat with scout llm and scout agent
+- Examples
+
+---
+
+## Data model and setup
+
+A Chat is an Array annotated with Chat, so every method mutates/appends to the underlying array.
+
+- Chat.setup([]) → an annotated empty conversation.
+- Each message appended is a Hash with keys:
+  - role: String or Symbol (e.g., 'user', 'system', 'assistant', …).
+  - content: String (or structured content passed through).
+
+Example
+```ruby
+chat = Chat.setup []
+chat.system "You are an assistant"
+chat.user   "Hi"
+puts chat.print
+```
+
+Chat uses Annotation. You can annotate/cloned arrays and preserve Chat behavior.
+
+---
+
+## Adding messages
+
+All helpers append a message:
+
+- message(role, content) — base append.
+- user(text), system(text), assistant(text).
+- import(file), continue(file) — declarative import markers for LLM.chat (see Interop).
+- file(path), directory(path) — inline content (see next section).
+- format(value) — set desired output format (e.g. :json, or JSON Schema Hash).
+- tool(workflow, task, inputs?) — register a Workflow task tool declaration (see Tools below).
+- task(workflow, task_name, inputs={}) — declare a Workflow task to run, converted to a job (Step) via LLM.chat.
+- inline_task(workflow, task_name, inputs={}) — like task but inlined result.
+- job(step), inline_job(step) — attach a precomputed Step’s result (file content or function call output).
+- association(name, path, options={}) — register a KnowledgeBase association (LLM will build a tool for it).
+
+Utilities:
+- tag(content, name=nil, tag=:file, role=:user) — wrap content in a tagged block (e.g., <file name="…">…</file>) and append it as role (default :user).
+
+---
+
+## Files, directories and tagging
+
+Use file/directory/tag to place content into the chat:
+
+- file(path) — appends the contents of path tagged as <file name="…">…</file>.
+- directory(path) — appends all files inside as a sequence of <file> tags.
+- tag(content, name=nil, tag=:file, role=:user) — manual variant to tag any text.
+
+Tagged content is respected by LLM.parse/LLM.chat and protected from unintended parsing/splitting.
+
+---
+
+## Declaring tools, tasks and jobs
+
+Chat supports wiring external tools into conversations:
+
+- tool workflow task [input options] — declares a callable tool from a Workflow task. Example:
+  ```ruby
+  chat.tool "Baking", "bake_muffin_tray"
+  ```
+  LLM.ask will export this task as a function tool; when the model calls it, the function block (or the default runner) will run the job and feed the result back to the model.
+
+- task workflow task [input options] — enqueues a Workflow job to be produced before the conversation proceeds; replaces itself with a job marker.
+
+- inline_task workflow task [input options] — like task, but inlines the result (as a file-like message) for immediate context.
+
+- job(step) / inline_job(step) — attach an existing Step result. job inserts a function_call + tool output pair so models can reason over the output; inline_job inserts the raw result file.
+
+- association name path [options] — registers a KnowledgeBase association file as a tool (e.g., undirected=true, source/target formats). LLM will add a “children” style tool or per-association tool definition.
+
+These declarations are processed by LLM.chat (see Interop) to produce steps and tools before the model is queried.
+
+---
+
+## Options, endpoint and formats
+
+Set transient options as messages; LLM.options will read them and merge into the ask options:
+
+- option(key, value) — arbitrary key/value (e.g., temperature).
+- endpoint(value) — named endpoint (merges with Scout.etc.AI[endpoint].yaml).
+- model(value) — backend model ID.
+- format(value) — request output format:
+  - :json or 'json_object' for JSON.
+  - JSON Schema Hash for structured object/array replies (Responses/OpenAI support).
+
+You can also insert a previous_response_id message to continue a Responses session.
+
+Note: Most options reset after an assistant turn, except previous_response_id which persists until overwritten.
+
+---
+
+## Running a chat: ask/chat/json/json_format
+
+- ask(…): returns the assistant reply (String) by calling LLM.ask(LLM.chat(self), …).
+- respond(…): equivalent to ask(current_chat, …) when used through an Agent.
+- chat(…): calls ask with return_messages: true, appends the assistant reply to the conversation, and returns the reply content.
+- json(…): sets format :json, calls ask, parses JSON, and returns:
+  - obj['content'] if the parsed object is {"content": …}, else the object.
+- json_format(schema, …): sets format to the provided schema (Hash), calls ask, and parses the JSON accordingly. Returns obj or obj['content'] when applicable.
+
+These helpers adapt the conversation to common usage patterns: accumulate messages, call the model, and parse outputs when needed.
+
+---
+
+## Persistence helpers and branching
+
+- print — pretty-prints the conversation (using LLM.print of the processed chat).
+- save(path, force=true) — writes print output. If path is a name (Symbol/String without path), it resolves via Scout.chats.
+- write(path, force=true) — alias writing with print content.
+- write_answer(path, force=true) — writes the last assistant answer only.
+- branch — returns a deep-annotated dup so you can explore branches without mutating the original.
+- shed — returns a Chat containing only the last message (useful for prompts that must include only the latest instruction).
+- answer — returns the content of the last message.
+
+---
+
+## Interop with LLM.chat
+
+A Chat instance can be passed to LLM.ask by Chat#ask; internally it runs:
+
+1) LLM.chat(self) — expands the Array of messages into a pipeline:
+   - imports, clear, clean,
+   - tasks → produces dependencies (Workflow.produce),
+   - jobs → turns Steps into function calls or inline files,
+   - files/directories → expand into <file> tags.
+
+2) LLM.options to collect endpoint/model/format/etc.
+
+3) Backend ask with optional tool wiring (from tool/association declarations).
+
+Thus, your Chat can be both a declarative script (like a “chat file”) and a runnable conversation object.
+
+---
+
+## CLI: using Chat with scout llm and scout agent
+
+The CLI uses the same message DSL and processing. Useful commands:
+
+- Ask an LLM:
+  - scout llm ask [options] [question]
+    - -t|--template <file_or_key> — load a prompt template; if it contains “???”, the trailing question replaces it; otherwise concatenates as a new user message.
+    - -c|--chat <chat_file> — open a conversation file; the response is appended to the file (using Chat.print).
+    - -i|--inline <file> — answer comments of the form “# ask: …” inside a source file; writes answers inline between “# Response start/end”.
+    - -f|--file <file> — prepend the file contents as a tagged <file> message; or embed STDIN/file where “...” appears in the question.
+    - -m|--model, -e|--endpoint, -b|--backend — select backend and model (merged with per-endpoint configs).
+    - -d|--dry_run — expand and print the conversation (LLM.print) without asking.
+
+- Ask via an Agent (workflow + knowledge base):
+  - scout agent ask [options] [agent_name] [question]
+    - Loads the agent from:
+      - Scout.workflows[agent_name] (workflow.rb) or
+      - Scout.chats[agent_name] or a directory with workflow.rb, knowledge_base/, start_chat.
+    - Same flags as llm ask; adds:
+      - -wt|--workflow_tasks list — export only these tasks to the agent as callable tools.
+
+- Auxiliary:
+  - scout llm template — list prompt templates (Scout.questions).
+  - scout llm server — minimal chat web UI over ./chats with a REST API (save/run lists).
+  - scout agent kb <agent_name> … — runs KnowledgeBase CLI pre-wired to the agent’s KB.
+
+This CLI uses the same LLM.chat pipeline and Chat.print/save semantics as the Ruby API.
+
+---
+
+## Examples
+
+Create and print a minimal conversation
+```ruby
+a = LLM::Agent.new
+a.start_chat.system 'you are a robot'
+a.user "hi"
+puts a.print
+```
+
+Compile messages with roles inline
+```ruby
+text = <<~EOF
+system:
+
+you are a terse assistant that only write in short sentences
+
+assistant:
+
+Here is some stuff
+
+user: feedback
+
+that continues here
+EOF
+LLM.chat(text)  # → messages array ready to ask
+```
+
+Register and use a Workflow tool
+```ruby
+chat = Chat.setup []
+chat.user "Use the provided tool to learn the instructions of baking a tray of muffins. Don't give me your own recipe."
+chat.tool "Baking", "bake_muffin_tray"
+LLM.ask(chat)
+```
+
+Declare KnowledgeBase associations and ask
+```ruby
+chat = Chat.setup []
+chat.system "Query the knowledge base of familiar relationships to answer the question"
+chat.user "Who is Miki's brother in law?"
+chat.message(:association, "brothers #{datafile_test(:person).brothers} undirected=true")
+chat.message(:association, "marriages #{datafile_test(:person).marriages} undirected=true source=\"=>Alias\" target=\"=>Alias\"")
+LLM.ask(chat)
+```
+
+Request structured JSON
+```ruby
+chat = Chat.setup []
+chat.system "Respond in json format with a hash of strings as keys and string arrays as values, at most three in length"
+chat.user "What other movies have the protagonists of the original gost busters played on, just the top."
+chat.format :json
+puts chat.ask
+```
+
+Iterate structured results with an Agent
+```ruby
+agent = LLM::Agent.new
+agent.iterate("List the 3 steps to bake bread") do |step|
+  puts "- #{step}"
+end
+```
+
+---
+
+Chat provides an ergonomic, declarative way to build conversations in code and on disk. It composes seamlessly with LLM.chat/ask, Workflows (as tools), KnowledgeBases (as associations), and the Scout CLI, making it easy to author, run, and persist agentic interactions.