scout-ai 1.2.3 → 2.0.0

data/doc/Improvements.md

@@ -0,0 +1,325 @@
+# Improvements Advisory
+
+This document catalogs known code issues, documentation gaps, architectural
+suggestions, and anti-patterns to avoid when contributing to Scout-AI. It is
+derived from the research artifacts in [../research/](../research/) and is
+intended as a living reference for maintainers and contributors.
+
+Each entry includes a priority to help triage effort:
+
+| Priority | Meaning |
+|---|---|
+| **High** | Correctness bug, security concern, or actively misleading behavior. Fix soon. |
+| **Medium** | Technical debt that hampers maintainability or extensibility. Address when touching the area. |
+| **Low** | Cleanup, deprecation, or polish. Good first issue or background work. |
+
+---
+
+## Code Issues
+
+### 1. ~~`prov` command monkey-patches the `Chat` class~~
+
+**Priority:** High
+
+> **Status: Resolved.** The `info` command has been removed entirely. Its
+> traversal logic is shared `Chat` library code (`Chat.traverse_provenance`),
+> and its flow-graph rendering lives inline in `scout_commands/llm/prov`;
+> there is no separate `Chat::ProvenanceFlow` class. The flow-graph
+> capabilities of `info` (imports, job deduplication, DOT, SVG/PNG/PDF) are
+> available via `scout-ai llm prov -f`, `--dot`, and `-p`/`--plot`.
+
+**Sources:** [../research/commands-analysis.md](../research/commands-analysis.md), [../research/provenance-analysis.md](../research/provenance-analysis.md).
+
+---
+
+### 2. ~~`prov` command has a hardcoded fallback path~~
+
+**Priority:** High
+
+> **Status: Resolved.** The hardcoded fallback path has been removed. The
+> `prov` command now raises `MissingParameterException` if no filename is
+> provided, matching the behavior of other CLI commands.
+
+**Sources:** [../research/commands-analysis.md](../research/commands-analysis.md).
+
+---
+
+## Documentation Gaps
+
+### D1. ~~Python integration not integrated into the new documentation structure~~
+
+**Priority:** Medium
+
+> **Status: Resolved.** See [user/Python.md](user/Python.md).
+
+**Sources:** [../research/synthesis-report.md](../research/synthesis-report.md).
+
+---
+
+### D2. Model subsystem documentation remains standalone
+
+**Priority:** Low
+
+**Problem:**
+`doc/Model.md` documents the `ScoutModel` / `PythonModel` / `TorchModel` /
+`HuggingfaceModel` subsystem — wrapping ML models for evaluation and training.
+This is tangential to the agent/LLM layer and is intentionally kept separate,
+but it is not linked from the new documentation structure.
+
+**Recommended action:**
+Keep `Model.md` as a standalone reference. Add a note in [StartHere.md](StartHere.md)
+pointing to it. Optionally move it to `doc/developer/Model.md` for structural
+consistency. Do not merge into the LLM docs unless explicitly requested.
+
+**Sources:** [../research/synthesis-report.md](../research/synthesis-report.md).
+
+---
+
+### D3. No dedicated getting-started / installation guide
+
+**Priority:** Low
+
+**Problem:**
+Installation, Gemfile setup, and first-endpoint configuration are covered in
+[user/GettingStarted.md](user/GettingStarted.md), but it may be too terse for
+users who want a guided walkthrough.
+
+**Recommended action:**
+Consider expanding the Getting Started guide with a more linear tutorial (install →
+configure endpoint → first `ask` → first chat file → first agent → first
+workflow). Low priority since the current guide covers the essentials.
+
+**Sources:** [../research/synthesis-report.md](../research/synthesis-report.md).
+
+---
+
+## Architectural Suggestions
+
+### A1. Promote ChatAnalyst provenance traversal from SC26 to the core library
+
+**Priority:** Medium
+
+**Problem:**
+The `ChatAnalyst` agent (currently in `~/git/workflows/SC26/Agent/ChatAnalyst/`)
+implements a `Session` class with BFS-based provenance discovery, token
+accounting, and edge-graph construction. The `info` CLI command independently
+implemented similar logic (`LLMInfoReport`). Having two implementations of the
+same traversal algorithm is a maintenance burden.
+
+> **Partial progress:** The `info` command has been removed, and `prov`
+> renders its flow graph inline in `scout_commands/llm/prov` on top of the
+> shared `Chat` traversal primitives; there is no separate
+> `Chat::ProvenanceFlow` class. ChatAnalyst should be updated to consume the
+> shared `Chat` primitives as well.
+
+**Recommended action:**
+Update ChatAnalyst's `Session` class to delegate to the shared `Chat`
+provenance primitives (`Chat.traverse_provenance`, `Chat.agent_meta_evidence`,
+`Chat.provenance_token_events`) instead of maintaining its own BFS traversal.
+
+**Sources:** [../research/provenance-analysis.md](../research/provenance-analysis.md), [../research/multi-agent-patterns-analysis.md](../research/multi-agent-patterns-analysis.md).
+
+---
+
+### A2. Wire up the custom prompt strategy registry (`REGISTERED_STRATEGIES`)
+
+**Priority:** Medium
+
+**Problem:**
+The prompt strategy system has a designed extension point
+(`REGISTERED_STRATEGIES`) that is not implemented. This prevents plugins or
+users from registering custom named strategies without modifying the source.
+
+**Recommended action:**
+Implement the registry: define `REGISTERED_STRATEGIES = {}` and add a
+`Chat.register_prompt_strategy(name, &block)` class method. Document the
+extension point in [developer/PromptProcessing.md](developer/PromptProcessing.md).
+
+**Sources:** [../research/prompt-strategies-analysis.md](../research/prompt-strategies-analysis.md).
+
+---
+
+### A3. ~~Remove `prov` command; let `info` subsume it entirely~~
+
+**Priority:** Medium
+
+> **Status: Resolved (inverted).** The `info` command has been removed instead.
+> Its flow-graph rendering lives inline in `scout_commands/llm/prov`, on top of
+> the shared `Chat` traversal primitives; there is no separate
+> `Chat::ProvenanceFlow` class. The `prov` command provides both the
+> text-tree report and the flow/DOT/SVG capabilities that were formerly in
+> `info`. `prov` is the sole provenance CLI command.
+
+**Sources:** [../research/commands-analysis.md](../research/commands-analysis.md), [../research/provenance-analysis.md](../research/provenance-analysis.md).
+
+---
+
+### A4. Ensure consistent endpoint configuration documentation
+
+**Priority:** Low
+
+**Problem:**
+Endpoint configuration (YAML keys, `~/.scout/etc/AI/<name>` format, config
+defaults via `~/.scout/etc/config`, environment variables) is documented in
+[user/GettingStarted.md](user/GettingStarted.md) and [developer/Backends.md](developer/Backends.md),
+but the two should be checked for consistency. The research artifacts noted
+that endpoint configuration was a HIGH-priority gap in the original docs.
+
+**Recommended action:**
+Review both documents to ensure the endpoint YAML examples, key names, and
+configuration precedence are identical. Cross-link them so readers can find
+the canonical reference.
+
+**Sources:** [../research/synthesis-report.md](../research/synthesis-report.md).
+
+---
+
+## Anti-patterns to Watch For
+
+These anti-patterns are drawn from the Scout-AI coding philosophy
+([../research/coding-philosophy-analysis.md](../research/coding-philosophy-analysis.md)).
+They are the most common ways that well-intentioned code fights the library
+instead of composing with it.
+
+---
+
+### AP1. Don't create wrapper classes for Chat
+
+**❌ Non-idiomatic:**
+```ruby
+class MyConversation
+  def initialize
+    @messages = []
+  end
+  def add_user(text)
+    @messages << { role: 'user', content: text }
+  end
+end
+```
+
+**✅ Idiomatic:**
+```ruby
+chat = Chat.setup([])
+chat.user("Hello")
+```
+
+**Why:** `Chat` is an annotation on a plain `Array`. Wrapping it in a custom
+class breaks serialization, composition, caching, and every helper that
+expects an Array. Use `Chat.setup(any_array)` and the DSL methods.
+
+---
+
+### AP2. Don't hardcode provider logic in `LLM.ask`
+
+**❌ Non-idiomatic:**
+```ruby
+def self.ask(question, options = {})
+  if options[:provider] == 'openai'
+    # 50 lines of OpenAI-specific code inline
+  end
+end
+```
+
+**✅ Idiomatic:**
+```ruby
+def self.ask(question, options = {})
+  options = IndiferentHash.setup(options)
+  backend = LLM.resolve_backend(options)
+  backend.ask(messages, options, &block)
+end
+```
+
+**Why:** Provider logic belongs in backend modules (composed via
+`prepend`/`include`). `LLM.ask` should dispatch, not implement.
+
+---
+
+### AP3. Don't use plain `Hash` for options that come from user input
+
+**❌ Non-idiomatic:**
+```ruby
+def ask(question, options = {})
+  model = options[:model]  # fails if user passed 'model' as a string key
+end
+```
+
+**✅ Idiomatic:**
+```ruby
+def ask(question, options = {})
+  options = IndiferentHash.setup(options)
+  model = options[:model]  # works for both :model and 'model'
+end
+```
+
+**Why:** Options arrive from YAML files, CLI flags, and Ruby hashes with
+inconsistent key types. `IndiferentHash` normalizes access. Always call
+`IndiferentHash.setup` on any options hash at the entry point.
+
+---
+
+### AP4. Don't subclass to add behavior
+
+**❌ Non-idiomatic:**
+```ruby
+class SpecialChat < Array
+  def user(content)
+    self << { role: 'user', content: content }
+  end
+end
+```
+
+**✅ Idiomatic:**
+```ruby
+module Chat
+  extend Annotation
+  def user(content)
+    message(:user, content)
+  end
+end
+# Then: Chat.setup(any_array)
+```
+
+**Why:** Subclassing creates a rigid hierarchy and breaks the "plain Array"
+contract. Annotation and module composition add behavior non-invasively.
+
+---
+
+### AP5. Don't scatter file I/O without `Path` / `Open`
+
+**❌ Non-idiomatic:**
+```ruby
+File.read("/hardcoded/path/#{name}")
+```
+
+**✅ Idiomatic:**
+```ruby
+path = Scout.var.Agent[name].start_chat
+content = Open.read(path.find) if path.exists?
+```
+
+**Why:** Scout's `Path` API handles convention-based resolution, annotation,
+and existence checks. `Open` provides atomic writes and encoding safety.
+Hardcoded paths break portability and testability.
+
+---
+
+### AP6. Don't define methods on `Agent` that duplicate `Chat`
+
+**❌ Non-idiomatic:**
+```ruby
+class Agent
+  def add_user_message(text)
+    current_chat << { role: 'user', content: text }
+  end
+end
+```
+
+**✅ Idiomatic:**
+```ruby
+agent.user(text)  # works automatically via method_missing → current_chat
+```
+
+**Why:** `Agent` already delegates unknown methods to `current_chat` via
+`method_missing`. Defining wrapper methods on `Agent` creates redundancy and
+maintenance overhead. If the method exists on `Chat`, it already works on
+`Agent`.

data/doc/StartHere.md

@@ -0,0 +1,110 @@
+# Scout-AI Documentation
+
+Scout-AI is an agent and LLM layer built on top of
+[Scout](https://github.com/mikisvaz/scout-gear). It provides a reproducible
+conversation format (`Chat`), tool calling backed by real Scout workflows,
+knowledge bases, and MCP servers, and multi-agent orchestration encoded as
+typed, inspectable workflow jobs.
+
+---
+
+## Choose your path
+
+### I want to build applications with Scout-AI
+
+→ Go to **[user/](user/)** documentation.
+
+The user documentation explains how to use Scout-AI to build agents, define
+tools, configure inference, and orchestrate multi-agent workflows. It is
+organized around concepts and tasks, not internal classes.
+
+**Start here:**
+1. [user/GettingStarted.md](user/GettingStarted.md) — install, configure, first conversation.
+2. [user/CoreConcepts.md](user/CoreConcepts.md) — the four building blocks.
+3. Then follow the topic guides as needed.
+
+### I want to extend or modify Scout-AI
+
+→ Go to **[developer/](developer/)** documentation.
+
+The developer documentation explains how Scout-AI is implemented: the
+architecture, the compilation pipeline, the backend abstraction, the
+delegation internals, and the provenance system. It is concise and links to
+[research/](../research/) for deep code investigations.
+
+**Start here:**
+1. [developer/Architecture.md](developer/Architecture.md) — subsystem map and data flow.
+2. [developer/DesignPrinciples.md](developer/DesignPrinciples.md) — coding philosophy and idioms.
+3. Then follow the topic guides as needed.
+
+### I need to understand how a subsystem works in detail
+
+→ Go to **[../research/](../research/)** investigation documents.
+
+These are architectural reports produced during code investigations. They are
+not maintained documentation and may be outdated, but they contain detailed
+call graphs, implementation discoveries, and design rationale.
+
+---
+
+## Reading paths for common tasks
+
+| Task | Reading path |
+|---|---|
+| **Build my first agent** | user/GettingStarted → user/CoreConcepts → user/BuildingAgents |
+| **Understand multi-agent systems** | user/Delegation → user/MultiAgentWorkflows → developer/DelegationInternals |
+| **Add tool support** | user/ToolCalling → (user/Python if needed) |
+| **Configure inference** | user/RunningInference → user/ManagingContext |
+| **Understand the internals** | developer/Architecture → developer/ChatLifecycle → developer/Backends |
+| **Track and inspect provenance** | developer/Provenance → research/provenance-analysis |
+| **Write idiomatic code** | developer/DesignPrinciples → research/coding-philosophy-analysis |
+| **Use the CLI** | user/Cookbook (quick reference) → research/commands-analysis (full detail) |
+
+---
+
+## Documentation structure
+
+```
+doc/
+├── StartHere.md          ← you are here
+├── Improvements.md       ← known issues and improvement advisory
+├── Model.md              ← ML model subsystem (separate from harness)
+├── user/                 ← building applications with Scout-AI
+│   ├── GettingStarted.md
+│   ├── CoreConcepts.md
+│   ├── WritingChats.md
+│   ├── BuildingAgents.md
+│   ├── ToolCalling.md
+│   ├── RunningInference.md
+│   ├── ManagingContext.md
+│   ├── Delegation.md
+│   ├── MultiAgentWorkflows.md
+│   ├── Python.md
+│   └── Cookbook.md
+├── developer/            ← extending and modifying Scout-AI
+│   ├── Architecture.md
+│   ├── ChatLifecycle.md
+│   ├── DesignPrinciples.md
+│   ├── PromptProcessing.md
+│   ├── Backends.md
+│   ├── DelegationInternals.md
+│   └── Provenance.md
+└── ../research/          ← architectural investigation reports
+    ├── chat-core-analysis.md
+    ├── prompt-strategies-analysis.md
+    ├── agent-delegation-analysis.md
+    ├── agent-workflow-analysis.md
+    ├── backends-analysis.md
+    ├── tools-system-analysis.md
+    ├── provenance-analysis.md
+    ├── multi-agent-patterns-analysis.md
+    ├── coding-philosophy-analysis.md
+    ├── commands-analysis.md
+    └── synthesis-report.md
+```
+
+Each layer becomes progressively more detailed and less stable:
+
+```
+Code → Investigation (research/) → Developer docs (doc/developer/) → User docs (doc/user/)
+```

data/doc/developer/Architecture.md

@@ -0,0 +1,126 @@
+# Architecture
+
+This document explains the overall system architecture of Scout-AI and how its
+subsystems interact. It is intended for framework contributors who need a
+mental map before diving into specific subsystems.
+
+For deeper investigation of any subsystem, see the corresponding
+[research document](../../research/).
+
+---
+
+## Subsystem map
+
+```
+                     ┌─────────────────────────────────────────────┐
+                     │                  LLM (module)                 │
+                     │  LLM.ask  — entry point for all inference     │
+                     │  LLM.chat — parse/compile chat files          │
+                     │  LLM.load_agent — resolve agent directories    │
+                     └───────────────┬───────────────────────────────┘
+                                     │
+                 ┌───────────────────┼───────────────────────┐
+                 ▼                   ▼                       ▼
+           ┌──────────┐       ┌──────────────┐       ┌──────────────┐
+           │ Backend  │       │  LLM::Agent  │       │    Tools     │
+           │ adapter  │       │ (stateful)   │       │ (WF/KB/MCP)  │
+           └──────────┘       └──────┬───────┘       └──────┬───────┘
+                                    │ holds                │
+                              ┌─────▼─────┐          ┌──────▼──────┐
+                              │   Chat    │◄────────►│  Workflow   │
+                              │ (Array +  │  task    │  tasks as   │
+                              │  DSL)     │  tools   │  tools      │
+                              └───────────┘          └─────────────┘
+                                    │ extends
+                              ┌─────▼─────┐
+                              │ Annotation│ (non-invasive mixin)
+                              └───────────┘
+```
+
+### The six core abstractions
+
+| Abstraction | Realized by | Responsibility |
+|---|---|---|
+| **Chat** | `Chat` module (Annotation on Array) | A conversation: a plain Array of message Hashes, annotated with DSL methods. |
+| **Agent** | `LLM::Agent` class | A stateful conversation holder with tools, a workflow, knowledge bases, and delegation capabilities. |
+| **AgentWorkflow** | `AgentWorkflow` mixin | A `Workflow` mixin that adds `chat_task`, `helper :agent`, and `helper :log_agent` for multi-agent strategies. `log_agent` delegates to `Agent#save`, so workflow jobs and manual agents write to one canonical layout (`<job>.files/<name>.chat` — `agent.chat` by default, `worker.chat`/`critic.chat` for named agents — plus the lazy `<name>.society/<Agent>/<conversation>/…` tree). |
+| **Backend** | `LLM::Backend` module + provider modules | Stateless adapter to a specific LLM provider API. Shares logic via `Backend::ClassMethods`, overrides via `prepend`. |
+| **Tools** | `LLM` module methods | Definition and execution of callable tools: workflow tasks, KB queries, MCP servers. |
+| **Annotation** | `Annotation` (from scout-essentials) | Non-invasive metadata injection onto existing objects without subclassing or wrapping. |
+
+---
+
+## Dependency direction
+
+```
+scout-ai.rb
+  └─ scout/llm/ask.rb       (requires scout, chat)
+  └─ scout/llm/chat.rb      (requires chat/annotation, parse, process, prompt, persist, tools, utils)
+  └─ scout/llm/agent.rb     (requires ask, agent/chat, iterate, delegate, workflow)
+  └─ scout/llm/embed.rb
+  └─ scout/llm/image.rb
+  └─ scout/llm/tools/       (workflow, knowledge_base, mcp, call)
+  └─ scout/llm/backends/    (default + provider adapters)
+```
+
+The key direction is **Agent → Chat → Annotation**.
+
+Backends depend on Chat and `Backend::ClassMethods`, not on Agent.
+AgentWorkflow depends on Agent and Chat, not on specific Backends.
+
+This means you can use Chat and Backends without ever instantiating an Agent,
+and you can use Agents without AgentWorkflow. Each layer adds capability
+without creating hard downward dependencies.
+
+---
+
+## How data flows through the system
+
+A single inference request flows through the layers as follows:
+
+1. **Entry**: `LLM.ask(messages, options)` or `Agent#ask(messages, options)`.
+2. **Chat compilation**: The messages (string, file, or Array) are compiled
+   into a canonical Array of Hashes via `Chat.parse`. Options embedded in the
+   chat (via `option:`, `model:`, `endpoint:` directives) are extracted into
+   the options hash.
+3. **Tool extraction**: Tool/introduce/association roles are extracted from
+   messages. Workflow tasks and KB databases are converted to tool definitions.
+4. **Prompt preparation**: The message array passes through
+   `Chat.prepare_prompt` which applies context-management strategies
+   (e.g., `shorten_tools`). This is **ephemeral** — the stored chat is never
+   mutated.
+5. **Backend dispatch**: `LLM.ask` selects the appropriate Backend module
+   (OpenAI, Anthropic, etc.) via a registry/case dispatch and calls its
+   `ask` method.
+6. **API call + tool loop**: The Backend formats the prompt for the provider,
+   calls the API, parses the response. If the model emitted a tool call, the
+   tool is executed and the result appended; then the Backend re-calls the
+   API with the growing message list (the `chain_tools` recursive loop).
+7. **Response**: The Backend returns the response as an annotated Chat (Array
+   of Hashes), with provenance metadata (`meta:` messages) interleaved.
+
+---
+
+## Extension points
+
+| To extend... | Where to add | Pattern |
+|---|---|---|
+| A new LLM provider | `lib/scout/llm/backends/<provider>.rb` | Define a module that `prepend`s `<Provider>Methods` and `include`s `Backend::ClassMethods`. Override `query`, `process_response`, `format_messages` as needed. |
+| A new tool type | `lib/scout/llm/tools/<type>.rb` | Define a module method that returns `{ name => [executor, definition] }` hashes. |
+| A new prompt strategy | Register in `REGISTERED_STRATEGIES` (currently undefined) or pass a `Proc` via `prompt_strategies:` option. |
+| A new agent type | `Agent/<Name>/` directory with `start_chat`, `agent.rb`, or `workflow.rb`. | Conventional discovery. |
+| A new chat-task workflow | `include_workflow AgentWorkflow` in your Workflow module. | Use `chat_task`, `helper :agent`, etc. |
+
+---
+
+## Cross-references
+
+- [ChatLifecycle.md](ChatLifecycle.md) — Chat data model, compilation, annotations.
+- [PromptProcessing.md](PromptProcessing.md) — Context management internals.
+- [Backends.md](Backends.md) — Backend abstraction and inference loop.
+- [DelegationInternals.md](DelegationInternals.md) — Multi-agent mechanics.
+- [Provenance.md](Provenance.md) — Provenance data model and traversal.
+- [DesignPrinciples.md](DesignPrinciples.md) — Coding philosophy and idioms.
+
+> For detailed code investigations of each subsystem, browse the
+> [research/](../../research/) directory.